-
Notifications
You must be signed in to change notification settings - Fork 9
feat: initial documentation for Logos Storage #166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gmega
wants to merge
10
commits into
main
Choose a base branch
from
feat/libstorage-tutorial
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+240
−0
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
ffbbcf5
feat: add initial welcome page + libstorage tutorial covering v0.1 us…
gmega 15d3c49
some rewording/clarifications
gmega ce403a9
complete links to module API tutorial and app skeleton; fix some mist…
gmega 53d98ca
update storage doc index
gmega 83bfabe
applied formatting to libstorage tutorial
kashepavadan 549e401
Apply suggestion from @weboko
kashepavadan a3915b4
changed docs to module
kashepavadan a5c1e4e
add version
kashepavadan b87b5c2
rename file
kashepavadan 759210f
chore: follow up to #166 (#284)
weboko File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Welcome to Logos Storage | ||
|
|
||
| Logos Storage aims at being a resilient, decentralized and censorship-resistant storage layer for | ||
| the Logos stack. At present, Logos Storage provides a basic filesharing layer, not unlike IPFS or | ||
| Bittorrent, which allows users to publish and share files with one another. | ||
|
|
||
| Moving forward, Logos Storage will enable full provider and downloader anonymity, as well as | ||
| anonymous persistence. Check out [our roadmap](https://roadmap.logos.co/storage/roadmap/) for more details. | ||
|
|
||
| ## Using Logos Storage | ||
|
|
||
| Logos Storage can currently be used in the following ways: | ||
|
|
||
| **Logos Storage UI App.** | ||
|
|
||
| * The [Logos Storage UI App](https://github.com/logos-co/logos-storage-ui) is a desktop application which allows one to upload and download files over the Logos Storage network. This is the easiest, simplest way to try out Logos Storage. | ||
|
|
||
| **Programmatically.** | ||
|
|
||
| * **Logos Module API.** The [Logos Storage Module](https://github.com/logos-co/logos-storage-module) - a high-level C++ API - is the recommended way of using Logos Storage in your app. Make sure to check: [[Build a CLI app with Logos Storage](./storage-tutorial.md) | [API reference](https://logos-co.github.io/logos-storage-module/api_reference.html)] | ||
|
|
||
| * **libstorage.** [libstorage](https://github.com/logos-storage/logos-storage-nim) is a lower-level C API that goes under the Logos Module API. If you need to construct bindings for a language other than C++, this is what you would use. We have [Go](https://github.com/logos-storage/logos-storage-go), [Rust](https://github.com/nipsysdev/storage-rust-bindings), and higher-level, simple [C](https://github.com/logos-storage/easylibstorage) bindings available. [[API reference](https://github.com/logos-storage/logos-storage-nim/blob/master/library/README.md)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,218 @@ | ||
| --- | ||
| title: Build a CLI app with Logos Storage | ||
| doc_type: procedure | ||
| product: storage | ||
| topics: [] | ||
| steps_layout: sectioned | ||
| authors: | ||
| owner: logos | ||
| doc_version: 1 | ||
| slug: storage-tutorial | ||
| --- | ||
|
|
||
| # Build a CLI app with Logos Storage | ||
|
|
||
| #### Get started building a CLI application that transfers files over the Logos Storage network. | ||
|
|
||
| This tutorial walks you through building a simple CLI application that uploads and downloads files over the Logos Storage network using the [Logos Storage Module API v0.3.2](https://logos-co.github.io/logos-storage-module/api_reference.html). It is intended for developers who are setting up a new application using the skeleton project and working through the module lifecycle for the first time. | ||
|
|
||
| The tutorial uses the [Logos Storage App Skeleton](https://github.com/logos-storage/logos-storage-app-skeleton), which provides a ready-made entry point at `app_main` that uses the `LogosModules` object to access the API. The skeleton also provides a set of Qt-compatible synchronization utilities. | ||
|
|
||
| **Before you start**, make sure you have the following: | ||
|
|
||
| - [Nix package manager](https://nixos.org/download/) | ||
| - Git | ||
|
|
||
| ## What to expect | ||
|
|
||
| - You can initialize, start, and cleanly shut down the storage module within your application. | ||
| - You can upload a file to the network and receive a Content Identifier (CID) that uniquely identifies it. | ||
| - You can download a file from the network using a Content Identifier (CID). | ||
|
|
||
| ## Step 1: Build the skeleton app | ||
|
|
||
| Clone the skeleton repository and compile the binary so you have a working entry point before adding any storage logic. | ||
|
|
||
| 1. Clone the skeleton repository: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/logos-storage/logos-storage-app-skeleton.git | ||
| cd logos-storage-app-skeleton | ||
| ``` | ||
|
|
||
| 1. Build with Nix: | ||
|
|
||
| ```bash | ||
| nix build | ||
| ``` | ||
|
|
||
| If you don't have flakes enabled globally, add experimental flags: | ||
|
|
||
| ```bash | ||
| nix build --extra-experimental-features 'nix-command flakes' | ||
| ``` | ||
|
|
||
| 1. Confirm the compiled binary is available at `./result/bin/storage-app`. | ||
|
|
||
| ## Step 2: Initialize the module | ||
|
|
||
| The skeleton's `app_main` already initializes and starts the module, so you have a working baseline. The snippets in the rest of this tutorial show the calls in isolation; extend `app/main.cpp` to add upload and download logic on top. | ||
|
|
||
| Call `init()` with your configuration once at startup, passing a JSON configuration string. See the [API Reference](https://logos-co.github.io/logos-storage-module/api_reference.html) for all available options. | ||
|
|
||
| ```cpp | ||
| const QString jsonConfig = "{" | ||
| "\"listen-addrs\": [\"/ip4/0.0.0.0/tcp/8000\"]," | ||
| "\"disc-port\": 9000," | ||
| "\"data-dir\": \"./app-data\"," | ||
| "\"nat\": \"none\"" | ||
| "}"; | ||
| bool result = m_logos->storage_module.init(jsonConfig); | ||
| ``` | ||
|
|
||
| > [!CAUTION] | ||
| > Do not call `init()` more than once per instance unless you call `destroy()` first. | ||
|
|
||
| ## Step 3: Start the node | ||
|
|
||
| Subscribe to the `storageStart` event and call `start()` afterward, to be able to detect failures. | ||
|
|
||
| ```cpp | ||
| m_logos->storage_module.on("storageStart", [this](const QVariantList& data) { | ||
| bool success = data[0].toBool(); | ||
| if (!success) { | ||
| QString error = data[1].toString(); | ||
| // Handle error | ||
| } | ||
| }); | ||
|
|
||
| bool result = m_logos->storage_module.start(); | ||
| ``` | ||
|
|
||
| ## Step 4: Upload a file | ||
|
|
||
| The Storage Module allows for two upload approaches. Choose `uploadUrl` for straightforward cases. Use the streaming API when you need fine-grained control over how data is sent. | ||
|
|
||
| ### Upload with `uploadUrl` | ||
|
|
||
| The simplest way to upload files is to subscribe to the upload events, then call `uploadUrl()` with the path to your file. The network returns a Content Identifier (CID) on success. | ||
|
|
||
| 1. Subscribe to the `storageUploadDone` and `storageUploadProgress` events: | ||
|
|
||
| ```cpp | ||
| //m_logos is the LogosModules object, used for API calls | ||
| m_logos->storage_module.on("storageUploadDone", [this](const QVariantList& data) { | ||
| bool success = data[0].toBool(); | ||
| QString sessionId = data[1].toString(); | ||
| QString cidOrError = data[2].toString(); | ||
|
|
||
| if (success) { | ||
| qDebug() << "Upload complete. CID:" << cidOrError; | ||
| } else { | ||
| qDebug() << "Upload failed:" << cidOrError; | ||
| } | ||
| }); | ||
|
|
||
| m_logos->storage_module.on("storageUploadProgress", [this](const QVariantList& data) { | ||
| bool success = data[0].toBool(); | ||
| QString sessionId = data[1].toString(); | ||
| int bytes = data[2].toInt(); | ||
| qDebug() << "Uploaded" << bytes << "bytes"; | ||
| }); | ||
| ``` | ||
|
|
||
| 1. Call `uploadUrl()` with the local file path: | ||
|
|
||
| ```cpp | ||
| QUrl fileUrl = QUrl::fromLocalFile("/path/to/myfile"); | ||
| LogosResult result = m_logos->storage_module.uploadUrl(fileUrl); | ||
| ``` | ||
|
|
||
| ### Upload with the streaming API | ||
|
|
||
| Use the streaming upload API for chunk-level control. | ||
|
|
||
| 1. Initialize the session: | ||
|
|
||
| ```cpp | ||
| LogosResult result = m_logos->storage_module.uploadInit(filename); | ||
| QString sessionId = result.getValue<QString>(); | ||
| ``` | ||
|
|
||
| 1. Upload chunks: | ||
|
|
||
| ```cpp | ||
| QFile file(filepath); | ||
| file.open(QIODevice::ReadOnly); | ||
| int chunkSize = 1024 * 64; | ||
| while (!file.atEnd()) { | ||
| QByteArray chunk = file.read(chunkSize); | ||
| result = m_logos->storage_module.uploadChunk(sessionId, chunk); | ||
| if (!result.success) { | ||
| // Handle error | ||
| break; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 1. Finalize: | ||
|
|
||
| ```cpp | ||
| result = m_logos->storage_module.uploadFinalize(sessionId); | ||
| if (result.success) { | ||
| QString cid = result.getValue<QString>(); | ||
| qDebug() << "CID:" << cid; | ||
| } | ||
| ``` | ||
|
|
||
| ## Step 5: Download a file | ||
|
|
||
| To download content, you need the CID returned during upload. The storage module discovers the content on the network using the CID. | ||
|
|
||
| 1. Subscribe to the `storageDownloadDone` and `storageDownloadProgress` events: | ||
|
|
||
| ```cpp | ||
| //m_logos is the LogosModules object, used for API calls | ||
| m_logos->storage_module.on("storageDownloadDone", [this](const QVariantList& data) { | ||
| bool success = data[0].toBool(); | ||
| QString message = data[1].toString(); | ||
| if (success) { | ||
| qDebug() << "Download complete"; | ||
| } else { | ||
| qDebug() << "Download failed:" << message; | ||
| } | ||
| }); | ||
|
|
||
| m_logos->storage_module.on("storageDownloadProgress", [this](const QVariantList& data) { | ||
| bool success = data[0].toBool(); | ||
| QString sessionId = data[1].toString(); | ||
| int size = data[2].toInt(); | ||
| qDebug() << "Downloaded" << size << "bytes"; | ||
| }); | ||
| ``` | ||
|
|
||
| 1. Call `downloadToUrl()` with the CID and the local destination path: | ||
|
|
||
| ```cpp | ||
| QUrl destination = QUrl::fromLocalFile("/path/to/output"); | ||
| LogosResult result = m_logos->storage_module.downloadToUrl(cid, destination /*, local = false*/); | ||
| ``` | ||
|
|
||
| - Set the `local` (third) parameter of `downloadToUrl` to `true` to retrieve only locally-cached data. | ||
| - Leave `local` as `false` (default) to fetch from the network. | ||
|
|
||
| ## Step 6: Stop and clean up | ||
|
|
||
| Always stop the node before destroying resources to avoid leaving sessions open. To do so, call `stop()` and wait for the `storageStop` event, then call `destroy()`: | ||
|
|
||
| ```cpp | ||
| LogosResult result = m_logos->storage_module.stop(); | ||
| // Wait for storageStop event... | ||
| result = m_logos->storage_module.destroy(); | ||
| ``` | ||
|
|
||
| ## Frequently asked questions | ||
|
|
||
| ### Can I run the storage module without a UI? | ||
|
|
||
| Yes. The storage module is a Qt plugin that can be loaded by any Logos Core host, including the headless [`logoscore`](https://github.com/logos-co/logos-logoscore-cli) CLI. See the [`logoscore` README](https://github.com/logos-co/logos-logoscore-cli) for headless usage and how to wire the storage module plugin into it. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this file is called
libstorage-tutorial.mdwhich is not correct anymoreThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, fixed