docs: add nydus v3 proposal#1903
Conversation
Signed-off-by: imeoer <yansong.ys@antgroup.com>
Fricounet
left a comment
Fricounet
left a comment
There was a problem hiding this comment.
A lot of good things I look forward to!
| ## Overall Improvements | ||
|
|
||
| - Consolidate and simplify command-line arguments, APIs, and configuration formats. | ||
| - Remove features that are no longer needed or have been effectively unmaintained. |
There was a problem hiding this comment.
do you have a list of what you have in mind there?
| ``` yaml | ||
| # Nydusd configuration | ||
|
|
||
| backend: | ||
| # Type: registry | localfs | oss | ||
| type: registry | ||
| config: | ||
| # Drop the read request once http request timeout, in seconds | ||
| timeout: 5 | ||
| # Drop the read request once http connection timeout, in seconds | ||
| connect_timeout: 5 | ||
| # Retry count when read request failed | ||
| retry_limit: 0 | ||
| proxy: | ||
| url: | ||
|
|
||
| cache: | ||
| # Type: blobcache | dummy | ||
| type: blobcache | ||
| config: | ||
| # Directory of cache files, only for local | ||
| work_dir: /cache | ||
|
|
||
| prefetch: | ||
| # Enable blob prefetch | ||
| enable: false | ||
| # Prefetch thread count | ||
| threads_count: 10 | ||
| # Limit prefetch bandwidth to 1MiB/s | ||
| bandwidth_rate: 1048576 | ||
| ``` |
There was a problem hiding this comment.
is that the complete config format envisioned there? Or are there other knobs which are missing?
| | FUSE + Virtiofs | Mount meta and blob files as FUSE, expose through Virtiofs to guest | Frequent kernel/user context switches, high file operation overhead | | ||
| | Virtiofs + Blobs | Expose blob directory via Virtiofs, guest mounts meta + blobs as EROFS | EROFS DAX integration causes compatibility issues | | ||
| | Virtio-blk + Blobs | Build meta + blobs into virtio-blk device, guest mounts as EROFS | No DAX support, causes guest/host double page cache waste | | ||
| | Virtio-pmem + Local Blobs | Map local meta + blob files to mmap region as virtio-pmem device | Requires pre-downloaded blobs, no on-demand fetching | |
There was a problem hiding this comment.
could we enrich this with how each approach can be done currently (ie using which config flags). I know this has been a massive source of confusion for me in the past so would love to see that clarified
| ### Functional Changes | ||
|
|
||
| - Simplify the API surface and adopt a more mature HTTP server framework to improve concurrency. | ||
| - Switch to YAML as the configuration format and simplify the configuration model. |
There was a problem hiding this comment.
definitely something I'd like to see! The current json config is hard to reason about and it's painful to figure out which option is doing what
| ### Functional Changes | ||
|
|
||
| - Embed CRC32 into chunk metadata by default. | ||
| - Support build-time deduplication based on chunk dictionaries. |
There was a problem hiding this comment.
just curious ,what's the benefit?
| - Introduce a stricter filesystem test suite. | ||
| - Reduce CI runtime for integration tests. | ||
|
|
||
| ## Nydus Build Tool (nydus-image) |
There was a problem hiding this comment.
the idea with nydus-image is that it's the low level tool that allows for interacting with nydus images but only locally right? For registry interactions, nydus-image rely on higher level tools (like nydusify) to get the image from the registry and present it locally in a way that's understandable for nydus-image right?
| --log-level <log-level> | ||
| Log level: [default: info] [possible values: trace, debug, info, warn, error] | ||
| --type <type> | ||
| Conversion type: [default: dir-nydus] [possible values: dir-nydus, tar-nydus] |
There was a problem hiding this comment.
potentially, would we consider supporting a diroci-nydus or taroci-nydus option which can work witch images if they are in an OCI tar format? (imo that's also fine to consider that nydus-image won't support this and it's the role of the caller to handle unpacking the tar image in a directory for nydus-image to consume)
| Usage: nydusd <COMMAND> [OPTIONS] | ||
|
|
||
| Commands: | ||
| fuse [default] Run as a FUSE daemon to mount nydus image |
|
|
||
| # For fuse, prefetch subcommand | ||
| --apisock <apisock> | ||
| Path to the nydus daemon administration API socket |
There was a problem hiding this comment.
| Path to the nydus daemon administration API socket | |
| Path where the nydus daemon administration API socket will be created |
small nit just so that it's clear that nydusd is creating the socket and does not expect it to be there. In opposition to the config and bootstrap flags where the files need to be present
|
|
||
| ### API | ||
|
|
||
| ``` shell |
There was a problem hiding this comment.
maybe consider adding the API that will be used to pass the config that shouldn't be written to disk?
2 participants
Overview
Nydus is considering a major refactor to version v3, and we've drafted a proposal for it.
Related Issues
Please link to the relevant issue. For example:
Fix #123orRelated #456.Change Details
Please describe your changes in detail:
Test Results
If you have any relevant screenshots or videos that can help illustrate your changes, please add them here.
Change Type
Please select the type of change your pull request relates to:
Self-Checklist
Before submitting a pull request, please ensure you have completed the following: