feat(dag): add --local-only to dag export and import#11229
feat(dag): add --local-only to dag export and import#11229ChayanDass wants to merge 6 commits intoipfs:masterfrom
Conversation
- Export: only export blocks present locally; skip missing (partial CAR). --local-only with --offline. Support both binary and base58 link keys. - Import: support partial CARs; --local-only with -- pin-roots=false (error if both --pin-roots and --local-only set). - Fix cidFromBinString to accept base58 key format from link implementations. Signed-off-by: Chayan Das <01chayandas@gmail.com>
2e5e224 to
9ad0c3b
Compare
- remove local replace directive for go-car/v2 - upgrade to v2.16.1-0.20260306172652-7d2f4aceb070
lidel
left a comment
There was a problem hiding this comment.
Thank you @ChayanDass, quick first pass review below.
Also, we need to wait for go-car release with your fix before this can be merged.
There was a problem hiding this comment.
Test guidance
@ChayanDass please add tests in test/cli, e.g. in dag_test.go. The existing tests there (TestDag, TestDagImportFastProvide) and the harness helpers IPFSDagExport/IPFSDagImport show the patterns to follow.
How to create partially local DAG
Perhaps useful: to test partial DAG export/import you need multi-block test data where some blocks are missing locally. The setup is:
ipfs addenough data (e.g. 300KB with--raw-leaves) to produce a root with multiple child blocks- Use
ipfs refs <root>to list child block CIDs - Use
ipfs block rm <child-cid>to remove one or more blocks from the local blockstore - Now the DAG is partially local --
dag export --offlineshould fail, butdag export --local-only --offlineshould succeed and produce a partial CAR
Specific tests to add
-
TestDagExportLocalOnly-- set up a partial DAG (add content, remove a child block). Confirmdag export --offlinefails. Confirmdag export --local-only --offlinesucceeds and the resulting CAR has fewer blocks than the full DAG (import on a second node with--pin-roots=false --statsto compare block counts). -
TestDagExportLocalOnlyRequiresOffline-- verify that--local-onlywithout--offlineeither errors or behaves as documented. Currently there is no enforcement, and without--offlinethe node may fetch missing blocks from the network, making--local-onlymisleading. -
TestDagImportPartialCAR-- export a partial CAR with--local-only --offline, import it on a fresh node with--pin-roots=false, verify success. -
TestDagImportLocalOnlyPinRootsConflict-- verifydag import --local-only --pin-rootserrors with the expected message.
Signed-off-by: Chayan Das <01chayandas@gmail.com>
Signed-off-by: Chayan Das <01chayandas@gmail.com>
Add
--local-onlytodag exportanddag importSummary
Adds support for partial CAR export and import: export only blocks that exist locally (skipping missing blocks), and import such partial CARs without pinning roots.
Changes
dag export--local-only(bool): When set, only blocks present in the local blockstore are exported; missing blocks are skipped (partial CAR). User must pass--offlineexplicitly for a local-only DAG walk; the command does not set offline automatically.dag import--pin-roots(bool, default true): Pin optional roots listed in the CAR headers after importing.--pin-roots=falseand--local-only(not both with pin-roots true).Usage
Export a partial CAR (local blocks only; user must pass both flags):
Import that partial CAR without pinning roots:
ipfs dag import --pin-roots=false partial.car # or ipfs dag import --local-only --pin-roots=false partial.carReferences
ipfs dag export --ignore-missing-blocks#10826, ipfs-webui#2380