feat(vm): dm-snapshot based block-level CoW for sandbox rootfs#207
Open
lucas77778 wants to merge 4 commits intomasterfrom
Open
feat(vm): dm-snapshot based block-level CoW for sandbox rootfs#207lucas77778 wants to merge 4 commits intomasterfrom
lucas77778 wants to merge 4 commits intomasterfrom
Conversation
Introduce `CowManager` in `snapshot_cow.rs` that uses Linux dm-snapshot to share a single read-only template ext4 image across sandboxes. Each sandbox gets a sparse COW file; only written blocks consume disk space. Key design decisions: - Uses `std::process::Command` + `spawn_blocking` instead of `tokio::process::Command` to avoid SIGCHLD conflicts with the PID-1 reaper in arcbox-agent. - Two-step busybox `losetup -f` + `losetup DEV FILE` serialized via `AsyncMutex` to prevent TOCTOU races on concurrent sandbox creation. - Synchronous `cleanup_stale_sync()` runs at init to remove orphaned dm devices and COW files from previous crashes. - Template loop devices are refcounted; detached when last sandbox using that template is removed.
In direct mode (jailer: None), `do_boot()` now calls
`cow_manager.setup()` to create a dm-snapshot device and passes
`/dev/mapper/arcbox-snap-{id}` to Firecracker as the rootfs block
device. On failure, falls back to using the rootfs image directly.
`remove_sandbox_impl()` tears down the dm-snapshot after the
Firecracker process exits. TTL expiry and restore paths also pass
the cow_manager through for proper cleanup.
Jailer mode and checkpoint/restore are unchanged (Phase 2/3).
The default EnvFilter only allowed `arcbox_agent=info`, which silently dropped all logs from the `arcbox_vm` crate (including dm-snapshot setup/teardown). Add `arcbox_vm=info` so CowManager diagnostics are visible in agent.log.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds dm-snapshot–backed block-level copy-on-write to avoid per-sandbox rootfs image copies when running nested Firecracker sandboxes inside the guest VM.
Changes:
- Introduces
CowManager/CowHandleto create and manage dm-snapshot devices backed by sparse COW files. - Wires CoW setup/teardown into sandbox boot and removal (direct mode; jailer mode deferred).
- Extends logging defaults in
arcbox-agentto includearcbox_vmatinfo.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| virt/arcbox-vm/src/snapshot_cow.rs | New dm-snapshot CoW implementation (loop attach, dmsetup create/remove, stale cleanup, small unit tests). |
| virt/arcbox-vm/src/sandbox.rs | Integrates CoW into sandbox lifecycle: setup during boot (direct mode) and teardown during remove. |
| virt/arcbox-vm/src/lib.rs | Exposes the new snapshot_cow module. |
| virt/arcbox-vm/src/error.rs | Adds VmmError::DeviceMapper for dm/loop/dmsetup-related failures. |
| guest/arcbox-agent/src/main.rs | Expands default tracing filter to include arcbox_vm=info. |
- teardown(): only delete COW file after both dm device removal and loop detach succeed; avoids unlinking a still-referenced backing file that would delay space reclamation. - Downgrade dm-snapshot fallback log from warn to debug in do_boot(); CowManager::new() already warns once at init when dmsetup is missing, so per-sandbox warnings are redundant noise.
lucas77778
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CowManager(snapshot_cow.rs) that uses Linux dm-snapshot to share a single read-only template ext4 image across sandboxes. Each sandbox gets a sparse COW file backed by a dm-snapshot device — only written blocks consume disk space, eliminating the need for full rootfs copies.jailer: None).do_boot()creates a snapshot device at/dev/mapper/arcbox-snap-{id}and passes it to Firecracker;remove_sandbox_impl()tears it down after the process exits. Falls back to direct rootfs usage if dm-snapshot is unavailable.arcbox_vm=infoso CowManager diagnostics are visible inagent.log.Key design decisions
std::process::Command+spawn_blockinginstead oftokio::process::Commandto avoid SIGCHLD conflicts with the PID-1 reaper in arcbox-agent.AsyncMutexserializes busyboxlosetup -f+losetup DEV FILEtwo-step operations to prevent TOCTOU races on concurrent sandbox creation.cleanup_stale_sync()runs synchronously atCowManager::new()to remove orphaned dm devices and COW files from previous crashes.teardown()— only unlinks the backing file after both dm device and loop device are successfully released.Prerequisites
CONFIG_DM_SNAPSHOT=yin guest kernel (merged in feat(config): enable dm-snapshot for sandbox CoW rootfs kernel#5)dmsetupbinary at/arcbox/bin/dmsetupvia VirtioFSBenchmark (1GB rootfs, 1536MB memory, 5 sandboxes)
Test plan
readystatedm-snapshot createdanddm-snapshot teardown completein agent.logcargo clippyandcargo fmtpass with zero warnings