Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f05cf30
feat(storage): add storage foundation module (PR1)
Jun 14, 2026
ddabd72
chore: add osx-arm64 platform to pixi workspace
Jun 14, 2026
53511ec
feat(storage): add reconcile diff-matrix healing primitive (PR2)
Jun 14, 2026
030f001
feat(storage): add sync upload core — backoff, dead-letter, upload se…
Jun 14, 2026
c4899c3
feat(storage): add `bubbaloop storage` CLI + config/backend factory (…
Jun 14, 2026
c791f59
fix(storage): address code-review findings (security, correctness, ef…
Jun 14, 2026
40403d0
feat(storage): add `storage profile` CLI verbs (PR2)
Jun 14, 2026
538e069
feat(storage): add `storage configure` CLI + config writer (PR2)
Jun 14, 2026
0b27545
feat(storage): add storage::discover + `storage topics` CLI (PR2)
Jun 14, 2026
690487c
feat(storage): add S3-compatible backend behind `s3` feature (PR2)
Jun 14, 2026
78d4645
docs(storage): update storage CLI module doc for the full PR2 verb su…
Jun 14, 2026
d1adc50
feat(storage): add replay + ring_buffer + CLI replay (PR3)
Jun 14, 2026
a16de97
fix(storage): address PR3 code-review findings
Jun 14, 2026
173e14c
feat(storage): add sync queue driver (PR4, §3.4.2)
Jun 14, 2026
ff2b041
feat(mcp): add read-only storage observability tools (PR4, §6)
Jun 14, 2026
cb5fb25
feat(daemon): wire background storage sync driver into daemon startup…
Jun 14, 2026
1266d5f
feat(mcp): add mutating + Zenoh + replay storage tools, completing §6…
Jun 14, 2026
733afba
docs(storage): add storage concepts + internals docs, update ARCHITEC…
Jun 14, 2026
774296f
fix(storage): address PR4 code-review findings
Jun 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,11 @@ If it's app-layer complexity → reject it. If it strengthens sensor drivers →
│ │ Telemetry watchdog (memory/CPU/disk monitoring) │ │
│ └──────────────────────┬─────────────────────────────┘ │
│ ┌──────────────────────┴─────────────────────────────┐ │
│ │ Storage (fleet recording subsystem) │ │
│ │ mcap-recorder sink | content-addressed sync │ │
│ │ reconcile | replay | S3-compatible backends │ │
│ └──────────────────────┬─────────────────────────────┘ │
│ ┌──────────────────────┴─────────────────────────────┐ │
│ │ Zenoh Data Plane (zero-copy, real-time) │ │
│ └──────┬───────────┬───────────┬─────────────────────┘ │
└─────────┼───────────┼───────────┼────────────────────────┘
Expand All @@ -68,6 +73,11 @@ If it's app-layer complexity → reject it. If it strengthens sensor drivers →
└────────┘ └────────┘ └────────┘
```

**Three pillars on the data plane:** the **node runtime** produces live data, the
**agent runtime** reasons over it, and **storage** captures the fleet's Zenoh
traffic to disk, ships it to an S3-compatible backend, and replays it
byte-identically. See [docs/concepts/storage.md](docs/concepts/storage.md).

**Entry points:**
- `bubbaloop agent setup` — interactive wizard: configure provider, model, and identity. No daemon needed.
- `bubbaloop agent chat` — thin Zenoh CLI client (LLM runs daemon-side)
Expand Down Expand Up @@ -304,6 +314,37 @@ The daemon runs a cross-platform resource watchdog (`daemon/telemetry/`) that pr

---

## Storage (Fleet Recording)

The third pillar alongside the node and agent runtimes. Storage records the
fleet's Zenoh traffic into self-describing recordings, syncs them to an
S3-compatible backend, and replays them. Full detail in
[docs/concepts/storage.md](docs/concepts/storage.md) and
[docs/concepts/storage-internals.md](docs/concepts/storage-internals.md).

**Model:** a recording is `manifest.json` (the source of truth — no SQLite index)
plus `chunks/` of MCAP (rosbag2 defaults: 786432-byte chunks, zstd, dual
`publish_time`/`log_time`). Object keys are content-addressed
(`{machine_id}/{recording}/chunk-{idx}-{sha256}`) so re-uploads are idempotent and
a bucket is the sharing unit.

| Concern | Implementation |
|---------|----------------|
| **Recorder** | `mcap-recorder` marketplace sink node (`role: sink`), one per fleet in `client` mode; idle until it receives a `start` command |
| **Sync** | Daemon background task, chunk-finalize-driven, bounded FIFO + shared backoff that pauses the queue on failure; dead-letter list persisted to disk |
| **Integrity** | Streaming SHA-256 per chunk — in filename, manifest, and `x-amz-checksum-sha256`; verified again on download. Mandatory, not configurable |
| **Reconcile** | Diffs local manifest vs local files vs remote objects; heals partial uploads; never deletes (idempotent, safe to re-run) |
| **Replay** | Pure planner + `ReplaySink` trait; paced Zenoh re-publish with `CongestionControl::Block` for faithful, lossless playback |
| **Secrets** | Cloud credentials only in `~/.bubbaloop/secrets.toml` (0600), never in `config.toml` or logs |

**Control surfaces:** the `bubbaloop storage` CLI
(`configure | topics | profile | list | info | upload | download | reconcile |
replay | rm`) and a paired `storage_*` MCP tool per verb (read = viewer,
day-to-day = operator, config/remote-delete = admin). The dashboard Recordings
tab is a control surface (list / start / stop / download), not a data aggregator.

---

## Topic Hierarchy

```
Expand Down
Loading