Skip to content

storage: SqliteAdapter (Phase 2 of #139) #141

@willwashburn

Description

@willwashburn

Phase 2 of #139. Depends on Phase 1 (StorageAdapter seam) being merged.

Goal

Implement SqliteAdapter — a single-file SQLite database that replaces the JSONL ledger + per-session content sidecars + dedup-hash files for users who opt in via RELAYBURN_STORAGE=sqlite.

Uses Node 22's built-in node:sqlite (already used by archive.ts). DB path defaults to ~/.relayburn/burn.sqlite, configurable via RELAYBURN_SQLITE_PATH.

Schema

Mirrors the existing archive.ts table layout (sessions, turns, tool_calls, tool_result_events, compactions) and extends it with the record types currently held in JSONL only: relationships, user_turns, stamps, content. Dedup hashes from index-sidecar.ts (turnIdHash, relationshipIdHash, toolResultEventIdHash, userTurnIdHash, turnContentFingerprint) become PRIMARY KEY / UNIQUE columns so INSERT … ON CONFLICT DO NOTHING collapses duplicates natively — no separate .idx sidecar.

Migration files live under packages/ledger/src/adapters/migrations/ (shared with the Postgres adapter via small dialect-templating, no ORM).

Implementation notes

  • WAL mode (PRAGMA journal_mode=WAL) for concurrent readers.
  • withLock(name, fn) uses a locks table + BEGIN IMMEDIATE so multiple processes (e.g. ingest + MCP query) on the same file serialize correctly.
  • Streaming query* paths use prepared statements with cursor iteration; no full-result materialization for large datasets.
  • pruneContent becomes a parameterized DELETE FROM content WHERE ….

Files

  • New: packages/ledger/src/adapters/sqlite-adapter.ts, packages/ledger/src/adapters/migrations/001_initial.sql, dialect-templating helper.
  • New: packages/ledger/src/adapters/adapter-suite.test.ts — parameterized contract tests (append/dedup/query/lock semantics) that run against both file and sqlite.
  • Modified: packages/ledger/src/adapters/factory.ts — wire up sqlite branch.

Verification

  • The new parameterized adapter test suite passes against both File and Sqlite (same scenarios, same expectations).
  • Manual: RELAYBURN_STORAGE=sqlite relayburn ingest against the same fixtures used in Phase 1 → relayburn analyze --json produces metrics matching the file-adapter run within rounding.
  • Concurrent-writer smoke test: two relayburn ingest processes against overlapping inputs into the same SQLite file produce no duplicates and no SQLITE_BUSY errors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions