Explore whether Dolt should become an optional structured data layer for Agents Remember memory.

[Foxfire1st]

Summary

Explore whether Dolt should become an optional structured data layer for Agents Remember memory.

This should not replace onboarding Markdown. The stronger architecture is:

canonical memory model
  -> validated / stored in Dolt tables
  -> rendered deterministically into Markdown
  -> committed as human-readable durable memory
  -> rebuildable from Markdown as fallback

The likely value is not “database instead of Markdown.” The likely value is using Dolt as an enforcement, query, branch, merge, dashboard, and performance layer underneath the existing by-path onboarding system.

---

Design Philosophy

Agents Remember is already close to a structured memory system:

• file-level onboarding maps 1:1 to source files
• route-local overviews govern mirrored source routes
• generated overview.index.json files provide routing metadata
• onboarding metadata has standardized fields
• references are separated into Docs / Repo-Internal / Cross-Repo sections
• Update History has ordering and append-only rules
• memory quality checks already parse Markdown and enforce style / drift rules

Dolt appears interesting because it can formalize that structure without giving up Markdown openness.

The desired operating model:

Markdown remains human-readable and Git-reviewable.
Dolt provides typed tables, constraints, SQL queries, branch-aware diffs, and structured merges.
MCP tools expose domain operations instead of letting the model hand-edit fragile Markdown conventions.
A deterministic renderer owns Markdown formatting.
A deterministic importer can rebuild the structured store from committed Markdown.

The core rule should be:

Core Agents Remember must still work with Markdown only. Dolt is an optional acceleration, enforcement, and review layer.

---

What was learned about Dolt so far

Based on the initial read of dolthub/dolt:

• Dolt describes itself as “Git for Data”: a SQL database that can fork, clone, branch, merge, push, and pull like a Git repository.
• It can be used through a MySQL-compatible SQL interface.
• Version-control functionality is exposed through SQL system tables, functions, and procedures.
• The CLI exposes Git-like operations such as init, status, add, commit, diff, branch, checkout, merge, clone, fetch, pull, and push.
• Dolt versions tables, not files.
• The README explicitly says Dolt is useful for agent memory, especially multi-agent / multi-machine workflows.
• The local data directory is .dolt/, with data stored under .dolt/noms in Dolt source.
• Dolt is a separate binary, approximately ~100MB according to the README, so it should not become a hard dependency for the core Markdown path.

Useful Dolt references:

• README: https://github.com/dolthub/dolt/blob/main/README.md
• Local .dolt data directory source: https://github.com/dolthub/dolt/blob/main/go/libraries/doltcore/dbfactory/file.go
• Dolt SQL version-control docs: https://www.dolthub.com/docs/sql-reference/version-control
• Dolt SQL procedures: https://www.dolthub.com/docs/sql-reference/version-control/dolt-sql-procedures
• Dolt diff functions / table functions: https://www.dolthub.com/docs/sql-reference/version-control/dolt-sql-functions

---

Current Agents Remember evidence / fit

The current repo already has the important seams:

• README.md defines the core model as durable, git-verified repo memory with by-path onboarding as the core and semantics / relationship providers as optional layers.
• skills/c-05-create-or-update-onboarding-files/templates/file-level-onboarding-template.md defines a canonical file-level onboarding content model with stable metadata fields and stable sections.
• skills/c-05-create-or-update-onboarding-files/SKILL.md already treats onboarding placement, route indexes, update history, and reference sections as structured rules.
• mcp/src/agents_remember/kernel/route_index.py already generates deterministic overview.index.json files from route overviews and sidecars.
• mcp/src/agents_remember/memory_quality/check.py already has a pluggable memory quality check runner.
• mcp/src/agents_remember/memory_quality/style/update_history/history_order.py already parses Markdown to enforce Update History ordering.
• mcp/src/agents_remember/memory_quality/integrity/onboarding_drift_check/discovery.py already parses onboarding table metadata from Markdown.
• mcp/src/agents_remember/controllers/context_packet.py already centralizes repo path, memory root, onboarding root, ledger path, providers, worktree status, and drift status.

That means Dolt would not be adding structure from nowhere. It would formalize structure the project already depends on.

---

Highest-leverage use case

The highest-leverage use case is to stop making agents hand-maintain fragile Markdown structure.

Today the agent has to remember rules like:

Append Update History at the top.
Use the exact section name.
Do not invent reference categories.
Preserve metadata fields.
Keep route indexes generated, not hand-edited.

With a structured layer, agents should call MCP tools like:

onboarding_history_add(...)
onboarding_section_upsert(...)
onboarding_reference_add(...)
onboarding_unit_update_metadata(...)
route_overview_upsert(...)

Then the system enforces:

• section kinds are known enum values
• doc types are known enum values
• reference kinds are known enum values
• source path uniqueness is enforced
• history entries are stored as rows and rendered newest-first
• metadata is complete before render / commit
• route indexes and Markdown tables are deterministic projections

This changes the model from “the LLM must format Markdown correctly” to “the LLM must call constrained domain operations correctly.”

---

Proposed architecture

Storage modes

Start with two explicit modes:

projection mode:
  Markdown -> parser -> Dolt tables
  Dolt is a rebuildable local query cache

structured-write mode:
  MCP operation -> Dolt tables -> deterministic Markdown renderer
  Markdown is committed after render
  manual Markdown edits are detected as out-of-band changes

Projection mode should be implemented first. Structured-write mode should only come after round-trip rendering is reliable.

Suggested memory layout

Do not Git-track .dolt/ by default.

Suggested layout:

ar-memory/
  onboarding/
    ... existing Markdown memory ...
  structured/
    schema.sql
    renderer-version.json
    export/
      onboarding_units.jsonl
      sections.jsonl
      history.jsonl
      references.jsonl
    .dolt/          # generated local Dolt DB, gitignored

Reasoning:

• .dolt/ is itself a versioned database store.
• Committing it into Git would likely create noisy diffs, repo bloat, and nested version-control confusion.
• Markdown + schema + deterministic exports should be enough to rebuild the Dolt database.
• If multi-machine structured sync is needed later, use Dolt’s native push / pull / clone model instead of storing .dolt/ in Git.

---

Possible schema sketch

Initial schema can be small:

repository (
  repo_id primary key,
  memory_root,
  onboarding_root,
  schema_version
);

onboarding_unit (
  unit_id primary key,
  repo_id,
  source_path,
  onboarding_path,
  doc_type,
  governing_overview_path,
  last_updated,
  last_verified_commit_hash,
  last_verified_commit_date,
  unique(repo_id, source_path, doc_type)
);

onboarding_section (
  unit_id,
  section_kind,
  body_markdown,
  ordinal,
  primary key(unit_id, section_kind)
);

history_entry (
  entry_id primary key,
  unit_id,
  timestamp,
  code_commit_hash,
  summary,
  actor,
  supersedes_entry_id
);

reference_entry (
  reference_id primary key,
  unit_id,
  reference_kind, -- docs | repo_internal | cross_repo
  finding,
  citations,
  source_path,
  health_status,
  last_checked_at
);

route_overview (
  route_id primary key,
  repo_id,
  source_route,
  overview_path,
  hot_path_summary
);

Likely future tables:

entity_catalog_entry
route_index_snapshot
memory_quality_finding
reference_health_check
provider_retrieval_event
structured_memory_revision

---

Proposed MCP tools

Bootstrap / maintenance

memory_db_status(repo_id)
memory_db_init(repo_id, dry_run=true)
memory_db_rebuild(repo_id, dry_run=true)
memory_db_validate(repo_id)
memory_db_export(repo_id)

Read tools

memory_unit_get(repo_id, source_path)
memory_section_get(repo_id, source_path, section)
memory_history_get(repo_id, source_path, limit=10)
memory_references_get(repo_id, source_path, kind=null)
memory_route_context_get(repo_id, source_path)
memory_coverage_report(repo_id)

Write tools, later only

onboarding_unit_create(...)
onboarding_unit_update_metadata(...)
onboarding_section_upsert(...)
onboarding_reference_add(...)
onboarding_reference_update_health(...)
onboarding_history_add(...)
route_overview_upsert(...)
entity_catalog_entry_upsert(...)

Important constraint:

Do not expose arbitrary SQL to the model as the normal interface. Expose domain-specific MCP tools backed by SQL.

---

Retrieval optimization

Dolt can become a fourth retrieval substrate alongside the existing router concepts:

Intent        -> path/onboarding truth
Semantics     -> concept search
Relationship  -> graph / call / dependency context
Structured    -> exact schema-backed memory query

Examples:

-- Get only invariants for a file instead of reading the whole sidecar.
select body_markdown
from onboarding_section
join onboarding_unit using (unit_id)
where repo_id = ?
  and source_path = ?
  and section_kind = 'invariants';

This should reduce token load because agents can fetch exact sections instead of whole Markdown files.

---

Branch / worktree optimization

Dolt is especially interesting because Agents Remember already has branch / worktree / ledger semantics.

Potential mapping:

Git code branch / worktree
  -> matching memory Git branch
  -> matching Dolt branch
  -> structured memory edits
  -> Markdown render
  -> Git memory commit
  -> Dolt memory commit
  -> closeout merge / carryover

Potential tools:

memory_branch_start(repo_id, code_branch)
memory_branch_status(repo_id)
memory_branch_diff(repo_id, base="main", head="task-branch")
memory_branch_merge_preview(repo_id)
memory_branch_closeout(repo_id)

Potential closeout review queries:

Show all invariant changes introduced by this task branch.
Show all cross-repo references changed since main.
Show all onboarding units whose lastVerifiedCommitHash changed.
Show the memory delta grouped by source route.
Show all reference health statuses that worsened.

This would be much more reviewable than raw Markdown diffs alone.

---

Dashboard / operations benefits

Dolt-backed views could power dashboard panels such as:

files missing onboarding
files with stale verification
history entries by route
cross-repo references by repo
docs references needing health checks
onboarding units changed by task branch
routes with low coverage
most-edited memory sections
memory churn over time
structured closeout diff

This would avoid repeated filesystem crawling and Markdown regex passes for dashboard/quality-control operations.

---

Implementation path

Phase 0 — feasibility spike

Goal: prove Dolt can be installed/discovered and initialized safely as an optional provider.

Likely tasks:

• Add optional Dolt detection.
• Add config gate, for example:

{
  "structuredMemory": {
    "enabled": true,
    "provider": "dolt",
    "mode": "projection",
    "dataRoot": "structured/.dolt"
  }
}

• Add schema.sql draft.
• Add memory_db_status and memory_db_init.
• Ensure failure is graceful when dolt is missing.
• Ensure core Markdown-only Agents Remember still works without Dolt.

Phase 1 — read-only structured projection

Goal: build Dolt tables from existing Markdown and query them.

Likely tasks:

• Parse current file-level onboarding metadata from Markdown.
• Parse stable sections into onboarding_section rows.
• Parse Update History into history_entry rows.
• Parse Docs / Repo-Internal / Cross-Repo reference tables into reference_entry rows.
• Mirror route overview / route index data into SQL views.
• Add query tools:
  • memory_section_get
  • memory_history_get
  • memory_references_get
  • memory_coverage_report
• Add tests against fixtures and, ideally, this repo’s own memory repo.

Phase 2 — deterministic Markdown renderer

Goal: prove DB -> Markdown render is stable before allowing agents to write structured memory.

Likely tasks:

• Implement renderer for canonical file-level onboarding.
• Implement renderer for route overview metadata if appropriate.
• Add round-trip test:

Markdown -> parse -> Dolt rows -> render -> normalize -> compare

• Define normalization rules for whitespace, empty tables, placeholder text, and section ordering.
• Keep Markdown output reviewable and close to current templates.

Phase 3 — schema-first writes

Goal: allow agents to write structured memory through constrained MCP tools.

Likely tasks:

• Add write tools for metadata, sections, references, and history.
• Add transaction wrapper:

validate payload
write Dolt rows
render Markdown
parse rendered Markdown
compare parsed rows
run memory quality checks
commit/update files only if clean

• Add out-of-band edit detection when Markdown has changed without corresponding structured state.

Phase 4 — branch/worktree integration

Goal: align Dolt branches with task/worktree memory branches.

Likely tasks:

• Sanitize Git branch names to Dolt branch-safe names.
• Create / checkout matching Dolt branch during task start.
• Include structured memory status in context_packet.
• Add branch diff / merge preview tools.
• Integrate with carryover / closeout.

Phase 5 — dashboard and operations layer

Goal: expose high-value SQL-derived operational views.

Likely tasks:

• Add dashboard-oriented query endpoints.
• Add memory health trend views.
• Add coverage-by-route views.
• Add structured closeout summaries.
• Add benchmark metrics for token savings and retrieval precision.

---

Acceptance criteria for the first useful spike

A good first PR should prove:

• Dolt is optional and disabled by default.
• memory_db_status(repo_id) reports whether Dolt is available and whether a structured DB exists.
• memory_db_rebuild(repo_id) can build a local Dolt DB from existing onboarding Markdown.
• At least metadata + sections + Update History are represented in tables.
• memory_section_get(repo_id, source_path, section) can return a section without reading the whole Markdown file.
• The DB can be deleted and rebuilt from committed Markdown.
• No existing Markdown-only workflow regresses.

---

Risks / non-goals

Risks

• Dolt is an additional binary and operational dependency.
• Tracking .dolt/ in Git would likely be a mistake unless proven otherwise.
• Dual-source drift is dangerous if Markdown and Dolt are both editable authorities.
• Branch-name mapping may need care.
• The renderer must be deterministic enough to avoid noisy Markdown churn.
• Arbitrary SQL exposed to agents could bypass domain invariants.

Non-goals for the first spike

• Do not replace Markdown.
• Do not require Dolt for normal Agents Remember usage.
• Do not expose raw SQL as the main agent-facing API.
• Do not implement structured writes before read-only projection and round-trip tests work.
• Do not commit .dolt/ into the memory repo by default.

---

Open questions

• Should structured/ live under every memory root, or under ar-coordination/providers/structured-memory/?
• Should deterministic JSONL exports be Git-tracked, or only schema.sql + Markdown?
• Should Dolt be treated as a provider, a memory-quality subsystem, or a core-but-optional structured memory layer?
• Should route indexes remain JSON files only, or become SQL views with JSON files rendered from SQL?
• What is the minimum schema needed before this becomes useful for dashboard work?
• Is Dolt’s branch/merge functionality worth integrating early, or should it wait until structured-write mode?

---

Current recommendation

Start with read-only projection mode.

Do not start with schema-first writes. First prove that current onboarding Markdown can be parsed into stable tables and queried usefully. Then prove deterministic rendering. Only then allow agents to write structured memory through MCP domain operations.

The ideal first milestone is:

Markdown remains source/fallback.
Dolt is generated locally.
Agents can query exact memory sections through MCP.
No existing workflow changes.

If that works, Dolt can later become the structured enforcement layer for memory writes, branch-aware memory diffs, dashboard views, and multi-agent closeout review.

[Foxfire1st]

Follow-up: Dolt as the enforcement layer, not just a projection cache

After the follow-up design discussion, the direction is clearer: the highest-value use of Dolt is not merely read-only projection from Markdown. The stronger architecture is to make Dolt the structured enforcement layer for memory writes, with Markdown as the deterministic human-readable publication artifact.

Proposed operating model:

agent intent
  -> constrained MCP memory-write tool
  -> typed Dolt transaction
  -> schema / constraint / invariant validation
  -> deterministic onboarding Markdown render
  -> memory quality gate
  -> Git-reviewed durable Markdown commit

The key shift is that agents should stop hand-editing fragile Markdown conventions. They should call domain-specific MCP tools such as:

onboarding_unit_create(...)
onboarding_section_upsert(...)
onboarding_reference_add(...)
onboarding_history_append(...)
onboarding_verification_update(...)
route_overview_upsert(...)

Dolt then becomes the place where the onboarding model is enforced through typed tables, required fields, uniqueness constraints, foreign keys, enum-like check constraints, transactions, and explicit validation queries. If an agent violates the memory model, the write should fail loudly before any Markdown is produced.

Provider/runtime model

Dolt should likely run like the other optional providers:

• Dockerized runtime, not a required host binary.
• Enabled/configured through the existing provider/settings model.
• Worktree/task isolation can map to provider runtime isolation later.
• The Dolt data directory can be mounted under the memory repo or memory-provider area, but this needs an explicit storage decision.
• Markdown output remains the Git-reviewed artifact.

The important rule remains: no dual authority. If structured-write mode is active, Markdown should be rendered output. Manual Markdown edits are out-of-band and need explicit reconciliation/import tooling rather than silent fallback behavior.

What Dolt adds beyond SQLite/JSON indexes

For simple exact reads, SQLite or JSON could work. Dolt becomes interesting because it combines structured SQL writes with versioned table state. That gives Agents Remember a review model based on memory meaning instead of only Markdown line diffs.

Dolt can expose row-level diffs between branches/commits, for example through dolt_diff('main', 'task-branch', 'onboarding_section'). Agents Remember can then interpret those row diffs through the onboarding schema and present semantic deltas such as:

Changed invariants
Added cross-repo references
Removed docs references
Verification metadata updates
Update History entries appended
Route coverage changes
Schema changes that should not have happened

This only works if the schema is granular enough. Storing a whole Markdown file as one blob would not provide meaningful semantic deltas. Storing onboarding units, sections, references, history entries, verification metadata, and route coverage as typed rows would.

Example review query shape:

select
  coalesce(to_source_path, from_source_path) as source_path,
  diff_type,
  from_body_markdown,
  to_body_markdown
from dolt_diff('main', 'task-branch', 'onboarding_section')
where coalesce(to_section_kind, from_section_kind) = 'invariants';

The MCP layer can turn that into a reviewer-facing summary:

Changed invariants:
- mcp/src/.../context_packet.py
  modified provider skipped-state invariant

Obvious advantages

• Bad memory writes fail loudly instead of becoming quiet Markdown rot.
• The renderer owns formatting, ordering, tables, headings, and route indexes.
• Agents interact with memory through intent-shaped tools, not broad text editing.
• Domain search becomes possible: exact sections, stale references, changed invariants, missing verification, route coverage, cross-repo contracts.
• Review becomes structured: memory deltas can be grouped by semantic category rather than Markdown file diff alone.

Less obvious advantages

• The schema becomes executable doctrine. Current prose rules become constraints and validation checks.
• The MCP tools become the stable public memory API, reducing model dependence on instructions.
• Memory writes can be transactional: validate, render, parse-back/check, quality-gate, and publish only if clean.
• Conflict handling can move from paragraph-level Markdown conflicts to row-level memory-model conflicts.
• Dashboards become cheaper and more honest because coverage, churn, stale references, and route health are queryable.
• Prompt-injection and accidental format drift risk is reduced because memory mutation flows through constrained tools.
• The same structured model can later power a dashboard, onboarding editor, closeout preview, and review UI.

Revised first milestone

The first useful spike should probably prove a tiny structured-write loop, not only read-only projection:

1. Add optional Dolt provider status/detection through the provider/runtime model.
2. Define a minimal schema for onboarding units, sections, history, references, and verification metadata.
3. Implement one or two constrained MCP write tools, for example onboarding_section_upsert and onboarding_history_append.
4. Enforce required fields, known section/reference kinds, uniqueness, and ordering rules in the structured layer.
5. Render one deterministic file-level onboarding Markdown document from Dolt rows.
6. Parse/check the rendered Markdown enough to prove round-trip stability.
7. Show a semantic diff between two Dolt revisions/branches, ideally changed invariants or added references.
8. Keep normal Markdown-only Agents Remember behavior available when the Dolt provider is disabled.

Non-goals for that spike:

• Do not expose arbitrary SQL to agents as the normal interface.
• Do not replace human-readable Markdown review.
• Do not wire Dolt merge behavior into closeout before the simple write/render/review loop is proven.
• Do not create silent compatibility/fallback write paths that make authority ambiguous.

This narrows the question from “should Dolt be a cache under Markdown?” to “can Dolt make onboarding writes sound by construction, then publish deterministic Markdown and semantic review deltas?”