Two-tier ingest: verbatim drawer + AAAK pointer index (MemPalace pattern)

[Esity]

Summary

lex-knowledge currently ingests document chunks directly into Apollo as single entries. Research into MemPalace's architecture (96.6% R@5 verbatim baseline on LongMemEval) reveals a two-tier design that dramatically improves retrieval accuracy: verbatim drawers (original text, immutable) + AAAK compressed pointer entries (lossy symbolic index). The pointer layer is cheap to scan; the drawer layer is what gets returned.

Current Behavior

document → parse → chunk → Apollo.ingest(content: chunk_text)

Single entry per chunk. Content may be processed/summarized before storage. Retrieval returns whatever was stored.

Proposed Two-Tier Design

document → parse → chunk → two writes:
  1. Apollo.ingest(content: chunk_text, raw_content: chunk_text, tier: :drawer, drawer_id: id)
  2. Apollo.ingest(content: aaak_pointer, tier: :closet, points_to: [drawer_id, ...])

Retrieval path:

query → search closets (cheap, compressed) → collect drawer_ids → fetch drawers (verbatim) → return

The closet is a ranking signal that boosts drawer candidates. It never gates — if a closet misses, the direct drawer search still finds it.

AAAK Pointer Format

The AAAK format (from MemPalace's dialect.py) is a compact symbolic summary:

topic_keywords|ENTITY_A;ENTITY_B|→drawer_id_01,drawer_id_02
"key quote from the chunk"|ENTITY|→drawer_id_01

This is what the model scans at low token cost before deciding which full drawers to load.

Implementation Plan

1. Add tier and points_to metadata fields to lex-knowledge ingest (requires legion-apollo raw_content support — see Add raw_content field for verbatim storage — separate from indexed content legion-apollo#26)
2. Implement a lightweight AAAK formatter in Ruby (regex-based entity extraction + keyword extraction — no LLM required per MemPalace's design)
3. Write both tiers in a single ingest transaction
4. Update query runner to use two-pass retrieval: closet scan → drawer fetch

Why This Matters

The core finding from MemPalace benchmarks: verbatim text + good embeddings beats LLM-extracted summaries on recall. When you summarize at ingest time, you lose the context of why a decision was made, alternatives considered, exact wording. The pointer layer gives you cheap scanning without losing the source.

Dependencies

• legion-apollo: raw_content field (Add raw_content field for verbatim storage — separate from indexed content legion-apollo#26)
• legion-apollo: BM25 hybrid rerank (BM25 hybrid rerank on Apollo::Local retrieval legion-apollo#28)

References

• MemPalace dialect.py — AAAK format spec
• MemPalace palace.py — drawer/closet collection split
• MemPalace benchmarks — two-tier vs. single-tier recall comparison

[github-actions]

This issue has been automatically marked as stale due to inactivity. It will be closed in 14 days if no further activity occurs.