Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
dceb967
feat(forgeteval): ForgetEval-Adv (64 cases) + LLM-optional adapter hook
WaylandYang May 13, 2026
77af629
feat(forgeteval): add LangGraph / Cognee / A-MEM adapters
WaylandYang May 13, 2026
8d4fcb3
feat(forgeteval): hybrid release + LLM-release hook
WaylandYang May 13, 2026
dd83928
feat(forgeteval): ForgetEval-Adv v0.3 — 2 new attack categories
WaylandYang May 13, 2026
17c48e5
feat(forgeteval): ForgetEval-Adv v0.4 — 32 cases added to high-varian…
WaylandYang May 13, 2026
3fe2222
docs: sync forgeteval docs + README to v0.4 numbers
WaylandYang May 13, 2026
2dd7fe0
docs(paper): refresh paper.pdf with v0.4 bench + 4-adapter comparison
WaylandYang May 13, 2026
de25154
docs(paper): paper.pdf — abstract & §6.10 fully synced to v0.4
WaylandYang May 13, 2026
df11de0
feat(forgeteval): refine LLM_PROMPT_SUPERSEDE with explicit bias + fe…
WaylandYang May 13, 2026
b750861
docs: add Lethe+LLM row to README + paper.pdf — 96.4% adversarial
WaylandYang May 13, 2026
174da93
docs(paper): add embedder ablation paragraph to §6.10
WaylandYang May 13, 2026
60d2d2a
chore: gitignore .coverage test artifact
WaylandYang Jun 14, 2026
34050d1
feat(schema): AUTOINCREMENT rowid for one-way-purge invariant
WaylandYang Jun 14, 2026
9bc228f
feat(forgeteval): Adapter Protocol — system-agnostic eval interface
WaylandYang Jun 14, 2026
9966c23
test: depth-physics smoke tests for Lethe v1
WaylandYang Jun 14, 2026
d5a8e47
feat(forgeteval): hand-crafted adversarial layer — 64 cases, 8 catego…
WaylandYang Jun 14, 2026
282d54b
feat(forgeteval): LLM-generated adversarial extension layer
WaylandYang Jun 14, 2026
62a0cbf
docs: add arXiv ID 2606.15903 to README badge + CITATION.cff
WaylandYang Jun 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ _pypi_smoke/
# Recipe outputs — receipts and demo files written when running recipes/
recipes/*.json
alice_purge_receipt.json
.coverage
24 changes: 24 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
cff-version: 1.2.0
title: "Lethe: AI memory built to forget"
message: "If you use Lethe or ForgetEval in your research, please cite the paper below."
type: software
authors:
- family-names: Yang
given-names: Dongxu
affiliation: DeepLethe
repository-code: "https://github.com/deeplethe/lethe"
license: MIT
preferred-citation:
type: article
title: "Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations"
authors:
- family-names: Yang
given-names: Dongxu
affiliation: DeepLethe
year: 2026
journal: "arXiv preprint"
url: "https://arxiv.org/abs/2606.15903"
identifiers:
- type: other
value: "arXiv:2606.15903"
description: "arXiv preprint identifier"
72 changes: 49 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
</p>

<p align="center">
<a href="https://arxiv.org/abs/2606.15903"><img src="https://img.shields.io/badge/arXiv-2606.15903-b31b1b?style=flat-square&logo=arxiv&logoColor=white" alt="arXiv" /></a>
<a href="https://pypi.org/project/pylethe/"><img src="https://img.shields.io/pypi/v/pylethe.svg?style=flat-square&color=blue&label=PyPI&logo=pypi&logoColor=white" alt="PyPI" /></a>
<a href="https://github.com/deeplethe/lethe/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/deeplethe/lethe/ci.yml?style=flat-square&label=tests" alt="tests" /></a>
<img src="https://img.shields.io/badge/python-3.10%2B-3776AB?style=flat-square&logo=python&logoColor=white" alt="python" />
Expand Down Expand Up @@ -107,29 +108,54 @@ production. Pass / fail is exact substring matching on top-k recall,
no LLM judge, deterministic. Full methodology:
**[docs/forgeteval.md](docs/forgeteval.md)**.

| System | super | decay | amnesia | purge | drift | Overall |
|---------------|------:|------:|--------:|------:|------:|------------------------------:|
| **Lethe v1** | 100% | 100% | 98% | 100% | 99% | **99.3%** (993 / 1000) |
| Mem0 (2.0.2) | 100% | 100% | 70% | 75% | 100% | 88.8% |
| MemPalace | 0% | 0% | 0% | 0% | 0% | 0% (no forgetting primitives) |

Mem0 ties on supersession / decay / drift but breaks at the precision
operations: forgetting one entity without bleeding into near-neighbors
(amnesia 70%) and deleting by identifier without over-pruning siblings
(purge 75%). MemPalace's zeros are not a benchmark failure — they
are an honest report that the library was built without `supersede`,
`release`, or `purge`.

In production this maps to real failures. **70% amnesia** means three
in ten *"forget this user"* requests leave fragments reachable to
other queries — a GDPR liability and a stale-context bug. **75%
purge** means one in four deletions either miss the target or take a
neighbor with them — the silent delete-by-similarity failure that
bricks compliance audits. **MemPalace's 0%** is the opposite failure:
a GDPR Article 17 right-to-be-forgotten request becomes a manual
data-migration project, not a one-line API call.

Reproduce: `py bench/forgeteval/run.py --adapter {lethe|mem0|mempalace} --scale 200`
| System | super | decay | amnesia | purge | drift | Overall |
|-------------------|------:|------:|--------:|------:|------:|------------------------------:|
| **Lethe v1** | 100% | 100% | 98% | 100% | 99% | **99.3%** (993 / 1000) |
| LangMem (LangGraph) | 100% | 100% | 98% | 100% | 99% | 99.5% (995 / 1000) |
| Mem0 (2.0.2) | 100% | 100% | 70% | 75% | 100% | 88.8% |
| MemPalace | 0% | 0% | 0% | 0% | 0% | 0% (no forgetting primitives) |

The template suite is near-saturated for vector-store-backed
systems with adaptive eviction: Lethe and LangGraph's `InMemoryStore`
both clear 99 %. Mem0's gap is on amnesia (70 %) and purge (75 %),
the two families that probe width-control and identifier precision.
MemPalace returns 0 because the API has no deletion primitives.

For more discriminative comparison we run **ForgetEval-Adv**, a
112-case hand-crafted layer covering 10 attack categories
(substring traps, prefix collisions, paraphrase supersession,
negation, temporal qualifiers, shared attributes, compound facts,
identifier obfuscation, cross-lingual identifiers, recursive
supersession). See [docs/forgeteval_adversarial.md](docs/forgeteval_adversarial.md).

| System | adversarial overall | wall / case | trade-off shape |
|---------------------|---------------------:|-------------:|------------------------------------------------|
| **Lethe v1** | 70 / 112 (62.5 %) | ~48 ms | 100 % prefix_collision, 0 % cross_lingual |
| Mem0 v2.0.2 | 76 / 112 (67.9 %) | ~527 ms | 50 % prefix_collision, 50 % cross_lingual |
| LangMem (LangGraph) | 69 / 112 (61.6 %) | ~56 ms | 94 % prefix_collision, 0 % cross_lingual |
| MemPalace | 0 / 112 ( 0.0 %) | ~167 ms | no deletion primitives |
| **Lethe + LLM** | **108 / 112 (96.4 %)** | ~2.2 s (mutations only) | 100 % cross_lingual, 100 % shared_attribute; 8 / 10 categories at 100 % |

The Lethe+LLM row uses the optional `llm: Callable[[str], str]`
hook on `LetheAdapter` wired to DeepSeek-V3 via SiliconFlow.
Cost: ~$0.05 for a full 112-case run. The recall hot path
remains LLM-free; only the three mutation operations (`supersede`,
`purge`, `release`) consult the model.

Statistically separated per-category claims at p < 0.05 (non-
overlapping Wilson 95 % CIs at n=16):
**Lethe > Mem0 on prefix_collision** (lexical-precise purge wins);
**Mem0 > Lethe / LangMem on cross_lingual_identifier** (vector-soft
matching wins). Overall Wilson CIs of the three deterministic
systems overlap — the bench reads the trade-off, not a winner.

For attack categories that need semantic understanding the engine
deliberately doesn't provide (compound_fact across all 3 systems,
identifier_obfuscation for Lethe / LangMem), the
`LetheAdapter(llm=...)` hook routes those decisions to an LLM at
mutation time; the recall hot path stays LLM-free.

Reproduce: `py bench/forgeteval/run.py --adapter {lethe|mem0|langmem|mempalace} --suite {template,adversarial}`

> ForgetEval is downstream of the depth model — and the depth model
> is downstream of ForgetEval. A failing `purge_gdpr` case in early
Expand Down
Loading
Loading