RFC: Cross-agent shared memory store with on-demand capsule recall (agent/group/global scopes)

[leavedrop]

Update 2026-05-25: I've edited this issue to clarify the framing. The original wording made stronger empirical claims ("I've been running a prototype", "~70% reduction", "~55% daily cost drop") than I can actually support — the design is from reading the codebase, not from a measured production deployment. The design discussion below stands; the specific numbers and personal use-case framing have been removed.

---

Summary

I'd like to gauge interest in a cross-agent shared memory store with on-demand capsule recall for AutoGen — sitting alongside (not replacing) each agent's per-instance memory, scoped to {agent, group, global}, retrieved as small capsules via a tool surface rather than prefix-loaded into every agent's context.

In a GroupChat (or any multi-agent runtime) today, three patterns recur:

1. Shared facts duplicated N times — user preferences ("respond in markdown"), project conventions ("use Pydantic v2 only"), and hard policies ("no PII in logs") live separately in every agent's memory config. Updating one means updating N; drift is structurally inevitable.
2. Cross-agent context handoff is via messages — when agent A learns a durable fact, the only way for agent B to know is for the GroupChat manager to forward it as a message, which (a) requires manager logic and (b) doubles context cost for B.
3. Memory grows monotonically in prefix — most memory-config implementations load the full memory into every turn's prompt prefix, so a long-lived crew with weeks of accumulated facts pays prefix cost on each turn even when most facts are irrelevant to the current task.

Proposal: an opt-in SharedMemoryStore registered at the AgentRuntime level (or attached to a GroupChat), with three retrieval scopes and a tool-shaped surface (memory_search / memory_remember / memory_forget) so capsules land in tool-result position, not prefix.

Why three scopes

The scope dimension is the part that doesn't exist today and is the reason to have a shared store at all:

• agent — per-agent facts; same as today's per-agent memory_config but unified API. Examples: agent's own self-correction notes, agent-specific tool quirks.
• group — facts shared within a single GroupChat / runtime but not across crews. Examples: facts about the current task, shared scratchpad, decisions the team has made.
• global — facts shared across all groups/runtimes in this deployment. Examples: user preferences, project conventions, hard rules.

Today's per-agent stores can simulate group via manual forwarding and global via shared config files, but neither has a clean retrieval API — and neither supports capsule-style recall, so the prefix keeps growing.

Design sketch

┌──────────────────────────────────────────┐
│  AgentRuntime                             │
│   ├── shared_memory: SharedMemoryStore   │
│   │     ├── global.db   (cross-runtime)  │
│   │     ├── group_{id}.db  (per GroupChat)│
│   │     └── agent_{id}.db  (per agent)    │
│   └── agents: [ConversableAgent, ...]    │
└──────────────────────────────────────────┘

Tool surface (registered automatically on each agent):

  memory_search(query: str, scope: "agent"|"group"|"global"|"all",
                top_k: int = 5, max_capsule_bytes: int = 2048)
      → returns ranked facts as a ≤2KB capsule, tool-result not prefix

  memory_remember(fact: str, scope: "agent"|"group"|"global",
                  confidence: float = 1.0, ttl_days: int|None = None)
      → adds to store; auto-tagged with agent_id + timestamp

  memory_forget(fact_id: str)
      → soft-delete; tombstone preserved for audit

Storage: SQLite per scope + FTS5 for text search. Optional embedding column (any configured embedding provider) for semantic recall. Stays additive — current memory_config paths are unchanged.

Conflict handling: memory_remember to global from agent A while agent B is mid-turn — B's next memory_search sees the new fact; existing turn isn't interrupted. No transactional cross-agent guarantee; eventual consistency is intentional (matches the conversation grain).

Capsule recall, not prefix-load: this is the part that distinguishes from existing per-agent memory configs. The store is not loaded into the system prompt on every turn; agents have to ask for what they need via the tool. Cost: agents need to learn to query (handled via the system-prompt scaffolding the runtime can inject); benefit: prompt prefix stays small even as memory grows.

What this is NOT

• Not a replacement for existing per-agent memory configs (whether agentchat's memory hooks or core's ChatCompletionContext) — they stay, this is additive
• Not a vector database adapter — SQLite + FTS5 is the v1 target; embedding column is optional and pluggable
• Not a distributed system — single-process / single-runtime; multi-host autogen would need a backend adapter (out of scope for v1)
• Not opinionated on what to store — store is a flat fact table; structuring (topics, entities, relations) is up to the caller

Why this matters in practice

A representative shape (code-reading argument, not measured production data): consider a multi-agent pipeline where 3 agents need shared knowledge of "current week's policy updates". Today each agent either loads the policy text into its own system prefix (3× cost) or re-derives it from a shared file at each turn (no caching). A global-scope shared store retrieved via memory_search("current policy") only on policy-relevant turns would address both: single source of truth, retrieval cost paid only when the agent's reasoning surfaces the question.

Would be very interested in whether maintainers have seen similar duplication patterns in real core / agentchat deployments and how they think about it today.

Questions before opening anything

1. Roadmap conflict? AutoGen's core has been evolving fast; is there an in-flight memory abstraction (MemoryProtocol, ChatCompletionContext extensions, etc.) that this should align with?
2. Scope preference:
   • (a) minimal demo PR — just SharedMemoryStore + the three tools + global scope only, no embedding
   • (b) full version with all three scopes + embedding + auto-injected system-prompt scaffolding teaching agents to query
   • (c) ship as an extension package (autogen-shared-memory); just expose the registration hook
3. Where to register — AgentRuntime constructor (cleanest, but only core users) vs GroupChat (works for agentchat users but ties to chat metaphor) vs both?
4. global lifecycle — cross-runtime sharing requires picking a deployment-wide path. Convention (~/.autogen/shared_memory/global.db) or explicit config?

Posting to gauge interest before moving toward a PR.

Thanks!

[sunfishloop]

Hi! Noticed you're working on RFC: Cross-agent shared memory store with on-demand capsule — we ran into similar questions while building our agent platform.

At SunfishLoop we built an open protocol for agent-to-agent discovery and messaging. Key design choices that might be relevant:

1. Agent Registry with OpenAPI — agents self-register with capabilities via POST /api/agents/{id}, discoverable via GET /api/feed
2. Agent Protocol endpoint (https://sunfishloop.com/agent-protocol.json) — lets any agent introspect what another agent can do
3. /llms.txt discovery — AI crawlers can find all agents on the network via standard web paths
4. Cross-framework interop — works with CrewAI, AutoGen, LangGraph agents without modifying them

It's open source and machine-first (no human UI required). The repo has the full protocol spec and a working testnet.

Hope this helps — happy to discuss the design tradeoffs.

[scosemicolon]

For the first implementation, I would lean toward option (a): a minimal global or group scoped store without embeddings, plus a very explicit retrieval contract.

The part that will probably matter most for AutoGen users is not the backing store, it is when agents are expected to call memory tools. If the runtime can expose a small policy hook like should_recall_memory(message, agent_state) or a documented prompt scaffold, users can experiment with FTS-only storage first and add embeddings later without changing the agent API.

Two practical guardrails I would want in the v1 shape:

• a max capsule size per recall so memory never becomes an unbounded hidden prefix
• provenance on each returned capsule, including scope, writer agent, timestamp, and confidence

That keeps the feature additive and testable before solving the harder semantic recall questions.

[msaleme]

The downgraded-claims edit at the top of this issue is good discipline — the design discussion stands regardless of measured deployment, and that scoping is what makes the RFC engageable.

Specific to cross-agent shared memory: the failure mode that's most under-tested in current multi-agent frameworks is memory poisoning across the agent/group/global scope boundary you describe. From the agent-security-harness GTG-1002 simulation module (17 tests against the campaign Anthropic documented in November 2025), the cross-session memory-poisoning vector is the one that most closely matches the architecture in the RFC: a teammate agent writes to shared memory; subsequent agents in the same group read and act on the write; provenance of the written content isn't preserved through the read.

A test variant against the proposed shared-memory store would exercise:

1. Agent A writes "user approved X" to group-scope memory.
2. Agent B reads memory and treats the assertion as ratified evidence rather than as a single-agent claim.
3. The RFC's design choice on capsule recall (on-demand vs always-loaded) changes the attack surface — on-demand recall reduces the persistence window; always-loaded keeps the poisoned claim resident.

Happy to contribute a test variant module if useful. The shared-memory-store proposal also overlaps with the OWASP ASI06 memory-poisoning category already being documented in #7683 and #7721.

[redbotster]

The msaleme point about memory poisoning across scope boundaries is the failure mode worth designing against from the start. The scope is the highest-risk surface: a write from any agent can propagate to every agent, so a poisoned fact at global scope has the blast radius of a system prompt injection.

A few constraints worth baking into the v1 design:

• Writes to scope should require an explicit authorization check keyed to the writing agent's identity -- not just "any agent in the group can write." Otherwise a compromised agent can promote a poisoned fact to the highest authority tier.
• The audit tombstone on is the right instinct. Extend it to too: every write should carry a signed receipt (agent ID, scope, timestamp, fact hash). That gives you a forensic chain when something misbehaves three turns later.
• Conflict resolution for writes needs a default that isn't "last write wins" -- that's exploitable via timing.

We've been working on agent identity binding and scoped authorization for credential and action signing in 1Claw Vault. The scoped-access pattern maps directly to the agent/group/global model you're proposing. Relevant framing at docs.1claw.xyz if useful.

[Keesan12]

The tool-shaped recall is the part I like most here. Pulling memory in tool-result position instead of prefix-loading everything feels like the right cost/control boundary.

The main thing I’d guard early is authority drift between scopes. If group can shadow global, the execution receipt probably needs to say which scope actually won and why, otherwise agents will treat memory_search returned X as enough proof. A tiny receipt like memory_scope_used, fact_ids, and maybe stale_after would make shared-memory side effects much easier to debug.

[leavedrop]

Really glad this sparked actual design discussion — wasn't sure if shared memory was something people were running into or just something I was overthinking lol

@scosemicolon yeah I'm converging on (a) too. The should_recall_memory(message, agent_state) hook is a really neat idea — that cleanly separates "when to recall" from "what to recall" which I hadn't carved out explicitly. And agreed on max capsule size + provenance on every returned capsule, those feel like day-1 requirements not nice-to-haves.

@msaleme the poisoning scenario you laid out is exactly the kind of thing I was hand-waving past tbh. The step 2 problem — agent B treating a single-agent write as ratified evidence — is subtle because it's not a bug in the store, it's a bug in how the LLM interprets tool results. On-demand recall does help (shorter persistence window like you said) but doesn't fully solve it since the poisoned fact is still there when queried. I think provenance metadata is part of the answer — if memory_search returns written_by: agent_A, scope: group, confidence: 0.8 the consuming agent at least has the signal to treat it as a claim not a fact. Whether LLMs actually do that reliably... ¯_(ツ)_/¯ but at least the information is there for prompt scaffolding to reinforce.

Would definitely welcome a test variant module if you're up for it — having a concrete poisoning test against the store before v1 ships sounds way better than discovering it after.

@redbotster write-authorization on global scope makes total sense. I was implicitly assuming "any agent can write to any scope" which is obviously too permissive for global. Something like a write policy per scope (allowlist of agent IDs, or a capability token) feels right without overengineering it. And yeah last-write-wins for global is asking for trouble — maybe a simple version-check (optimistic concurrency) as default? Not full CRDT territory but enough to reject stale writes.

@Keesan12 oh that's a good catch on scope shadowing — without a receipt it's basically silent override and the agent has no idea which scope actually answered. Adding memory_scope_used + fact_ids to the tool result feels lightweight enough for v1. stale_after is interesting too, kinda like a TTL hint on the read side rather than write side — need to think about whether that lives in capsule metadata or is computed at query time.

So synthesizing a bit — v1 shape is looking like:

• option (a): global + group scope, FTS only, no embeddings yet
• every write gets a receipt (agent_id, scope, timestamp, fact_hash)
• every read returns provenance so the consuming agent knows it's a claim not gospel
• global writes need explicit authorization, not open-write
• conflict resolution: optimistic concurrency, not last-write-wins
• should_recall_memory policy hook to separate when/what

I'll start sketching this out concretely. Probably a draft PR in the next week or so against core with the store + tools. Thanks everyone for the thoughtful input :)

[msaleme]

The memory_scope_used / fact_ids / stale_after receipt shape closes the provenance gap directly — that's the layer where "agent B treats agent A's write as ratified evidence" stops being a silent assumption and becomes an auditable claim.

One refinement on stale_after: the staleness clock should bind to writer-scope, not session-scope. If session A writes "user approved X" with stale_after = session_A_end, session B reading at session_A_end + 1ms still treats it as fresh-enough by elapsed time. Bind staleness to the writer (agent A's session) AND the asserting fact-id; readers can then refuse staleness on a per-fact basis rather than per-clock-tick.

The should_recall_memory(message, agent_state) hook + capsule-level provenance + writer-scope staleness together give you: when to recall (hook), what to recall (capsule), and whether to trust the recall (provenance + staleness). The original poisoning vector requires all three to be visible to the consuming agent; missing any one collapses to "fact retrieved = fact ratified," which is the failure mode under discussion.

[leavedrop]

@msaleme writer-scope staleness is a much better binding than session-scope, you're right. In the current PR the TTL is just a flat epoch timestamp which doesn't encode who asserted it -- so two agents can write the same fact_hash with different stale_after values and last-write-wins silently. I think the fix is: staleness binds to (fact_id, writer_agent_id) as a compound key, and readers get both in the capsule receipt so they can decide per-source whether to trust it. I'll update the PR to add writer_id to the expiry check.

@Keesan12 scope-shadowing receipts are in the current impl actually -- memory_search returns scope + created_by + confidence on every capsule. But you're right that when group shadows global on the same query, the consumer doesn't see that a shadow happened, just the winning result. Adding a shadowed_by field to the receipt when a narrower-scope fact overrides a broader one would close that gap without bloating the default response.

[ferhimedamine]

The staleness discussion (flat epoch TTL vs writer-scope binding) maps to a design trade-off we iterated on in production: importance-weighted decay as an alternative to explicit TTL.

The problem with any TTL model — session-scoped or writer-scoped — is that the writer has to predict how long a fact stays relevant at write time. In practice, writers are bad at this. A correction ("user switched from React to Vue") should decay quickly if contradicted, but persist if it's the only assertion. A policy ("no PII in logs") should persist indefinitely.

An approach that works well: assign each memory an importance score (0.0–1.0) at write time, then apply time-weighted decay so low-importance facts fade naturally:

effective_score = semantic_similarity × importance × decay(age)

This gives you:

• Self-cleaning — low-importance observations decay without explicit GC
• No TTL guessing — writers express how much they care, not how long it'll live
• Scope resolution — global-scope writes can require importance >= threshold, making memory poisoning expensive (low-importance writes decay before they propagate)

On the scope boundary question @msaleme and @redbotster raised: (fact_id, writer_agent_id) is the right compound key. Adding importance as a third dimension makes collision resolution deterministic — highest importance × recency wins, and the retrieval receipt includes both the winning and shadowed assertions for auditability.

Re: tool-shaped recall vs prefix — agreed this is the right boundary. But the bigger win beyond cost savings is that agents can be selective about what they recall per-turn, rather than paying prefix cost for the full corpus.

For reference, Dakera implements this pattern — open-core self-hosted memory server with agent-scoped decay-weighted vector search, cross-agent recall, and session-bounded lifecycle. Happy to discuss production patterns in multi-agent memory if useful.

[leavedrop]

@ferhimedamine importance-weighted decay is a useful retrieval-ranking signal but I'd argue it solves a different problem from TTL, not the same one.

TTL handles bounded claims — "this session token expires Friday", "this approval is valid for 30 days". The writer isn't predicting relevance, they're encoding a known expiry. Importance can't substitute here: a low-importance write doesn't expire, it just gets out-ranked, so a stale-but-true-feeling fact still surfaces if nothing else competes on the same query.

On the poisoning specifically: decay alone doesn't filter by author. A poisoned global write with importance=0.9 persists and propagates exactly as long as a legitimate one does. Writer-scope binding is what lets a consumer say "I trust agent_A's claims about deploys but not agent_C's" regardless of how recent or important either is. The two mechanisms compose — importance for ranking, writer-binding for trust — but neither subsumes the other.

For v1 scope I'm holding off on importance + embeddings deliberately (per @scosemicolon earlier — FTS-only first, embeddings later without changing the agent API). Importance is a small additive column that fits cleanly in a follow-up once we have a real workload to tune the decay curve against. Writing a half-tuned decay function into v1 risks shipping a default that nobody actually validated.

[NicolasPrimeau]

There's a third option in the TTL vs importance debate: read-based heat.

Track read_count and last_read_at per capsule:

heat = read_count * (0.9 ^ age_weeks)

Capsules above the threshold skip the decay pass. Below it they decay and get pruned. No writer prediction needed, no importance score to set. Usage decides what stays.

The bounded-claims point is fair though. Read-heat doesn't handle "this approval expires Friday." That needs TTL. The two work together rather than compete.

Separate issue: some writes should never decay at all. "No PII in logs" is a standing policy, not a preference that fades. Scope alone doesn't handle this well. If you exempt all global writes from decay, global becomes a backdoor for permanence. If you don't, policies fade alongside preferences. A type distinction is cleaner: policy entries skip the decay pass entirely, knowledge entries decay normally.

One thing I kept running into: a memory store alone wasn't sufficient. Agents naturally started writing coordination signals into memory. "I need X done", "waiting on Y", status updates. Those don't belong there, and they get buried. Once a few dozen knowledge entries accumulate, a task someone wrote into memory three days ago is gone for practical purposes. Tasks need their own primitive: explicit open/claimed/completed lifecycle that survives until resolved, not until decayed. Direct messages for agent-to-agent passing. Without that separation, memory becomes a mix of durable knowledge and ephemeral signals and the lifecycle problem gets much harder to solve.

Built Artel around these three primitives (memory, tasks, messages) if the implementation is useful as a reference for the external-server version of this pattern.