diff --git a/docs/enhancements/enh-005-directional-graph-expansion.md b/docs/enhancements/enh-005-directional-graph-expansion.md new file mode 100644 index 0000000..dae1ac3 --- /dev/null +++ b/docs/enhancements/enh-005-directional-graph-expansion.md @@ -0,0 +1,228 @@ +# enh-005: directional graph expansion for GraphRAG + +> Improves a *shipped* feature (feat-023, GraphRAG hybrid retrieval). Not a +> defect — graph expansion works as designed — this adds a `direction` knob so +> expansion can follow **asymmetric** edges the right way (callers vs callees, +> who-cites vs what-it-cites). It stays entirely within the locked `GraphStore` +> contract, which already exposes `get_edges(direction=...)`, so there is **no +> ABC change**. + +--- + +## Metadata + +| Field | Value | +|---|---| +| **ID** | enh-005 | +| **Title** | `direction` on `GraphExpansion` (in / out / any) | +| **Status** | accepted (targeted at the 0.4 train) | +| **Owner** | kjoshi | +| **Created** | 2026-06-17 | +| **Target version** | 0.4 | +| **Languages** | `python` (TS deferred) | +| **Improves** | feat-023 (GraphRAG hybrid retrieval) | +| **Module package(s)** | `agentforge-core` (`GraphExpansion` value + config schema), `agentforge` (`Retriever` expansion) | + +--- + +## 1. Why this enhancement + +feat-023 expands a retrieved seed by walking the graph up to `max_hops`, +optionally filtered to `edge_types` (so expansion is already **typed**). But it +walks those edges **without a direction** — `GraphStore.traverse()` has no +direction parameter, so expansion treats every edge as undirected. + +For **symmetric** relationships (`RELATED_TO`, `SIMILAR`) that's fine. For +**asymmetric** ones it loses precision, because the two directions answer +different questions: + +| Relationship | Out-edges (`X → ?`) | In-edges (`? → X`) | +|---|---|---| +| `CITES` | what X cites | who cites X | +| `CALLS` | what X calls (callees) | who calls X (callers) | +| `IMPORTS` | X's dependencies | X's dependents | +| `REPORTS_TO` | X's manager chain | X's reports | + +Today an agent asking "who cites this paper?" gets X's *own* citations mixed in +— noise that dilutes the candidate set and the reranker's budget. The fix is a +single **`direction`** control on the expansion policy. + +## 2. Why it must ship as framework + +- **Expansion is a retrieval-time policy decision** — the same argument feat-023 + makes for owning graph expansion at all. *Direction* is part of that policy; + pushing it into agent code re-forks the merge/decay/dedup logic the framework + deliberately centralised. +- **It needs cross-driver consistency.** "In/out/any" must mean the same thing + over Neo4j, SurrealDB, and the embedded driver. The framework already defines + that vocabulary on `GraphStore.get_edges(direction=...)`; this surfaces it at + the retrieval layer. +- **It stays inside the locked contract.** `get_edges(direction=...)` is already + part of the v0.1 `GraphStore` ABC — so the capability is reachable with **no + ABC change and no major-version bump**. This is the clean, contract-respecting + way to add it. +- **Without framework ownership:** every agent post-filters expansion results by + direction itself — re-implementing (often incorrectly) what the store can do + natively, and losing it the moment they switch graph backends. + +## 3. How derived agents benefit + +One new optional key; no code: + +```yaml +retrieval: + graph_expansion: + store: { driver: kuzu, config: { path: .ckg } } + edge_types: [CITES] + direction: in # ← NEW: "who cites this", not "what this cites" + max_hops: 2 +``` + +- Precise expansion for asymmetric graphs (citation networks, call graphs, + supply chains, org charts). +- **Fully backward compatible:** `direction` defaults to `"any"`, which is + exactly today's behaviour — existing configs and `Retriever(...)` calls are + unchanged, bit for bit. + +## 4. Feature specifications + +### 4.1 User-facing experience +- `retrieval.graph_expansion` gains an optional `direction: in | out | any` + (default `any`). +- `GraphExpansion(direction=...)` constructs the same policy programmatically. + +### 4.2 Public API / contract + +```python +# agentforge_core/values/retrieval.py +class GraphExpansion(BaseModel): + model_config = ConfigDict(frozen=True, strict=True, arbitrary_types_allowed=True) + store: GraphStore + max_hops: int = 2 + edge_types: tuple[str, ...] | None = None + direction: Literal["out", "in", "any"] = "any" # NEW + text_property: str = "text" + decay: float = 0.5 + +# agentforge_core/config/schema.py +class GraphExpansionConfig(BaseModel): + model_config = ConfigDict(strict=True, extra="forbid") + store: ModuleEntry + max_hops: int = Field(default=2, ge=1) + edge_types: tuple[str, ...] | None = None + direction: Literal["out", "in", "any"] = "any" # NEW + text_property: str = "text" + decay: float = Field(default=0.5, gt=0.0, le=1.0) +``` + +**No change to the `GraphStore` ABC.** Direction is consumed via the existing +`get_edges(node_id, *, edge_type, direction)` method. + +### 4.3 Internal mechanics +The `Retriever`'s `_expand_via_graph` (feat-023) gains a directional path: + +- **`direction == "any"`** → unchanged: call `store.traverse(start_id, + edge_types, max_depth, limit)` exactly as today. Existing behaviour preserved. +- **`direction in {"out", "in"}`** → directional BFS in the retriever using the + locked primitive: for each hop up to `max_hops`, expand the frontier via + `store.get_edges(node_id, edge_type=t, direction=direction)` for each `t` in + `edge_types` (or all types when `edge_types is None`), collecting the + neighbour on the appropriate end (`dst` for `out`, `src` for `in`). +- **Neighbour synthesis is identical to feat-023** — per neighbour at `depth`: + `score = seed.score * decay**depth`, metadata carries + `agentforge.expanded_from` + `agentforge.hop`; dedup by id with direct hits + winning. Only *which* neighbours are gathered changes; the merge/decay/dedup + pipeline is reused verbatim. + +This keeps `direction="any"` on the optimized native `traverse()` path while +giving precise directional control where it matters — without touching the ABC. + +### 4.4 Module packaging +- `GraphExpansion` value + `GraphExpansionConfig` change in `agentforge-core`. +- `Retriever` expansion change in `agentforge`. +- **No new package** (same footprint as feat-023). + +### 4.5 Configuration +```yaml +retrieval: + graph_expansion: + store: { driver: neo4j, config: { uri: bolt://localhost:7687 } } + edge_types: [CALLS] + direction: in # callers; omit or set "any" for current behaviour + max_hops: 3 + decay: 0.5 +``` + +## 5. Plug-and-play & upgrade story +- `direction` defaults to `"any"` → **zero behavioural change** for existing + agents; existing YAML and `Retriever(...)` calls keep identical results. +- Opt in by adding the one key. Works across every `graph_stores` driver, + because it rides on the contract's `get_edges(direction=...)`. + +## 6. Cross-language parity +TypeScript port deferred (mirrors feat-023). The direction semantics map onto +the same `get_edges` direction vocabulary; the TS port mirrors 1:1 when +scheduled. + +## 7. Test strategy +- **Value validation** — `direction` accepts only `in|out|any`; default `any`. +- **Expansion correctness** — over a fixture graph with asymmetric edges: + `direction="in"` returns predecessors (callers), `"out"` returns successors + (callees), `"any"` returns the prior (undirected) set. +- **Combination** — `direction` + `edge_types` together (e.g. in-edges of type + `CALLS` only). +- **Backward compatibility** — `direction="any"` (and omitted) produce results + byte-identical to the pre-change `traverse()` path. +- **Decay / dedup unchanged** — same scores and dedup outcomes as feat-023 for + the neighbours gathered. +- Uses the in-memory reference graph store; fully offline. + +## 8. Risks & open questions +- **`traverse()` vs `get_edges` performance.** For `direction="any"` we keep the + (possibly DB-optimized) `traverse()` path; directional expansion does its BFS + via `get_edges`, which may issue more calls on dense graphs. Bounded by + `max_hops` + the existing candidate cap; documented. +- **Per-edge-type direction.** Some questions want different directions per edge + type in one pass (e.g. `CALLS` in **and** `IMPORTS` out). v1 keeps a single + `direction` for the whole expansion; per-type direction is a future + extension (an agent can run two expansions meanwhile). + +## 9. Out of scope +- A `direction` parameter on `GraphStore.traverse()` — that would change the + **locked** ABC (major bump); deliberately avoided by using `get_edges`. +- Per-edge-type direction maps. +- Native single-query directional expansion inside Neo4j/SurrealDB. +- TypeScript port. + +## 10. References +- feat-023 (GraphRAG hybrid retrieval — the feature this improves). +- feat-027 (embedded `KuzuGraphStore` — ships on the same 0.4 train; its + `get_edges(direction=...)` is the embedded path this rides). +- `GraphStore` contract — `get_edges(direction=...)` (the locked primitive used). + +## 11. Implementation status (Python) +**Status: accepted, not yet implemented.** Suggested chunking: +1. Spec + catalogue/roadmap pointer. +2. `direction` on `GraphExpansion` + `GraphExpansionConfig`; `_expand_via_graph` + directional branch via `get_edges`; unit tests incl. backward-compat. +3. Status flip + CHANGELOG + runbook note on feat-023. + +## 12. Runbook + +### How do I expand only along callers / dependents / who-cites? +Set `direction: in` in the `graph_expansion` block (with the relevant +`edge_types`). For callees / dependencies / what-it-cites, use `direction: out`. +Omit it (or `any`) to keep the original undirected expansion. + +```yaml +retrieval: + graph_expansion: + store: { driver: kuzu, config: { path: .ckg } } + edge_types: [CALLS] + direction: in # callers of the seed, up to max_hops + max_hops: 2 +``` + +### Will this change my existing agent's results? +No. `direction` defaults to `any`, which is the pre-enhancement behaviour. You +opt in per retrieval config. diff --git a/docs/features/README.md b/docs/features/README.md index e0fed45..b9a6f70 100644 --- a/docs/features/README.md +++ b/docs/features/README.md @@ -42,6 +42,7 @@ | **feat-008** | Findings & output shapes (Simple/Patch/Narrative/MultiSpan + renderers) | shipped (Python) | 0.1 | both | `agentforge`, `agentforge-core` | | **feat-009** | Observability — structured logging (JSON) + distributed tracing (OTel) + hook fan-out; vendor backends (Langfuse / Phoenix / Evidently / StatsD) slated for v0.2 | shipped (Python, OTel only) | 0.1 (framework + OTel — shipped), 0.2 (vendor backends) | both | `agentforge`, `agentforge-otel`, future `agentforge-langfuse`, `agentforge-phoenix`, `agentforge-evidently`, `agentforge-statsd` | | **feat-026** | Application config extension — reserved `app:` namespace + typed `app_as()` accessor (Phase 1), registered typed sections validated via the module-schema engine + entry points (Phase 2), pluggable config sources / separate files via `imports:` (Phase 3). Lets agents built on AgentForge reuse the config machinery (interpolation, layering, `--resolved`, uniform validation). Reported via #86 | shipped (all 3 phases → 0.3.0) | 0.3.0 | both | `agentforge-core`, `agentforge` | +| **feat-027** | Embedded `GraphStore` — `KuzuGraphStore`, a zero-ops, file-backed, in-process graph driver (the graph analogue of the SQLite `MemoryStore`). Implements the locked `GraphStore` ABC and passes `run_graph_conformance`, so it is swap-compatible with Neo4j/SurrealDB; `path: .ckg` and the store exists — no server. Makes the whole graph + GraphRAG path testable offline. Drives the `agentforge-graph` code-graph dogfood | accepted | 0.4 | python (TS deferred) | new `agentforge-memory-kuzu` | ### Safety & security diff --git a/docs/features/feat-027-embedded-graph-store.md b/docs/features/feat-027-embedded-graph-store.md new file mode 100644 index 0000000..a7942f7 --- /dev/null +++ b/docs/features/feat-027-embedded-graph-store.md @@ -0,0 +1,235 @@ +# feat-027: Embedded GraphStore (file-backed, zero-ops graph driver) + +## Metadata + +| Field | Value | +|---|---| +| **ID** | feat-027 | +| **Title** | `KuzuGraphStore` — embedded, file-backed `GraphStore` driver | +| **Status** | accepted (targeted at the 0.4 train) | +| **Owner** | kjoshi | +| **Created** | 2026-06-17 | +| **Target version** | 0.4 | +| **Languages** | `python` (TS deferred) | +| **Module package(s)** | new `agentforge-memory-kuzu` (registers `KuzuGraphStore` under the `graph_stores` entry-point category) | +| **Depends on** | feat-005 (`GraphStore` ABC + conformance), ADR-0007 (locked surface) | +| **Blocks** | none | + +--- + +## 1. Why this feature + +Every shipped `GraphStore` driver — `Neo4jGraphStore`, `SurrealGraphStore` — +requires a **separate database server** to be provisioned, networked, and +authenticated before an agent can store a single node. There is no graph store +that lives at a **local file path**, in-process, the way SQLite serves the +relational `MemoryStore` and the way `InMemoryGraphStore` serves tests (but +without persistence). + +That gap is felt the moment an agent wants a real graph for **local +development, CI, a single-host deployment, or an embedded product**: the only +options today are "stand up Neo4j" (operationally heavy) or "hand-roll a +store" (re-inventing the contract). The natural configuration for a local graph +is simply a path — + +```yaml +store: { driver: kuzu, config: { path: .ckg } } +``` + +— and nothing currently satisfies it. An **embedded, file-backed +`GraphStore`** closes the gap: a persistent property graph in a single +directory, no server, no network, started in milliseconds. + +## 2. Why it must ship as framework + +- **`GraphStore` is a framework-locked ABC (ADR-0007).** Adding a driver is the + standard contribution shape; every agent benefits from one more backend + behind the same surface. +- **The `graph_stores` entry-point category already exists** (Neo4j, SurrealDB, + and the in-memory reference register there). An embedded, persistent driver + is the missing **zero-ops** member of that set — the graph analogue of the + SQLite `MemoryStore`. +- **Cross-driver conformance guarantees behaviour.** The driver must pass + `run_graph_conformance`, so it is swap-compatible with Neo4j/SurrealDB — + upsert idempotency, edge-readback, pattern match, depth-bounded traversal, + and cascade-delete semantics are identical. +- **Offline-first is a framework value.** An embedded store makes the *entire* + graph + GraphRAG path testable with no server and no credentials, matching + the framework's offline-replay ethos. +- **Without framework ownership:** every agent that wants a local graph either + pulls in a heavyweight server or writes a bespoke store that drifts from the + contract and can't be swapped. + +## 3. How derived agents benefit + +A scaffolded agent gets a persistent graph with **no infrastructure**, by config +alone: + +```yaml +# agentforge.yaml +retrieval: + graph_expansion: + store: { driver: kuzu, config: { path: .ckg } } # was: a Neo4j server + max_hops: 2 +``` + +- **Zero setup** — `path: .ckg` and the store exists; nothing to install or run. +- **Swap-by-config, no code** — because it obeys the same `GraphStore` contract, + an agent can develop on `kuzu` locally and switch to `neo4j` for a shared + deployment by changing one line. +- **GraphRAG plugs in unchanged** — feat-023 consumes any `graph_stores` driver, + so an embedded store works with graph expansion immediately. +- **Fully offline tests** — agents can build and traverse a real persisted graph + in CI with no service dependency. + +## 4. Feature specifications + +### 4.1 User-facing experience +- `KuzuGraphStore.from_path(path)` — ergonomic construction at a directory + (mirrors `SqliteMemoryStore.from_path`). The store opens (or creates) an + embedded database under `path`. +- `from_config(*, path)` — the standard module-construction convention, so the + driver builds from a `config:` block. +- Implements the full locked `GraphStore` surface (`add_node`, `add_edge`, + `get_edges`, `match`, `traverse`, `delete_node`, `delete_edge`, `close`). +- Declares optional `capabilities()` it can honour (candidates: `"transactions"`, + `"fulltext"`; `"vector"` only if a `KuzuVectorStore` is added later — out of + scope here). + +### 4.2 Public API / contract + +```python +class KuzuGraphStore(GraphStore): + def __init__(self, *, connection: KuzuConnection) -> None: ... + + @classmethod + async def from_path(cls, path: str | Path) -> "KuzuGraphStore": + """Open or create an embedded graph database under `path`.""" + + @classmethod + async def from_config(cls, *, path: str | Path) -> "KuzuGraphStore": ... + + # all GraphStore abstract methods, conformance-verified +``` + +- **Node/edge mapping.** `GraphNode(id, labels, properties)` → + a node record keyed on `id`; `GraphEdge(src, dst, edge_type, properties)` → + a typed relationship. `add_node`/`add_edge` are **idempotent upserts** (re-add + by `id` / `(src, dst, edge_type)` replaces `properties`), per the ABC. +- **Edge integrity.** `add_edge` raises `ValueError` if `src`/`dst` are unknown + nodes — the contract's well-formedness rule. +- **`get_edges(direction=...)`** maps to the embedded engine's directional edge + lookup (the primitive feat-023's directional expansion — enh-005 — relies on). + +### 4.3 Internal mechanics +- **Storage.** A single embedded database directory at `path`; opened in-process, + no server. Schema (node table + a generic typed-relationship table) is created + lazily on first write. +- **Upserts.** Implemented via the engine's merge/replace semantics keyed on the + stable ids, matching the ABC's idempotency invariant. +- **`traverse`.** Variable-length / recursive query bounded by `max_depth` and + `limit`; returns `Path` objects with `len(edges) == len(nodes) - 1` in chain + order (conformance-checked). +- **`match`.** Pattern evaluated by the engine's query language; same `Path` + return shape as the Cypher/SurrealQL drivers. +- **`delete_node(cascade)`.** `cascade=True` removes incident edges; `cascade=False` + raises if edges remain (no orphaned edges) — the contract rule. +- **Concurrency.** Embedded single-writer; documented (see Risks). `close()` + releases the file handle. + +### 4.4 Module packaging +- New sister package **`agentforge-memory-kuzu`** (mirrors the + `agentforge-memory-neo4j` / `-surrealdb` packages that host graph drivers). +- `pyproject.toml` registers + `[project.entry-points."agentforge.graph_stores"]` → `kuzu = "agentforge_memory_kuzu:KuzuGraphStore"`. +- Versioned in lockstep on the coordinated release train (ADR-0015). + +### 4.5 Configuration +```yaml +# anywhere a graph_stores driver is accepted (e.g. retrieval.graph_expansion.store) +store: + driver: kuzu + config: + path: .ckg # directory; created if absent +``` + +## 5. Plug-and-play & upgrade story +- Purely additive — a new entry-point member. Existing configs and drivers are + untouched. +- Any feature that consumes a `graph_stores` driver (notably GraphRAG, feat-023) + works with `kuzu` immediately, no changes. +- Switching to/from another graph driver is a one-line config change; data + migration between backends is a separate concern (out of scope). + +## 6. Cross-language parity +TypeScript port deferred (same posture as the other graph/retrieval features). +The contract surface and conformance suite are language-agnostic; the TS port +mirrors this 1:1 when scheduled. + +## 7. Test strategy +- **Conformance** — `run_graph_conformance` against a real embedded database in + a temp dir. This is the headline: a persistent driver fully exercised + **offline, no server, no credentials**. +- **Unit** — `from_path`/`from_config` construction; upsert idempotency; + edge-integrity `ValueError`; `get_edges` direction; cascade-delete; `close`. +- **Live gating** — **not required** (embedded), unlike Neo4j/SurrealDB which + gate on `RUN_LIVE_*`. This is a deliberate advantage. +- **Cross-platform** — note the native dependency (see Risks); CI matrix must + cover the supported wheels. + +## 8. Risks & open questions +- **Native dependency.** The embedded engine ships a compiled component; the + package must pin to platforms with available wheels and document the support + matrix. Mitigation: keep it an opt-in sister package (installing + `agentforge` core is unaffected). +- **Concurrency model.** Embedded engines are typically single-writer. Document + the constraint; an agent doing concurrent writes (e.g. a file-watcher + re-indexer) must serialise. Aligns with how SQLite is treated. +- **File locking under watch loops.** Long-lived readers + a re-indexing writer + need a clear locking story; surfaced in the runbook. + +## 9. Out of scope +- `KuzuVectorStore` (the engine can index embeddings) — a natural follow-up so + one embedded file serves graph **and** vector, but a separate spec. +- Cross-backend data migration / export-import. +- TypeScript port. + +## 10. References +- `GraphStore` contract + ADR-0007 (locked surface). +- `agentforge_core.testing.run_graph_conformance` (the cross-driver suite this + driver must pass). +- feat-005 (GraphStore + drivers), feat-023 (GraphRAG — the primary consumer). +- enh-005 (directional graph expansion — relies on this driver's + `get_edges(direction=...)` for the embedded path). + +## 11. Implementation status (Python) +**Status: accepted, not yet implemented.** Suggested chunking when built: +1. Spec + catalogue row + roadmap pointer. +2. `KuzuGraphStore` class + entry-point registration + `from_path`/`from_config` + + conformance pass (offline). +3. Capability gating (`fulltext`/`transactions` as honoured) + runbook. +4. Status flip + catalogue + roadmap + CHANGELOG. + +## 12. Runbook + +### How do I use an embedded graph store? +```python +from agentforge_memory_kuzu import KuzuGraphStore + +async with await KuzuGraphStore.from_path(".ckg") as store: + await store.add_node(GraphNode(id="a", labels=("Func",))) + await store.add_node(GraphNode(id="b", labels=("Func",))) + await store.add_edge(GraphEdge(src="b", dst="a", edge_type="CALLS")) + callers = await store.get_edges("a", edge_type="CALLS", direction="in") +``` +Via YAML: set `driver: kuzu, config: { path: .ckg }` anywhere a `graph_stores` +driver is accepted. + +### When should I NOT use it? +- Multi-writer / shared-service deployments → use `neo4j`. +- You already run Neo4j/SurrealDB for other stores → reuse it for one DB. + +### How does it compare to the in-memory store? +`InMemoryGraphStore` is ephemeral (test fixture). This driver **persists** to a +path and survives process restarts — the embedded, production-capable local +option.