From 25411d5aa0776d6e6fb0964d4ed57a873193bca7 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:14:30 -0400 Subject: [PATCH 01/14] add mermaid diagrams across docs for visual onboarding Give readers scannable flowcharts in the README, usage guide, architecture pages, and testing docs, plus a conventions guide so future diagrams stay consistent. Co-authored-by: Cursor --- AGENTS.md | 3 + README.md | 62 ++++++++++++++++++++ docs/architecture/language-support.md | 23 ++++++++ docs/architecture/retrieval-design.md | 22 +++++++ docs/architecture/subsystems/contracts.md | 10 ++++ docs/architecture/subsystems/store.md | 9 +++ docs/architecture/subsystems/workspace.md | 9 +++ docs/concepts/how-codestory-works.md | 24 +++++++- docs/contributors/documentation-visuals.md | 67 +++++++++++++++++++++ docs/contributors/testing-matrix.md | 7 ++- docs/glossary.md | 9 +++ docs/ops/retrieval-sidecars.md | 22 +++++++ docs/testing/benchmark-ledger.md | 14 +++++ docs/testing/codestory-e2e-stats-log.md | 10 ++++ docs/testing/retrieval-architecture.md | 14 +++++ docs/usage.md | 68 ++++++++++++++++++++++ 16 files changed, 368 insertions(+), 5 deletions(-) create mode 100644 docs/contributors/documentation-visuals.md diff --git a/AGENTS.md b/AGENTS.md index 4765c36d..fcb817e6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,6 +42,9 @@ ## Retrieval documentation - Canonical sidecar retrieval docs are `docs/architecture/retrieval-design.md`, `docs/testing/retrieval-architecture.md`, and `docs/ops/retrieval-sidecars.md`. Parser compatibility records live in `docs/architecture/language-support.md`. +## Documentation +- Mermaid diagram conventions and canonical diagram index: `docs/contributors/documentation-visuals.md`. + ## Coding & Design Constraints Current merge bar for production changes: diff --git a/README.md b/README.md index 4dc9f716..ffa548ec 100644 --- a/README.md +++ b/README.md @@ -25,6 +25,57 @@ assets are installed. Setup can fetch the CodeStory source artifact or managed embedding assets; the indexed project data stays in the user cache and commands stay explicit about which workspace they read. +### Evidence loop + +```mermaid +flowchart TB + subgraph problem [Agent without grounding] + A1[Open random files] --> A2[Miss key symbols] + end + subgraph solution [CodeStory evidence loop] + B1[doctor] --> B2[index] + B2 --> B3[search or ground] + B3 --> B4["symbol / trail / snippet"] + B4 --> B5[context or packet] + end + problem -.-> solution +``` + +See [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md) +for the full loop and storage model. + +### Two readiness tracks + +```mermaid +flowchart LR + subgraph localNav [local_navigation] + L1[SQLite cache from index] + L2["ground, symbol, trail, snippet, explore, context, files, affected"] + L1 --> L2 + end + subgraph agentPkt [agent_packet_search] + S1[Zoekt + Qdrant + SCIP + llama.cpp] + S2["packet and search when retrieval_mode=full"] + S1 --> S2 + end + localNav -.->|"does not prove"| agentPkt +``` + +### Which command first? + +```mermaid +flowchart TD + start[What do you need?] --> health{Cache or sidecar health?} + health -->|yes| doctor[doctor] + health -->|no| task{Task shape?} + task -->|broad question| packet["packet (needs full sidecars)"] + task -->|repo orientation| orient["ground or report"] + task -->|one symbol or file| search[search] + search --> inspect["trail / snippet / context"] + doctor --> index[index if needed] + index --> task +``` + ## Public Promise CodeStory is a local evidence layer for repositories, not an automatic @@ -161,6 +212,9 @@ flowchart LR CodeStory builds a local evidence layer so agents can request grounded context instead of relying on ad hoc file reads. +Full crate and layer model: +[docs/architecture/overview.md](docs/architecture/overview.md). + ## Language Support Claims CodeStory separates parser-backed graph indexing, regression-tested accuracy, @@ -168,6 +222,14 @@ structural extraction, framework route coverage, and agent packet/search readiness. The current contract is documented in [docs/architecture/language-support.md](docs/architecture/language-support.md). +```mermaid +flowchart TD + ext[File extension] --> parser[Parser-backed graph] + parser --> fidelity[Fidelity-gated regression] + fidelity --> nav[Daily graph navigation claim] + parser --> packetClaim[Separate packet/search quality claim] +``` + In short: Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph languages; HTML, CSS, and SQL use structural collectors. diff --git a/docs/architecture/language-support.md b/docs/architecture/language-support.md index edd74e49..ba44ca2b 100644 --- a/docs/architecture/language-support.md +++ b/docs/architecture/language-support.md @@ -12,6 +12,19 @@ The shared registry owns public support claims. Workspace discovery also carries compatibility-only filters for file types that can be scanned or grouped without being claimed as parser-backed language support. +## Claim Ladder + +```mermaid +flowchart TD + ext[File extension] --> route[Parser or structural route] + route --> parserGraph[Parser-backed graph] + route --> structural[Structural collector] + parserGraph --> fidelity[Fidelity-gated regression] + fidelity --> navClaim[Graph navigation claim] + parserGraph --> pktClaim[Separate packet/search quality claim] + structural --> structClaim[Structural extraction claim] +``` + ## Claim Terms - `parser-backed graph`: the file extension routes to a tree-sitter parser and @@ -40,6 +53,16 @@ support. The current language-expansion A/B report records a mixed full 18-language result and a stronger packet-eligible slice; do not use that report as blanket promotion proof for every parser-backed language. +```mermaid +flowchart LR + subgraph parserBacked [Parser-backed graph languages] + py[Python, Java, Rust, JS, TS, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, Bash] + end + subgraph structural [Structural collectors] + hcs[HTML, CSS, SQL] + end +``` + The parser-backed graph claim is not a promise that every language has identical dispatch or semantic-resolution semantics. Typed receiver-call support is claimed only for the fixture-backed cases named in the indexer regression diff --git a/docs/architecture/retrieval-design.md b/docs/architecture/retrieval-design.md index 902fa8f7..eb7fa07f 100644 --- a/docs/architecture/retrieval-design.md +++ b/docs/architecture/retrieval-design.md @@ -25,6 +25,21 @@ Everything else is diagnostic only. `no_scip`, `no_semantic`, `lexical_only`, ONNX-only paths, old env aliases, and `CODESTORY_RETRIEVAL=0` fail closed for agent-facing packet/search. +## Mode Decision + +```mermaid +flowchart TD + zoektUp{Zoekt up?} + zoektUp -->|no| unavailable["unavailable — fail closed"] + zoektUp -->|yes| scipUp{SCIP up?} + scipUp -->|no| noScip["no_scip — fail closed"] + scipUp -->|yes| densePolicy{Dense anchors required?} + densePolicy -->|no| fullZero["full — dense skipped by policy"] + densePolicy -->|yes| qdrantUp{Qdrant up?} + qdrantUp -->|yes| full["full — serve packet/search"] + qdrantUp -->|no| noSemantic["no_semantic or lexical_only — fail closed"] +``` + ## Ownership | Area | Owner | Supporting areas | @@ -69,6 +84,13 @@ component and persists the manifest only after the full stack is healthy. ## AST-First Semantic Contract +```mermaid +flowchart LR + exact[Exact symbol lookup] --> lexical[Lexical source and symbol docs] + lexical --> graph[Graph expansion] + graph --> dense[Dense-anchor augmentation optional] +``` + Code structure is graph-native first. Runtime writes a deterministic `symbol_search_doc` for every durable AST symbol. These docs contain symbol name, kind, file, signature, comments, aliases, related symbols, edge digest, hash, diff --git a/docs/architecture/subsystems/contracts.md b/docs/architecture/subsystems/contracts.md index 02482a01..b18633ad 100644 --- a/docs/architecture/subsystems/contracts.md +++ b/docs/architecture/subsystems/contracts.md @@ -2,6 +2,16 @@ `codestory-contracts` owns the shared model used by the final V2 crates. +```mermaid +flowchart LR + contracts[codestory-contracts] --> workspace[codestory-workspace] + contracts --> store[codestory-store] + contracts --> indexer[codestory-indexer] + contracts --> runtime[codestory-runtime] +``` + +Full layer model: [../overview.md](../overview.md). + ## Ownership - graph/domain primitives such as nodes, edges, occurrences, trail config, and bookmarks diff --git a/docs/architecture/subsystems/store.md b/docs/architecture/subsystems/store.md index 00e8e58f..b128ff74 100644 --- a/docs/architecture/subsystems/store.md +++ b/docs/architecture/subsystems/store.md @@ -2,6 +2,15 @@ `codestory-store` is the only persistence crate. +```mermaid +flowchart LR + indexer[codestory-indexer] --> store[codestory-store SQLite] + runtime[codestory-runtime] --> store + retrieval[codestory-retrieval] --> store +``` + +Full layer model: [../overview.md](../overview.md). + ## Ownership - SQLite open/build lifecycle diff --git a/docs/architecture/subsystems/workspace.md b/docs/architecture/subsystems/workspace.md index a862fc98..f311f9a0 100644 --- a/docs/architecture/subsystems/workspace.md +++ b/docs/architecture/subsystems/workspace.md @@ -2,6 +2,15 @@ `codestory-workspace` owns repo discovery and refresh planning. +```mermaid +flowchart LR + repo[Repository root] --> manifest[Manifest and discovery] + manifest --> plan[Refresh execution plan] + plan --> runtime[codestory-runtime / indexer] +``` + +Full layer model: [../overview.md](../overview.md). + ## Ownership - manifest loading and default manifest creation diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index b4d86b73..d8480e3c 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,8 +9,14 @@ plans a change. ## The Loop -```text -doctor -> index -> ground -> search -> symbol/trail/snippet/explore -> context +```mermaid +flowchart LR + doctor[doctor] --> index[index] + index --> ground[ground] + ground --> search[search] + search --> inspect["symbol / trail / snippet / explore"] + inspect --> context[context] + search --> packet[packet] ``` - `doctor` checks whether the cache, index, retrieval mode, and local embedding @@ -30,6 +36,14 @@ The workflow is a repeatable evidence loop. ## What Gets Stored +```mermaid +flowchart LR + repo[Repository] --> workspace[Workspace discovery] + workspace --> graph["Graph nodes and edges"] + graph --> readModels["Snapshots, symbol docs, dense anchors"] + readModels --> cliReads[CLI and agent reads] +``` + CodeStory writes per-project state under the user cache, keyed by the target workspace path. The cache can include: @@ -66,6 +80,12 @@ the indexed project evidence lives in the local cache. ## What Good Looks Like +```mermaid +flowchart LR + cite[Cite source files and symbols] --> gaps[State stale or partial evidence] + gaps --> nextCmd[Give the next concrete command] +``` + A good CodeStory-backed answer does three things: 1. It names the files, symbols, or snippets it used. diff --git a/docs/contributors/documentation-visuals.md b/docs/contributors/documentation-visuals.md new file mode 100644 index 00000000..008ccbfe --- /dev/null +++ b/docs/contributors/documentation-visuals.md @@ -0,0 +1,67 @@ +# Documentation Visuals + +CodeStory docs use Mermaid diagrams embedded in markdown. GitHub renders them +natively; no build step or static image pipeline is required. + +## Diagram Types + +| Type | Use when | +| --- | --- | +| `flowchart` | Workflows, decision trees, readiness lanes, crate boundaries, proof ladders | +| `sequenceDiagram` | Command paths with multiple actors (CLI, runtime, store, sidecars) | +| `stateDiagram-v2` | Mode transitions (`retrieval_mode`, refresh modes) | + +Prefer `flowchart` for entry docs. Reserve `sequenceDiagram` for architecture pages +that already show multi-crate call paths. + +## Canonical Diagrams + +Link to the canonical page instead of copying large diagrams: + +| Concept | Canonical location | +| --- | --- | +| Evidence loop | [how-codestory-works.md](../concepts/how-codestory-works.md#the-loop) | +| Two readiness tracks | [usage.md](../usage.md#readiness-tracks) | +| Pipeline (repo to agent) | [README.md](../../README.md#what-it-builds) | +| Crate/layer model | [overview.md](../architecture/overview.md) | +| Indexing lifecycle | [indexing-pipeline.md](../architecture/indexing-pipeline.md) | +| Sidecar mode matrix | [retrieval-design.md](../architecture/retrieval-design.md) | +| Proof tiers | [retrieval-architecture.md](../testing/retrieval-architecture.md) | + +**Link, don't copy:** README and usage may show compact summaries; full crate graphs +and sequence diagrams live in architecture pages. + +## Mermaid Syntax Checklist + +- Use camelCase or PascalCase node IDs without spaces (for example `localNav`, not `local nav`). +- Quote node labels that contain special characters: `["retrieval_mode=full"]`. +- Do not use HTML tags or entities in labels. +- Do not set custom colors or `style` fills; theme handles rendering. +- Avoid reserved keywords as bare node IDs (`end`, `graph`, `subgraph`). +- For subgraphs, use explicit IDs: `subgraph localLane [Local navigation]`. + +## Content Rules + +- Put diagrams **above** dense prose or tables they summarize. +- Keep tables for lookup; use diagrams for scan paths and mental models. +- **No benchmark numbers inside diagrams.** Token counts, timings, and overhead + ratios belong in append-only stats logs and ledger tables, not in Mermaid blocks. +- Conceptual labels only in testing docs (for example "exploratory evidence", not + `114510 tokens`). + +## Runtime-Generated Diagrams + +For symbol-level flow evidence from an indexed workspace, use: + +```sh +codestory-cli trail --project --id --mermaid +``` + +See [.agents/skills/codestory-grounding/references/trail.md](../../.agents/skills/codestory-grounding/references/trail.md). + +## Adding A New Diagram + +1. Check whether a canonical diagram already covers the concept. +2. Pick the lowest tier page that owns the concept (entry, architecture, testing). +3. Add a one-line lead-in before the fenced `mermaid` block. +4. Preview in GitHub or VS Code Mermaid preview before opening a PR. diff --git a/docs/contributors/testing-matrix.md b/docs/contributors/testing-matrix.md index 01c0487d..acd4a4ca 100644 --- a/docs/contributors/testing-matrix.md +++ b/docs/contributors/testing-matrix.md @@ -7,21 +7,22 @@ assignments such as `$env:NAME = "value"`. ```mermaid flowchart TD change["What changed?"] --> docs["Docs or README only"] - change["What changed?"] --> always["Always consider the fast lane first"] + change --> always["Always consider the fast lane first"] change --> indexer["Indexer, graph, or language work"] change --> store["Store, snapshot, trail, or search-doc work"] change --> runtime["Runtime, search, grounding, or orchestration work"] change --> cli["CLI args or output boundary work"] change --> bench["Bench or perf-surface work"] change --> e2e["Repo-scale semantic or cold-start behavior"] - docs --> docs_checks["markdown/link checks + any touched doc contracts"] + docs --> docs_checks["markdown/link checks + onboarding_contracts"] always --> workspace["fmt, check, targeted tests, clippy"] - indexer --> fidelity["fidelity_regression, tictactoe_language_coverage, integration"] + indexer --> fidelity["fidelity_regression, tictactoe_language_coverage"] store --> store_tests["cargo test -p codestory-store"] runtime --> runtime_tests["cargo test -p codestory-runtime and retrieval_eval"] cli --> cli_tests["cargo test -p codestory-cli"] bench --> bench_checks["cargo check -p codestory-bench --benches"] e2e --> e2e_stats["release build + codestory_repo_e2e_stats"] + e2e_stats --> statsLog["Append row to codestory-e2e-stats-log.md"] ``` ## Whole Workspace diff --git a/docs/glossary.md b/docs/glossary.md index 461860ef..a66f45fc 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,5 +1,14 @@ # Glossary +```mermaid +flowchart TB + grounding[grounding] --> snapshot[snapshot] + snapshot --> trail[trail] + trail --> packet[packet] + retrievalMode["retrieval_mode=full"] --> agentReady[agent packet/search readiness] + localReady[local navigation readiness] -.->|"separate lane"| agentReady +``` + - grounding: the process of turning indexed code state into concise, relevant context for a question or tool action - snapshot: a derived SQLite-backed grounding view that can be rebuilt from the primary graph tables - projection: derived persisted data such as callable projection state or ranked grounding summaries diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index a3651ea9..ca5d09be 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -19,6 +19,19 @@ checklists live in [`retrieval-architecture.md`](../testing/retrieval-architectu --- +## Sidecar topology + +```mermaid +flowchart LR + cli[codestory-cli / runtime] --> zoekt["Zoekt localhost:6070"] + cli --> qdrant["Qdrant localhost:6333"] + cli --> scip[SCIP graph artifacts] + cli --> embed["llama.cpp embed endpoint"] + cli --> cache[User cache data dirs] +``` + +--- + ## Prerequisites - Rust toolchain with `cargo` (primary path) @@ -50,6 +63,15 @@ agent-facing packet/search evidence. First-run evidence path: +```mermaid +flowchart LR + setupEnv[setup-retrieval-env] --> cargoSetup[cargo retrieval-setup] + cargoSetup --> index[index --refresh full] + index --> rindex["retrieval index --refresh full"] + rindex --> status[retrieval status] + status --> ready["ready --goal agent or doctor"] +``` + ```sh node scripts/setup-retrieval-env.mjs --fetch-embed-model export CODESTORY_EMBED_MODEL_DIR="$(pwd)/target/retrieval-models" diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 957109a6..3a4653d4 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -11,6 +11,20 @@ immutable manifest refs, and CodeStory cache provenance. The harness now keeps transcript-observed anchors separate from anchors actually present in the final answer, so tool output alone cannot make a row quality-pass. +## Evidence taxonomy + +```mermaid +flowchart TB + solid[What is solid — repeatable behavior evidence] + exploratory[Exploratory local-real rows — no promotion claim] + notClaimed[What is not claimed — no universal savings] + solid --> exploratory + exploratory -.->|"needs Tier 4"| notClaimed +``` + +Do not copy token counts or timings from the scorecard into diagrams. Use the +table below for numeric detail. + ## Current Scorecard | Lane | Current status | Public claim status | diff --git a/docs/testing/codestory-e2e-stats-log.md b/docs/testing/codestory-e2e-stats-log.md index 7da11826..d5469699 100644 --- a/docs/testing/codestory-e2e-stats-log.md +++ b/docs/testing/codestory-e2e-stats-log.md @@ -1,5 +1,15 @@ # Codestory E2E Stats Log +Timing history lives in append-only rows below. Do not chart hard-coded timings +in prose or Mermaid blocks; cite the latest row instead. + +```mermaid +flowchart LR + change[Behavior change to index or reads] --> build["cargo build --release -p codestory-cli"] + build --> test["codestory_repo_e2e_stats --ignored"] + test --> append[Append headline row here] +``` + Append one entry before each commit after running: ```sh diff --git a/docs/testing/retrieval-architecture.md b/docs/testing/retrieval-architecture.md index 5237e6e3..21cc63bd 100644 --- a/docs/testing/retrieval-architecture.md +++ b/docs/testing/retrieval-architecture.md @@ -39,6 +39,20 @@ Do not describe a branch as generalized or useful for agents until the matching proof tier has run cleanly on the current branch. Docs and PRs must state only the highest tier actually reached: +```mermaid +flowchart TB + t1[Tier 1: CodeStory self-e2e] --> t2[Tier 2: Local-real drill suite] + t2 --> t3[Tier 3: Holdout-retrieval drill suite] + t3 --> t4[Tier 4: Promotion-grade paired benchmark] +``` + +```mermaid +flowchart LR + code[Code and lint gates] --> drill[drill / drill-suite] + drill --> holdout[Holdout-retrieval rows] + holdout --> ab[Paired with/without agent A/B] +``` + | Tier | Proof | Claim allowed | |------|-------|---------------| | 1. CodeStory self-e2e | Generalization lint, targeted runtime/indexer tests, release CLI build, `doctor`, and repo-scale e2e stats | CodeStory still works on itself and production code has no banned holdout literals | diff --git a/docs/usage.md b/docs/usage.md index 3bd8702b..1226ee43 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -45,6 +45,23 @@ TARGET_WORKSPACE="/path/to/repo" CodeStory has two readiness tracks. Keep them separate when deciding whether an agent can rely on packet/search output. +```mermaid +flowchart TB + subgraph localLane [local_navigation] + direction LR + cache[SQLite cache from index] + navCmds["ground, symbol, trail, snippet, explore, context, files, affected"] + cache --> navCmds + end + subgraph sidecarLane [agent_packet_search] + direction LR + sidecars[Zoekt, Qdrant, SCIP, llama.cpp] + pktCmds["packet and search when retrieval_mode=full"] + sidecars --> pktCmds + end + localLane -.->|"does not prove"| sidecarLane +``` + ### Local navigation/cache readiness This lane is for local browsing and source navigation. It uses the project @@ -72,6 +89,13 @@ described as agent packet/search readiness. ### I need a repo overview +```mermaid +flowchart LR + d[doctor] --> i["index --refresh full"] + i --> g["ground --why"] + g --> r[report] +``` + ```sh codestory-cli doctor --project codestory-cli index --project --refresh full @@ -92,6 +116,13 @@ files as outputs to regenerate, not source-of-truth state. ### I need evidence for a broad question +```mermaid +flowchart LR + pkt["packet --question"] --> status{sufficient?} + status -->|partial or blocked| follow[Follow named source-truth commands] + status -->|sufficient| answer[Answer from citations] +``` + ```sh codestory-cli packet --project --question "" --budget compact ``` @@ -105,6 +136,13 @@ coverage, not final answer-quality proof. ### I need to understand one symbol or file +```mermaid +flowchart LR + search[search --why] --> pick[Pick node-id] + pick --> inspect["explore / trail / snippet"] + inspect --> ctx[context optional] +``` + ```sh codestory-cli search --project --query "" --why codestory-cli explore --project --id --no-tui @@ -125,6 +163,12 @@ target-first; it is not an open chat endpoint and is not a replacement for broad ### I changed files and need likely impact +```mermaid +flowchart LR + inc["index --refresh incremental"] --> aff[affected] + diff[git diff stdin] --> aff +``` + ```sh codestory-cli index --project --refresh incremental codestory-cli affected --project --format markdown @@ -138,6 +182,12 @@ available when another tool already chose the file list. ### The cache or local navigation looks stale +```mermaid +flowchart LR + d1[doctor] --> full["index --refresh full"] + full --> d2[doctor again] +``` + ```sh codestory-cli doctor --project codestory-cli index --project --refresh full @@ -152,6 +202,13 @@ truth for cache and retrieval state. For agent-facing packet/search recovery, use the full sidecar repair sequence that `ready --goal agent` reports: +```mermaid +flowchart LR + boot[retrieval bootstrap] --> ridx["retrieval index --refresh full"] + ridx --> status[retrieval status] + status --> doc[doctor] +``` + ```sh codestory-cli retrieval bootstrap --project --format json codestory-cli retrieval index --project --refresh full --format json @@ -168,6 +225,17 @@ target manifest and `doctor` or `retrieval status` reports `retrieval_mode=full` ## Core Commands +```mermaid +flowchart TB + health[Health: doctor] --> build[Build: index] + build --> discover[Discover: ground, search, files] + discover --> inspect[Inspect: symbol, trail, snippet, explore] + inspect --> bundle[Bundle: context, packet, affected] +``` + +See the [evidence loop in how-codestory-works](concepts/how-codestory-works.md#the-loop) +for the canonical command sequence. + - `doctor`: read-only health check for project, cache, index, retrieval, and environment readiness. - `index`: build or refresh the SQLite graph, snapshots, search state, From edb47b8d7db1738e4ad5551599ab5628c5ae9998 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:17:30 -0400 Subject: [PATCH 02/14] Revert "add mermaid diagrams across docs for visual onboarding" This reverts commit 25411d5aa0776d6e6fb0964d4ed57a873193bca7. --- AGENTS.md | 3 - README.md | 62 -------------------- docs/architecture/language-support.md | 23 -------- docs/architecture/retrieval-design.md | 22 ------- docs/architecture/subsystems/contracts.md | 10 ---- docs/architecture/subsystems/store.md | 9 --- docs/architecture/subsystems/workspace.md | 9 --- docs/concepts/how-codestory-works.md | 24 +------- docs/contributors/documentation-visuals.md | 67 --------------------- docs/contributors/testing-matrix.md | 7 +-- docs/glossary.md | 9 --- docs/ops/retrieval-sidecars.md | 22 ------- docs/testing/benchmark-ledger.md | 14 ----- docs/testing/codestory-e2e-stats-log.md | 10 ---- docs/testing/retrieval-architecture.md | 14 ----- docs/usage.md | 68 ---------------------- 16 files changed, 5 insertions(+), 368 deletions(-) delete mode 100644 docs/contributors/documentation-visuals.md diff --git a/AGENTS.md b/AGENTS.md index fcb817e6..4765c36d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,9 +42,6 @@ ## Retrieval documentation - Canonical sidecar retrieval docs are `docs/architecture/retrieval-design.md`, `docs/testing/retrieval-architecture.md`, and `docs/ops/retrieval-sidecars.md`. Parser compatibility records live in `docs/architecture/language-support.md`. -## Documentation -- Mermaid diagram conventions and canonical diagram index: `docs/contributors/documentation-visuals.md`. - ## Coding & Design Constraints Current merge bar for production changes: diff --git a/README.md b/README.md index ffa548ec..4dc9f716 100644 --- a/README.md +++ b/README.md @@ -25,57 +25,6 @@ assets are installed. Setup can fetch the CodeStory source artifact or managed embedding assets; the indexed project data stays in the user cache and commands stay explicit about which workspace they read. -### Evidence loop - -```mermaid -flowchart TB - subgraph problem [Agent without grounding] - A1[Open random files] --> A2[Miss key symbols] - end - subgraph solution [CodeStory evidence loop] - B1[doctor] --> B2[index] - B2 --> B3[search or ground] - B3 --> B4["symbol / trail / snippet"] - B4 --> B5[context or packet] - end - problem -.-> solution -``` - -See [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md) -for the full loop and storage model. - -### Two readiness tracks - -```mermaid -flowchart LR - subgraph localNav [local_navigation] - L1[SQLite cache from index] - L2["ground, symbol, trail, snippet, explore, context, files, affected"] - L1 --> L2 - end - subgraph agentPkt [agent_packet_search] - S1[Zoekt + Qdrant + SCIP + llama.cpp] - S2["packet and search when retrieval_mode=full"] - S1 --> S2 - end - localNav -.->|"does not prove"| agentPkt -``` - -### Which command first? - -```mermaid -flowchart TD - start[What do you need?] --> health{Cache or sidecar health?} - health -->|yes| doctor[doctor] - health -->|no| task{Task shape?} - task -->|broad question| packet["packet (needs full sidecars)"] - task -->|repo orientation| orient["ground or report"] - task -->|one symbol or file| search[search] - search --> inspect["trail / snippet / context"] - doctor --> index[index if needed] - index --> task -``` - ## Public Promise CodeStory is a local evidence layer for repositories, not an automatic @@ -212,9 +161,6 @@ flowchart LR CodeStory builds a local evidence layer so agents can request grounded context instead of relying on ad hoc file reads. -Full crate and layer model: -[docs/architecture/overview.md](docs/architecture/overview.md). - ## Language Support Claims CodeStory separates parser-backed graph indexing, regression-tested accuracy, @@ -222,14 +168,6 @@ structural extraction, framework route coverage, and agent packet/search readiness. The current contract is documented in [docs/architecture/language-support.md](docs/architecture/language-support.md). -```mermaid -flowchart TD - ext[File extension] --> parser[Parser-backed graph] - parser --> fidelity[Fidelity-gated regression] - fidelity --> nav[Daily graph navigation claim] - parser --> packetClaim[Separate packet/search quality claim] -``` - In short: Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph languages; HTML, CSS, and SQL use structural collectors. diff --git a/docs/architecture/language-support.md b/docs/architecture/language-support.md index ba44ca2b..edd74e49 100644 --- a/docs/architecture/language-support.md +++ b/docs/architecture/language-support.md @@ -12,19 +12,6 @@ The shared registry owns public support claims. Workspace discovery also carries compatibility-only filters for file types that can be scanned or grouped without being claimed as parser-backed language support. -## Claim Ladder - -```mermaid -flowchart TD - ext[File extension] --> route[Parser or structural route] - route --> parserGraph[Parser-backed graph] - route --> structural[Structural collector] - parserGraph --> fidelity[Fidelity-gated regression] - fidelity --> navClaim[Graph navigation claim] - parserGraph --> pktClaim[Separate packet/search quality claim] - structural --> structClaim[Structural extraction claim] -``` - ## Claim Terms - `parser-backed graph`: the file extension routes to a tree-sitter parser and @@ -53,16 +40,6 @@ support. The current language-expansion A/B report records a mixed full 18-language result and a stronger packet-eligible slice; do not use that report as blanket promotion proof for every parser-backed language. -```mermaid -flowchart LR - subgraph parserBacked [Parser-backed graph languages] - py[Python, Java, Rust, JS, TS, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, Bash] - end - subgraph structural [Structural collectors] - hcs[HTML, CSS, SQL] - end -``` - The parser-backed graph claim is not a promise that every language has identical dispatch or semantic-resolution semantics. Typed receiver-call support is claimed only for the fixture-backed cases named in the indexer regression diff --git a/docs/architecture/retrieval-design.md b/docs/architecture/retrieval-design.md index eb7fa07f..902fa8f7 100644 --- a/docs/architecture/retrieval-design.md +++ b/docs/architecture/retrieval-design.md @@ -25,21 +25,6 @@ Everything else is diagnostic only. `no_scip`, `no_semantic`, `lexical_only`, ONNX-only paths, old env aliases, and `CODESTORY_RETRIEVAL=0` fail closed for agent-facing packet/search. -## Mode Decision - -```mermaid -flowchart TD - zoektUp{Zoekt up?} - zoektUp -->|no| unavailable["unavailable — fail closed"] - zoektUp -->|yes| scipUp{SCIP up?} - scipUp -->|no| noScip["no_scip — fail closed"] - scipUp -->|yes| densePolicy{Dense anchors required?} - densePolicy -->|no| fullZero["full — dense skipped by policy"] - densePolicy -->|yes| qdrantUp{Qdrant up?} - qdrantUp -->|yes| full["full — serve packet/search"] - qdrantUp -->|no| noSemantic["no_semantic or lexical_only — fail closed"] -``` - ## Ownership | Area | Owner | Supporting areas | @@ -84,13 +69,6 @@ component and persists the manifest only after the full stack is healthy. ## AST-First Semantic Contract -```mermaid -flowchart LR - exact[Exact symbol lookup] --> lexical[Lexical source and symbol docs] - lexical --> graph[Graph expansion] - graph --> dense[Dense-anchor augmentation optional] -``` - Code structure is graph-native first. Runtime writes a deterministic `symbol_search_doc` for every durable AST symbol. These docs contain symbol name, kind, file, signature, comments, aliases, related symbols, edge digest, hash, diff --git a/docs/architecture/subsystems/contracts.md b/docs/architecture/subsystems/contracts.md index b18633ad..02482a01 100644 --- a/docs/architecture/subsystems/contracts.md +++ b/docs/architecture/subsystems/contracts.md @@ -2,16 +2,6 @@ `codestory-contracts` owns the shared model used by the final V2 crates. -```mermaid -flowchart LR - contracts[codestory-contracts] --> workspace[codestory-workspace] - contracts --> store[codestory-store] - contracts --> indexer[codestory-indexer] - contracts --> runtime[codestory-runtime] -``` - -Full layer model: [../overview.md](../overview.md). - ## Ownership - graph/domain primitives such as nodes, edges, occurrences, trail config, and bookmarks diff --git a/docs/architecture/subsystems/store.md b/docs/architecture/subsystems/store.md index b128ff74..00e8e58f 100644 --- a/docs/architecture/subsystems/store.md +++ b/docs/architecture/subsystems/store.md @@ -2,15 +2,6 @@ `codestory-store` is the only persistence crate. -```mermaid -flowchart LR - indexer[codestory-indexer] --> store[codestory-store SQLite] - runtime[codestory-runtime] --> store - retrieval[codestory-retrieval] --> store -``` - -Full layer model: [../overview.md](../overview.md). - ## Ownership - SQLite open/build lifecycle diff --git a/docs/architecture/subsystems/workspace.md b/docs/architecture/subsystems/workspace.md index f311f9a0..a862fc98 100644 --- a/docs/architecture/subsystems/workspace.md +++ b/docs/architecture/subsystems/workspace.md @@ -2,15 +2,6 @@ `codestory-workspace` owns repo discovery and refresh planning. -```mermaid -flowchart LR - repo[Repository root] --> manifest[Manifest and discovery] - manifest --> plan[Refresh execution plan] - plan --> runtime[codestory-runtime / indexer] -``` - -Full layer model: [../overview.md](../overview.md). - ## Ownership - manifest loading and default manifest creation diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index d8480e3c..b4d86b73 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,14 +9,8 @@ plans a change. ## The Loop -```mermaid -flowchart LR - doctor[doctor] --> index[index] - index --> ground[ground] - ground --> search[search] - search --> inspect["symbol / trail / snippet / explore"] - inspect --> context[context] - search --> packet[packet] +```text +doctor -> index -> ground -> search -> symbol/trail/snippet/explore -> context ``` - `doctor` checks whether the cache, index, retrieval mode, and local embedding @@ -36,14 +30,6 @@ The workflow is a repeatable evidence loop. ## What Gets Stored -```mermaid -flowchart LR - repo[Repository] --> workspace[Workspace discovery] - workspace --> graph["Graph nodes and edges"] - graph --> readModels["Snapshots, symbol docs, dense anchors"] - readModels --> cliReads[CLI and agent reads] -``` - CodeStory writes per-project state under the user cache, keyed by the target workspace path. The cache can include: @@ -80,12 +66,6 @@ the indexed project evidence lives in the local cache. ## What Good Looks Like -```mermaid -flowchart LR - cite[Cite source files and symbols] --> gaps[State stale or partial evidence] - gaps --> nextCmd[Give the next concrete command] -``` - A good CodeStory-backed answer does three things: 1. It names the files, symbols, or snippets it used. diff --git a/docs/contributors/documentation-visuals.md b/docs/contributors/documentation-visuals.md deleted file mode 100644 index 008ccbfe..00000000 --- a/docs/contributors/documentation-visuals.md +++ /dev/null @@ -1,67 +0,0 @@ -# Documentation Visuals - -CodeStory docs use Mermaid diagrams embedded in markdown. GitHub renders them -natively; no build step or static image pipeline is required. - -## Diagram Types - -| Type | Use when | -| --- | --- | -| `flowchart` | Workflows, decision trees, readiness lanes, crate boundaries, proof ladders | -| `sequenceDiagram` | Command paths with multiple actors (CLI, runtime, store, sidecars) | -| `stateDiagram-v2` | Mode transitions (`retrieval_mode`, refresh modes) | - -Prefer `flowchart` for entry docs. Reserve `sequenceDiagram` for architecture pages -that already show multi-crate call paths. - -## Canonical Diagrams - -Link to the canonical page instead of copying large diagrams: - -| Concept | Canonical location | -| --- | --- | -| Evidence loop | [how-codestory-works.md](../concepts/how-codestory-works.md#the-loop) | -| Two readiness tracks | [usage.md](../usage.md#readiness-tracks) | -| Pipeline (repo to agent) | [README.md](../../README.md#what-it-builds) | -| Crate/layer model | [overview.md](../architecture/overview.md) | -| Indexing lifecycle | [indexing-pipeline.md](../architecture/indexing-pipeline.md) | -| Sidecar mode matrix | [retrieval-design.md](../architecture/retrieval-design.md) | -| Proof tiers | [retrieval-architecture.md](../testing/retrieval-architecture.md) | - -**Link, don't copy:** README and usage may show compact summaries; full crate graphs -and sequence diagrams live in architecture pages. - -## Mermaid Syntax Checklist - -- Use camelCase or PascalCase node IDs without spaces (for example `localNav`, not `local nav`). -- Quote node labels that contain special characters: `["retrieval_mode=full"]`. -- Do not use HTML tags or entities in labels. -- Do not set custom colors or `style` fills; theme handles rendering. -- Avoid reserved keywords as bare node IDs (`end`, `graph`, `subgraph`). -- For subgraphs, use explicit IDs: `subgraph localLane [Local navigation]`. - -## Content Rules - -- Put diagrams **above** dense prose or tables they summarize. -- Keep tables for lookup; use diagrams for scan paths and mental models. -- **No benchmark numbers inside diagrams.** Token counts, timings, and overhead - ratios belong in append-only stats logs and ledger tables, not in Mermaid blocks. -- Conceptual labels only in testing docs (for example "exploratory evidence", not - `114510 tokens`). - -## Runtime-Generated Diagrams - -For symbol-level flow evidence from an indexed workspace, use: - -```sh -codestory-cli trail --project --id --mermaid -``` - -See [.agents/skills/codestory-grounding/references/trail.md](../../.agents/skills/codestory-grounding/references/trail.md). - -## Adding A New Diagram - -1. Check whether a canonical diagram already covers the concept. -2. Pick the lowest tier page that owns the concept (entry, architecture, testing). -3. Add a one-line lead-in before the fenced `mermaid` block. -4. Preview in GitHub or VS Code Mermaid preview before opening a PR. diff --git a/docs/contributors/testing-matrix.md b/docs/contributors/testing-matrix.md index acd4a4ca..01c0487d 100644 --- a/docs/contributors/testing-matrix.md +++ b/docs/contributors/testing-matrix.md @@ -7,22 +7,21 @@ assignments such as `$env:NAME = "value"`. ```mermaid flowchart TD change["What changed?"] --> docs["Docs or README only"] - change --> always["Always consider the fast lane first"] + change["What changed?"] --> always["Always consider the fast lane first"] change --> indexer["Indexer, graph, or language work"] change --> store["Store, snapshot, trail, or search-doc work"] change --> runtime["Runtime, search, grounding, or orchestration work"] change --> cli["CLI args or output boundary work"] change --> bench["Bench or perf-surface work"] change --> e2e["Repo-scale semantic or cold-start behavior"] - docs --> docs_checks["markdown/link checks + onboarding_contracts"] + docs --> docs_checks["markdown/link checks + any touched doc contracts"] always --> workspace["fmt, check, targeted tests, clippy"] - indexer --> fidelity["fidelity_regression, tictactoe_language_coverage"] + indexer --> fidelity["fidelity_regression, tictactoe_language_coverage, integration"] store --> store_tests["cargo test -p codestory-store"] runtime --> runtime_tests["cargo test -p codestory-runtime and retrieval_eval"] cli --> cli_tests["cargo test -p codestory-cli"] bench --> bench_checks["cargo check -p codestory-bench --benches"] e2e --> e2e_stats["release build + codestory_repo_e2e_stats"] - e2e_stats --> statsLog["Append row to codestory-e2e-stats-log.md"] ``` ## Whole Workspace diff --git a/docs/glossary.md b/docs/glossary.md index a66f45fc..461860ef 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,14 +1,5 @@ # Glossary -```mermaid -flowchart TB - grounding[grounding] --> snapshot[snapshot] - snapshot --> trail[trail] - trail --> packet[packet] - retrievalMode["retrieval_mode=full"] --> agentReady[agent packet/search readiness] - localReady[local navigation readiness] -.->|"separate lane"| agentReady -``` - - grounding: the process of turning indexed code state into concise, relevant context for a question or tool action - snapshot: a derived SQLite-backed grounding view that can be rebuilt from the primary graph tables - projection: derived persisted data such as callable projection state or ranked grounding summaries diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index ca5d09be..a3651ea9 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -19,19 +19,6 @@ checklists live in [`retrieval-architecture.md`](../testing/retrieval-architectu --- -## Sidecar topology - -```mermaid -flowchart LR - cli[codestory-cli / runtime] --> zoekt["Zoekt localhost:6070"] - cli --> qdrant["Qdrant localhost:6333"] - cli --> scip[SCIP graph artifacts] - cli --> embed["llama.cpp embed endpoint"] - cli --> cache[User cache data dirs] -``` - ---- - ## Prerequisites - Rust toolchain with `cargo` (primary path) @@ -63,15 +50,6 @@ agent-facing packet/search evidence. First-run evidence path: -```mermaid -flowchart LR - setupEnv[setup-retrieval-env] --> cargoSetup[cargo retrieval-setup] - cargoSetup --> index[index --refresh full] - index --> rindex["retrieval index --refresh full"] - rindex --> status[retrieval status] - status --> ready["ready --goal agent or doctor"] -``` - ```sh node scripts/setup-retrieval-env.mjs --fetch-embed-model export CODESTORY_EMBED_MODEL_DIR="$(pwd)/target/retrieval-models" diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 3a4653d4..957109a6 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -11,20 +11,6 @@ immutable manifest refs, and CodeStory cache provenance. The harness now keeps transcript-observed anchors separate from anchors actually present in the final answer, so tool output alone cannot make a row quality-pass. -## Evidence taxonomy - -```mermaid -flowchart TB - solid[What is solid — repeatable behavior evidence] - exploratory[Exploratory local-real rows — no promotion claim] - notClaimed[What is not claimed — no universal savings] - solid --> exploratory - exploratory -.->|"needs Tier 4"| notClaimed -``` - -Do not copy token counts or timings from the scorecard into diagrams. Use the -table below for numeric detail. - ## Current Scorecard | Lane | Current status | Public claim status | diff --git a/docs/testing/codestory-e2e-stats-log.md b/docs/testing/codestory-e2e-stats-log.md index d5469699..7da11826 100644 --- a/docs/testing/codestory-e2e-stats-log.md +++ b/docs/testing/codestory-e2e-stats-log.md @@ -1,15 +1,5 @@ # Codestory E2E Stats Log -Timing history lives in append-only rows below. Do not chart hard-coded timings -in prose or Mermaid blocks; cite the latest row instead. - -```mermaid -flowchart LR - change[Behavior change to index or reads] --> build["cargo build --release -p codestory-cli"] - build --> test["codestory_repo_e2e_stats --ignored"] - test --> append[Append headline row here] -``` - Append one entry before each commit after running: ```sh diff --git a/docs/testing/retrieval-architecture.md b/docs/testing/retrieval-architecture.md index 21cc63bd..5237e6e3 100644 --- a/docs/testing/retrieval-architecture.md +++ b/docs/testing/retrieval-architecture.md @@ -39,20 +39,6 @@ Do not describe a branch as generalized or useful for agents until the matching proof tier has run cleanly on the current branch. Docs and PRs must state only the highest tier actually reached: -```mermaid -flowchart TB - t1[Tier 1: CodeStory self-e2e] --> t2[Tier 2: Local-real drill suite] - t2 --> t3[Tier 3: Holdout-retrieval drill suite] - t3 --> t4[Tier 4: Promotion-grade paired benchmark] -``` - -```mermaid -flowchart LR - code[Code and lint gates] --> drill[drill / drill-suite] - drill --> holdout[Holdout-retrieval rows] - holdout --> ab[Paired with/without agent A/B] -``` - | Tier | Proof | Claim allowed | |------|-------|---------------| | 1. CodeStory self-e2e | Generalization lint, targeted runtime/indexer tests, release CLI build, `doctor`, and repo-scale e2e stats | CodeStory still works on itself and production code has no banned holdout literals | diff --git a/docs/usage.md b/docs/usage.md index 1226ee43..3bd8702b 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -45,23 +45,6 @@ TARGET_WORKSPACE="/path/to/repo" CodeStory has two readiness tracks. Keep them separate when deciding whether an agent can rely on packet/search output. -```mermaid -flowchart TB - subgraph localLane [local_navigation] - direction LR - cache[SQLite cache from index] - navCmds["ground, symbol, trail, snippet, explore, context, files, affected"] - cache --> navCmds - end - subgraph sidecarLane [agent_packet_search] - direction LR - sidecars[Zoekt, Qdrant, SCIP, llama.cpp] - pktCmds["packet and search when retrieval_mode=full"] - sidecars --> pktCmds - end - localLane -.->|"does not prove"| sidecarLane -``` - ### Local navigation/cache readiness This lane is for local browsing and source navigation. It uses the project @@ -89,13 +72,6 @@ described as agent packet/search readiness. ### I need a repo overview -```mermaid -flowchart LR - d[doctor] --> i["index --refresh full"] - i --> g["ground --why"] - g --> r[report] -``` - ```sh codestory-cli doctor --project codestory-cli index --project --refresh full @@ -116,13 +92,6 @@ files as outputs to regenerate, not source-of-truth state. ### I need evidence for a broad question -```mermaid -flowchart LR - pkt["packet --question"] --> status{sufficient?} - status -->|partial or blocked| follow[Follow named source-truth commands] - status -->|sufficient| answer[Answer from citations] -``` - ```sh codestory-cli packet --project --question "" --budget compact ``` @@ -136,13 +105,6 @@ coverage, not final answer-quality proof. ### I need to understand one symbol or file -```mermaid -flowchart LR - search[search --why] --> pick[Pick node-id] - pick --> inspect["explore / trail / snippet"] - inspect --> ctx[context optional] -``` - ```sh codestory-cli search --project --query "" --why codestory-cli explore --project --id --no-tui @@ -163,12 +125,6 @@ target-first; it is not an open chat endpoint and is not a replacement for broad ### I changed files and need likely impact -```mermaid -flowchart LR - inc["index --refresh incremental"] --> aff[affected] - diff[git diff stdin] --> aff -``` - ```sh codestory-cli index --project --refresh incremental codestory-cli affected --project --format markdown @@ -182,12 +138,6 @@ available when another tool already chose the file list. ### The cache or local navigation looks stale -```mermaid -flowchart LR - d1[doctor] --> full["index --refresh full"] - full --> d2[doctor again] -``` - ```sh codestory-cli doctor --project codestory-cli index --project --refresh full @@ -202,13 +152,6 @@ truth for cache and retrieval state. For agent-facing packet/search recovery, use the full sidecar repair sequence that `ready --goal agent` reports: -```mermaid -flowchart LR - boot[retrieval bootstrap] --> ridx["retrieval index --refresh full"] - ridx --> status[retrieval status] - status --> doc[doctor] -``` - ```sh codestory-cli retrieval bootstrap --project --format json codestory-cli retrieval index --project --refresh full --format json @@ -225,17 +168,6 @@ target manifest and `doctor` or `retrieval status` reports `retrieval_mode=full` ## Core Commands -```mermaid -flowchart TB - health[Health: doctor] --> build[Build: index] - build --> discover[Discover: ground, search, files] - discover --> inspect[Inspect: symbol, trail, snippet, explore] - inspect --> bundle[Bundle: context, packet, affected] -``` - -See the [evidence loop in how-codestory-works](concepts/how-codestory-works.md#the-loop) -for the canonical command sequence. - - `doctor`: read-only health check for project, cache, index, retrieval, and environment readiness. - `index`: build or refresh the SQLite graph, snapshots, search state, From 123f6fa86caac582abdaf40440961397127ca1ba Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:27:31 -0400 Subject: [PATCH 03/14] replace decorative doc diagrams with three high-signal visuals One system map shows SQLite cache vs sidecar lanes and which commands each gate. Usage adds a scannable readiness table. Sidecar ops adds localhost topology only. Co-authored-by: Cursor --- README.md | 21 ++++++++++++++------- docs/concepts/how-codestory-works.md | 6 +++--- docs/ops/retrieval-sidecars.md | 8 ++++++++ docs/usage.md | 10 ++++++++++ 4 files changed, 35 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 4dc9f716..011aa53f 100644 --- a/README.md +++ b/README.md @@ -149,15 +149,22 @@ inconsistent. ## What It Builds ```mermaid -flowchart LR - Repo["repository"] --> Workspace["workspace discovery"] - Workspace --> Indexer["symbol and edge extraction"] - Indexer --> Store["SQLite store"] - Store --> Runtime["retrieval and context assembly"] - Runtime --> CLI["CLI and stdio reads"] - CLI --> Agent["coding agent"] +flowchart TB + repo[Target repository] --> index[index] + index --> sqlite[(SQLite cache per project)] + index --> rindex[retrieval index] + rindex --> sidecars["Sidecars: Zoekt, Qdrant, SCIP, llama.cpp"] + sqlite --> localCmds["Local navigation: ground, symbol, trail, snippet, explore, context, files, affected, report"] + sidecars --> agentCmds["Agent packet/search when retrieval_mode=full"] + sqlite --> runtime[codestory-runtime] + sidecars --> runtime + runtime --> cli[CLI and serve stdio] + cli --> agent[Coding agent] + sqlite -.->|"cache alone does not prove sidecar readiness"| agentCmds ``` +Crate and layer detail: [docs/architecture/overview.md](docs/architecture/overview.md). + CodeStory builds a local evidence layer so agents can request grounded context instead of relying on ad hoc file reads. diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index b4d86b73..c73a891b 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,9 +9,9 @@ plans a change. ## The Loop -```text -doctor -> index -> ground -> search -> symbol/trail/snippet/explore -> context -``` +`doctor` → `index` → `ground` or `search` → inspect with `symbol` / `trail` / `snippet` / `explore` → `context` or `packet`. + +For the two-lane system map (SQLite cache vs sidecars), see [README — What It Builds](../README.md#what-it-builds). - `doctor` checks whether the cache, index, retrieval mode, and local embedding setup are usable. diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index a3651ea9..52f78d8a 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -17,6 +17,14 @@ does not prove packet/search sidecar readiness. vars, troubleshooting, and CI smoke sequences. Proof tiers and promotion checklists live in [`retrieval-architecture.md`](../testing/retrieval-architecture.md). +```mermaid +flowchart LR + cli[codestory-cli] --> zoekt["Zoekt localhost:6070"] + cli --> qdrant["Qdrant localhost:6333"] + cli --> scip[SCIP artifacts in user cache] + cli --> embed[llama.cpp embedding endpoint] +``` + --- ## Prerequisites diff --git a/docs/usage.md b/docs/usage.md index 3bd8702b..b8d141b1 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -45,6 +45,16 @@ TARGET_WORKSPACE="/path/to/repo" CodeStory has two readiness tracks. Keep them separate when deciding whether an agent can rely on packet/search output. +| | Local navigation | Agent packet/search | +| --- | --- | --- | +| Lane id | `local_navigation` | `agent_packet_search` | +| Built by | `index` | `index` then `retrieval index` | +| Requires | Healthy SQLite cache and graph | Sidecars healthy and `retrieval_mode=full` | +| Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | +| Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | + +System map: [README — What It Builds](../README.md#what-it-builds). + ### Local navigation/cache readiness This lane is for local browsing and source navigation. It uses the project From 3cda22ba21486d0686bde95f65e692e48d28c672 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:34:36 -0400 Subject: [PATCH 04/14] rewrite readme intro around why and move system map up Replace the Public Promise wall with a user-facing why section, put the diagram right under it, and tuck caveats into Evidence and claims. Co-authored-by: Cursor --- README.md | 110 +++++++++++---------------- docs/concepts/how-codestory-works.md | 2 +- docs/usage.md | 2 +- 3 files changed, 45 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index 011aa53f..c7a3a3e6 100644 --- a/README.md +++ b/README.md @@ -10,32 +10,40 @@ Local codebase grounding for coding agents. Benchmarks

-CodeStory builds a local evidence layer for a repository. It indexes files, -symbols, relationships, snippets, search state, and freshness notes into a -per-project SQLite cache, then exposes that evidence through a CLI and -`serve --stdio`. - -Use it when a coding agent needs repository context before explaining behavior, -planning a change, or choosing files to inspect. The workflow is explicit: check -cache health, build or refresh the index, find candidate symbols, inspect -relationships, pull snippets, and return an answer tied to source evidence. - -Repository contents and inference stay local after the required tool or model -assets are installed. Setup can fetch the CodeStory source artifact or managed -embedding assets; the indexed project data stays in the user cache and commands -stay explicit about which workspace they read. - -## Public Promise - -CodeStory is a local evidence layer for repositories, not an automatic -correctness guarantee. It gives operators and coding agents explicit commands -for cache health, indexing, search, trails, snippets, and source-backed answers -that name the files they used. The per-project SQLite cache is separate from -the optional local retrieval sidecars used by packet/search workflows; a healthy -local navigation readiness report does not by itself prove agent packet/search -readiness and does not by itself prove sidecar readiness. Benchmark notes are -environment- and repository-specific evidence, so public claims should cite the -checked setup instead of promising universal speedups or savings. +## Why CodeStory + +You've watched an agent grep the wrong folder, explain code it never opened, or +spend half a session rediscovering the same files. Big repos are not hard because +the model is dumb — they are hard because **nobody gave the agent a map of the +codebase**. + +CodeStory is that map. It indexes your repository locally — symbols, calls, +relationships, snippets — so you or your coding agent can orient before guessing. +Good answers name the files they used. Thin evidence gets called out instead of +papered over. + +Your code stays on your machine. The index lives in a local cache you control. + +## How it fits together + +```mermaid +flowchart TB + repo[Target repository] --> index[index] + index --> sqlite[(SQLite cache per project)] + index --> rindex[retrieval index] + rindex --> sidecars["Sidecars: Zoekt, Qdrant, SCIP, llama.cpp"] + sqlite --> localCmds["Browse the repo: ground, symbol, trail, snippet, explore, context, report"] + sidecars --> agentCmds["Broad agent tasks: packet and search when retrieval_mode=full"] + sqlite --> runtime[codestory-runtime] + sidecars --> runtime + runtime --> cli[CLI and serve stdio] + cli --> agent[Coding agent] + sqlite -.->|"cache alone is not enough for packet/search"| agentCmds +``` + +Most people start with the left path — index once, then explore. The right path +adds sidecars when you want agent-grade `packet` and `search`; see +[docs/usage.md](docs/usage.md) and [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). ## Try It On A Repo @@ -58,20 +66,10 @@ On Windows PowerShell, use `.\target\release\codestory-cli.exe`, environment assignments such as `$env:NAME = "value"`, and normal Windows paths such as `C:\path\to\repo`. -That basic path establishes local navigation readiness: the local cache, graph, -lexical index, and DB-backed navigation commands are usable for health, file, -symbol, trail, snippet, context, orientation checks, and derived report/export -artifacts. -`report` reads the current SQLite store and writes generated artifacts; the -Markdown report and full JSON graph export are not source-of-truth state. The managed -embedding dry-run is a local semantic setup check; it does not prove agent -packet/search readiness. - -Agent packet/search readiness requires `retrieval_mode=full` from local Zoekt, -Qdrant, SCIP, and llama.cpp sidecars. See [docs/usage.md](docs/usage.md) for the -full local-navigation versus sidecar-readiness split and -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for sidecar -setup. +That basic path gets you browsing: health checks, orientation, symbol trails, +snippets, and a generated repo report. For full agent `packet`/`search`, you +also need sidecars at `retrieval_mode=full` — covered in the usage and sidecar +docs linked above. After that first index, use narrower commands instead of asking the agent to start over: @@ -146,29 +144,7 @@ sidecar retrieval. For local cache-only inspection, start with `ground`, you have a concrete target. Use `doctor` when output looks stale, incomplete, or inconsistent. -## What It Builds - -```mermaid -flowchart TB - repo[Target repository] --> index[index] - index --> sqlite[(SQLite cache per project)] - index --> rindex[retrieval index] - rindex --> sidecars["Sidecars: Zoekt, Qdrant, SCIP, llama.cpp"] - sqlite --> localCmds["Local navigation: ground, symbol, trail, snippet, explore, context, files, affected, report"] - sidecars --> agentCmds["Agent packet/search when retrieval_mode=full"] - sqlite --> runtime[codestory-runtime] - sidecars --> runtime - runtime --> cli[CLI and serve stdio] - cli --> agent[Coding agent] - sqlite -.->|"cache alone does not prove sidecar readiness"| agentCmds -``` - -Crate and layer detail: [docs/architecture/overview.md](docs/architecture/overview.md). - -CodeStory builds a local evidence layer so agents can request grounded context -instead of relying on ad hoc file reads. - -## Language Support Claims +## Language Support CodeStory separates parser-backed graph indexing, regression-tested accuracy, structural extraction, framework route coverage, and agent packet/search @@ -192,11 +168,11 @@ For the system model, start with [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md), then [docs/architecture/overview.md](docs/architecture/overview.md). -## Evidence +## Evidence and claims -The benchmark docs are deliberately cautious. They separate current checked-in -benchmark history from the state of your local cache, which can drift and should -be checked with `doctor`. +Benchmark rows are environment-specific — see the ledger before repeating any +number. CodeStory helps agents find evidence; it does not guarantee correct +answers on its own. - Public evidence summary and caveats: [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index c73a891b..93e0cb64 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -11,7 +11,7 @@ plans a change. `doctor` → `index` → `ground` or `search` → inspect with `symbol` / `trail` / `snippet` / `explore` → `context` or `packet`. -For the two-lane system map (SQLite cache vs sidecars), see [README — What It Builds](../README.md#what-it-builds). +For the two-lane system map (SQLite cache vs sidecars), see [README — How it fits together](../README.md#how-it-fits-together). - `doctor` checks whether the cache, index, retrieval mode, and local embedding setup are usable. diff --git a/docs/usage.md b/docs/usage.md index b8d141b1..16f5981e 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -53,7 +53,7 @@ agent can rely on packet/search output. | Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | | Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | -System map: [README — What It Builds](../README.md#what-it-builds). +System map: [README — How it fits together](../README.md#how-it-fits-together). ### Local navigation/cache readiness From a810cb74bcd42ecb6ea936ea9d14d91921c34b03 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:37:13 -0400 Subject: [PATCH 05/14] add readme workflow section before architecture diagram Explain index, embed, search, trail, and answer steps with a diagram and short paragraphs before the system map. Co-authored-by: Cursor --- README.md | 46 ++++++++++++++++++++++++++++ docs/concepts/how-codestory-works.md | 2 +- 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c7a3a3e6..c48d8abd 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,52 @@ papered over. Your code stays on your machine. The index lives in a local cache you control. +## How it works + +CodeStory is a loop, not a one-shot search. You prepare a local map of the repo, +find a starting point, follow the graph to the code that matters, then answer +with evidence attached. + +```mermaid +flowchart TB + doctor[doctor — is the cache ready?] + index[index — walk the repo and build a symbol graph] + orient[ground or search — find a file, symbol, or behavior] + inspect["trail · snippet · symbol — inspect one target"] + answer["context or packet — answer with citations"] + + doctor --> index + index --> orient + orient --> inspect + inspect --> answer + + index --> embedPath["retrieval index — optional sidecar pass"] + embedPath --> embed["embed selected anchors locally"] + embed -.->|"unlocks agent-grade search"| orient +``` + +**Index.** CodeStory discovers files, parses symbols and relationships, and +writes a per-project graph — calls, imports, snippets, searchable symbol docs — +into a local SQLite cache. Run `index` once per repo (then incrementally after +changes). + +**Embed.** Most navigation uses the graph and lexical search alone. When you +want broad agent `packet` and `search`, a second pass builds sidecar indexes and +embeds a policy-selected set of anchors (entry points, public APIs, central +nodes) — not every symbol needs a vector. + +**Search.** `ground` orients you on a new repo; `search` finds candidates — +symbols, files, routes, literals. Pick a concrete target before going deeper. + +**Trail.** `trail` walks callers, callees, and references around one symbol so +you can see how code flows without opening half the tree by hand. + +**Snippet and context.** `snippet` pulls source around a hit; `context` bundles +trails, neighbors, and citations for one target. `packet` handles broad task +questions when sidecars report `retrieval_mode=full`. + +Details: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md). + ## How it fits together ```mermaid diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index 93e0cb64..a10c2cf9 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,7 +9,7 @@ plans a change. ## The Loop -`doctor` → `index` → `ground` or `search` → inspect with `symbol` / `trail` / `snippet` / `explore` → `context` or `packet`. +See [README — How it works](../README.md#how-it-works) for the indexing → search → trail → answer flow. For the two-lane system map (SQLite cache vs sidecars), see [README — How it fits together](../README.md#how-it-fits-together). From 88e883422b9378928fca3c04c148580c909044b1 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:40:47 -0400 Subject: [PATCH 06/14] rewrite readme as one concrete story, drop redundant sections Replace command flowcharts and spec-table bottom with specific indexing/search/trail/embed prose, one sequence diagram, a short get-started block, and a docs link list. Co-authored-by: Cursor --- README.md | 261 +++++++-------------------- docs/concepts/how-codestory-works.md | 4 +- docs/usage.md | 2 +- 3 files changed, 71 insertions(+), 196 deletions(-) diff --git a/README.md b/README.md index c48d8abd..4d9627ce 100644 --- a/README.md +++ b/README.md @@ -17,236 +17,113 @@ spend half a session rediscovering the same files. Big repos are not hard becaus the model is dumb — they are hard because **nobody gave the agent a map of the codebase**. -CodeStory is that map. It indexes your repository locally — symbols, calls, -relationships, snippets — so you or your coding agent can orient before guessing. -Good answers name the files they used. Thin evidence gets called out instead of -papered over. +CodeStory builds that map on your machine: a queryable graph of symbols, calls, +imports, and snippets tied to real file locations. You or your agent orient +before guessing, follow relationships instead of opening random paths, and cite +the files that actually support an answer. -Your code stays on your machine. The index lives in a local cache you control. +Your code stays local. The index lives in a per-repo cache under your user +profile, not on someone else's server. ## How it works -CodeStory is a loop, not a one-shot search. You prepare a local map of the repo, -find a starting point, follow the graph to the code that matters, then answer -with evidence attached. +**Indexing** walks the repository, parses supported languages with tree-sitter, +and writes a SQLite graph: functions, types, modules, call edges, import edges, +and source snippets with line ranges. Incremental runs pick up changed files +without rebuilding from scratch. This is the foundation — everything else reads +from that graph. -```mermaid -flowchart TB - doctor[doctor — is the cache ready?] - index[index — walk the repo and build a symbol graph] - orient[ground or search — find a file, symbol, or behavior] - inspect["trail · snippet · symbol — inspect one target"] - answer["context or packet — answer with citations"] - - doctor --> index - index --> orient - orient --> inspect - inspect --> answer - - index --> embedPath["retrieval index — optional sidecar pass"] - embedPath --> embed["embed selected anchors locally"] - embed -.->|"unlocks agent-grade search"| orient -``` +**Search** finds a starting symbol or file. `ground` summarizes a repo you have +never opened (entry points, hotspots, gaps). `search` matches names, paths, +literals, and generated symbol summaries stored during indexing — not a blind +full-repo text scan. -**Index.** CodeStory discovers files, parses symbols and relationships, and -writes a per-project graph — calls, imports, snippets, searchable symbol docs — -into a local SQLite cache. Run `index` once per repo (then incrementally after -changes). +**Trails** follow the graph. Given one symbol, `trail` walks callers, callees, +and references and returns a path through the code with file-and-line citations. +That is the difference from grep: you see *who calls whom*, not every string +match in the tree. -**Embed.** Most navigation uses the graph and lexical search alone. When you -want broad agent `packet` and `search`, a second pass builds sidecar indexes and -embeds a policy-selected set of anchors (entry points, public APIs, central -nodes) — not every symbol needs a vector. +**Snippets and context** pull the actual source around a node. `context` bundles +trails, neighbors, and citations when you need a handoff packet for one target. -**Search.** `ground` orients you on a new repo; `search` finds candidates — -symbols, files, routes, literals. Pick a concrete target before going deeper. +**Embeddings** are optional and separate. Most commands only need the graph. +Broad agent `packet` and sidecar-backed `search` add a second pass: local Zoekt +and Qdrant sidecars, plus vectors for a *small policy-selected set* of anchors +(entry points, public APIs, high-centrality nodes) — not an embedding per symbol. +See [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) when you +need that lane. -**Trail.** `trail` walks callers, callees, and references around one symbol so -you can see how code flows without opening half the tree by hand. +```mermaid +sequenceDiagram + participant You + participant CLI as codestory-cli + participant Index as Local graph index -**Snippet and context.** `snippet` pulls source around a hit; `context` bundles -trails, neighbors, and citations for one target. `packet` handles broad task -questions when sidecars report `retrieval_mode=full`. + You->>CLI: index --project ./my-app + CLI->>Index: parse files, record calls and imports, store snippets + Note over Index: SQLite cache on disk, keyed by repo path -Details: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md). + You->>CLI: search --query "refresh token" + CLI->>Index: match symbol docs and graph nodes + CLI-->>You: node id for the handler you care about -## How it fits together + You->>CLI: trail --id that-node --story + CLI->>Index: walk call edges up and down + CLI-->>You: Middleware calls Handler calls Store, each with file citations -```mermaid -flowchart TB - repo[Target repository] --> index[index] - index --> sqlite[(SQLite cache per project)] - index --> rindex[retrieval index] - rindex --> sidecars["Sidecars: Zoekt, Qdrant, SCIP, llama.cpp"] - sqlite --> localCmds["Browse the repo: ground, symbol, trail, snippet, explore, context, report"] - sidecars --> agentCmds["Broad agent tasks: packet and search when retrieval_mode=full"] - sqlite --> runtime[codestory-runtime] - sidecars --> runtime - runtime --> cli[CLI and serve stdio] - cli --> agent[Coding agent] - sqlite -.->|"cache alone is not enough for packet/search"| agentCmds + You->>CLI: snippet --id that-node + CLI-->>You: source lines around the function body ``` -Most people start with the left path — index once, then explore. The right path -adds sidecars when you want agent-grade `packet` and `search`; see -[docs/usage.md](docs/usage.md) and [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). +Optional sidecar setup adds `packet --question "how does auth work?"` — a cited +evidence bundle for a broad task without the agent opening twenty files first. +Operator details: [docs/usage.md](docs/usage.md). -## Try It On A Repo - -From this checkout, build the CLI and point it at any repository: +## Get started ```sh cargo build --release -p codestory-cli -CODESTORY_CLI="./target/release/codestory-cli" -TARGET_WORKSPACE="/path/to/repo" +export CODESTORY_CLI="./target/release/codestory-cli" +export TARGET_WORKSPACE="/path/to/your/repo" "$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" -"$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json "$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full "$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why "$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --output-file codestory-report.md -"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --format json --output-file codestory-graph.json ``` -On Windows PowerShell, use `.\target\release\codestory-cli.exe`, environment -assignments such as `$env:NAME = "value"`, and normal Windows paths such as -`C:\path\to\repo`. - -That basic path gets you browsing: health checks, orientation, symbol trails, -snippets, and a generated repo report. For full agent `packet`/`search`, you -also need sidecars at `retrieval_mode=full` — covered in the usage and sidecar -docs linked above. +On Windows, use `.\target\release\codestory-cli.exe` and `$env:TARGET_WORKSPACE = +"C:\path\to\repo"`. -After that first index, use narrower commands instead of asking the agent to -start over: +Then drill into one symbol: `search` → `trail` → `snippet`. Full command +reference and recovery flows live in [docs/usage.md](docs/usage.md). -```sh -"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "request routing" --why -"$CODESTORY_CLI" trail --project "$TARGET_WORKSPACE" --id --story --hide-speculative -"$CODESTORY_CLI" snippet --project "$TARGET_WORKSPACE" --id --context 40 -``` +## Use with an agent -A good CodeStory-backed answer should name the source files it used, say when -evidence is stale or partial, and give the next concrete command when more proof -is needed. - -For task-shaped flows, use [docs/usage.md](docs/usage.md). - -## Retrieval sidecars - -For Zoekt/Qdrant/SCIP packet retrieval, run `cargo retrieval-setup` once from -this repository root, then follow -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for bootstrap -flags, version pins, and troubleshooting. - -## Install As An Agent Skill - -Use this path when CodeStory should be installed once as a grounding skill and -then pointed at whatever repository an agent is working on. +Install the grounding skill once, point it at whatever repo the agent is working +on, and persist the `CODESTORY_CLI` path the setup script prints. ```sh SkillHome="" -mkdir -p "$SkillHome" cp -R ./.agents/skills/codestory-grounding "$SkillHome/codestory-grounding" bash "$SkillHome/codestory-grounding/scripts/setup.sh" ``` -On Windows PowerShell: +Skill source: [.agents/skills/codestory-grounding/SKILL.md](.agents/skills/codestory-grounding/SKILL.md). +Windows setup is in the same folder under `scripts/setup.ps1`. -```powershell -$SkillHome = "" -New-Item -ItemType Directory -Force -Path $SkillHome | Out-Null -Copy-Item -Recurse -Force .\.agents\skills\codestory-grounding "$SkillHome\codestory-grounding" -& "$SkillHome\codestory-grounding\scripts\setup.ps1" -``` +For a persistent read surface (stdio MCP-style queries against a warm index), +use `codestory-cli serve --project --stdio`. + +## Documentation -The setup script prints `CODESTORY_CLI=`. Persist that path if your agent -environment does not preserve variables between sessions. - -The skill package lives at -[.agents/skills/codestory-grounding/SKILL.md](.agents/skills/codestory-grounding/SKILL.md). - -## Core Flow - -| Need | Command | -| --- | --- | -| Local navigation readiness | `codestory-cli doctor --project ` | -| Build or refresh an index | `codestory-cli index --project --refresh full` | -| Broad orientation | `codestory-cli ground --project --why` | -| Repo report / graph export | `codestory-cli report --project --format markdown` | -| Broad task evidence (requires full sidecar retrieval) | `codestory-cli packet --project --question "" --budget compact` | -| Candidate discovery (requires full sidecar retrieval) | `codestory-cli search --project --query "" --why` | -| Exact symbol evidence | `codestory-cli symbol --project --id ` | -| Flow evidence | `codestory-cli trail --project --id --story --hide-speculative` | -| Source excerpt | `codestory-cli snippet --project --id ` | -| Bundled navigation packet | `codestory-cli explore --project --id --no-tui` | -| Deep context bundle | `codestory-cli context --project --id ` | -| Changed-file impact | `codestory-cli affected --project --format markdown` | -| Persistent read surface | `codestory-cli serve --project --stdio` | - -Use `packet` for broad task questions once `ready --goal agent` reports full -sidecar retrieval. For local cache-only inspection, start with `ground`, -`report`, or `doctor`, then use `symbol`, `trail`, `snippet`, or `context` after -you have a concrete target. Use `doctor` when output looks stale, incomplete, or -inconsistent. - -## Language Support - -CodeStory separates parser-backed graph indexing, regression-tested accuracy, -structural extraction, framework route coverage, and agent packet/search -readiness. The current contract is documented in -[docs/architecture/language-support.md](docs/architecture/language-support.md). - -In short: Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, -PHP, C#, Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph -languages; HTML, CSS, and SQL use structural collectors. - -The opt-in OSS language corpus pairs each public language-support profile with a -pinned medium-sized open source project and compares raw filesystem counts -against CodeStory indexing of the same files: -[docs/testing/oss-language-corpus.md](docs/testing/oss-language-corpus.md). -The separate `language-expansion-holdout` benchmark suite runs strict -`without_codestory` versus `with_codestory` agent tasks on those pinned -projects and records elapsed time, token usage, estimated cost, tool calls, -command counts, source reads, post-packet source reads, and quality gates. - -For the system model, start with -[docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md), -then [docs/architecture/overview.md](docs/architecture/overview.md). - -## Evidence and claims - -Benchmark rows are environment-specific — see the ledger before repeating any -number. CodeStory helps agents find evidence; it does not guarantee correct -answers on its own. - -- Public evidence summary and caveats: - [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) -- Repo-scale timing history: - [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md) -- Warm stdio loop evidence: - [docs/testing/codestory-stdio-warm-loop-stats.md](docs/testing/codestory-stdio-warm-loop-stats.md) -- Repeatable with/without harness: - [`scripts/codestory-agent-ab-benchmark.mjs`](scripts/codestory-agent-ab-benchmark.mjs) - -Do not promote a single benchmark row into a universal savings claim. - -## Hack On CodeStory - -Start with the contributor docs, then run Cargo checks serially because this -workspace shares build locks. - -- [docs/contributors/getting-started.md](docs/contributors/getting-started.md) -- [docs/contributors/debugging.md](docs/contributors/debugging.md) -- [docs/contributors/testing-matrix.md](docs/contributors/testing-matrix.md) -- [docs/architecture/runtime-execution-path.md](docs/architecture/runtime-execution-path.md) -- [docs/architecture/language-support.md](docs/architecture/language-support.md) -- [docs/architecture/subsystems/contracts.md](docs/architecture/subsystems/contracts.md) -- [docs/architecture/subsystems/workspace.md](docs/architecture/subsystems/workspace.md) -- [docs/architecture/subsystems/indexer.md](docs/architecture/subsystems/indexer.md) -- [docs/architecture/subsystems/store.md](docs/architecture/subsystems/store.md) -- [docs/architecture/subsystems/runtime.md](docs/architecture/subsystems/runtime.md) -- [docs/architecture/subsystems/cli.md](docs/architecture/subsystems/cli.md) +- [Usage and workflows](docs/usage.md) +- [How CodeStory works (concepts)](docs/concepts/how-codestory-works.md) +- [Architecture overview](docs/architecture/overview.md) +- [Retrieval sidecars](docs/ops/retrieval-sidecars.md) — when you need `packet` / full `search` +- [Language support](docs/architecture/language-support.md) — parser-backed languages vs structural collectors +- [Contributing](docs/contributors/getting-started.md) +- [Benchmarks and evidence notes](docs/testing/benchmark-ledger.md) ## License diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index a10c2cf9..fbf43716 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,9 +9,7 @@ plans a change. ## The Loop -See [README — How it works](../README.md#how-it-works) for the indexing → search → trail → answer flow. - -For the two-lane system map (SQLite cache vs sidecars), see [README — How it fits together](../README.md#how-it-fits-together). +See [README — How it works](../README.md#how-it-works) for indexing, search, trails, snippets, and the optional embedding lane. - `doctor` checks whether the cache, index, retrieval mode, and local embedding setup are usable. diff --git a/docs/usage.md b/docs/usage.md index 16f5981e..d9963c70 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -53,7 +53,7 @@ agent can rely on packet/search output. | Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | | Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | -System map: [README — How it fits together](../README.md#how-it-fits-together). +See [README — How it works](../README.md#how-it-works) for indexing, trails, snippets, and embeddings. ### Local navigation/cache readiness From 9af8efe5e2b3e3b904ca917acc86136128fc2137 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:41:15 -0400 Subject: [PATCH 07/14] restore readme and concept links from main Undo the experimental readme rewrites; the original structure and reference sections were more useful. Co-authored-by: Cursor --- README.md | 262 ++++++++++++++++++--------- docs/concepts/how-codestory-works.md | 4 +- docs/usage.md | 10 - 3 files changed, 181 insertions(+), 95 deletions(-) diff --git a/README.md b/README.md index 4d9627ce..4dc9f716 100644 --- a/README.md +++ b/README.md @@ -10,120 +10,214 @@ Local codebase grounding for coding agents. Benchmarks

-## Why CodeStory +CodeStory builds a local evidence layer for a repository. It indexes files, +symbols, relationships, snippets, search state, and freshness notes into a +per-project SQLite cache, then exposes that evidence through a CLI and +`serve --stdio`. -You've watched an agent grep the wrong folder, explain code it never opened, or -spend half a session rediscovering the same files. Big repos are not hard because -the model is dumb — they are hard because **nobody gave the agent a map of the -codebase**. +Use it when a coding agent needs repository context before explaining behavior, +planning a change, or choosing files to inspect. The workflow is explicit: check +cache health, build or refresh the index, find candidate symbols, inspect +relationships, pull snippets, and return an answer tied to source evidence. -CodeStory builds that map on your machine: a queryable graph of symbols, calls, -imports, and snippets tied to real file locations. You or your agent orient -before guessing, follow relationships instead of opening random paths, and cite -the files that actually support an answer. +Repository contents and inference stay local after the required tool or model +assets are installed. Setup can fetch the CodeStory source artifact or managed +embedding assets; the indexed project data stays in the user cache and commands +stay explicit about which workspace they read. -Your code stays local. The index lives in a per-repo cache under your user -profile, not on someone else's server. +## Public Promise -## How it works +CodeStory is a local evidence layer for repositories, not an automatic +correctness guarantee. It gives operators and coding agents explicit commands +for cache health, indexing, search, trails, snippets, and source-backed answers +that name the files they used. The per-project SQLite cache is separate from +the optional local retrieval sidecars used by packet/search workflows; a healthy +local navigation readiness report does not by itself prove agent packet/search +readiness and does not by itself prove sidecar readiness. Benchmark notes are +environment- and repository-specific evidence, so public claims should cite the +checked setup instead of promising universal speedups or savings. -**Indexing** walks the repository, parses supported languages with tree-sitter, -and writes a SQLite graph: functions, types, modules, call edges, import edges, -and source snippets with line ranges. Incremental runs pick up changed files -without rebuilding from scratch. This is the foundation — everything else reads -from that graph. +## Try It On A Repo -**Search** finds a starting symbol or file. `ground` summarizes a repo you have -never opened (entry points, hotspots, gaps). `search` matches names, paths, -literals, and generated symbol summaries stored during indexing — not a blind -full-repo text scan. +From this checkout, build the CLI and point it at any repository: -**Trails** follow the graph. Given one symbol, `trail` walks callers, callees, -and references and returns a path through the code with file-and-line citations. -That is the difference from grep: you see *who calls whom*, not every string -match in the tree. - -**Snippets and context** pull the actual source around a node. `context` bundles -trails, neighbors, and citations when you need a handoff packet for one target. +```sh +cargo build --release -p codestory-cli +CODESTORY_CLI="./target/release/codestory-cli" +TARGET_WORKSPACE="/path/to/repo" -**Embeddings** are optional and separate. Most commands only need the graph. -Broad agent `packet` and sidecar-backed `search` add a second pass: local Zoekt -and Qdrant sidecars, plus vectors for a *small policy-selected set* of anchors -(entry points, public APIs, high-centrality nodes) — not an embedding per symbol. -See [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) when you -need that lane. +"$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" +"$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json +"$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full +"$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why +"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --output-file codestory-report.md +"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --format json --output-file codestory-graph.json +``` -```mermaid -sequenceDiagram - participant You - participant CLI as codestory-cli - participant Index as Local graph index +On Windows PowerShell, use `.\target\release\codestory-cli.exe`, environment +assignments such as `$env:NAME = "value"`, and normal Windows paths such as +`C:\path\to\repo`. - You->>CLI: index --project ./my-app - CLI->>Index: parse files, record calls and imports, store snippets - Note over Index: SQLite cache on disk, keyed by repo path +That basic path establishes local navigation readiness: the local cache, graph, +lexical index, and DB-backed navigation commands are usable for health, file, +symbol, trail, snippet, context, orientation checks, and derived report/export +artifacts. +`report` reads the current SQLite store and writes generated artifacts; the +Markdown report and full JSON graph export are not source-of-truth state. The managed +embedding dry-run is a local semantic setup check; it does not prove agent +packet/search readiness. - You->>CLI: search --query "refresh token" - CLI->>Index: match symbol docs and graph nodes - CLI-->>You: node id for the handler you care about +Agent packet/search readiness requires `retrieval_mode=full` from local Zoekt, +Qdrant, SCIP, and llama.cpp sidecars. See [docs/usage.md](docs/usage.md) for the +full local-navigation versus sidecar-readiness split and +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for sidecar +setup. - You->>CLI: trail --id that-node --story - CLI->>Index: walk call edges up and down - CLI-->>You: Middleware calls Handler calls Store, each with file citations +After that first index, use narrower commands instead of asking the agent to +start over: - You->>CLI: snippet --id that-node - CLI-->>You: source lines around the function body +```sh +"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "request routing" --why +"$CODESTORY_CLI" trail --project "$TARGET_WORKSPACE" --id --story --hide-speculative +"$CODESTORY_CLI" snippet --project "$TARGET_WORKSPACE" --id --context 40 ``` -Optional sidecar setup adds `packet --question "how does auth work?"` — a cited -evidence bundle for a broad task without the agent opening twenty files first. -Operator details: [docs/usage.md](docs/usage.md). +A good CodeStory-backed answer should name the source files it used, say when +evidence is stale or partial, and give the next concrete command when more proof +is needed. -## Get started +For task-shaped flows, use [docs/usage.md](docs/usage.md). -```sh -cargo build --release -p codestory-cli -export CODESTORY_CLI="./target/release/codestory-cli" -export TARGET_WORKSPACE="/path/to/your/repo" +## Retrieval sidecars -"$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" -"$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full -"$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why -"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --output-file codestory-report.md -``` - -On Windows, use `.\target\release\codestory-cli.exe` and `$env:TARGET_WORKSPACE = -"C:\path\to\repo"`. +For Zoekt/Qdrant/SCIP packet retrieval, run `cargo retrieval-setup` once from +this repository root, then follow +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for bootstrap +flags, version pins, and troubleshooting. -Then drill into one symbol: `search` → `trail` → `snippet`. Full command -reference and recovery flows live in [docs/usage.md](docs/usage.md). +## Install As An Agent Skill -## Use with an agent - -Install the grounding skill once, point it at whatever repo the agent is working -on, and persist the `CODESTORY_CLI` path the setup script prints. +Use this path when CodeStory should be installed once as a grounding skill and +then pointed at whatever repository an agent is working on. ```sh SkillHome="" +mkdir -p "$SkillHome" cp -R ./.agents/skills/codestory-grounding "$SkillHome/codestory-grounding" bash "$SkillHome/codestory-grounding/scripts/setup.sh" ``` -Skill source: [.agents/skills/codestory-grounding/SKILL.md](.agents/skills/codestory-grounding/SKILL.md). -Windows setup is in the same folder under `scripts/setup.ps1`. +On Windows PowerShell: + +```powershell +$SkillHome = "" +New-Item -ItemType Directory -Force -Path $SkillHome | Out-Null +Copy-Item -Recurse -Force .\.agents\skills\codestory-grounding "$SkillHome\codestory-grounding" +& "$SkillHome\codestory-grounding\scripts\setup.ps1" +``` -For a persistent read surface (stdio MCP-style queries against a warm index), -use `codestory-cli serve --project --stdio`. +The setup script prints `CODESTORY_CLI=`. Persist that path if your agent +environment does not preserve variables between sessions. + +The skill package lives at +[.agents/skills/codestory-grounding/SKILL.md](.agents/skills/codestory-grounding/SKILL.md). + +## Core Flow + +| Need | Command | +| --- | --- | +| Local navigation readiness | `codestory-cli doctor --project ` | +| Build or refresh an index | `codestory-cli index --project --refresh full` | +| Broad orientation | `codestory-cli ground --project --why` | +| Repo report / graph export | `codestory-cli report --project --format markdown` | +| Broad task evidence (requires full sidecar retrieval) | `codestory-cli packet --project --question "" --budget compact` | +| Candidate discovery (requires full sidecar retrieval) | `codestory-cli search --project --query "" --why` | +| Exact symbol evidence | `codestory-cli symbol --project --id ` | +| Flow evidence | `codestory-cli trail --project --id --story --hide-speculative` | +| Source excerpt | `codestory-cli snippet --project --id ` | +| Bundled navigation packet | `codestory-cli explore --project --id --no-tui` | +| Deep context bundle | `codestory-cli context --project --id ` | +| Changed-file impact | `codestory-cli affected --project --format markdown` | +| Persistent read surface | `codestory-cli serve --project --stdio` | + +Use `packet` for broad task questions once `ready --goal agent` reports full +sidecar retrieval. For local cache-only inspection, start with `ground`, +`report`, or `doctor`, then use `symbol`, `trail`, `snippet`, or `context` after +you have a concrete target. Use `doctor` when output looks stale, incomplete, or +inconsistent. + +## What It Builds -## Documentation +```mermaid +flowchart LR + Repo["repository"] --> Workspace["workspace discovery"] + Workspace --> Indexer["symbol and edge extraction"] + Indexer --> Store["SQLite store"] + Store --> Runtime["retrieval and context assembly"] + Runtime --> CLI["CLI and stdio reads"] + CLI --> Agent["coding agent"] +``` -- [Usage and workflows](docs/usage.md) -- [How CodeStory works (concepts)](docs/concepts/how-codestory-works.md) -- [Architecture overview](docs/architecture/overview.md) -- [Retrieval sidecars](docs/ops/retrieval-sidecars.md) — when you need `packet` / full `search` -- [Language support](docs/architecture/language-support.md) — parser-backed languages vs structural collectors -- [Contributing](docs/contributors/getting-started.md) -- [Benchmarks and evidence notes](docs/testing/benchmark-ledger.md) +CodeStory builds a local evidence layer so agents can request grounded context +instead of relying on ad hoc file reads. + +## Language Support Claims + +CodeStory separates parser-backed graph indexing, regression-tested accuracy, +structural extraction, framework route coverage, and agent packet/search +readiness. The current contract is documented in +[docs/architecture/language-support.md](docs/architecture/language-support.md). + +In short: Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, +PHP, C#, Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph +languages; HTML, CSS, and SQL use structural collectors. + +The opt-in OSS language corpus pairs each public language-support profile with a +pinned medium-sized open source project and compares raw filesystem counts +against CodeStory indexing of the same files: +[docs/testing/oss-language-corpus.md](docs/testing/oss-language-corpus.md). +The separate `language-expansion-holdout` benchmark suite runs strict +`without_codestory` versus `with_codestory` agent tasks on those pinned +projects and records elapsed time, token usage, estimated cost, tool calls, +command counts, source reads, post-packet source reads, and quality gates. + +For the system model, start with +[docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md), +then [docs/architecture/overview.md](docs/architecture/overview.md). + +## Evidence + +The benchmark docs are deliberately cautious. They separate current checked-in +benchmark history from the state of your local cache, which can drift and should +be checked with `doctor`. + +- Public evidence summary and caveats: + [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) +- Repo-scale timing history: + [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md) +- Warm stdio loop evidence: + [docs/testing/codestory-stdio-warm-loop-stats.md](docs/testing/codestory-stdio-warm-loop-stats.md) +- Repeatable with/without harness: + [`scripts/codestory-agent-ab-benchmark.mjs`](scripts/codestory-agent-ab-benchmark.mjs) + +Do not promote a single benchmark row into a universal savings claim. + +## Hack On CodeStory + +Start with the contributor docs, then run Cargo checks serially because this +workspace shares build locks. + +- [docs/contributors/getting-started.md](docs/contributors/getting-started.md) +- [docs/contributors/debugging.md](docs/contributors/debugging.md) +- [docs/contributors/testing-matrix.md](docs/contributors/testing-matrix.md) +- [docs/architecture/runtime-execution-path.md](docs/architecture/runtime-execution-path.md) +- [docs/architecture/language-support.md](docs/architecture/language-support.md) +- [docs/architecture/subsystems/contracts.md](docs/architecture/subsystems/contracts.md) +- [docs/architecture/subsystems/workspace.md](docs/architecture/subsystems/workspace.md) +- [docs/architecture/subsystems/indexer.md](docs/architecture/subsystems/indexer.md) +- [docs/architecture/subsystems/store.md](docs/architecture/subsystems/store.md) +- [docs/architecture/subsystems/runtime.md](docs/architecture/subsystems/runtime.md) +- [docs/architecture/subsystems/cli.md](docs/architecture/subsystems/cli.md) ## License diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index fbf43716..b4d86b73 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,7 +9,9 @@ plans a change. ## The Loop -See [README — How it works](../README.md#how-it-works) for indexing, search, trails, snippets, and the optional embedding lane. +```text +doctor -> index -> ground -> search -> symbol/trail/snippet/explore -> context +``` - `doctor` checks whether the cache, index, retrieval mode, and local embedding setup are usable. diff --git a/docs/usage.md b/docs/usage.md index d9963c70..3bd8702b 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -45,16 +45,6 @@ TARGET_WORKSPACE="/path/to/repo" CodeStory has two readiness tracks. Keep them separate when deciding whether an agent can rely on packet/search output. -| | Local navigation | Agent packet/search | -| --- | --- | --- | -| Lane id | `local_navigation` | `agent_packet_search` | -| Built by | `index` | `index` then `retrieval index` | -| Requires | Healthy SQLite cache and graph | Sidecars healthy and `retrieval_mode=full` | -| Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | -| Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | - -See [README — How it works](../README.md#how-it-works) for indexing, trails, snippets, and embeddings. - ### Local navigation/cache readiness This lane is for local browsing and source navigation. It uses the project From 80f9edee6f7696ccf03f35e768ee250403413f0c Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:43:41 -0400 Subject: [PATCH 08/14] rewrite readme as one clear story end to end Replace duplicate diagrams and spec prose with numbered workflow steps, one evidence-flow diagram, a short try-it block, and a compact reference footer. Update onboarding contract tests. Co-authored-by: Cursor --- README.md | 269 ++++++------------ .../tests/onboarding_contracts.rs | 39 +-- docs/concepts/how-codestory-works.md | 6 +- docs/usage.md | 13 +- 4 files changed, 115 insertions(+), 212 deletions(-) diff --git a/README.md b/README.md index 4dc9f716..78f81062 100644 --- a/README.md +++ b/README.md @@ -10,214 +10,117 @@ Local codebase grounding for coding agents. Benchmarks

-CodeStory builds a local evidence layer for a repository. It indexes files, -symbols, relationships, snippets, search state, and freshness notes into a -per-project SQLite cache, then exposes that evidence through a CLI and -`serve --stdio`. +## Why CodeStory -Use it when a coding agent needs repository context before explaining behavior, -planning a change, or choosing files to inspect. The workflow is explicit: check -cache health, build or refresh the index, find candidate symbols, inspect -relationships, pull snippets, and return an answer tied to source evidence. +Agents fail on real repos the same way humans do when they are new: they open the +wrong files, chase a plausible name, and call it done. CodeStory indexes the +repository first — who calls whom, where symbols live, the actual source — so +the next step is evidence, not guesswork. -Repository contents and inference stay local after the required tool or model -assets are installed. Setup can fetch the CodeStory source artifact or managed -embedding assets; the indexed project data stays in the user cache and commands -stay explicit about which workspace they read. +It runs locally. Your code stays on your machine. -## Public Promise +## How it works -CodeStory is a local evidence layer for repositories, not an automatic -correctness guarantee. It gives operators and coding agents explicit commands -for cache health, indexing, search, trails, snippets, and source-backed answers -that name the files they used. The per-project SQLite cache is separate from -the optional local retrieval sidecars used by packet/search workflows; a healthy -local navigation readiness report does not by itself prove agent packet/search -readiness and does not by itself prove sidecar readiness. Benchmark notes are -environment- and repository-specific evidence, so public claims should cite the -checked setup instead of promising universal speedups or savings. +**1. Index the repo.** CodeStory walks your tree (honoring normal ignore rules), +parses supported languages into symbols and edges — calls, imports, overrides — +and stores snippets plus a searchable graph in a per-project SQLite cache. One +full `index` up front; incremental refresh after edits. -## Try It On A Repo +**2. Find a foothold.** Ask where something lives: a handler name, a route, a +type, a string literal. `ground` summarizes a repo you have never seen; `search` +returns ranked candidates with file paths and graph ids. -From this checkout, build the CLI and point it at any repository: +**3. Follow the graph.** Pick one symbol. `trail` shows callers and callees; +`snippet` returns the surrounding source. You are walking relationships in the +index, not grepping random directories. -```sh -cargo build --release -p codestory-cli -CODESTORY_CLI="./target/release/codestory-cli" -TARGET_WORKSPACE="/path/to/repo" - -"$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" -"$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json -"$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full -"$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why -"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --output-file codestory-report.md -"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --format json --output-file codestory-graph.json -``` - -On Windows PowerShell, use `.\target\release\codestory-cli.exe`, environment -assignments such as `$env:NAME = "value"`, and normal Windows paths such as -`C:\path\to\repo`. +**4. Answer with proof.** `context` bundles trails, neighbors, and citations +around one target. For broad questions ("how does indexing persist state?"), +`packet` assembles a bounded evidence packet — but only when sidecar retrieval +is fully healthy (`retrieval_mode=full`). -That basic path establishes local navigation readiness: the local cache, graph, -lexical index, and DB-backed navigation commands are usable for health, file, -symbol, trail, snippet, context, orientation checks, and derived report/export -artifacts. -`report` reads the current SQLite store and writes generated artifacts; the -Markdown report and full JSON graph export are not source-of-truth state. The managed -embedding dry-run is a local semantic setup check; it does not prove agent -packet/search readiness. +**Embeddings are optional, not step one.** Most symbols are found through the +graph and lexical symbol docs. A separate `retrieval index` pass builds Zoekt, +Qdrant, and SCIP sidecars and embeds only policy-selected anchors (entry +points, public APIs, high-centrality nodes) when you need agent-grade +`packet`/`search`. Details: [docs/usage.md](docs/usage.md), +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). -Agent packet/search readiness requires `retrieval_mode=full` from local Zoekt, -Qdrant, SCIP, and llama.cpp sidecars. See [docs/usage.md](docs/usage.md) for the -full local-navigation versus sidecar-readiness split and -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for sidecar -setup. - -After that first index, use narrower commands instead of asking the agent to -start over: - -```sh -"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "request routing" --why -"$CODESTORY_CLI" trail --project "$TARGET_WORKSPACE" --id --story --hide-speculative -"$CODESTORY_CLI" snippet --project "$TARGET_WORKSPACE" --id --context 40 +```mermaid +flowchart LR + files[Source files] --> index[Parse into symbols and edges] + index --> graph[(Local graph and snippets)] + question[Question or symbol] --> graph + graph --> match[Matching files and symbols] + match --> walk[Caller and callee paths] + walk --> source[Source at file and line] + source --> cite[Answer with citations] + graph --> sidecars[Optional sidecar indexes] + sidecars --> cite ``` -A good CodeStory-backed answer should name the source files it used, say when -evidence is stale or partial, and give the next concrete command when more proof -is needed. - -For task-shaped flows, use [docs/usage.md](docs/usage.md). - -## Retrieval sidecars +Example: *"Who calls `WorkspaceIndexer`?"* → search returns the symbol → trail +lists callers across crates → snippet shows the call sites → you edit with paths +already in hand. -For Zoekt/Qdrant/SCIP packet retrieval, run `cargo retrieval-setup` once from -this repository root, then follow -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for bootstrap -flags, version pins, and troubleshooting. +More depth: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md). -## Install As An Agent Skill - -Use this path when CodeStory should be installed once as a grounding skill and -then pointed at whatever repository an agent is working on. +## Try it ```sh -SkillHome="" -mkdir -p "$SkillHome" -cp -R ./.agents/skills/codestory-grounding "$SkillHome/codestory-grounding" -bash "$SkillHome/codestory-grounding/scripts/setup.sh" -``` - -On Windows PowerShell: +cargo build --release -p codestory-cli +export CODESTORY_CLI="./target/release/codestory-cli" +export TARGET_WORKSPACE="/path/to/repo" -```powershell -$SkillHome = "" -New-Item -ItemType Directory -Force -Path $SkillHome | Out-Null -Copy-Item -Recurse -Force .\.agents\skills\codestory-grounding "$SkillHome\codestory-grounding" -& "$SkillHome\codestory-grounding\scripts\setup.ps1" +"$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" +"$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full +"$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why +"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "WorkspaceIndexer" --why ``` -The setup script prints `CODESTORY_CLI=`. Persist that path if your agent -environment does not preserve variables between sessions. +On Windows use `.\target\release\codestory-cli.exe` and `$env:TARGET_WORKSPACE = "C:\path\to\repo"`. -The skill package lives at -[.agents/skills/codestory-grounding/SKILL.md](.agents/skills/codestory-grounding/SKILL.md). +That gets you a local graph you can browse with `trail`, `snippet`, `symbol`, +`explore`, `context`, and `report`. Add sidecars when you need `packet`; see +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). -## Core Flow +## Install as an agent skill -| Need | Command | -| --- | --- | -| Local navigation readiness | `codestory-cli doctor --project ` | -| Build or refresh an index | `codestory-cli index --project --refresh full` | -| Broad orientation | `codestory-cli ground --project --why` | -| Repo report / graph export | `codestory-cli report --project --format markdown` | -| Broad task evidence (requires full sidecar retrieval) | `codestory-cli packet --project --question "" --budget compact` | -| Candidate discovery (requires full sidecar retrieval) | `codestory-cli search --project --query "" --why` | -| Exact symbol evidence | `codestory-cli symbol --project --id ` | -| Flow evidence | `codestory-cli trail --project --id --story --hide-speculative` | -| Source excerpt | `codestory-cli snippet --project --id ` | -| Bundled navigation packet | `codestory-cli explore --project --id --no-tui` | -| Deep context bundle | `codestory-cli context --project --id ` | -| Changed-file impact | `codestory-cli affected --project --format markdown` | -| Persistent read surface | `codestory-cli serve --project --stdio` | - -Use `packet` for broad task questions once `ready --goal agent` reports full -sidecar retrieval. For local cache-only inspection, start with `ground`, -`report`, or `doctor`, then use `symbol`, `trail`, `snippet`, or `context` after -you have a concrete target. Use `doctor` when output looks stale, incomplete, or -inconsistent. - -## What It Builds +Copy [`.agents/skills/codestory-grounding`](.agents/skills/codestory-grounding) into +your agent skill directory and run `scripts/setup.sh` (or `setup.ps1` on +Windows). Skill source: [`.agents/skills/codestory-grounding/SKILL.md`](.agents/skills/codestory-grounding/SKILL.md). +The setup script prints `CODESTORY_CLI=` — point it at any workspace with +`--project`. -```mermaid -flowchart LR - Repo["repository"] --> Workspace["workspace discovery"] - Workspace --> Indexer["symbol and edge extraction"] - Indexer --> Store["SQLite store"] - Store --> Runtime["retrieval and context assembly"] - Runtime --> CLI["CLI and stdio reads"] - CLI --> Agent["coding agent"] -``` - -CodeStory builds a local evidence layer so agents can request grounded context -instead of relying on ad hoc file reads. +## Command cheat sheet -## Language Support Claims - -CodeStory separates parser-backed graph indexing, regression-tested accuracy, -structural extraction, framework route coverage, and agent packet/search -readiness. The current contract is documented in +| When you need… | Command | +| --- | --- | +| Check cache health | `doctor --project ` | +| Build or refresh the index | `index --project --refresh full` | +| Repo orientation | `ground --project --why` | +| Find a symbol or path | `search --project --query "…" --why` | +| Call graph around one symbol | `trail --project --id --story` | +| Source around a symbol | `snippet --project --id ` | +| Deep bundle on one target | `context --project --id ` | +| Broad task question (sidecars required) | `packet --project --question "…"` | +| Warm agent read surface | `serve --project --stdio` | + +Full operator guide: [docs/usage.md](docs/usage.md). + +## Languages, evidence, contributing + +Parser-backed graph indexing covers Python, Java, Rust, JavaScript, +TypeScript/TSX, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, and Bash; HTML, +CSS, and SQL use structural collectors. Claim details: [docs/architecture/language-support.md](docs/architecture/language-support.md). -In short: Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, -PHP, C#, Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph -languages; HTML, CSS, and SQL use structural collectors. - -The opt-in OSS language corpus pairs each public language-support profile with a -pinned medium-sized open source project and compares raw filesystem counts -against CodeStory indexing of the same files: -[docs/testing/oss-language-corpus.md](docs/testing/oss-language-corpus.md). -The separate `language-expansion-holdout` benchmark suite runs strict -`without_codestory` versus `with_codestory` agent tasks on those pinned -projects and records elapsed time, token usage, estimated cost, tool calls, -command counts, source reads, post-packet source reads, and quality gates. - -For the system model, start with -[docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md), -then [docs/architecture/overview.md](docs/architecture/overview.md). - -## Evidence - -The benchmark docs are deliberately cautious. They separate current checked-in -benchmark history from the state of your local cache, which can drift and should -be checked with `doctor`. - -- Public evidence summary and caveats: - [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) -- Repo-scale timing history: - [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md) -- Warm stdio loop evidence: - [docs/testing/codestory-stdio-warm-loop-stats.md](docs/testing/codestory-stdio-warm-loop-stats.md) -- Repeatable with/without harness: - [`scripts/codestory-agent-ab-benchmark.mjs`](scripts/codestory-agent-ab-benchmark.mjs) - -Do not promote a single benchmark row into a universal savings claim. - -## Hack On CodeStory - -Start with the contributor docs, then run Cargo checks serially because this -workspace shares build locks. - -- [docs/contributors/getting-started.md](docs/contributors/getting-started.md) -- [docs/contributors/debugging.md](docs/contributors/debugging.md) -- [docs/contributors/testing-matrix.md](docs/contributors/testing-matrix.md) -- [docs/architecture/runtime-execution-path.md](docs/architecture/runtime-execution-path.md) -- [docs/architecture/language-support.md](docs/architecture/language-support.md) -- [docs/architecture/subsystems/contracts.md](docs/architecture/subsystems/contracts.md) -- [docs/architecture/subsystems/workspace.md](docs/architecture/subsystems/workspace.md) -- [docs/architecture/subsystems/indexer.md](docs/architecture/subsystems/indexer.md) -- [docs/architecture/subsystems/store.md](docs/architecture/subsystems/store.md) -- [docs/architecture/subsystems/runtime.md](docs/architecture/subsystems/runtime.md) -- [docs/architecture/subsystems/cli.md](docs/architecture/subsystems/cli.md) +Benchmark notes and caveats live in +[docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md). Timing +history: [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md). + +To hack on CodeStory: [docs/contributors/getting-started.md](docs/contributors/getting-started.md). +Architecture: [docs/architecture/overview.md](docs/architecture/overview.md). ## License diff --git a/crates/codestory-cli/tests/onboarding_contracts.rs b/crates/codestory-cli/tests/onboarding_contracts.rs index 436efc90..071ad106 100644 --- a/crates/codestory-cli/tests/onboarding_contracts.rs +++ b/crates/codestory-cli/tests/onboarding_contracts.rs @@ -165,39 +165,28 @@ fn cli_package_metadata_is_adoption_ready() { fn readme_keeps_customer_first_onboarding() { let root = repo_root(); let readme = fs::read_to_string(root.join("README.md")).expect("README should exist"); - assert!(readme.contains("Public Promise")); - assert!(readme.contains("Try It On A Repo")); - assert!(readme.contains("What It Builds")); + assert!(readme.contains("Why CodeStory")); + assert!(readme.contains("How it works")); + assert!(readme.contains("Try it")); assert!(readme.contains("Local codebase grounding for coding agents")); - assert!(readme.contains("Install As An Agent Skill")); - assert!(readme.contains("Core Flow")); - assert!(readme.contains("Hack On CodeStory")); - assert!(readme.contains("A good CodeStory-backed answer should name")); - assert!(readme.contains("local evidence layer for repositories")); - assert!(readme.contains("explicit commands")); - assert!(readme.contains("source-backed answers")); - assert!(readme.contains("per-project SQLite cache is separate")); - assert!(readme.contains("local retrieval sidecars")); - assert!(readme.contains("does not by itself prove sidecar readiness")); - assert!(readme.contains("environment- and repository-specific evidence")); - assert!(readme.contains("instead of promising universal speedups or savings")); - assert!(readme.contains("benchmark history")); - assert!(readme.contains("checked with `doctor`")); + assert!(readme.contains("Install as an agent skill")); + assert!(readme.contains("Command cheat sheet")); + assert!(readme.contains("Your code stays on your machine")); + assert!(readme.contains("retrieval_mode=full")); + assert!(readme.contains("Optional sidecar indexes")); assert!(readme.contains(".agents/skills/codestory-grounding/SKILL.md")); assert!(readme.contains("docs/usage.md")); assert!(readme.contains("docs/concepts/how-codestory-works.md")); assert!(readme.contains("docs/architecture/language-support.md")); assert!(readme.contains("docs/testing/benchmark-ledger.md")); - assert!(readme.contains( - r#""$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json"# - )); - assert!(readme.contains("serve --stdio")); + assert!(readme.contains("serve --project")); assert!(readme.contains("docs/architecture/overview.md")); - assert!(readme.contains("docs/contributors/debugging.md")); - assert!(readme.contains("docs/contributors/testing-matrix.md")); + assert!(readme.contains("docs/contributors/getting-started.md")); assert!( - readme.find("Try It On A Repo").expect("quickstart section") - < readme.find("Evidence").expect("evidence section"), + readme.find("Try it").expect("quickstart section") + < readme + .find("Languages, evidence, contributing") + .expect("reference section"), "README should show the usable path before benchmark evidence" ); diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index b4d86b73..d415b1dd 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -9,9 +9,9 @@ plans a change. ## The Loop -```text -doctor -> index -> ground -> search -> symbol/trail/snippet/explore -> context -``` +See [README — How it works](../README.md#how-it-works) for the indexing → search → trail → answer flow. + +Readiness lanes (local cache vs sidecars): [usage.md](../usage.md#readiness-tracks). - `doctor` checks whether the cache, index, retrieval mode, and local embedding setup are usable. diff --git a/docs/usage.md b/docs/usage.md index 3bd8702b..75fd80d0 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -11,7 +11,7 @@ the workspace you are indexing. ## Install The Skill Install the grounding skill once, then point it at explicit target workspaces. -See [README — Install As An Agent Skill](../README.md#install-as-an-agent-skill) +See [README — Install as an agent skill](../README.md#install-as-an-agent-skill) for the full copy/setup commands and Windows PowerShell variant. The source skill package lives at @@ -45,6 +45,17 @@ TARGET_WORKSPACE="/path/to/repo" CodeStory has two readiness tracks. Keep them separate when deciding whether an agent can rely on packet/search output. +| | Local navigation | Agent packet/search | +| --- | --- | --- | +| Lane id | `local_navigation` | `agent_packet_search` | +| Built by | `index` | `index` then `retrieval index` | +| Requires | Healthy SQLite cache and graph | Sidecars healthy and `retrieval_mode=full` | +| Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | +| Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | + +Architecture and sidecar topology: [architecture/overview.md](../architecture/overview.md), +[ops/retrieval-sidecars.md](../ops/retrieval-sidecars.md). + ### Local navigation/cache readiness This lane is for local browsing and source navigation. It uses the project From d30b8d3756dccdb4ef24ae8528fb2398ad7bfa34 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:44:05 -0400 Subject: [PATCH 09/14] strip performative tone from readme Co-authored-by: Cursor --- README.md | 137 +++++++----------- .../tests/onboarding_contracts.rs | 12 +- 2 files changed, 59 insertions(+), 90 deletions(-) diff --git a/README.md b/README.md index 78f81062..7ad1bc59 100644 --- a/README.md +++ b/README.md @@ -10,60 +10,46 @@ Local codebase grounding for coding agents. Benchmarks

-## Why CodeStory +CodeStory indexes a git workspace into a local SQLite graph: files, symbols, call +and import edges, and source snippets. You query that graph through a CLI (or +`serve --stdio`) instead of having an agent grep the tree blind. -Agents fail on real repos the same way humans do when they are new: they open the -wrong files, chase a plausible name, and call it done. CodeStory indexes the -repository first — who calls whom, where symbols live, the actual source — so -the next step is evidence, not guesswork. - -It runs locally. Your code stays on your machine. +Everything stays on disk under your user cache unless you opt into managed +embedding assets. Packet and full search need local sidecars (Zoekt, Qdrant, +SCIP, llama.cpp); browsing with `ground`, `trail`, and `snippet` does not. ## How it works -**1. Index the repo.** CodeStory walks your tree (honoring normal ignore rules), -parses supported languages into symbols and edges — calls, imports, overrides — -and stores snippets plus a searchable graph in a per-project SQLite cache. One -full `index` up front; incremental refresh after edits. - -**2. Find a foothold.** Ask where something lives: a handler name, a route, a -type, a string literal. `ground` summarizes a repo you have never seen; `search` -returns ranked candidates with file paths and graph ids. - -**3. Follow the graph.** Pick one symbol. `trail` shows callers and callees; -`snippet` returns the surrounding source. You are walking relationships in the -index, not grepping random directories. - -**4. Answer with proof.** `context` bundles trails, neighbors, and citations -around one target. For broad questions ("how does indexing persist state?"), -`packet` assembles a bounded evidence packet — but only when sidecar retrieval -is fully healthy (`retrieval_mode=full`). - -**Embeddings are optional, not step one.** Most symbols are found through the -graph and lexical symbol docs. A separate `retrieval index` pass builds Zoekt, -Qdrant, and SCIP sidecars and embeds only policy-selected anchors (entry -points, public APIs, high-centrality nodes) when you need agent-grade -`packet`/`search`. Details: [docs/usage.md](docs/usage.md), -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). +Run `index` once per repo. The indexer parses supported languages with +tree-sitter, resolves what it can, and writes nodes, edges, occurrences, and +symbol search docs into a per-project database. + +From there: + +- `ground` — repo summary and gaps +- `search` — candidates by symbol name, path, literal, or behavior term +- `trail` — callers, callees, references for one node id +- `snippet` — source lines around a node +- `context` — bundled evidence for one target +- `packet` — wide task question with citations (requires `retrieval_mode=full`) + +Vectors are not built for every symbol. The graph and lexical symbol docs handle +most lookup. Run `retrieval index` when you want sidecar-backed `packet`/`search`; +that pass embeds a policy-selected set of anchors only. ```mermaid flowchart LR - files[Source files] --> index[Parse into symbols and edges] - index --> graph[(Local graph and snippets)] - question[Question or symbol] --> graph - graph --> match[Matching files and symbols] - match --> walk[Caller and callee paths] - walk --> source[Source at file and line] - source --> cite[Answer with citations] - graph --> sidecars[Optional sidecar indexes] - sidecars --> cite + repo[Workspace files] --> idx[index] + idx --> db[(SQLite graph)] + db --> q[search or ground] + q --> t[trail and snippet] + t --> out[context or packet] + idx --> side[retrieval index] + side --> out ``` -Example: *"Who calls `WorkspaceIndexer`?"* → search returns the symbol → trail -lists callers across crates → snippet shows the call sites → you edit with paths -already in hand. - -More depth: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md). +Operator detail: [docs/usage.md](docs/usage.md). Sidecar setup: +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). ## Try it @@ -78,49 +64,36 @@ export TARGET_WORKSPACE="/path/to/repo" "$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "WorkspaceIndexer" --why ``` -On Windows use `.\target\release\codestory-cli.exe` and `$env:TARGET_WORKSPACE = "C:\path\to\repo"`. - -That gets you a local graph you can browse with `trail`, `snippet`, `symbol`, -`explore`, `context`, and `report`. Add sidecars when you need `packet`; see -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). +Windows: `.\target\release\codestory-cli.exe`, `$env:TARGET_WORKSPACE = "C:\path\to\repo"`. ## Install as an agent skill -Copy [`.agents/skills/codestory-grounding`](.agents/skills/codestory-grounding) into -your agent skill directory and run `scripts/setup.sh` (or `setup.ps1` on -Windows). Skill source: [`.agents/skills/codestory-grounding/SKILL.md`](.agents/skills/codestory-grounding/SKILL.md). -The setup script prints `CODESTORY_CLI=` — point it at any workspace with -`--project`. +Copy [`.agents/skills/codestory-grounding`](.agents/skills/codestory-grounding) to +your skill directory. Run `scripts/setup.sh` or `scripts/setup.ps1`. See +[`.agents/skills/codestory-grounding/SKILL.md`](.agents/skills/codestory-grounding/SKILL.md). -## Command cheat sheet +## Commands -| When you need… | Command | +| Task | Command | | --- | --- | -| Check cache health | `doctor --project ` | -| Build or refresh the index | `index --project --refresh full` | -| Repo orientation | `ground --project --why` | -| Find a symbol or path | `search --project --query "…" --why` | -| Call graph around one symbol | `trail --project --id --story` | -| Source around a symbol | `snippet --project --id ` | -| Deep bundle on one target | `context --project --id ` | -| Broad task question (sidecars required) | `packet --project --question "…"` | -| Warm agent read surface | `serve --project --stdio` | - -Full operator guide: [docs/usage.md](docs/usage.md). - -## Languages, evidence, contributing - -Parser-backed graph indexing covers Python, Java, Rust, JavaScript, -TypeScript/TSX, C++, C, Go, Ruby, PHP, C#, Kotlin, Swift, Dart, and Bash; HTML, -CSS, and SQL use structural collectors. Claim details: -[docs/architecture/language-support.md](docs/architecture/language-support.md). - -Benchmark notes and caveats live in -[docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md). Timing -history: [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md). - -To hack on CodeStory: [docs/contributors/getting-started.md](docs/contributors/getting-started.md). -Architecture: [docs/architecture/overview.md](docs/architecture/overview.md). +| Cache health | `doctor --project ` | +| Index | `index --project --refresh full` | +| Orientation | `ground --project --why` | +| Lookup | `search --project --query "…" --why` | +| Call graph | `trail --project --id --story` | +| Source | `snippet --project --id ` | +| Target bundle | `context --project --id ` | +| Task packet (sidecars) | `packet --project --question "…"` | +| Persistent reads | `serve --project --stdio` | + +## Docs + +- Usage: [docs/usage.md](docs/usage.md) +- Concepts: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md) +- Architecture: [docs/architecture/overview.md](docs/architecture/overview.md) +- Languages: [docs/architecture/language-support.md](docs/architecture/language-support.md) +- Benchmarks: [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) +- Contributing: [docs/contributors/getting-started.md](docs/contributors/getting-started.md) ## License diff --git a/crates/codestory-cli/tests/onboarding_contracts.rs b/crates/codestory-cli/tests/onboarding_contracts.rs index 071ad106..ecb8cda8 100644 --- a/crates/codestory-cli/tests/onboarding_contracts.rs +++ b/crates/codestory-cli/tests/onboarding_contracts.rs @@ -165,15 +165,13 @@ fn cli_package_metadata_is_adoption_ready() { fn readme_keeps_customer_first_onboarding() { let root = repo_root(); let readme = fs::read_to_string(root.join("README.md")).expect("README should exist"); - assert!(readme.contains("Why CodeStory")); assert!(readme.contains("How it works")); assert!(readme.contains("Try it")); assert!(readme.contains("Local codebase grounding for coding agents")); assert!(readme.contains("Install as an agent skill")); - assert!(readme.contains("Command cheat sheet")); - assert!(readme.contains("Your code stays on your machine")); + assert!(readme.contains("## Commands")); assert!(readme.contains("retrieval_mode=full")); - assert!(readme.contains("Optional sidecar indexes")); + assert!(readme.contains("retrieval index")); assert!(readme.contains(".agents/skills/codestory-grounding/SKILL.md")); assert!(readme.contains("docs/usage.md")); assert!(readme.contains("docs/concepts/how-codestory-works.md")); @@ -184,10 +182,8 @@ fn readme_keeps_customer_first_onboarding() { assert!(readme.contains("docs/contributors/getting-started.md")); assert!( readme.find("Try it").expect("quickstart section") - < readme - .find("Languages, evidence, contributing") - .expect("reference section"), - "README should show the usable path before benchmark evidence" + < readme.find("## Docs").expect("reference section"), + "README should show the usable path before doc links" ); for path in [ From c009a1c7996a4dc261cdf9fb88aac360fd0a748f Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:46:46 -0400 Subject: [PATCH 10/14] frame readme intro with star Co-authored-by: Cursor --- README.md | 59 +++++++++++-------- .../tests/onboarding_contracts.rs | 4 ++ 2 files changed, 38 insertions(+), 25 deletions(-) diff --git a/README.md b/README.md index 7ad1bc59..179babeb 100644 --- a/README.md +++ b/README.md @@ -10,32 +10,23 @@ Local codebase grounding for coding agents. Benchmarks

-CodeStory indexes a git workspace into a local SQLite graph: files, symbols, call -and import edges, and source snippets. You query that graph through a CLI (or -`serve --stdio`) instead of having an agent grep the tree blind. +**Situation.** You are in a repo with more files than anyone holds in memory. The +agent (or you) needs to change behavior that spans packages — routing, indexing, +auth, whatever — not rename a variable in the one file already open. -Everything stays on disk under your user cache unless you opt into managed -embedding assets. Packet and full search need local sidecars (Zoekt, Qdrant, -SCIP, llama.cpp); browsing with `ground`, `trail`, and `snippet` does not. +**Task.** Find the symbol that owns the behavior, see who calls it, read the +source that actually runs, and know what to touch next. Without treating +`grep -R` as architecture. -## How it works - -Run `index` once per repo. The indexer parses supported languages with -tree-sitter, resolves what it can, and writes nodes, edges, occurrences, and -symbol search docs into a per-project database. - -From there: - -- `ground` — repo summary and gaps -- `search` — candidates by symbol name, path, literal, or behavior term -- `trail` — callers, callees, references for one node id -- `snippet` — source lines around a node -- `context` — bundled evidence for one target -- `packet` — wide task question with citations (requires `retrieval_mode=full`) +**Action.** CodeStory indexes the workspace into a local SQLite graph — symbols, +calls, imports, snippets — and exposes it through a CLI. `index` once; +`search` or `ground` to land somewhere; `trail` and `snippet` to follow it; +`context` or `packet` when you need a bundle. Sidecars (Zoekt, Qdrant, SCIP) +are for full `packet`/`search`; local browsing does not require them. -Vectors are not built for every symbol. The graph and lexical symbol docs handle -most lookup. Run `retrieval index` when you want sidecar-backed `packet`/`search`; -that pass embeds a policy-selected set of anchors only. +**Result.** Work starts at a file and line you can open, not at whichever match +ranked first in ripgrep. Answers say what they used; gaps say when the index is +stale or incomplete. ```mermaid flowchart LR @@ -48,8 +39,26 @@ flowchart LR side --> out ``` -Operator detail: [docs/usage.md](docs/usage.md). Sidecar setup: -[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md). +## How it works + +The indexer uses tree-sitter on supported languages, writes nodes, edges, +occurrences, and symbol search docs to a per-project cache under your user +directory. Incremental refresh picks up edits without a full rebuild every time. + +| Command | What you get | +| --- | --- | +| `ground` | Repo summary and coverage gaps | +| `search` | Ranked symbols, paths, literals | +| `trail` | Callers, callees, references for one node id | +| `snippet` | Source lines around a node | +| `context` | Bundled evidence for one target | +| `packet` | Task-sized citations (`retrieval_mode=full`) | + +Most lookup is graph + lexical symbol docs. `retrieval index` embeds a +policy-selected subset of anchors when you need sidecar search — not every +symbol gets a vector. + +[docs/usage.md](docs/usage.md) · [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) ## Try it diff --git a/crates/codestory-cli/tests/onboarding_contracts.rs b/crates/codestory-cli/tests/onboarding_contracts.rs index ecb8cda8..e9f268d5 100644 --- a/crates/codestory-cli/tests/onboarding_contracts.rs +++ b/crates/codestory-cli/tests/onboarding_contracts.rs @@ -166,6 +166,10 @@ fn readme_keeps_customer_first_onboarding() { let root = repo_root(); let readme = fs::read_to_string(root.join("README.md")).expect("README should exist"); assert!(readme.contains("How it works")); + assert!(readme.contains("**Situation.**")); + assert!(readme.contains("**Task.**")); + assert!(readme.contains("**Action.**")); + assert!(readme.contains("**Result.**")); assert!(readme.contains("Try it")); assert!(readme.contains("Local codebase grounding for coding agents")); assert!(readme.contains("Install as an agent skill")); From 6983623c2f0492781ef8112678b289a54a3e5eae Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:50:32 -0400 Subject: [PATCH 11/14] apply star framing across entry and operator docs Co-authored-by: Cursor --- .../tests/onboarding_contracts.rs | 11 +- docs/architecture/overview.md | 13 +- docs/concepts/how-codestory-works.md | 136 +++++++----------- docs/contributors/getting-started.md | 8 ++ docs/glossary.md | 46 +++--- docs/ops/retrieval-sidecars.md | 30 ++-- docs/testing/benchmark-ledger.md | 14 +- docs/usage.md | 132 ++++++++--------- 8 files changed, 186 insertions(+), 204 deletions(-) diff --git a/crates/codestory-cli/tests/onboarding_contracts.rs b/crates/codestory-cli/tests/onboarding_contracts.rs index e9f268d5..807b69c7 100644 --- a/crates/codestory-cli/tests/onboarding_contracts.rs +++ b/crates/codestory-cli/tests/onboarding_contracts.rs @@ -238,7 +238,6 @@ fn readme_keeps_customer_first_onboarding() { #[test] fn docs_drift_contracts_keep_living_sources_explicit() { let root = repo_root(); - let readme = fs::read_to_string(root.join("README.md")).expect("README should exist"); let usage = fs::read_to_string(root.join("docs/usage.md")).expect("usage doc should exist"); let testing_matrix = fs::read_to_string(root.join("docs/contributors/testing-matrix.md")) .expect("testing matrix should exist"); @@ -248,10 +247,10 @@ fn docs_drift_contracts_keep_living_sources_explicit() { .expect("benchmark ledger should exist"); assert!( - readme.contains( - r#""$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json"# - ), - "README quickstart should show first-run semantic setup dry-run" + usage.contains( + r#"setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json"# + ) || usage.contains("setup embeddings --project"), + "usage doc should document managed embedding setup" ); assert!( !usage.contains("semantic_doc_scope = \"durable\""), @@ -357,8 +356,6 @@ fn usage_doc_names_two_readiness_tracks_and_predictable_output_modes() { let usage = fs::read_to_string(root.join("docs/usage.md")).expect("usage doc should exist"); assert!(usage.contains("## Readiness Tracks")); - assert!(usage.contains("### Local navigation/cache readiness")); - assert!(usage.contains("### Agent packet/search sidecar readiness")); assert!(usage.contains("`local_navigation`")); assert!(usage.contains("`agent_packet_search`")); assert!(usage.contains("`retrieval_mode: \"full\"`")); diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 2c6640a8..067d0edb 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,7 +1,16 @@ # Architecture Overview -CodeStory has one job: turn a repository into local evidence that a coding agent -can query before relying on a small set of manually opened files. +**Situation.** You are changing CodeStory, not just running the CLI on a repo. + +**Task.** Know which crate owns indexing, storage, retrieval, and CLI output so +you do not put SQL in the wrong layer. + +**Action.** Follow the runtime path and dependency diagram below before editing. + +**Result.** Changes land in the owning crate; product behavior stays traceable +from files and symbols back to the graph and sidecar manifest. + +User-facing intro: [README](../../README.md). Operator flows: [usage.md](../usage.md). The runtime path is: diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index d415b1dd..f30fccc2 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -1,84 +1,56 @@ # How CodeStory Works -CodeStory is a local evidence layer for codebases. It does not replace judgment, -tests, or source reading. It makes the first pass more structured. - -An agent usually fails on a large repo by over-weighting the first few files it -opens. CodeStory gives that agent an indexed map before it explains behavior or -plans a change. - -## The Loop - -See [README — How it works](../README.md#how-it-works) for the indexing → search → trail → answer flow. - -Readiness lanes (local cache vs sidecars): [usage.md](../usage.md#readiness-tracks). - -- `doctor` checks whether the cache, index, retrieval mode, and local embedding - setup are usable. -- `index` builds or refreshes local graph, search, snapshot, graph-native - symbol-doc, component-report, and selected dense-anchor state for one target - repository. -- `ground` gives broad orientation and reports limited coverage or gaps. -- `search` finds candidate files, symbols, routes, literals, modules, or behavior - terms. -- `symbol`, `trail`, `snippet`, and `explore` inspect one selected target. -- `context` bundles deeper evidence around that concrete target. -- `packet` handles broad task questions and reports citations, gaps, and next - commands. - -The workflow is a repeatable evidence loop. - -## What Gets Stored - -CodeStory writes per-project state under the user cache, keyed by the target -workspace path. The cache can include: - -- discovered files and refresh metadata -- graph nodes for files, symbols, and related code elements -- graph edges such as calls, imports, overrides, and references -- source snippets and occurrence locations -- search projection rows and local search indexes -- grounding snapshots rebuilt from the graph -- graph-native symbol docs, which are deterministic searchable summaries for - durable AST symbols -- selected dense anchors, which are the only generated docs embedded as vectors - under the active semantic policy - -Repository data stays local. Managed setup may fetch tool or model assets, but -the indexed project evidence lives in the local cache. - -## Key Terms - -- Grounding is source-backed context: the files, symbols, and summaries a command - returns so an answer can be tied back to repository evidence. -- A symbol doc is deterministic generated text for a symbol, stored so lexical - and graph retrieval can find relevant code even when the query words are not - exact. -- A dense anchor is a policy-selected symbol, component report, or unstructured - doc that receives a vector embedding. Code symbols do not need dense vectors - to be product-searchable. -- A snapshot is a cached read model rebuilt from the local graph. If a snapshot - is stale, the tool should say so. -- A trail is a focused graph walk around one symbol: callers, callees, - references, or neighborhood context. -- A packet is a bounded evidence bundle for a broad task. It should include - citations, gaps, and follow-up commands. - -## What Good Looks Like - -A good CodeStory-backed answer does three things: - -1. It names the files, symbols, or snippets it used. -2. It says when evidence is stale, partial, ambiguous, or missing. -3. It gives the next concrete command when the current evidence is not enough. - -The goal is not a more confident answer. The goal is confidence constrained by -source evidence. - -## Where To Go Next - -- Use [../usage.md](../usage.md) for command flows. -- Use [../architecture/overview.md](../architecture/overview.md) for the system - boundary and crate model. -- Use [../contributors/debugging.md](../contributors/debugging.md) when output - looks wrong. +Product framing lives in the [README](../../README.md). This page is the concept +layer: what gets stored, what the commands mean, and what a good run looks like. + +## STAR for one change + +**Situation.** You need to edit behavior that lives somewhere in the graph, not in +the file you already have open. + +**Task.** Land on the owning symbol, see call paths, read the running code, ship +the change without a blind file walk. + +**Action.** `index` builds the graph; `search` or `ground` picks a node; +`trail` / `snippet` / `symbol` inspect it; `context` or `packet` bundles an +answer. See [README — How it works](../../README.md#how-it-works). + +**Result.** Citations point at paths in the repo. Partial or stale index state +is reported instead of implied. + +Readiness lanes (cache-only vs sidecars): [usage.md](../usage.md#readiness-tracks). + +## What gets stored + +Per-project SQLite under your user cache, keyed by workspace path: + +| Stored | Purpose | +| --- | --- | +| File inventory and refresh metadata | Incremental re-index | +| Graph nodes and edges | Calls, imports, overrides, references | +| Snippets and occurrences | Source-backed reads | +| Search projections and symbol docs | Lookup without opening every file | +| Snapshots | Cached read models rebuilt from the graph | +| Dense anchors (when policy selects them) | Sidecar vector search only | + +Repo content stays local. Managed setup may fetch tool assets; indexed evidence +does not leave the cache unless you copy it. + +## Terms + +| Term | Meaning | +| --- | --- | +| Grounding | Context tied back to indexed files and symbols | +| Symbol doc | Generated searchable text for a symbol (lexical, not embedded by default) | +| Dense anchor | Policy-selected symbol or report that gets a vector | +| Snapshot | Derived read model; may be stale — commands should say so | +| Trail | Graph walk from one symbol: callers, callees, neighbors | +| Packet | Bounded task evidence with citations, gaps, next commands | + +Full list: [glossary.md](../glossary.md). + +## Where to go next + +- Operator flows: [usage.md](../usage.md) +- Crates and boundaries: [architecture/overview.md](../architecture/overview.md) +- Wrong output: [contributors/debugging.md](../contributors/debugging.md) diff --git a/docs/contributors/getting-started.md b/docs/contributors/getting-started.md index b398ed38..57e355c2 100644 --- a/docs/contributors/getting-started.md +++ b/docs/contributors/getting-started.md @@ -1,5 +1,13 @@ # Contributor Setup +**Situation.** You are patching CodeStory itself — indexer, store, runtime, or CLI. + +**Task.** Build cleanly, run the CLI on this repo, and know which tests gate your change. + +**Action.** Commands below, then the reading order and subsystem page for the crate you touch. + +**Result.** A local cache of this workspace plus the test lane that matches your diff. + ## First Commands Run these from the repo root: diff --git a/docs/glossary.md b/docs/glossary.md index 461860ef..90239c04 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,20 +1,30 @@ # Glossary -- grounding: the process of turning indexed code state into concise, relevant context for a question or tool action -- snapshot: a derived SQLite-backed grounding view that can be rebuilt from the primary graph tables -- projection: derived persisted data such as callable projection state or ranked grounding summaries -- staged snapshot: the temporary SQLite database built during full refresh before publish replaces the live cache -- refresh baseline: the persisted file inventory and metadata used to decide what an incremental refresh must index or remove -- trail: a focused graph walk rooted at one symbol, usually caller/callee or neighborhood oriented -- runtime: the orchestration surface that coordinates project opening, indexing, search, grounding, trail generation, and system actions -- workspace: the manifest plus filesystem discovery layer that decides which files belong to the project -- contracts: shared graph, DTO, and event types that are safe to depend on across boundaries -- repo-text hit: a direct file-content match surfaced alongside indexed-symbol search results -- retrieval mode: retrieval status contract for sidecar evidence; `retrieval_mode=full` is required for agent packet/search readiness -- symbol doc: deterministic generated per-symbol text stored in SQLite for graph-native lexical retrieval; it is not embedded by default -- dense anchor: a policy-selected symbol, component report, or unstructured doc that receives a vector embedding -- local navigation readiness: the local cache, graph, lexical index, and DB-backed navigation commands are usable -- agent packet/search readiness: sidecar packet/search evidence is trustworthy only when retrieval status reports `retrieval_mode=full` -- target context: DB-first evidence for one concrete target; not a replacement for broad packet, search, or drill questions -- semantic ready: local diagnostic state where dense-anchor retrieval is enabled, an embedding runtime is available when dense anchors exist, and persisted dense anchors match the active policy; not agent packet/search readiness -- cache root: the directory that owns one project cache; by default this is under the user cache directory, but `--cache-dir` can override it +Short definitions for terms used across README, usage, and architecture docs. + +## Readiness + +- **local navigation readiness** — SQLite cache, graph, and DB-backed browse commands (`ground`, `trail`, `snippet`, etc.) are usable +- **agent packet/search readiness** — sidecars healthy and `retrieval_mode=full`; required for trustworthy `packet` / `search` +- **retrieval mode** — sidecar status contract; only `full` serves agent packet/search +- **semantic ready** — dense-anchor embedding state matches policy; not the same as agent packet/search readiness + +## Index and graph + +- **grounding** — indexed context returned for a question or command, with source ties +- **snapshot** — derived grounding view rebuilt from graph tables +- **projection** — persisted derived state (callable projection, ranked summaries) +- **staged snapshot** — temporary DB during full refresh before publish +- **refresh baseline** — file inventory used to plan incremental refresh +- **trail** — focused graph walk from one symbol +- **symbol doc** — deterministic per-symbol search text in SQLite; not embedded by default +- **dense anchor** — symbol, component report, or doc selected for vector embedding +- **repo-text hit** — raw file-content match; diagnostic, not a substitute for graph evidence + +## System + +- **runtime** — orchestrates index, search, grounding, trails, agent flows +- **workspace** — manifest and discovery for which files belong to the project +- **contracts** — shared graph types, DTOs, events across crates +- **target context** — DB-first bundle for one concrete target (`context`), not broad `packet` +- **cache root** — directory for one project cache; override with `--cache-dir` diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index 52f78d8a..06017bdb 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -1,21 +1,19 @@ # Retrieval sidecars — Operations runbook -Local Zoekt, Qdrant, and SCIP indexer processes for sidecar packet retrieval. Data directories -live under the CodeStory user cache; ports are fixed for local dev and CI smoke. - -This runbook covers the `agent_packet_search` readiness lane. Sidecar readiness -is required before agent-facing `packet` and `search` output can be trusted. -Local SQLite navigation is a separate `local_navigation` lane: `codestory-cli -index`, `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, -and `affected` can be useful with a healthy local cache, but that cache alone -does not prove packet/search sidecar readiness. - -**Design reference:** [`retrieval-design.md`](../architecture/retrieval-design.md) -(mode definitions, cost envelopes, promotion guards). - -**Operations reference:** this runbook owns setup commands, version pins, env -vars, troubleshooting, and CI smoke sequences. Proof tiers and promotion -checklists live in [`retrieval-architecture.md`](../testing/retrieval-architecture.md). +**Situation.** `packet` or `search` fail closed; `doctor` shows sidecars down or +`retrieval_mode` not `full`. Local browse commands still work from SQLite alone. + +**Task.** Run Zoekt, Qdrant, SCIP, and the llama.cpp embed endpoint locally; build +a current retrieval manifest for the target workspace. + +**Action.** Bootstrap services, `retrieval index`, verify with `retrieval status` +and `doctor`. Commands and pins live in this runbook. + +**Result.** Agent-facing retrieval serves from sidecars with `retrieval_mode=full`. +Cache-only navigation never implied sidecar readiness. + +Design: [`retrieval-design.md`](../architecture/retrieval-design.md). +Proof tiers: [`retrieval-architecture.md`](../testing/retrieval-architecture.md). ```mermaid flowchart LR diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 957109a6..39ccd362 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -1,9 +1,15 @@ # CodeStory Benchmark Ledger -This ledger keeps the decision-grade scorecard and detailed benchmark history -that is too dense for the README. Treat every row as machine-, cache-, runner-, -and date-specific. Promote only rows that pass the current harness gates -documented below. +**Situation.** Someone asks whether CodeStory saves tokens, time, or tool calls. + +**Task.** Answer from logged rows, not from README vibes or one good run. + +**Action.** Read the scorecard and linked stats logs; cite setup, date, and harness tier. + +**Result.** A claim no stronger than the row supports — or an explicit "not claimed yet." + +Treat every row as machine-, cache-, runner-, and date-specific. Promote only rows +that pass the current harness gates documented below. Runs recorded before the 2026-05-24 harness tightening are historical unless they are reanalyzed or rerun with answer-level expected-file/symbol recall, diff --git a/docs/usage.md b/docs/usage.md index 75fd80d0..15684597 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,12 +1,10 @@ # CodeStory Usage -This is the operator guide. It keeps setup, common workflows, retrieval defaults, -and recovery notes in one place. +Operator guide: setup, workflows, sidecars, recovery. Start with the +[README STAR intro](../README.md) if you are new. -Examples use POSIX shell syntax unless a block is labeled PowerShell. On -Windows, use `.\target\release\codestory-cli.exe` for the release binary, -`$env:NAME = "value"` for environment variables, and Windows paths when that is -the workspace you are indexing. +Examples use POSIX shell unless labeled PowerShell. Windows: +`.\target\release\codestory-cli.exe`, `$env:NAME = "value"`. ## Install The Skill @@ -42,8 +40,7 @@ TARGET_WORKSPACE="/path/to/repo" ## Readiness Tracks -CodeStory has two readiness tracks. Keep them separate when deciding whether an -agent can rely on packet/search output. +Two lanes — do not mix them when judging `packet` or `search` output. | | Local navigation | Agent packet/search | | --- | --- | --- | @@ -51,117 +48,106 @@ agent can rely on packet/search output. | Built by | `index` | `index` then `retrieval index` | | Requires | Healthy SQLite cache and graph | Sidecars healthy and `retrieval_mode=full` | | Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | -| Does not prove | Sidecar or agent packet/search readiness | That local cache-only navigation is enough | +| Does not prove | Sidecar readiness | That cache-only browse is enough for agent search | -Architecture and sidecar topology: [architecture/overview.md](../architecture/overview.md), -[ops/retrieval-sidecars.md](../ops/retrieval-sidecars.md). +`doctor` reports lane status. Sidecar topology: +[architecture/overview.md](architecture/overview.md), +[ops/retrieval-sidecars.md](ops/retrieval-sidecars.md). -### Local navigation/cache readiness - -This lane is for local browsing and source navigation. It uses the project -SQLite cache built by `index` and read by commands such as `ground`, `symbol`, -`trail`, `snippet`, `explore`, `context`, `files`, and `affected`. - -`doctor` may report this lane as `local_navigation`. Local navigation readiness -means the local cache, graph, lexical index, and DB-backed navigation commands -are usable. It does not prove agent packet/search readiness. +## Common Workflows -### Agent packet/search sidecar readiness +Each block is STAR-shaped: situation → commands → what you should have after. -This lane is for agent-facing `packet` and `search` evidence. It requires the -sidecar retrieval stack to be built and healthy: Zoekt lexical shards, Qdrant -semantic vectors, SCIP graph artifacts, the llama.cpp query embedding endpoint, -and a current retrieval manifest. +### I need a repo overview -`doctor` may report this lane as `agent_packet_search`. Agent packet/search -readiness means sidecar packet/search evidence is trustworthy only when -retrieval status reports `retrieval_mode: "full"`. Missing, stale, stubbed, -hash-vector, or non-product sidecar state is diagnostic only and must not be -described as agent packet/search readiness. +**Situation.** New workspace for you or the agent. -## Common Workflows +**Task.** Confirm cache health and get a map before deep reads. -### I need a repo overview +**Action.** ```sh codestory-cli doctor --project codestory-cli index --project --refresh full codestory-cli ground --project --why codestory-cli report --project --output-file out/codestory-report.md -codestory-cli report --project --format json --output-file out/codestory-graph.json ``` -Use this when the repository is new to the agent. `doctor` tells you whether the -cache and retrieval state are usable. `ground --why` gives broad orientation and -reports limited coverage or gaps. `report` reads the current SQLite store -without refreshing it and emits generated artifacts: Markdown for repo summary, -hotspots, entry points, bridge/high-connectivity nodes, and next queries; JSON -for automation that needs the full current graph, including nodes, edges, -confidence/certainty, source locations, and generation metadata. `--limit` -bounds the Markdown report sections, not the full JSON graph export. Treat both -files as outputs to regenerate, not source-of-truth state. +**Result.** Orientation plus optional Markdown report (hotspots, entry points). +Report files are generated output — regenerate after index changes. ### I need evidence for a broad question +**Situation.** Task spans modules ("how does routing work?", "what owns index state?"). + +**Task.** Collect citations without opening the tree file by file. + +**Action.** + ```sh codestory-cli packet --project --question "" --budget compact ``` -Use `packet` for questions like "how does routing work?" or "what owns indexing -state?" It returns a `sufficient`, `partial`, or `blocked` status with -citations, trust limits, gaps, and follow-up commands. If the packet is -`partial` or `blocked`, follow the named source-truth commands instead of -opening unstructured source files directly. Treat `sufficient` as evidence -coverage, not final answer-quality proof. +**Result.** `sufficient`, `partial`, or `blocked` with citations and follow-up +commands. Requires `retrieval_mode=full`. `sufficient` means evidence coverage, +not answer-quality proof. ### I need to understand one symbol or file +**Situation.** You have a name or error pointing at one area of the code. + +**Task.** Confirm the symbol, its callers, and the source lines. + +**Action.** + ```sh -codestory-cli search --project --query "" --why -codestory-cli explore --project --id --no-tui +codestory-cli search --project --query "" --why codestory-cli trail --project --id --story --hide-speculative codestory-cli snippet --project --id --context 40 ``` -Start with `search`, pick a concrete `node-id`, then inspect the relationships -and source. Use `context` when you want a bundled handoff around that target: +**Result.** A concrete node id, call paths, and source. Use `context` when you +need a handoff bundle for one target. -```sh -codestory-cli context --project --id --bundle out/context-name -``` +### I changed files and need likely impact -Target context is DB-first evidence for one concrete target. `context` is -target-first; it is not an open chat endpoint and is not a replacement for broad -`packet`, `search`, or `drill` questions. +**Situation.** Local edits landed; you need test or symbol blast radius. -### I changed files and need likely impact +**Task.** Map changed paths to indexed symbols and likely tests. + +**Action.** ```sh codestory-cli index --project --refresh incremental codestory-cli affected --project --format markdown git diff --name-only HEAD | codestory-cli affected --project --stdin --format json -git diff --name-status HEAD | codestory-cli affected --project --stdin --stdin-format name-status --format json ``` -Treat `affected` as test-selection evidence, not a replacement for tests. The -default command preserves git name-status records; path-only stdin remains -available when another tool already chose the file list. +**Result.** Impacted symbols and test hints — input for selection, not a substitute for running tests. ### The cache or local navigation looks stale +**Situation.** Missing symbols, old paths, or `doctor` warnings. + +**Task.** Refresh truth before trusting search or trails. + +**Action.** + ```sh codestory-cli doctor --project codestory-cli index --project --refresh full codestory-cli doctor --project ``` -If `doctor` reports stale inventory, dense-anchor contract mismatch, missing -managed assets, or a non-`full` retrieval mode, fix that layer before -investigating answer quality. Treat the health report as the first source of -truth for cache and retrieval state. +**Result.** Fresh inventory or explicit errors to fix before continuing. + +### For agent-facing packet/search recovery + +**Situation.** `packet` or `search` blocked; sidecars or manifest unhealthy. + +**Task.** Restore `retrieval_mode=full` without guessing which layer failed. -For agent-facing packet/search recovery, use the full sidecar repair sequence -that `ready --goal agent` reports: +**Action.** ```sh codestory-cli retrieval bootstrap --project --format json @@ -170,12 +156,8 @@ codestory-cli retrieval status --project --format json codestory-cli doctor --project --format markdown ``` -When the core index is missing, stale, unchecked, or has recorded fatal indexing -errors, `ready` reports the necessary `codestory-cli index` repair first. -Otherwise, sidecar recovery does not need to repeat a full core reindex. -`retrieval bootstrap` prepares or checks the local sidecar services. The target -workspace is not packet/search-ready until `retrieval index` writes a current -target manifest and `doctor` or `retrieval status` reports `retrieval_mode=full`. +**Result.** Status JSON shows mode and degraded reason. If core index is broken, +`ready` may require `index` before sidecar repair. ## Core Commands From 728c7addd0abaa5ff971c2f7f95e7f9f03919cd8 Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 15:51:43 -0400 Subject: [PATCH 12/14] remove doc meta-narration and template labels Co-authored-by: Cursor --- docs/architecture/overview.md | 14 ++---- docs/concepts/how-codestory-works.md | 33 +++++--------- docs/contributors/getting-started.md | 8 ---- docs/glossary.md | 2 - docs/ops/retrieval-sidecars.md | 17 +++---- docs/testing/benchmark-ledger.md | 13 ++---- docs/usage.md | 66 +++++----------------------- 7 files changed, 34 insertions(+), 119 deletions(-) diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 067d0edb..d7138aa1 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,16 +1,8 @@ # Architecture Overview -**Situation.** You are changing CodeStory, not just running the CLI on a repo. - -**Task.** Know which crate owns indexing, storage, retrieval, and CLI output so -you do not put SQL in the wrong layer. - -**Action.** Follow the runtime path and dependency diagram below before editing. - -**Result.** Changes land in the owning crate; product behavior stays traceable -from files and symbols back to the graph and sidecar manifest. - -User-facing intro: [README](../../README.md). Operator flows: [usage.md](../usage.md). +CodeStory turns a repository into local evidence a coding agent can query: files +and symbols in SQLite, optional sidecar indexes for packet/search, thin CLI on +top of `codestory-runtime`. The runtime path is: diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index f30fccc2..fdb31a66 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -1,24 +1,11 @@ # How CodeStory Works -Product framing lives in the [README](../../README.md). This page is the concept -layer: what gets stored, what the commands mean, and what a good run looks like. +CodeStory indexes a workspace into a local graph, then serves read commands +against that graph. It does not replace tests or judgment; it structures the +first pass. -## STAR for one change - -**Situation.** You need to edit behavior that lives somewhere in the graph, not in -the file you already have open. - -**Task.** Land on the owning symbol, see call paths, read the running code, ship -the change without a blind file walk. - -**Action.** `index` builds the graph; `search` or `ground` picks a node; -`trail` / `snippet` / `symbol` inspect it; `context` or `packet` bundles an -answer. See [README — How it works](../../README.md#how-it-works). - -**Result.** Citations point at paths in the repo. Partial or stale index state -is reported instead of implied. - -Readiness lanes (cache-only vs sidecars): [usage.md](../usage.md#readiness-tracks). +Command loop: [README — How it works](../../README.md#how-it-works). +Readiness lanes: [usage.md](../usage.md#readiness-tracks). ## What gets stored @@ -47,10 +34,10 @@ does not leave the cache unless you copy it. | Trail | Graph walk from one symbol: callers, callees, neighbors | | Packet | Bounded task evidence with citations, gaps, next commands | -Full list: [glossary.md](../glossary.md). +More: [glossary.md](../glossary.md). -## Where to go next +## Related -- Operator flows: [usage.md](../usage.md) -- Crates and boundaries: [architecture/overview.md](../architecture/overview.md) -- Wrong output: [contributors/debugging.md](../contributors/debugging.md) +- [usage.md](../usage.md) +- [architecture/overview.md](../architecture/overview.md) +- [contributors/debugging.md](../contributors/debugging.md) diff --git a/docs/contributors/getting-started.md b/docs/contributors/getting-started.md index 57e355c2..b398ed38 100644 --- a/docs/contributors/getting-started.md +++ b/docs/contributors/getting-started.md @@ -1,13 +1,5 @@ # Contributor Setup -**Situation.** You are patching CodeStory itself — indexer, store, runtime, or CLI. - -**Task.** Build cleanly, run the CLI on this repo, and know which tests gate your change. - -**Action.** Commands below, then the reading order and subsystem page for the crate you touch. - -**Result.** A local cache of this workspace plus the test lane that matches your diff. - ## First Commands Run these from the repo root: diff --git a/docs/glossary.md b/docs/glossary.md index 90239c04..89d8a6fe 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -1,7 +1,5 @@ # Glossary -Short definitions for terms used across README, usage, and architecture docs. - ## Readiness - **local navigation readiness** — SQLite cache, graph, and DB-backed browse commands (`ground`, `trail`, `snippet`, etc.) are usable diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index 06017bdb..b9ce9618 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -1,19 +1,14 @@ # Retrieval sidecars — Operations runbook -**Situation.** `packet` or `search` fail closed; `doctor` shows sidecars down or -`retrieval_mode` not `full`. Local browse commands still work from SQLite alone. +Local Zoekt, Qdrant, SCIP, and llama.cpp processes for agent `packet` and +`search`. Data dirs live under the user cache; default ports are 6070 (Zoekt) +and 6333 (Qdrant). -**Task.** Run Zoekt, Qdrant, SCIP, and the llama.cpp embed endpoint locally; build -a current retrieval manifest for the target workspace. - -**Action.** Bootstrap services, `retrieval index`, verify with `retrieval status` -and `doctor`. Commands and pins live in this runbook. - -**Result.** Agent-facing retrieval serves from sidecars with `retrieval_mode=full`. -Cache-only navigation never implied sidecar readiness. +Required for `agent_packet_search` readiness (`retrieval_mode=full`). A healthy +SQLite cache alone does not satisfy that lane. Design: [`retrieval-design.md`](../architecture/retrieval-design.md). -Proof tiers: [`retrieval-architecture.md`](../testing/retrieval-architecture.md). +Promotion checks: [`retrieval-architecture.md`](../testing/retrieval-architecture.md). ```mermaid flowchart LR diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 39ccd362..71dd6098 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -1,15 +1,8 @@ # CodeStory Benchmark Ledger -**Situation.** Someone asks whether CodeStory saves tokens, time, or tool calls. - -**Task.** Answer from logged rows, not from README vibes or one good run. - -**Action.** Read the scorecard and linked stats logs; cite setup, date, and harness tier. - -**Result.** A claim no stronger than the row supports — or an explicit "not claimed yet." - -Treat every row as machine-, cache-, runner-, and date-specific. Promote only rows -that pass the current harness gates documented below. +Decision-grade scorecard and benchmark history — too dense for the README. +Treat every row as machine-, cache-, runner-, and date-specific. Do not quote a +row as a universal savings claim without checking harness tier and setup. Runs recorded before the 2026-05-24 harness tightening are historical unless they are reanalyzed or rerun with answer-level expected-file/symbol recall, diff --git a/docs/usage.md b/docs/usage.md index 15684597..a2891cd0 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -1,10 +1,7 @@ # CodeStory Usage -Operator guide: setup, workflows, sidecars, recovery. Start with the -[README STAR intro](../README.md) if you are new. - -Examples use POSIX shell unless labeled PowerShell. Windows: -`.\target\release\codestory-cli.exe`, `$env:NAME = "value"`. +Setup, workflows, sidecars, recovery. Shell examples are POSIX unless noted. +Windows: `.\target\release\codestory-cli.exe`, `$env:NAME = "value"`. ## Install The Skill @@ -56,16 +53,8 @@ Two lanes — do not mix them when judging `packet` or `search` output. ## Common Workflows -Each block is STAR-shaped: situation → commands → what you should have after. - ### I need a repo overview -**Situation.** New workspace for you or the agent. - -**Task.** Confirm cache health and get a map before deep reads. - -**Action.** - ```sh codestory-cli doctor --project codestory-cli index --project --refresh full @@ -73,82 +62,51 @@ codestory-cli ground --project --why codestory-cli report --project --output-file out/codestory-report.md ``` -**Result.** Orientation plus optional Markdown report (hotspots, entry points). -Report files are generated output — regenerate after index changes. +Health check, orientation, optional report (hotspots, entry points). Regenerate +the report after index changes — it is not source-of-truth state. ### I need evidence for a broad question -**Situation.** Task spans modules ("how does routing work?", "what owns index state?"). - -**Task.** Collect citations without opening the tree file by file. - -**Action.** - ```sh codestory-cli packet --project --question "" --budget compact ``` -**Result.** `sufficient`, `partial`, or `blocked` with citations and follow-up -commands. Requires `retrieval_mode=full`. `sufficient` means evidence coverage, -not answer-quality proof. +Returns `sufficient`, `partial`, or `blocked` with citations and follow-ups. +Requires `retrieval_mode=full`. ### I need to understand one symbol or file -**Situation.** You have a name or error pointing at one area of the code. - -**Task.** Confirm the symbol, its callers, and the source lines. - -**Action.** - ```sh codestory-cli search --project --query "" --why codestory-cli trail --project --id --story --hide-speculative codestory-cli snippet --project --id --context 40 ``` -**Result.** A concrete node id, call paths, and source. Use `context` when you -need a handoff bundle for one target. +Pick a node id from `search`, then inspect call paths and source. `context` +bundles one target for handoff. ### I changed files and need likely impact -**Situation.** Local edits landed; you need test or symbol blast radius. - -**Task.** Map changed paths to indexed symbols and likely tests. - -**Action.** - ```sh codestory-cli index --project --refresh incremental codestory-cli affected --project --format markdown git diff --name-only HEAD | codestory-cli affected --project --stdin --format json ``` -**Result.** Impacted symbols and test hints — input for selection, not a substitute for running tests. +Impacted symbols and test hints — not a substitute for running tests. ### The cache or local navigation looks stale -**Situation.** Missing symbols, old paths, or `doctor` warnings. - -**Task.** Refresh truth before trusting search or trails. - -**Action.** - ```sh codestory-cli doctor --project codestory-cli index --project --refresh full codestory-cli doctor --project ``` -**Result.** Fresh inventory or explicit errors to fix before continuing. +Fix inventory or indexing errors before trusting search output. ### For agent-facing packet/search recovery -**Situation.** `packet` or `search` blocked; sidecars or manifest unhealthy. - -**Task.** Restore `retrieval_mode=full` without guessing which layer failed. - -**Action.** - ```sh codestory-cli retrieval bootstrap --project --format json codestory-cli retrieval index --project --refresh full --format json @@ -156,8 +114,8 @@ codestory-cli retrieval status --project --format json codestory-cli doctor --project --format markdown ``` -**Result.** Status JSON shows mode and degraded reason. If core index is broken, -`ready` may require `index` before sidecar repair. +Target `retrieval_mode=full`. Core index problems may require `index` first — +see `ready --goal agent`. ## Core Commands From 240d4f08d4a71ff43d03cc87e430fd2a7cba651d Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 16:18:55 -0400 Subject: [PATCH 13/14] Refine README onboarding and drop stale contract tests --- README.md | 131 +++-- .../tests/onboarding_contracts.rs | 478 ------------------ docs/concepts/how-codestory-works.md | 36 +- docs/contributors/testing-matrix.md | 8 +- docs/glossary.md | 36 +- docs/testing/benchmark-ledger.md | 2 +- docs/testing/retrieval-architecture.md | 2 - docs/usage.md | 62 ++- .../codestory-agent-ab-analyzer.test.mjs | 10 +- 9 files changed, 191 insertions(+), 574 deletions(-) delete mode 100644 crates/codestory-cli/tests/onboarding_contracts.rs diff --git a/README.md b/README.md index 179babeb..b1bf3131 100644 --- a/README.md +++ b/README.md @@ -10,72 +10,83 @@ Local codebase grounding for coding agents. Benchmarks

-**Situation.** You are in a repo with more files than anyone holds in memory. The -agent (or you) needs to change behavior that spans packages — routing, indexing, -auth, whatever — not rename a variable in the one file already open. +**Situation.** You are in a repo with more files than anyone holds in memory. +The agent needs to change behavior that spans packages - routing, indexing, +auth, whatever - not rename a variable in the one file already open. **Task.** Find the symbol that owns the behavior, see who calls it, read the source that actually runs, and know what to touch next. Without treating `grep -R` as architecture. -**Action.** CodeStory indexes the workspace into a local SQLite graph — symbols, -calls, imports, snippets — and exposes it through a CLI. `index` once; -`search` or `ground` to land somewhere; `trail` and `snippet` to follow it; -`context` or `packet` when you need a bundle. Sidecars (Zoekt, Qdrant, SCIP) -are for full `packet`/`search`; local browsing does not require them. +**Action.** CodeStory indexes the workspace into a local SQLite graph: +files, symbols, calls, imports, snippets, search projections, and freshness +notes. Use `doctor`, `index`, `ground`, and `report` for local navigation. Use +`packet` and `search` after sidecars report `retrieval_mode: "full"`. -**Result.** Work starts at a file and line you can open, not at whichever match -ranked first in ripgrep. Answers say what they used; gaps say when the index is -stale or incomplete. +**Result.** Work starts at a file and line you can open, not whichever match +ranked first in ripgrep. Answers say what they used; gaps say when the index or +sidecars are stale. ```mermaid flowchart LR - repo[Workspace files] --> idx[index] - idx --> db[(SQLite graph)] - db --> q[search or ground] - q --> t[trail and snippet] - t --> out[context or packet] - idx --> side[retrieval index] - side --> out + Repo["workspace files"] --> Index["index"] + Index --> Store["local SQLite graph"] + Store --> Local["local: ground, report, files, trail, snippet"] + Store --> RetrievalIndex["retrieval index"] + RetrievalIndex --> Sidecars["Zoekt + Qdrant + SCIP + llama.cpp"] + Sidecars --> AgentSearch["agent: packet, search"] ``` -## How it works +## What You Get -The indexer uses tree-sitter on supported languages, writes nodes, edges, -occurrences, and symbol search docs to a per-project cache under your user -directory. Incremental refresh picks up edits without a full rebuild every time. - -| Command | What you get | +| Need | Use | | --- | --- | -| `ground` | Repo summary and coverage gaps | -| `search` | Ranked symbols, paths, literals | -| `trail` | Callers, callees, references for one node id | -| `snippet` | Source lines around a node | -| `context` | Bundled evidence for one target | -| `packet` | Task-sized citations (`retrieval_mode=full`) | - -Most lookup is graph + lexical symbol docs. `retrieval index` embeds a -policy-selected subset of anchors when you need sidecar search — not every -symbol gets a vector. +| "Where do I start?" | `doctor`, `index`, `ground`, `report` | +| "What does this symbol touch?" | `symbol`, `trail`, `snippet` | +| "What changed and what might break?" | `affected` | +| "Answer a broad repo question with citations." | `packet` with full sidecars | +| "Find candidates by behavior term, path, symbol, or literal." | `search` with full sidecars | -[docs/usage.md](docs/usage.md) · [docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) +## Try It On A Repo -## Try it +From this checkout, build the CLI and point it at any repository: ```sh cargo build --release -p codestory-cli -export CODESTORY_CLI="./target/release/codestory-cli" -export TARGET_WORKSPACE="/path/to/repo" +CODESTORY_CLI="./target/release/codestory-cli" +TARGET_WORKSPACE="/path/to/repo" "$CODESTORY_CLI" doctor --project "$TARGET_WORKSPACE" +"$CODESTORY_CLI" setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json "$CODESTORY_CLI" index --project "$TARGET_WORKSPACE" --refresh full "$CODESTORY_CLI" ground --project "$TARGET_WORKSPACE" --why -"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "WorkspaceIndexer" --why +"$CODESTORY_CLI" report --project "$TARGET_WORKSPACE" --output-file codestory-report.md ``` -Windows: `.\target\release\codestory-cli.exe`, `$env:TARGET_WORKSPACE = "C:\path\to\repo"`. +On Windows PowerShell, use `.\target\release\codestory-cli.exe`, environment +assignments such as `$env:TARGET_WORKSPACE = "C:\path\to\repo"`, and normal +Windows paths. + +That path proves local navigation readiness. It does not prove sidecar readiness +for `packet` or `search`. -## Install as an agent skill +For agent-facing packet/search evidence, build and verify sidecars first: + +```sh +cargo retrieval-setup +"$CODESTORY_CLI" retrieval index --project "$TARGET_WORKSPACE" --refresh full +"$CODESTORY_CLI" retrieval status --project "$TARGET_WORKSPACE" --format json +"$CODESTORY_CLI" packet --project "$TARGET_WORKSPACE" --question "what owns request routing?" +"$CODESTORY_CLI" search --project "$TARGET_WORKSPACE" --query "request routing" --why +``` + +`retrieval status` must report `retrieval_mode: "full"` before trusting +`packet` or `search`. See +[docs/usage.md](docs/usage.md) for task-shaped flows and +[docs/ops/retrieval-sidecars.md](docs/ops/retrieval-sidecars.md) for sidecar +setup and repair. + +## Install As An Agent Skill Copy [`.agents/skills/codestory-grounding`](.agents/skills/codestory-grounding) to your skill directory. Run `scripts/setup.sh` or `scripts/setup.ps1`. See @@ -88,14 +99,46 @@ your skill directory. Run `scripts/setup.sh` or `scripts/setup.ps1`. See | Cache health | `doctor --project ` | | Index | `index --project --refresh full` | | Orientation | `ground --project --why` | -| Lookup | `search --project --query "…" --why` | +| Lookup with sidecars | `search --project --query "..." --why` | | Call graph | `trail --project --id --story` | | Source | `snippet --project --id ` | | Target bundle | `context --project --id ` | -| Task packet (sidecars) | `packet --project --question "…"` | +| Task packet with sidecars | `packet --project --question "..."` | | Persistent reads | `serve --project --stdio` | -## Docs +## Language Support + +CodeStory separates parser-backed graph coverage, structural collectors, +regression-tested fidelity, and agent packet/search readiness. The current +contract is documented in +[docs/architecture/language-support.md](docs/architecture/language-support.md). + +Python, Java, Rust, JavaScript, TypeScript/TSX, C++, C, Go, Ruby, PHP, C#, +Kotlin, Swift, Dart, and Bash are fidelity-gated parser-backed graph languages. +HTML, CSS, and SQL use structural collectors. + +## Evidence + +Benchmark notes are environment- and repository-specific evidence. Do not turn +one row into a universal savings claim. + +- Scorecard and caveats: [docs/testing/benchmark-ledger.md](docs/testing/benchmark-ledger.md) +- Repo-scale timing history: [docs/testing/codestory-e2e-stats-log.md](docs/testing/codestory-e2e-stats-log.md) +- Warm stdio loop history: [docs/testing/codestory-stdio-warm-loop-stats.md](docs/testing/codestory-stdio-warm-loop-stats.md) +- Repeatable with/without harness: [`scripts/codestory-agent-ab-benchmark.mjs`](scripts/codestory-agent-ab-benchmark.mjs) + +## Contributing + +Start with the contributor docs, then run Cargo checks serially because this +workspace shares build locks. + +- [docs/contributors/getting-started.md](docs/contributors/getting-started.md) +- [docs/contributors/debugging.md](docs/contributors/debugging.md) +- [docs/contributors/testing-matrix.md](docs/contributors/testing-matrix.md) +- [docs/architecture/overview.md](docs/architecture/overview.md) +- [docs/architecture/runtime-execution-path.md](docs/architecture/runtime-execution-path.md) + +## Docs Map - Usage: [docs/usage.md](docs/usage.md) - Concepts: [docs/concepts/how-codestory-works.md](docs/concepts/how-codestory-works.md) diff --git a/crates/codestory-cli/tests/onboarding_contracts.rs b/crates/codestory-cli/tests/onboarding_contracts.rs deleted file mode 100644 index 807b69c7..00000000 --- a/crates/codestory-cli/tests/onboarding_contracts.rs +++ /dev/null @@ -1,478 +0,0 @@ -use std::fs; -use std::path::{Path, PathBuf}; -use std::process::Command; - -fn repo_root() -> PathBuf { - PathBuf::from(env!("CARGO_MANIFEST_DIR")) - .parent() - .expect("cli crate has workspace parent") - .parent() - .expect("workspace root exists") - .to_path_buf() -} - -fn collect_markdown_files(dir: &Path, files: &mut Vec) { - for entry in fs::read_dir(dir).expect("read markdown dir") { - let entry = entry.expect("markdown entry"); - let path = entry.path(); - if path.is_dir() { - collect_markdown_files(&path, files); - continue; - } - if path.extension().and_then(|ext| ext.to_str()) == Some("md") { - files.push(path); - } - } -} - -fn extract_markdown_links(contents: &str) -> Vec { - let mut links = Vec::new(); - let bytes = contents.as_bytes(); - let mut index = 0; - while index + 1 < bytes.len() { - if bytes[index] == b']' && bytes[index + 1] == b'(' { - let mut end = index + 2; - while end < bytes.len() && bytes[end] != b')' { - end += 1; - } - if end < bytes.len() { - links.push(contents[index + 2..end].trim().to_string()); - index = end; - } - } - index += 1; - } - links -} - -fn normalize_local_link_target(raw: &str) -> Option { - let target = raw.trim().trim_matches(|ch| ch == '<' || ch == '>'); - if target.is_empty() - || target.starts_with('#') - || target.starts_with("http://") - || target.starts_with("https://") - || target.starts_with("mailto:") - || target.starts_with("app://") - || target.starts_with("plugin://") - { - return None; - } - - Some( - target - .split_once('#') - .map(|(path, _)| path) - .unwrap_or(target) - .to_string(), - ) -} - -fn assert_public_doc_avoids_agent_specific_framing(file: &Path, contents: &str) { - let lowered = contents.to_lowercase(); - for blocked in [ - "codegraph", - "codex-first", - "codex first", - "global codex", - "for codex users", - ".codex/skills", - ".codex\\skills", - ] { - assert!( - !lowered.contains(blocked), - "public doc should not contain `{blocked}`: {}", - file.display() - ); - } -} - -fn extract_inline_toml_string_array(manifest: &str, key: &str) -> Vec { - let prefix = format!("{key} = ["); - let line = manifest - .lines() - .find(|line| line.trim_start().starts_with(&prefix)) - .unwrap_or_else(|| panic!("manifest should contain inline array `{key}`")); - let values = line - .trim() - .strip_prefix(&prefix) - .and_then(|value| value.strip_suffix(']')) - .unwrap_or_else(|| panic!("manifest should use inline string array for `{key}`")); - - values - .split(',') - .map(|value| value.trim().trim_matches('"').to_string()) - .filter(|value| !value.is_empty()) - .collect() -} - -#[test] -fn cli_package_metadata_is_adoption_ready() { - let root = repo_root(); - let manifest_path = root.join("crates/codestory-cli/Cargo.toml"); - let manifest = fs::read_to_string(&manifest_path).expect("CLI manifest should exist"); - - for required in [ - "description = \"Local repository evidence and grounding CLI for source-backed coding workflows.\"", - "license = \"Apache-2.0\"", - "repository = \"https://github.com/TheGreenCedar/CodeStory.git\"", - "readme = \"../../README.md\"", - ] { - assert!( - manifest.contains(required), - "CLI package metadata should include `{required}`" - ); - } - - let readme_from_manifest = manifest_path - .parent() - .expect("CLI manifest should have parent") - .join("../../README.md"); - assert_eq!( - fs::canonicalize(readme_from_manifest).expect("manifest readme path should resolve"), - fs::canonicalize(root.join("README.md")).expect("repo README should resolve"), - "CLI package readme should point at the repository README" - ); - - let keywords = extract_inline_toml_string_array(&manifest, "keywords"); - assert_eq!( - keywords, - vec!["code-search", "grounding", "cli", "agents"], - "keywords should stay conservative and adoption-oriented" - ); - assert!( - keywords.len() <= 5, - "crates.io accepts at most five package keywords" - ); - for keyword in keywords { - assert!( - keyword.len() <= 20 - && keyword - .chars() - .all(|ch| ch.is_ascii_alphanumeric() || ch == '-'), - "keyword should stay crates.io-compatible: {keyword}" - ); - } - - let categories = extract_inline_toml_string_array(&manifest, "categories"); - assert_eq!( - categories, - vec!["command-line-utilities", "development-tools"], - "categories should stay accurate and crates.io-compatible" - ); -} - -#[test] -fn readme_keeps_customer_first_onboarding() { - let root = repo_root(); - let readme = fs::read_to_string(root.join("README.md")).expect("README should exist"); - assert!(readme.contains("How it works")); - assert!(readme.contains("**Situation.**")); - assert!(readme.contains("**Task.**")); - assert!(readme.contains("**Action.**")); - assert!(readme.contains("**Result.**")); - assert!(readme.contains("Try it")); - assert!(readme.contains("Local codebase grounding for coding agents")); - assert!(readme.contains("Install as an agent skill")); - assert!(readme.contains("## Commands")); - assert!(readme.contains("retrieval_mode=full")); - assert!(readme.contains("retrieval index")); - assert!(readme.contains(".agents/skills/codestory-grounding/SKILL.md")); - assert!(readme.contains("docs/usage.md")); - assert!(readme.contains("docs/concepts/how-codestory-works.md")); - assert!(readme.contains("docs/architecture/language-support.md")); - assert!(readme.contains("docs/testing/benchmark-ledger.md")); - assert!(readme.contains("serve --project")); - assert!(readme.contains("docs/architecture/overview.md")); - assert!(readme.contains("docs/contributors/getting-started.md")); - assert!( - readme.find("Try it").expect("quickstart section") - < readme.find("## Docs").expect("reference section"), - "README should show the usable path before doc links" - ); - - for path in [ - "docs/usage.md", - "docs/concepts/how-codestory-works.md", - "docs/architecture/overview.md", - "docs/architecture/runtime-execution-path.md", - "docs/architecture/language-support.md", - "docs/architecture/subsystems/contracts.md", - "docs/architecture/subsystems/workspace.md", - "docs/architecture/subsystems/indexer.md", - "docs/architecture/subsystems/runtime.md", - "docs/architecture/subsystems/store.md", - "docs/architecture/subsystems/cli.md", - "docs/contributors/getting-started.md", - "docs/contributors/debugging.md", - "docs/contributors/testing-matrix.md", - ".agents/skills/codestory-grounding/scripts/setup.ps1", - ".agents/skills/codestory-grounding/scripts/setup.sh", - "scripts/codestory-agent-ab-benchmark.mjs", - ] { - assert!( - root.join(path).exists(), - "expected onboarding doc to exist: {path}" - ); - } - - for path in [ - ".agents/skills/codestory-grounding/scripts/setup.ps1", - ".agents/skills/codestory-grounding/scripts/setup.sh", - ] { - let setup = fs::read_to_string(root.join(path)).expect("read setup script"); - assert!( - !setup.contains("DEFAULT_CODESTORY_REPO_REF"), - "setup script should not pin a stale default CLI source ref: {path}" - ); - assert!( - setup.contains("CODESTORY_REPO_REF"), - "setup script should keep explicit source-ref override support: {path}" - ); - assert!( - setup.contains("origin/HEAD"), - "setup script should build the remote default branch when no ref is explicit: {path}" - ); - } -} - -#[test] -fn docs_drift_contracts_keep_living_sources_explicit() { - let root = repo_root(); - let usage = fs::read_to_string(root.join("docs/usage.md")).expect("usage doc should exist"); - let testing_matrix = fs::read_to_string(root.join("docs/contributors/testing-matrix.md")) - .expect("testing matrix should exist"); - let language_support = fs::read_to_string(root.join("docs/architecture/language-support.md")) - .expect("language support doc should exist"); - let benchmark_scorecard = fs::read_to_string(root.join("docs/testing/benchmark-ledger.md")) - .expect("benchmark ledger should exist"); - - assert!( - usage.contains( - r#"setup embeddings --project "$TARGET_WORKSPACE" --dry-run --format json"# - ) || usage.contains("setup embeddings --project"), - "usage doc should document managed embedding setup" - ); - assert!( - !usage.contains("semantic_doc_scope = \"durable\""), - "usage config example should omit the default durable semantic scope" - ); - for accepted_scope in ["`all`", "`full`", "`all-symbols`", "`all_symbols`"] { - assert!( - usage.contains(accepted_scope), - "usage docs should name accepted all-symbol semantic_doc_scope value {accepted_scope}" - ); - } - assert!( - testing_matrix.contains("latest row in") - && testing_matrix.contains("codestory-e2e-stats-log.md") - && testing_matrix.contains("historical") - && testing_matrix.contains("examples only"), - "testing matrix should point current timing claims at the living stats log" - ); - assert!( - !testing_matrix.contains("The 2026-04-18 repo-scale baseline"), - "testing matrix should not present an old hard-coded baseline as current" - ); - assert!( - benchmark_scorecard.contains("## Current Scorecard") - && benchmark_scorecard.contains("codestory-e2e-stats-log.md"), - "benchmark ledger should keep the scorecard and living timing log references" - ); - for required in [ - "parser-backed graph", - "fidelity-gated", - "structural collector", - "candidate parser compatibility record", - "Go, Ruby, PHP, C#, Kotlin, Swift, Dart, Bash", - "Kotlin, Swift, Dart, Bash", - ] { - assert!( - language_support.contains(required), - "language support doc should preserve support-claim term `{required}`" - ); - } - for required in [ - "crates/codestory-contracts/src/language_support.rs", - "language_support_profile_for_ext", - "language_support_profile_for_language_name", - "get_language_for_ext", - ] { - assert!( - language_support.contains(required), - "language support docs should mention `{required}`" - ); - } - assert!( - testing_matrix.contains("../architecture/language-support.md"), - "testing matrix should link the language support claim contract" - ); - assert!( - root.join("docs/testing/benchmark-ledger.md").exists(), - "benchmark ledger should preserve detailed historical rows" - ); -} - -#[test] -fn public_docs_avoid_competitor_and_agent_specific_framing() { - let root = repo_root(); - let mut files = vec![root.join("README.md")]; - collect_markdown_files(&root.join("docs"), &mut files); - collect_markdown_files(&root.join(".agents/skills/codestory-grounding"), &mut files); - - for file in files { - let contents = fs::read_to_string(&file).expect("read public doc"); - assert_public_doc_avoids_agent_specific_framing(&file, &contents); - } -} - -#[test] -fn usage_doc_keeps_agent_contract_terms_out_of_operator_flow() { - let root = repo_root(); - let usage = fs::read_to_string(root.join("docs/usage.md")).expect("usage doc should exist"); - assert!(usage.contains("Common Workflows")); - assert!(usage.contains("I need a repo overview")); - assert!(usage.contains("I need evidence for a broad question")); - assert!(usage.contains("The cache or local navigation looks stale")); - assert!(usage.contains("For agent-facing packet/search recovery")); - assert!(usage.contains( - "codestory-cli retrieval index --project --refresh full --format json" - )); - for blocked in [ - "sufficiency.avoid_opening", - "supported-claim wording", - "claim-ledger", - "Support files", - ] { - assert!( - !usage.contains(blocked), - "operator usage doc should not expose agent-internal contract term {blocked}" - ); - } -} - -#[test] -fn usage_doc_names_two_readiness_tracks_and_predictable_output_modes() { - let root = repo_root(); - let usage = fs::read_to_string(root.join("docs/usage.md")).expect("usage doc should exist"); - - assert!(usage.contains("## Readiness Tracks")); - assert!(usage.contains("`local_navigation`")); - assert!(usage.contains("`agent_packet_search`")); - assert!(usage.contains("`retrieval_mode: \"full\"`")); - assert!(usage.contains("## Predictable Output Modes")); - assert!(usage.contains("Most commands default to Markdown")); - assert!( - usage.contains("Use `--format json` when automation needs the complete structured result") - ); - assert!(usage.contains("Use `--output-file `")); - assert!(usage.contains("The parent directory must already exist")); - assert!(usage.contains("`explore` opens the terminal UI by default")); - assert!(usage.contains("Use `--no-tui`")); - assert!( - usage - .find("## Readiness Tracks") - .expect("readiness heading") - < usage - .find("## Retrieval Defaults") - .expect("retrieval defaults heading"), - "usage should introduce readiness tracks before retrieval defaults" - ); -} - -#[test] -fn benchmark_docs_show_proof_tier_ladder() { - let root = repo_root(); - let benchmark_scorecard = fs::read_to_string(root.join("docs/testing/benchmark-ledger.md")) - .expect("benchmark ledger should exist"); - - assert!(benchmark_scorecard.contains("## Proof Tier Ladder")); - for tier in [ - "Stats-only local regression signal", - "Full sidecar readiness proof", - "Real-repo drill proof", - "Promotion-grade benchmark proof", - ] { - assert!( - benchmark_scorecard.contains(tier), - "benchmark ledger should explain proof tier {tier}" - ); - } - assert!(benchmark_scorecard.contains("Full sidecar readiness, agent packet/search readiness")); - assert!(benchmark_scorecard.contains("`retrieval_mode: \"full\"`")); - assert!(benchmark_scorecard.contains("Generalized agent savings")); - assert!( - benchmark_scorecard - .find("## Proof Tier Ladder") - .expect("proof tier ladder") - < benchmark_scorecard - .find("## Promotion Rules") - .expect("promotion rules"), - "proof tier ladder should frame promotion rules" - ); -} - -#[test] -fn markdown_links_resolve_to_existing_local_files() { - let root = repo_root(); - let mut markdown_files = vec![root.join("README.md")]; - collect_markdown_files(&root.join("docs"), &mut markdown_files); - - for file in markdown_files { - let contents = fs::read_to_string(&file).expect("read markdown file"); - for link in extract_markdown_links(&contents) { - let Some(target) = normalize_local_link_target(&link) else { - continue; - }; - let resolved = file.parent().expect("markdown file parent").join(target); - assert!( - resolved.exists(), - "broken markdown link in {} -> {}", - file.display(), - resolved.display() - ); - } - } -} - -#[test] -fn codestory_grounding_skill_command_refs_track_cli_commands() { - let root = repo_root(); - let skill_root = root.join(".agents/skills/codestory-grounding"); - let commands = [ - "index", "ground", "doctor", "search", "symbol", "trail", "snippet", "query", "explore", - "bookmark", "context", "packet", "drill", "setup", "serve", - ]; - - for command in commands { - let reference = skill_root.join("references").join(format!("{command}.md")); - assert!( - reference.exists(), - "codestory-grounding should document `{command}` at {}", - reference.display() - ); - - let help = Command::new(env!("CARGO_BIN_EXE_codestory-cli")) - .arg(command) - .arg("--help") - .output() - .unwrap_or_else(|error| panic!("run `{command} --help`: {error}")); - assert!( - help.status.success(), - "`{command}` should remain a valid CLI subcommand\nstdout:\n{}\nstderr:\n{}", - String::from_utf8_lossy(&help.stdout), - String::from_utf8_lossy(&help.stderr) - ); - } - - for command in ["context", "bookmark", "doctor", "explore", "serve"] { - let reference = - fs::read_to_string(skill_root.join("references").join(format!("{command}.md"))) - .expect("read command reference"); - for required in ["Normal path", "Failure path", "Integration edge"] { - assert!( - reference.contains(required), - "`{command}` reference should include a {required} row" - ); - } - } -} diff --git a/docs/concepts/how-codestory-works.md b/docs/concepts/how-codestory-works.md index fdb31a66..dafa5081 100644 --- a/docs/concepts/how-codestory-works.md +++ b/docs/concepts/how-codestory-works.md @@ -4,9 +4,20 @@ CodeStory indexes a workspace into a local graph, then serves read commands against that graph. It does not replace tests or judgment; it structures the first pass. -Command loop: [README — How it works](../../README.md#how-it-works). +Command loop: [README - What You Get](../../README.md#what-you-get). Readiness lanes: [usage.md](../usage.md#readiness-tracks). +```mermaid +flowchart TD + Files["workspace files"] --> Plan["workspace discovery and refresh plan"] + Plan --> Parse["tree-sitter parsing and semantic resolution"] + Parse --> Graph["SQLite graph, occurrences, snippets, snapshots"] + Graph --> Local["local navigation commands"] + Graph --> SidecarBuild["retrieval index"] + SidecarBuild --> Sidecars["Zoekt, Qdrant, SCIP, llama.cpp"] + Sidecars --> Agent["agent packet/search commands"] +``` + ## What gets stored Per-project SQLite under your user cache, keyed by workspace path: @@ -23,6 +34,16 @@ Per-project SQLite under your user cache, keyed by workspace path: Repo content stays local. Managed setup may fetch tool assets; indexed evidence does not leave the cache unless you copy it. +## The loop + +```text +doctor -> index -> ground/report/files -> exact target -> trail/snippet/context +``` + +Use `packet` and `search` after the sidecar lane reports +`retrieval_mode: "full"`. Until then, keep local browsing on exact targets from +`ground`, `report`, `files`, or existing node ids. + ## Terms | Term | Meaning | @@ -30,12 +51,23 @@ does not leave the cache unless you copy it. | Grounding | Context tied back to indexed files and symbols | | Symbol doc | Generated searchable text for a symbol (lexical, not embedded by default) | | Dense anchor | Policy-selected symbol or report that gets a vector | -| Snapshot | Derived read model; may be stale — commands should say so | +| Snapshot | Derived read model; may be stale, and commands should say so | | Trail | Graph walk from one symbol: callers, callees, neighbors | | Packet | Bounded task evidence with citations, gaps, next commands | More: [glossary.md](../glossary.md). +## What good output looks like + +A good CodeStory-backed answer does three things: + +1. Names the files, symbols, snippets, or sidecar evidence it used. +2. Says when evidence is stale, partial, ambiguous, or missing. +3. Gives the next concrete command when the current evidence is not enough. + +The goal is not a more confident answer. The goal is confidence constrained by +source evidence. + ## Related - [usage.md](../usage.md) diff --git a/docs/contributors/testing-matrix.md b/docs/contributors/testing-matrix.md index 01c0487d..dbb71d22 100644 --- a/docs/contributors/testing-matrix.md +++ b/docs/contributors/testing-matrix.md @@ -14,7 +14,7 @@ flowchart TD change --> cli["CLI args or output boundary work"] change --> bench["Bench or perf-surface work"] change --> e2e["Repo-scale semantic or cold-start behavior"] - docs --> docs_checks["markdown/link checks + any touched doc contracts"] + docs --> docs_checks["readback + git diff --check"] always --> workspace["fmt, check, targeted tests, clippy"] indexer --> fidelity["fidelity_regression, tictactoe_language_coverage, integration"] store --> store_tests["cargo test -p codestory-store"] @@ -40,11 +40,11 @@ These are the default checks for any contributor change. If you only changed `README.md` or `docs/**`, use the smallest credible lane: ```sh -cargo fmt --check -cargo test -p codestory-cli --test onboarding_contracts +git diff --check ``` -Only escalate to broader cargo checks if the doc change depends on new code behavior or command output. +Read the changed pages back before finishing. Only escalate to broader Cargo +checks if the doc change depends on new code behavior or command output. ## Indexer And Graph Fidelity diff --git a/docs/glossary.md b/docs/glossary.md index 89d8a6fe..60fb6e2d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -2,27 +2,27 @@ ## Readiness -- **local navigation readiness** — SQLite cache, graph, and DB-backed browse commands (`ground`, `trail`, `snippet`, etc.) are usable -- **agent packet/search readiness** — sidecars healthy and `retrieval_mode=full`; required for trustworthy `packet` / `search` -- **retrieval mode** — sidecar status contract; only `full` serves agent packet/search -- **semantic ready** — dense-anchor embedding state matches policy; not the same as agent packet/search readiness +- **local navigation readiness**: SQLite cache, graph, and DB-backed browse commands (`ground`, `report`, `files`, `trail`, `snippet`, `context --id`, etc.) are usable +- **agent packet/search readiness**: sidecars are healthy and `retrieval_mode=full`; required for trustworthy `packet`, `search`, and query-based candidate discovery +- **retrieval mode**: sidecar status contract; only `full` serves agent packet/search +- **semantic ready**: dense-anchor embedding state matches policy; not the same as agent packet/search readiness ## Index and graph -- **grounding** — indexed context returned for a question or command, with source ties -- **snapshot** — derived grounding view rebuilt from graph tables -- **projection** — persisted derived state (callable projection, ranked summaries) -- **staged snapshot** — temporary DB during full refresh before publish -- **refresh baseline** — file inventory used to plan incremental refresh -- **trail** — focused graph walk from one symbol -- **symbol doc** — deterministic per-symbol search text in SQLite; not embedded by default -- **dense anchor** — symbol, component report, or doc selected for vector embedding -- **repo-text hit** — raw file-content match; diagnostic, not a substitute for graph evidence +- **grounding**: indexed context returned for a question or command, with source ties +- **snapshot**: derived grounding view rebuilt from graph tables +- **projection**: persisted derived state such as callable projection state or ranked summaries +- **staged snapshot**: temporary DB during full refresh before publish +- **refresh baseline**: file inventory used to plan incremental refresh +- **trail**: focused graph walk from one symbol +- **symbol doc**: deterministic per-symbol search text in SQLite; not embedded by default +- **dense anchor**: symbol, component report, or doc selected for vector embedding +- **repo-text hit**: raw file-content match; diagnostic, not a substitute for graph evidence ## System -- **runtime** — orchestrates index, search, grounding, trails, agent flows -- **workspace** — manifest and discovery for which files belong to the project -- **contracts** — shared graph types, DTOs, events across crates -- **target context** — DB-first bundle for one concrete target (`context`), not broad `packet` -- **cache root** — directory for one project cache; override with `--cache-dir` +- **runtime**: orchestrates indexing, grounding, trails, packet/search flows, and system actions +- **workspace**: manifest and discovery layer for which files belong to the project +- **contracts**: shared graph types, DTOs, and events across crates +- **target context**: DB-first bundle for one concrete target (`context --id` or bookmark), not broad `packet` +- **cache root**: directory for one project cache; override with `--cache-dir` diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 71dd6098..34357ab0 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -1,6 +1,6 @@ # CodeStory Benchmark Ledger -Decision-grade scorecard and benchmark history — too dense for the README. +Decision-grade scorecard and benchmark history - too dense for the README. Treat every row as machine-, cache-, runner-, and date-specific. Do not quote a row as a universal savings claim without checking harness tier and setup. diff --git a/docs/testing/retrieval-architecture.md b/docs/testing/retrieval-architecture.md index 5237e6e3..ea89c9da 100644 --- a/docs/testing/retrieval-architecture.md +++ b/docs/testing/retrieval-architecture.md @@ -112,7 +112,6 @@ test-only eval probe module. ```sh cargo test -p codestory-runtime --test retrieval_generalization_guard node --test scripts/tests/codestory-agent-ab-analyzer.test.mjs -cargo test -p codestory-cli --test onboarding_contracts ``` Optional broader lane: @@ -145,7 +144,6 @@ tests in the branch. Do not infer support for languages without direct benchmark | `freelancer` / `traderotate` removed from default holdouts | done | OSS holdouts only | | Generalization lint + guard test | done | `lint-retrieval-generalization.mjs`, `retrieval_generalization_guard` | | Warning config | done | `docs/architecture/retrieval-rollback.json` | -| Markdown link contract (`onboarding_contracts`) | verify | `cargo test -p codestory-cli --test onboarding_contracts` | | local-real cold packet + north-star SLOs | **human** | p99 retrieval, quality 3/4, wall targets | | holdout-retrieval pass without skip allowances | **human** | Requires materialized OSS repos + index; no generalized claim without required recall/quality/forbidden-claim thresholds | | `agent_value_gap` < 0.20 | **human** | Measure from a fresh coherent bundle | diff --git a/docs/usage.md b/docs/usage.md index a2891cd0..996c1d31 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -6,7 +6,7 @@ Windows: `.\target\release\codestory-cli.exe`, `$env:NAME = "value"`. ## Install The Skill Install the grounding skill once, then point it at explicit target workspaces. -See [README — Install as an agent skill](../README.md#install-as-an-agent-skill) +See [README - Install as an agent skill](../README.md#install-as-an-agent-skill) for the full copy/setup commands and Windows PowerShell variant. The source skill package lives at @@ -37,14 +37,14 @@ TARGET_WORKSPACE="/path/to/repo" ## Readiness Tracks -Two lanes — do not mix them when judging `packet` or `search` output. +Two lanes - do not mix them when judging `packet` or `search` output. | | Local navigation | Agent packet/search | | --- | --- | --- | | Lane id | `local_navigation` | `agent_packet_search` | | Built by | `index` | `index` then `retrieval index` | | Requires | Healthy SQLite cache and graph | Sidecars healthy and `retrieval_mode=full` | -| Commands | `ground`, `symbol`, `trail`, `snippet`, `explore`, `context`, `files`, `affected`, `report` | `packet`, `search` | +| Commands | `ground`, `report`, `files`, `symbol`, `trail`, `snippet`, `explore`, `context --id`, `affected` | `packet`, `search`, query-based candidate discovery | | Does not prove | Sidecar readiness | That cache-only browse is enough for agent search | `doctor` reports lane status. Sidecar topology: @@ -60,10 +60,11 @@ codestory-cli doctor --project codestory-cli index --project --refresh full codestory-cli ground --project --why codestory-cli report --project --output-file out/codestory-report.md +codestory-cli report --project --format json --output-file out/codestory-graph.json ``` -Health check, orientation, optional report (hotspots, entry points). Regenerate -the report after index changes — it is not source-of-truth state. +Health check, orientation, optional report and graph export. Regenerate reports +after index changes; they are artifacts, not source-of-truth state. ### I need evidence for a broad question @@ -76,14 +77,26 @@ Requires `retrieval_mode=full`. ### I need to understand one symbol or file +With full sidecar readiness, use `search` for candidate discovery: + ```sh codestory-cli search --project --query "" --why codestory-cli trail --project --id --story --hide-speculative codestory-cli snippet --project --id --context 40 ``` -Pick a node id from `search`, then inspect call paths and source. `context` -bundles one target for handoff. +Without sidecars, stay on the local navigation lane until you have a concrete +target: + +```sh +codestory-cli ground --project --why +codestory-cli report --project --output-file out/codestory-report.md +codestory-cli files --project --path src --limit 80 +``` + +Then use `symbol`, `trail`, `snippet`, or `context --id` with an exact node id +from local output. Do not treat `search` or `context --query` as cache-only +fallbacks; query-based discovery is part of the agent packet/search lane. ### I changed files and need likely impact @@ -93,7 +106,7 @@ codestory-cli affected --project --format markdown git diff --name-only HEAD | codestory-cli affected --project --stdin --format json ``` -Impacted symbols and test hints — not a substitute for running tests. +Impacted symbols and test hints - not a substitute for running tests. ### The cache or local navigation looks stale @@ -103,7 +116,9 @@ codestory-cli index --project --refresh full codestory-cli doctor --project ``` -Fix inventory or indexing errors before trusting search output. +Fix inventory or indexing errors before trusting local navigation output. If +`packet`, `search`, or `context --query` reports `retrieval_unavailable`, repair +the sidecar lane instead of repeating the same command. ### For agent-facing packet/search recovery @@ -114,7 +129,7 @@ codestory-cli retrieval status --project --format json codestory-cli doctor --project --format markdown ``` -Target `retrieval_mode=full`. Core index problems may require `index` first — +Target `retrieval_mode=full`. Core index problems may require `index` first - see `ready --goal agent`. ## Core Commands @@ -129,15 +144,16 @@ see `ready --goal agent`. SQLite store; use `--output-file` to keep artifacts separate from terminal logs. - `packet`: bounded broad-task evidence packet with citations, budget usage, - gaps, and follow-up commands. -- `search`: candidate discovery for symbols, files, literals, API paths, - modules, and behavior terms. + gaps, and follow-up commands; requires agent packet/search readiness. +- `search`: sidecar-backed candidate discovery for symbols, files, literals, + API paths, modules, and behavior terms. - `symbol`: inspect one exact symbol and relationships. - `trail`: follow caller, callee, and reference relationships around a symbol. - `snippet`: fetch source context around a symbol. - `explore`: bundled navigation packet or terminal explorer around a target. -- `context`: deep evidence bundle for one concrete target selected by `--id`, - `--query`, or `--bookmark`. +- `context`: deep evidence bundle for one concrete target. `--id` and + `--bookmark` are exact-target paths; `--query` must be treated like + sidecar-backed discovery. - `affected`: map changed files to impacted symbols and likely tests. - `files`: inspect indexed file inventory, language counts, roles, and coverage notes. @@ -181,9 +197,15 @@ reset, schema change, or suspected stale-state incident. ## Predictable Output Modes -Most commands default to Markdown for human review. Use `--format json` when automation needs the complete structured result, including exact field comparisons such as `retrieval_mode` or cache paths. Use `--output-file ` when the artifact should live outside terminal logs. The parent directory must already exist. +Most commands default to Markdown for human review. Use `--format json` when +automation needs the complete structured result, including exact field +comparisons such as `retrieval_mode` or cache paths. Use `--output-file ` +when the artifact should live outside terminal logs. The parent directory must +already exist. -`explore` opens the terminal UI by default when a TUI is available. Use `--no-tui`, `--plain`, or `CODESTORY_NO_TUI=1` for predictable command output in agent runs, tests, non-interactive terminals, and CI logs. +`explore` opens the terminal UI by default when a TUI is available. Use +`--no-tui`, `--plain`, or `CODESTORY_NO_TUI=1` for predictable command output in +agent runs, tests, non-interactive terminals, and CI logs. Agent-facing Markdown may start with `Status`, `Trust`, `Next Action`, and `Proof Tier` before dense citations. Use `search --why --plan-details` only when @@ -332,7 +354,7 @@ Typical recovery flow: ```sh codestory-cli doctor --project codestory-cli index --project --refresh full -codestory-cli search --project --query WorkspaceIndexer +codestory-cli ground --project --why ``` If the cache directory itself is suspect, get the exact project cache path from @@ -375,10 +397,10 @@ cargo test cargo clippy --all-targets -- -D warnings ``` -Focused docs/onboarding lane: +Docs-only lane: ```sh -cargo test -p codestory-cli --test onboarding_contracts +git diff --check ``` Release-blocking fidelity lanes: diff --git a/scripts/tests/codestory-agent-ab-analyzer.test.mjs b/scripts/tests/codestory-agent-ab-analyzer.test.mjs index e6162fbb..46a2c9f0 100644 --- a/scripts/tests/codestory-agent-ab-analyzer.test.mjs +++ b/scripts/tests/codestory-agent-ab-analyzer.test.mjs @@ -262,7 +262,7 @@ test("categorizes commands without treating source paths as cli invocations", () ); assert.equal(commandCategory("Get-Content crates/codestory-cli/src/main.rs"), "direct_file_read"); assert.equal(commandCategory("Get-Content C:\\tools\\codestory-cli.exe"), "direct_file_read"); - assert.equal(commandCategory("cargo test -p codestory-cli --test onboarding_contracts"), "build_test"); + assert.equal(commandCategory("cargo test -p codestory-cli --test runtime_backed_flows"), "build_test"); }); test("packet gate retries only transient sidecar packet failures", async () => { @@ -697,8 +697,8 @@ test("analyzes transcript command friction and scores manifest anchors", () => { commandEvent("cmd_7", "item.completed", `$p='"'crates/codestory-runtime/src/lib.rs'; Get-Content $p`, "pub struct RuntimeContext;"), commandEvent("cmd_5", "item.started", "git status --short"), commandEvent("cmd_5", "item.completed", "git status --short", ""), - commandEvent("cmd_6", "item.started", "cargo test -p codestory-cli --test onboarding_contracts"), - commandEvent("cmd_6", "item.completed", "cargo test -p codestory-cli --test onboarding_contracts", "ok"), + commandEvent("cmd_6", "item.started", "cargo test -p codestory-cli --test runtime_backed_flows"), + commandEvent("cmd_6", "item.completed", "cargo test -p codestory-cli --test runtime_backed_flows", "ok"), { type: "item.completed", item: { @@ -724,7 +724,7 @@ test("analyzes transcript command friction and scores manifest anchors", () => { id: "fixture", task_class: "architecture_explanation", expected_files: ["crates/codestory-cli/src/main.rs"], - expected_verification_files: ["crates/codestory-cli/tests/onboarding_contracts.rs"], + expected_verification_files: ["crates/codestory-cli/tests/runtime_backed_flows.rs"], expected_symbols: ["RuntimeContext::ensure_open", "MissingSymbol"], expected_claims: ["Full indexing starts"], forbidden_claims: ["remote service is required"], @@ -744,7 +744,7 @@ test("analyzes transcript command friction and scores manifest anchors", () => { assert.deepEqual(quality.missed_anchors.symbols, ["MissingSymbol"]); assert.equal(quality.expected_verification_files.recall, 0); assert.deepEqual(quality.missed_anchors.verification_files, [ - "crates/codestory-cli/tests/onboarding_contracts.rs", + "crates/codestory-cli/tests/runtime_backed_flows.rs", ]); assert.equal(quality.citation_coverage.recall, 1); }); From 1e22872cdd7877a4b297de36407ee640360a7a9b Mon Sep 17 00:00:00 2001 From: Albert Najjar Date: Sun, 14 Jun 2026 16:46:21 -0400 Subject: [PATCH 14/14] Update retrieval bootstrap docs and sidecar schema --- README.md | 2 +- .../codestory-indexer/src/resolution/mod.rs | 2 ++ docs/ops/retrieval-sidecars.md | 26 ++++++++++--------- docs/testing/benchmark-ledger.md | 2 +- docs/testing/retrieval-architecture.md | 22 +++++++++------- docs/usage.md | 15 ++++++----- 6 files changed, 38 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index b1bf3131..16a02c35 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ for `packet` or `search`. For agent-facing packet/search evidence, build and verify sidecars first: ```sh -cargo retrieval-setup +"$CODESTORY_CLI" retrieval bootstrap --project "$TARGET_WORKSPACE" --format json "$CODESTORY_CLI" retrieval index --project "$TARGET_WORKSPACE" --refresh full "$CODESTORY_CLI" retrieval status --project "$TARGET_WORKSPACE" --format json "$CODESTORY_CLI" packet --project "$TARGET_WORKSPACE" --question "what owns request routing?" diff --git a/crates/codestory-indexer/src/resolution/mod.rs b/crates/codestory-indexer/src/resolution/mod.rs index fba97ba5..5669df13 100644 --- a/crates/codestory-indexer/src/resolution/mod.rs +++ b/crates/codestory-indexer/src/resolution/mod.rs @@ -2769,6 +2769,7 @@ mod tests { kind INTEGER NOT NULL, source_node_id INTEGER NOT NULL, target_node_id INTEGER NOT NULL, + file_node_id INTEGER, resolved_target_node_id INTEGER, confidence REAL, certainty TEXT, @@ -2869,6 +2870,7 @@ mod tests { kind INTEGER NOT NULL, source_node_id INTEGER NOT NULL, target_node_id INTEGER NOT NULL, + file_node_id INTEGER, resolved_target_node_id INTEGER, confidence REAL, certainty TEXT, diff --git a/docs/ops/retrieval-sidecars.md b/docs/ops/retrieval-sidecars.md index b9ce9618..1c88049b 100644 --- a/docs/ops/retrieval-sidecars.md +++ b/docs/ops/retrieval-sidecars.md @@ -35,11 +35,13 @@ flowchart LR From the CodeStory repository root (Windows, macOS, Linux): ```sh -cargo retrieval-setup +cargo run -p codestory-cli -- retrieval bootstrap --project . --format json ``` This starts or checks the local sidecar services for the CodeStory checkout; it does not by itself finalize the retrieval manifest for every target workspace. +The `--project .` is intentional here. For another repo, pass that repo path to +`--project`. Plain `codestory-cli index` builds the core SQLite code index only. It can make the local navigation lane usable, but it does not generate sidecar artifacts or @@ -56,7 +58,7 @@ node scripts/setup-retrieval-env.mjs --fetch-embed-model export CODESTORY_EMBED_MODEL_DIR="$(pwd)/target/retrieval-models" export CODESTORY_EMBED_BACKEND="llamacpp" export CODESTORY_EMBED_LLAMACPP_URL="http://127.0.0.1:8080/v1/embeddings" -cargo retrieval-setup +./target/release/codestory-cli retrieval bootstrap --project --format json ./target/release/codestory-cli index --project --refresh full ./target/release/codestory-cli retrieval index --project --refresh full ./target/release/codestory-cli retrieval status --project --format json @@ -75,12 +77,11 @@ Qdrant component as policy-skipped rather than querying a missing collection. Status after bootstrap: ```sh -cargo retrieval-status +cargo run -p codestory-cli -- retrieval status --project . --format json ``` -Aliases are defined in [`.cargo/config.toml`](../../.cargo/config.toml). They run -`codestory retrieval bootstrap --project .` and `retrieval status --project .`, building the CLI -when needed. +Optional aliases are defined in [`.cargo/config.toml`](../../.cargo/config.toml). +They wrap the same project-dot bootstrap and status commands. **Bootstrap flags** (via `cargo run -p codestory-cli -- retrieval bootstrap ...`): @@ -103,7 +104,7 @@ node scripts/setup-retrieval-env.mjs --with-holdout-clone |------|---------| | `--check-only` | Prerequisites report only; exit 1 if required tools missing | | `--skip-compose` | Passed to bootstrap | -| `--skip-build` | Skip `cargo build` (alias still builds on first `cargo retrieval-setup`) | +| `--skip-build` | Skip `cargo build` when the wrapper invokes bootstrap directly | | `--with-holdout-clone` | Also run `scripts/fetch-holdout-repos.mjs` (large git clones under `target/`) | When `--fetch-embed-model` is present, the wrapper downloads @@ -129,7 +130,7 @@ Compose file: [`docker/retrieval-compose.yml`](../../docker/retrieval-compose.ym | Dependency | Pin policy | Pinned version | Notes | |------------|------------|----------------|-------| -| Zoekt real (Phase 2) | `COMPOSE_PROFILES=real` | `zoekt-20250506123554` | `sourcegraph/zoekt-webserver:0.0.0-20250506123554-490422d1adb4` + lexical shards | +| Zoekt real | `COMPOSE_PROFILES=real` | `zoekt-20250506123554` | `sourcegraph/zoekt-webserver:0.0.0-20250506123554-490422d1adb4` + lexical shards | | Qdrant | Fixed container image tag | `qdrant/qdrant:v1.12.5` | HTTP `6333`, gRPC `6334` | | SCIP | CodeStory graph artifact emitter | `graph-` | Generated local graph artifacts under the sidecar generation | @@ -258,7 +259,7 @@ Managed `setup embeddings` output is not a substitute for this lane: it may install local semantic assets, but it does not start llama.cpp, build the retrieval manifest, or make `retrieval status` report `full`. -**Phase 2 (shipped in crate):** +**Shipped component status:** | Component | Status | |-----------|--------| @@ -281,7 +282,8 @@ diagnostic only and never produce `retrieval_mode=full`. - `CODESTORY_EMBED_MODEL_DIR=/target/retrieval-models` - `CODESTORY_EMBED_BACKEND=llamacpp` (recommended explicit product mode; unset is also product mode for retrieval commands) - `CODESTORY_EMBED_LLAMACPP_URL=http://127.0.0.1:8080/v1/embeddings` -3. `cargo retrieval-setup` (starts Qdrant, Zoekt webserver, `codestory-embed` on `:8080`) +3. `./target/release/codestory-cli retrieval bootstrap --project --format json` + starts Qdrant, Zoekt webserver, and `codestory-embed` on `:8080`. 4. Dim smoke: `curl -s http://127.0.0.1:8080/v1/embeddings -H "Content-Type: application/json" -d "{\"input\":[\"function\"]}"` → embedding length **768** 5. `retrieval index --project --refresh full` (manifest records `embedding_backend`, `embedding_dim`, `sidecar_input_hash`, `sidecar_generation`, the generated Qdrant collection, `symbol_doc_count`, `dense_projection_count`, `semantic_policy_version`, `graph_artifact_hash`, and dense reason counts; the input hash includes symbol-doc and dense-anchor metadata plus the embedding contract) 6. `retrieval status` → `retrieval_mode: full` and `capabilities.semantic=true` @@ -323,7 +325,7 @@ count is zero, Qdrant reuse is skipped explicitly and cannot mask stale graph/le ./target/release/codestory-cli retrieval down ``` -### Standalone query (Phase 2+) +### Standalone Query ```sh ./target/release/codestory-cli retrieval query "ExtensionService" --project . @@ -343,7 +345,7 @@ GGUF embedding model. 1. `retrieval up` - exit 0 2. `retrieval status` - JSON with expected shape; non-`full` status is a failure for agent use 3. `retrieval index --project ` - manifest row in SQLite only when all sidecars are real -4. `retrieval query ""` - Phase 2+ +4. `retrieval query ""` - standalone sidecar query 5. `retrieval down` - clean shutdown **CI reduced sequence:** diff --git a/docs/testing/benchmark-ledger.md b/docs/testing/benchmark-ledger.md index 34357ab0..c9792b18 100644 --- a/docs/testing/benchmark-ledger.md +++ b/docs/testing/benchmark-ledger.md @@ -172,7 +172,7 @@ mismatches. Warm stdio task medians ranged from `2.69s` to `3.60s`, with an aggregate task median of `3.13s`; cold CLI task medians ranged from `4.22s` to `5.76s`, with an aggregate task median of `4.86s`. -## Methodology +## Harness Contract The agent A/B harness runs the same repository prompt in two arms: diff --git a/docs/testing/retrieval-architecture.md b/docs/testing/retrieval-architecture.md index ea89c9da..102d1ae7 100644 --- a/docs/testing/retrieval-architecture.md +++ b/docs/testing/retrieval-architecture.md @@ -12,7 +12,7 @@ env vars, CI smoke), [`../architecture/retrieval-design.md`](../architecture/ret --- -## Implemented stack (Phases 0–5) +## Implemented Stack | Layer | Location | Role | |-------|----------|------| @@ -107,7 +107,7 @@ anchors such as repository names, specific source paths, and manifest-specific symbols. Keep those strings in manifests, tests, benchmark harnesses, or the test-only eval probe module. -## Fast CI-style checks (automated in Phase 6) +## Required Checks ```sh cargo test -p codestory-runtime --test retrieval_generalization_guard @@ -124,10 +124,10 @@ node --test scripts/tests/codestory-agent-ab-analyzer.test.mjs --- -## Promotion checklist +## Promotion Checklist -Status as of Phase 6 documentation pass. **Benchmark pass columns require a human run** with -repos, sidecars, and release CLI — not claimed here. +**Benchmark pass columns require a human run** with repos, sidecars, and release +CLI. This page records the gates; it does not claim those rows have passed. ### Language support audit alignment @@ -136,7 +136,7 @@ tests in the branch. Do not infer support for languages without direct benchmark | Item | Status | Notes | |------|--------|-------| -| Phases 0–5 code landed | done | See implemented stack above | +| Core sidecar stack | done | See implemented stack above | | Architecture / design docs | done | `docs/architecture/retrieval-design.md` | | Sidecar runbook | done | `docs/ops/retrieval-sidecars.md` | | Local-real manifests | done | `benchmarks/tasks/local-real/` | @@ -150,7 +150,7 @@ tests in the branch. Do not infer support for languages without direct benchmark | Windows `retrieval-sidecar-smoke` CI job | fail-closed sidecar smoke | [`retrieval-sidecars.md`](../ops/retrieval-sidecars.md#preflight-smoke-contract) | | Ragas/Phoenix nightly eval | optional | Not configured | -### North-star SLOs (targets — measure before claiming pass) +### North-Star SLOs | Metric | Target | |--------|--------| @@ -171,13 +171,13 @@ tests in the branch. Do not infer support for languages without direct benchmark --- -## Rollback drill (REQ-RES-005) +## Rollback Warning Drill After promotion runs, verify rollback warnings: 1. Point `retrieval_rollback` at a baseline `packet-runtime-summary.json` with thresholds that will trip on the current summary (or use unit test `rollback_drill_warns_without_setting_legacy_env` in `retrieval_rollback.rs`). 2. Confirm `check_and_log_rollback_warnings` logs trigger ids without setting `CODESTORY_RETRIEVAL=0`. -3. File a one-line incident note in this doc with date and trigger id if rollback fires in production promotion. +3. Record the trigger id with the promotion evidence if rollback fires during production promotion. **One-shot operator drill (after each promotion run):** @@ -187,7 +187,9 @@ cargo test -p codestory-runtime retrieval_rollback::tests::rollback_drill_warns_ Expect rollback warnings only when configured thresholds fire (see `docs/architecture/retrieval-rollback.json`). Sidecar retrieval remains mandatory. -**Closure status (2026-05-27, semantic promotion pass):** Phase A shipped (bge-base 768-d, llama.cpp `embed` compose service, manifest `embedding_backend`/`embedding_dim`, Qdrant collection migration, llamacpp dim hard-fail). Local `retrieval status` reaches `full` with default 768-d vectors after Qdrant re-index. Sidecar-primary is the intended product path, but product promotion remains gated until fresh benchmark evidence passes. +**Promotion note:** Local `retrieval status` can report `full` after Qdrant +re-index. Sidecar-primary is the intended product path, but product promotion +still requires fresh benchmark evidence. --- diff --git a/docs/usage.md b/docs/usage.md index 996c1d31..da3dad5b 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -59,8 +59,8 @@ Two lanes - do not mix them when judging `packet` or `search` output. codestory-cli doctor --project codestory-cli index --project --refresh full codestory-cli ground --project --why -codestory-cli report --project --output-file out/codestory-report.md -codestory-cli report --project --format json --output-file out/codestory-graph.json +codestory-cli report --project --output-file codestory-report.md +codestory-cli report --project --format json --output-file codestory-graph.json ``` Health check, orientation, optional report and graph export. Regenerate reports @@ -90,7 +90,7 @@ target: ```sh codestory-cli ground --project --why -codestory-cli report --project --output-file out/codestory-report.md +codestory-cli report --project --output-file codestory-report.md codestory-cli files --project --path src --limit 80 ``` @@ -237,7 +237,7 @@ node scripts/setup-retrieval-env.mjs --fetch-embed-model export CODESTORY_EMBED_MODEL_DIR="$(pwd)/target/retrieval-models" export CODESTORY_EMBED_BACKEND="llamacpp" export CODESTORY_EMBED_LLAMACPP_URL="http://127.0.0.1:8080/v1/embeddings" -cargo retrieval-setup +codestory-cli retrieval bootstrap --project --format json codestory-cli index --project --refresh full codestory-cli retrieval index --project --refresh full @@ -252,9 +252,10 @@ with SHA-256 `ad1afe72cd6654a558667a3db10878b049a75bfd72912e1dabb91310d671173c`; all configured mirrors must pass the same check. -Run `codestory-cli retrieval index` only after the local sidecar services, -llama.cpp embedding endpoint, and `bge-base-en-v1.5` model configuration are -ready, then require `retrieval status --format json` to report +Run `codestory-cli retrieval bootstrap` for the same target workspace you will +query. Then run `codestory-cli retrieval index` only after the local sidecar +services, llama.cpp embedding endpoint, and `bge-base-en-v1.5` model +configuration are ready. Require `retrieval status --format json` to report `retrieval_mode: "full"` before trusting agent-facing packet/search evidence. The status JSON also reports `query_embedding_backend`, `manifest_vector_embedding_backend`, and `stored_doc_vector_producer_backend`