From 986e6e96b1907ec85c0568637fc6404914c3e05b Mon Sep 17 00:00:00 2001 From: Osho Emmanuel Date: Tue, 14 Jul 2026 13:22:30 +0100 Subject: [PATCH 1/4] docs: document bounded harness architecture - Replace stale agent-mode guidance with single/agent/auto model-call docs. - Add the current harness architecture spec and migration guide. - Commit shared domain language and update AGENTS.md for the review harness. --- AGENTS.md | 55 ++++- CONTEXT.md | 151 +++++++++++++ README.md | 34 ++- ...0-vectorlint-harness-architecture-audit.md | 5 + docs/best-practices.mdx | 22 +- docs/cli-reference.mdx | 33 ++- docs/docs.json | 4 +- docs/how-it-works.mdx | 84 +++++--- docs/introduction.mdx | 2 +- ...3-31-agent-mode-implementation-plan.log.md | 8 + docs/migration-from-agent-mode.mdx | 51 +++++ docs/model-calls.mdx | 75 +++++++ docs/rubric-scoring.mdx | 23 +- docs/specs/2026-07-10-harness-architecture.md | 199 ++++++++++++++++++ docs/troubleshooting.mdx | 18 ++ docs/use-cases.mdx | 4 +- 16 files changed, 695 insertions(+), 73 deletions(-) create mode 100644 CONTEXT.md create mode 100644 docs/migration-from-agent-mode.mdx create mode 100644 docs/model-calls.mdx create mode 100644 docs/specs/2026-07-10-harness-architecture.md diff --git a/AGENTS.md b/AGENTS.md index 2d696a8..0604a5b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,46 @@ # Repository Guidelines -This repository implements VectorLint — a prompt‑driven, structured‑output content evaluator. Use this guide to navigate the codebase, run it locally, and contribute safely. +This repository implements VectorLint — a prompt‑driven, structured‑output content review harness. Use this guide to navigate the codebase, run it locally, and contribute safely. + +Use [`CONTEXT.md`](./CONTEXT.md) as the shared VectorLint domain language across code, docs, tests, and agent work. Prefer its terms when naming modules, writing tests, drafting docs, and describing architecture. + +## Agent Behavior Guidelines + +These guidelines reduce common LLM coding mistakes. They bias toward caution over speed; use judgment for trivial tasks. + +### Think Before Coding + +- State assumptions explicitly before implementing. +- If multiple interpretations exist, present them instead of picking silently. +- If a simpler approach exists, say so and push back when warranted. +- If something is unclear, stop, name the confusion, and ask. + +### Simplicity First + +- Write the minimum code that solves the problem. +- Do not add features beyond what was asked. +- Do not introduce abstractions for single-use code. +- Do not add flexibility or configurability that was not requested. +- Do not add error handling for impossible scenarios. +- If a change becomes larger than needed, simplify before finishing. + +### Surgical Changes + +- Touch only the files and lines required by the request. +- Do not improve adjacent code, comments, or formatting unless required. +- Match existing style, even when another style is tempting. +- If you notice unrelated dead code, mention it instead of deleting it. +- Remove imports, variables, functions, or files made unused by your own changes. +- Do not remove pre-existing dead code unless asked. + +### Goal-Driven Execution + +- Turn requests into verifiable goals before implementing. +- For bug fixes, write or identify a reproduction, then make it pass. +- For validation changes, cover invalid inputs, then make the checks pass. +- For refactors, verify behavior before and after the change. +- For multi-step tasks, state a brief plan with verification for each step. +- Loop until the success criteria are verified or a blocker is clear. ## Project Structure & Module Organization @@ -15,9 +55,12 @@ This repository implements VectorLint — a prompt‑driven, structured‑output - `output/` — TTY formatting (reporter, evidence location, line numbering) - `prompts/` — YAML frontmatter parsing, schema validation, directive loading - `providers/` — LLM abstractions (OpenAI, Anthropic, Azure, Gemini), request builder, provider factory + - `review/` — neutral review contract, boundary, budget, schemas, and model-call selection + - `executors/` — bounded `single` and `agent` model-call executors behind `ReviewExecutor` + - `findings/` — shared finding verification, filtering, scoring, diagnostics, and result assembly - `scan/` — file discovery (fast‑glob) honoring config and exclusions - `schemas/` — Zod schemas for all external data (API responses, config, CLI, env) - - `scoring/` — score calculation (density-based for check, rubric-based for judge) + - `scoring/` — score calculation for objective violation checks - `types/` — TypeScript type definitions - `presets/` — bundled rule packs (e.g., `VectorLint/`) - `tests/` — Vitest specs for config, scanning, evaluation, providers @@ -126,12 +169,14 @@ For documents >600 words, VectorLint automatically chunks content: ## Architecture & Principles - Boundary validation: all external data (files, CLI, env, APIs) validated at system boundaries using Zod schemas +- Bounded harness model: callers own exploration and context gathering; VectorLint owns constrained review through `single`/`agent`/`auto` model calls behind the `ReviewExecutor` contract +- On-page boundary: executors review target content plus explicit review context only; do not add workspace search, arbitrary file reads, model-authored rule overrides, or autonomous agent behavior - Type safety: strict TypeScript with no `any`; use `unknown` + schema validation for external data -- Dependency inversion: depend on `LLMProvider` and `SearchProvider` interfaces; keep providers thin (transport only) +- Dependency inversion: depend on `StructuredModelClient`, `ToolCallingModelClient`, and `SearchProvider` interfaces; keep providers thin (transport only) - Dependency injection: inject `RequestBuilder` via provider constructor to avoid coupling -- Separation of concerns: rules define rubric; schemas enforce structure; CLI orchestrates; evaluators process; reporters format +- Separation of concerns: rules define observable violation checks; schemas enforce structure; CLI orchestrates; executors run model calls; findings process results; reporters format output - Separation of concerns: when a file starts combining contracts, orchestration, and utility logic, extract shared helpers and types into focused modules -- Extensibility: add providers by implementing `LLMProvider` or `SearchProvider`; add evaluators via registry pattern +- Extensibility: add model providers by implementing the structured/tool-calling model client interfaces or search by implementing `SearchProvider` - Error handling: prefer the repository's custom error hierarchy over native `Error`; catch blocks use `unknown` type and extend existing custom error types before introducing raw exceptions - Shared domain constants: avoid magic strings for core runtime concepts; define shared constants, enums, or types and import them where needed - Naming: choose domain-accurate names that reflect the real abstraction level; avoid use-case-specific terminology in shared runtime code diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..ee0d542 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,151 @@ +# VectorLint + +VectorLint is a content review harness. This language file defines the terms used to describe reviews, rules, findings, scoring, and execution so product, documentation, tests, and code can share the same vocabulary. + +## Language + +### Review Model + +**Content Review Harness**: +A system that reviews supplied target content against source-backed rules and returns structured review results. The harness does not own exploration, research, workspace search, or content rewriting. +_Avoid_: Autonomous agent, workspace agent, agent mode + +**Review**: +One evaluation of target content against one or more rules. A review produces findings, scores, diagnostics, usage metadata, and output. +_Avoid_: Lint run when referring to the domain process, agent session + +### Review Inputs + +**Caller**: +The person or system that invokes VectorLint and supplies the target, target content, and any allowed review context. A caller does not define rules or configuration as part of the review invocation. +_Avoid_: Agent, user agent, workspace owner + +**Target**: +The subject of a review. A target identifies what VectorLint is reviewing, regardless of whether it came from a file, memory, or another caller-provided source. +_Avoid_: File when the review subject may not be a file, page when the content may not be documentation + +**Target File**: +A file used as the source of a target. A target file is one way to provide target content, but not every target must come from a file. +_Avoid_: Target when the distinction between the file and the review subject matters + +**Target Content**: +The actual text or content body reviewed for a target. Findings and finding evidence are grounded in target content. +_Avoid_: Target file, page content + +**Review Context**: +Additional content explicitly supplied to VectorLint for a review. Review context is in scope only because it was allowed as part of the review input. +_Avoid_: Caller context, workspace context, discovered context, ambient context + +**On-Page Boundary**: +The rule that VectorLint reviews only target content and review context. VectorLint must not discover arbitrary workspace files or expand the review scope on its own. +_Avoid_: Workspace scope, project-wide scope, cross-file scope + +### Rules + +**Rule**: +A source-backed instruction that defines observable violation conditions VectorLint should look for. Rules exist before a review is invoked. +_Avoid_: Prompt when referring to the domain concept, model instruction when implying the model authored it + +**Via Negativa Review**: +A review approach that looks for evidence a rule was violated rather than evidence the content aligns with an ideal. VectorLint rules should be written so a model can answer whether an observable violation condition is present. +_Avoid_: Alignment review, subjective assessment, rubric judgment + +**Violation Condition**: +An observable yes/no condition that counts as a rule violation when present in target content. A violation condition should be specific enough to ground a finding in finding evidence. +_Avoid_: Criterion when it implies subjective grading, preference, guideline + +**Rule Pack**: +A named collection of related rules that can be applied together. +_Avoid_: Preset when describing the domain concept, folder + +**Review Configuration**: +Pre-existing settings that determine how VectorLint runs reviews, selects rules, and formats behavior. Configuration is not the same as target content or review context. +_Avoid_: Caller input when referring to review invocation, target configuration + +**Check Rule**: +A rule expressed through observable violation conditions. This is the only future-facing rule style in VectorLint. +_Avoid_: Direct rule, standard rule + +### Execution + +**Model Call**: +The call shape VectorLint uses to run a reviewer model against target content during a review. +_Avoid_: Execution strategy, content access, process mode, evaluator type, rule type, mode + +**Single Model Call**: +A model call where VectorLint supplies the review request and target content without giving the model a read tool. If VectorLint chunks a large target, each chunk is still reviewed through a single model call. +_Avoid_: Direct strategy, standard mode, check path + +**Agent Model Call**: +A bounded model call where the model may request sections of the target content through a single target-scoped read capability. This is for context management during large or context-sensitive reviews; it is not workspace exploration. +_Avoid_: Autonomous agent mode, workspace agent, file reader + +**Auto Model Call**: +The model call value where VectorLint deterministically chooses between single and agent. +_Avoid_: Smart mode, agent fallback + +**Target Read Capability**: +The bounded ability to read line ranges from target content. It cannot read arbitrary files, search the workspace, rewrite rules, or create top-level workspace findings. +_Avoid_: Tool suite, workspace tools, read_file + +### Findings + +**Candidate Finding**: +A potential issue raised before VectorLint has decided whether it should be reported. Candidate findings may be filtered out or become diagnostics. +_Avoid_: Violation when the issue has not been accepted, result + +**Verified Finding**: +A finding that VectorLint has accepted for reporting because it is grounded in the target content and passes review filters. +_Avoid_: Issue when finding-evidence status matters, raw finding + +**Finding**: +A reported content issue produced by a review. When precision matters, use candidate finding or verified finding. +_Avoid_: Violation as the default user-facing term, problem + +**Finding Evidence**: +The exact target-content text or surrounding context that supports a finding. +_Avoid_: Quote when the text may include surrounding context, match when referring to the concept rather than a located span + +**Finding Evidence Verification**: +The act of confirming that the finding evidence for a candidate finding can be located in the target content. Unverified finding evidence should not become a verified finding. +_Avoid_: Line fallback, model-provided location + +**Finding Processing**: +The review step that turns candidate findings into verified findings, diagnostics, scores, and final review output. +_Avoid_: Projection, result processing, output processing + +**Diagnostic**: +A structured note about review execution or finding processing, especially when something affects trust, completeness, or interpretation but is not itself a content finding. +_Avoid_: Finding, warning when referring to the structured domain object + +### Scoring And Output + +**Severity**: +The impact level attached to a finding or rule outcome. +_Avoid_: Priority, importance + +**Score**: +A normalized quality measurement for a rule or review. Scores come from verified finding count or density for objective violation checks. +_Avoid_: Grade when referring to the domain object, rating + +**Review Result**: +The structured outcome of a review: verified findings, scores, diagnostics, usage metadata, and operational status. +_Avoid_: Projection result, formatter result, raw model output + +**Output Format**: +The representation used to present a review result to a human or machine, such as line output or JSON. +_Avoid_: Review result, report type + +**Review Budget**: +The explicit limits for a review, such as model calls, target size, review context size, chunks per rule, and duration. A review budget limits work, not the number of verified findings emitted. +_Avoid_: Rate limit, timeout when referring to the full budget concept + +### Historical Terms + +**Autonomous Agent Mode**: +The old VectorLint direction where VectorLint exposed workspace tools to a model and let it explore beyond the target content. This is historical language only; current domain language should use single model call, agent model call, caller, and content review harness. +_Avoid_: Agent mode as a current feature, workspace-agent review + +**Judge Rule**: +The historical rubric-style assessment rule where a model graded content against criteria. This rule style is removed from the future-facing product model. +_Avoid_: Rubric mode, subjective check, evaluator kind diff --git a/README.md b/README.md index 662ac7f..989399b 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # VectorLint: Prompt it, Lint it! [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -VectorLint is a command-line tool that evaluates and scores content using LLMs. It uses [LLM-as-a-Judge](https://en.wikipedia.org/wiki/LLM-as-a-Judge) to catch terminology, technical accuracy, and style issues that require contextual understanding. +VectorLint is a command-line content review harness that uses LLMs to find observable terminology, technical accuracy, and style violations that require contextual understanding. ![VectorLint Screenshot](./assets/VectorLint_screenshot.jpeg) @@ -144,16 +144,32 @@ Notes: - Prompts and outputs are recorded when Langfuse observability is enabled. - Do not send secrets, credentials, or PII unless your policy explicitly allows observability tooling to access that data. -## Agent Mode (under review) +## How VectorLint Runs Reviews -The `--mode agent` flag is under active rework. It currently enables an -autonomous cross-file review mode that is being **removed** in favor of a -bounded harness model. See -[`docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md`](docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md) -for the decision and the refactor plan. +VectorLint is a bounded review harness, not a workspace agent. Callers choose +the target content and any review context; VectorLint reviews that supplied +content against source-backed rules and returns findings, scores, diagnostics, +and usage metadata. -Until the refactor lands, `--mode agent` prints a deprecation warning and -falls back to standard mode. Do not build integrations against it. +Use `--model-call` to choose how the reviewer model is invoked: + +```bash +vectorlint doc.md --model-call single +vectorlint doc.md --model-call agent +vectorlint doc.md --model-call auto +``` + +- `single` runs structured model calls without tools. This is best for normal + documents and straightforward rule checks. +- `agent` runs a bounded target-only model call that can page through the + current target with `read_target_section`. It cannot search the workspace, + read arbitrary files, or override rules. +- `auto` is the default. VectorLint chooses `single` for normal inputs and + `agent` for large or multi-rule reviews that benefit from target paging. + +See the current +[`docs/specs/2026-07-10-harness-architecture.md`](docs/specs/2026-07-10-harness-architecture.md) +for the contract and boundary details. ## Contributing diff --git a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md index 51161ce..356e150 100644 --- a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md +++ b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md @@ -1,5 +1,10 @@ # VectorLint Harness Architecture Audit +> **HISTORICAL AUDIT.** This audit records the July 2026 decision that led to +> the bounded harness refactor. Some "current state" observations below +> describe the pre-refactor code. For the shipped architecture, see +> [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md). + Date: 2026-07-10 Status: Objective system audit plus product-direction audit. Request changes before shipping the current autonomous workspace-agent surface as a public contract. diff --git a/docs/best-practices.mdx b/docs/best-practices.mdx index 05c6ad8..a4e7276 100644 --- a/docs/best-practices.mdx +++ b/docs/best-practices.mdx @@ -51,23 +51,19 @@ LLMs evaluate content relative to an implied standard. Make that standard explic A grammar rule without context produces generic grammar findings. The same rule with a developer audience context produces findings calibrated to technical writing conventions. -## Use weights that reflect real priorities +## Write observable violation conditions -In judge rules, weights are the single most important configuration decision. They determine which criteria actually control the final score. Treat them as a statement of your content team's values — not arbitrary numbers. +VectorLint works best when each rule names what counts as a violation. Treat a rule as a checklist of conditions the reviewer can confirm from target evidence. -```yaml -criteria: - - name: Technical Accuracy - weight: 40 # Factual errors erode user trust — this matters most - - name: Clarity - weight: 30 # Unclear docs generate support tickets - - name: Tone - weight: 20 # Important but recoverable in editing - - name: SEO - weight: 10 # Nice to have, never at the expense of the above +```markdown +Flag a headline when it: +1. Omits the product or feature being discussed +2. Promises a benefit that the article does not support +3. Uses vague value words such as "seamless", "powerful", or "robust" +4. Exceeds 70 characters ``` -If everything has the same weight, nothing is prioritized. +Observable conditions make findings easier to verify and reduce subjective disagreements during review. ## Tier strictness by content type diff --git a/docs/cli-reference.mdx b/docs/cli-reference.mdx index febe11c..128c7ff 100644 --- a/docs/cli-reference.mdx +++ b/docs/cli-reference.mdx @@ -58,10 +58,35 @@ vectorlint --help ## Flags -| Flag | Description | -|------|-------------| -| `--help` | Print help and exit | -| `--version` | Print the installed VectorLint version and exit | +| Flag | Default | Description | +|------|---------|-------------| +| `--help` | — | Print help and exit | +| `--version` | — | Print the installed VectorLint version and exit | +| `--config ` | `.vectorlint.ini` | Path to a custom project config file | +| `--debug-json` | `false` | Write debug JSON artifacts for surfaced findings and review metadata | +| `--model-call ` | `auto` | Model-call strategy: `single`, `agent`, or `auto` | +| `--output ` | `line` | Output format: `line`, `json`, `vale-json`, or `rdjson` | +| `--show-prompt` | `false` | Print full prompt and injected content | +| `--show-prompt-trunc` | `false` | Print truncated prompt/content previews | +| `--verbose` | `false` | Enable verbose logging | + +### `--model-call` + +`--model-call` controls how VectorLint invokes the reviewer model: + +```bash +vectorlint doc.md --model-call single +vectorlint doc.md --model-call agent +vectorlint doc.md --model-call auto +``` + +| Value | Behavior | +| --- | --- | +| `single` | Runs structured model calls without tools | +| `agent` | Runs bounded target-only paging with `read_target_section` | +| `auto` | Default; chooses `single` or `agent` based on target size and rule count | + +See [Model calls](/model-calls) for usage guidance. ## Output diff --git a/docs/docs.json b/docs/docs.json index 5b7727c..f8ee7c1 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -20,7 +20,8 @@ "pages": [ "introduction", "use-cases", - "how-it-works" + "how-it-works", + "model-calls" ] }, { @@ -71,6 +72,7 @@ "configuration-schema", "env-variables", "frontmatter-fields", + "migration-from-agent-mode", "troubleshooting" ] } diff --git a/docs/how-it-works.mdx b/docs/how-it-works.mdx index 93e42fc..d587087 100644 --- a/docs/how-it-works.mdx +++ b/docs/how-it-works.mdx @@ -1,65 +1,83 @@ --- title: How it works -description: How VectorLint evaluates content using LLM-as-a-Judge, filtering, and two scoring methods. +description: How VectorLint reviews supplied content through a bounded harness. --- -VectorLint evaluates your content by sending it to a large language model (LLM) along with a structured prompt that defines your quality criteria. The LLM acts as a judge — reading your content, applying the criteria, and returning findings. VectorLint then filters those findings through a deterministic pipeline before surfacing violations in the CLI. +VectorLint is a content review harness. You give it target content, source-backed rules, provider credentials, and optional review context. VectorLint runs a bounded review and returns verified findings, scores, diagnostics, and usage metadata. -If you haven't run your first check yet, start with the [Quickstart](/quickstart). This page is for understanding the architecture behind what you're running. +If you haven't run your first check yet, start with the [Quickstart](/quickstart). This page explains the architecture behind that command. -## The evaluation pipeline +## The review pipeline -Every time you run `vectorlint doc.md`, three things happen in sequence. +Every time you run `vectorlint doc.md`, VectorLint follows the same high-level path. -### 1. Rule resolution +### 1. Resolve targets and rules -VectorLint reads your `.vectorlint.ini` to determine which rule packs apply to the file. It loads those rule files, prepends the contents of `VECTORLINT.md` (if present) to each rule's system prompt, and assembles the evaluation context. +VectorLint reads your project configuration to determine which rule packs apply to each target file. It loads rule files, applies `VECTORLINT.md` as global review context when present, and builds source-backed rules. -If no `.vectorlint.ini` exists, VectorLint detects `VECTORLINT.md` and creates a synthetic "Style Guide Compliance" rule automatically. This is the zero-config path — the fastest way to get from nothing to a working evaluation without writing any rule pack files. +If no `.vectorlint.ini` exists, VectorLint detects `VECTORLINT.md` and creates a synthetic style-guide rule automatically. This is the zero-config path. -### 2. LLM evaluation +### 2. Build a review request -VectorLint sends your content to the configured LLM provider with each rule's prompt. Depending on the rule type, the model returns either: +The CLI builds a `ReviewRequest` with: -- **A list of specific violations** (check rules) — for countable errors like grammar mistakes or banned terms -- **A scored rubric** (judge rules) — for quality dimensions like tone, clarity, or technical accuracy rated on a 1–4 scale +- target URI, content, content type, and byte length +- source-backed rules +- optional caller-supplied review context +- review budget +- output policy +- model-call strategy -Rules run concurrently up to the `Concurrency` limit set in `.vectorlint.ini`. +The review contract is intentionally neutral. It does not include workspace exploration tools, model-authored rule overrides, or subjective rubric scoring. -### 3. Filtering +### 3. Select a model call -Raw model output contains noise — potential violations that don't hold up under scrutiny. VectorLint reduces false positives through a two-phase filtering process: +VectorLint supports three model-call values: -1. **Candidate generation** — the model returns all potential violations, each tagged with required fields: rule support, exact evidence, context support, plausible non-violation, and fix quality. -2. **Deterministic surfacing** — VectorLint applies a strict filter and only surfaces violations that pass all required gates. +| Model call | Use it for | +| --- | --- | +| `single` | Normal reviews using structured model calls without tools | +| `agent` | Large or context-heavy reviews where the model needs target-section paging | +| `auto` | Default behavior; VectorLint chooses based on target size and rule count | -The CLI output is intentionally stricter than raw model candidates with fewer results and higher confidence. +`agent` is bounded to the current target. The only tool it can use is `read_target_section`, which reads line ranges from the target content already supplied to VectorLint. -You can tune how aggressively the filter operates with `CONFIDENCE_THRESHOLD` (default: `0.75`). Lower values surface more findings with higher recall; higher values surface fewer findings with higher precision. See [Configuring LLM providers](/llm-providers) for details. +See [Model calls](/model-calls) for CLI examples and selection details. -## Scoring +### 4. Process findings -VectorLint uses two scoring methods depending on the rule type. +Both model-call paths send raw candidate findings through the same finding-processing pipeline: -**Density-based scoring** (check rules) calculates error density, violations per 100 words. VectorLint weights a single error in a short document more heavily than the same error in a long one. This normalizes quality assessment across documents of any length. +1. filter candidates through the evidence gate +2. verify finding evidence against target content +3. deduplicate verified findings +4. score by verified finding count and density +5. resolve severity +6. assemble findings, scores, diagnostics, and usage metadata -**Rubric-based scoring** (judge rules) normalizes the LLM's 1–4 rating to a 1–10 scale and computes a weighted average across all criteria. Criteria weights reflect your real-world priorities — technical accuracy might carry a weight of 40 while SEO carries a weight of 10. +Unlocatable quoted evidence becomes a diagnostic and does not become a finding. -See [Quality scoring](/quality-scoring) for the full scoring reference. +### 5. Format output -## Two ways to define quality +The final `ReviewResult` is routed to the requested output format: -VectorLint gives you two complementary tools for expressing your content standards: +- `line` for human-readable terminal output +- `json` for VectorLint's native machine-readable output +- `vale-json` for Vale-compatible integrations +- `rdjson` for reviewdog-compatible integrations -- **`VECTORLINT.md`** — plain-language style instructions that apply globally to every evaluation. The fastest path to useful output: no rule files, no configuration syntax, just plain English instructions that the LLM uses as context for every check. -- **Rule pack files** — structured LLM prompts for specific, measurable checks with weighted scoring criteria. Use these when you need reproducible, auditable results on a particular dimension of quality. +## Boundaries -The two work together: `VECTORLINT.md` sets the baseline context, and rule pack files enforce specific criteria on top of it. +VectorLint reviews only the target content and explicit review context supplied to it. It does not search your workspace, read arbitrary files, or expand scope on its own. -See [Defining your style rules](/style-guide) for how to create both. +This makes VectorLint safe to call from external agents and CI systems: the caller owns exploration, and VectorLint owns constrained review. + +## More detail + +Read the current [harness architecture spec](https://github.com/TRocket-Labs/vectorlint/blob/main/docs/specs/2026-07-10-harness-architecture.md) for the code-level contract and removed surfaces. ## Next steps -- [Quickstart](/quickstart) — run your first evaluation in minutes -- [Defining your style rules](/style-guide) — create VECTORLINT.md and write custom rules -- [Configuring a project](/project-config) — map files to rule packs \ No newline at end of file +- [Model calls](/model-calls) — choose `single`, `agent`, or `auto` +- [Defining your style rules](/style-guide) — create `VECTORLINT.md` and custom rules +- [CLI reference](/cli-reference) — confirm command syntax and flags diff --git a/docs/introduction.mdx b/docs/introduction.mdx index 02197ad..0122713 100644 --- a/docs/introduction.mdx +++ b/docs/introduction.mdx @@ -5,7 +5,7 @@ description: "VectorLint is a large language model (LLM)-based prose linter that ## What is VectorLint? -VectorLint is a command-line tool that evaluates and scores documentation using large language models (LLMs). Instead of regex patterns that can only catch surface-level issues, VectorLint uses an [LLM-as-a-Judge](https://en.wikipedia.org/wiki/LLM-as-a-Judge) approach to catch terminology misuse, technical inaccuracies, and style inconsistencies that require contextual understanding to detect. +VectorLint is a command-line content review harness that uses large language models (LLMs) to evaluate documentation. Instead of regex patterns that can only catch surface-level issues, VectorLint looks for observable terminology, technical accuracy, and style violations that require contextual understanding to detect. If you can write a prompt for it, you can lint it with VectorLint. diff --git a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md b/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md index 4c19efb..641d780 100644 --- a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md +++ b/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md @@ -1,5 +1,13 @@ # Execution Log +> **SUPERSEDED (2026-07-10).** This log describes the removed autonomous +> agent-mode implementation. VectorLint is now a bounded review harness with +> `single`/`agent`/`auto` model calls. See +> [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md) +> for the current architecture and +> [`../migration-from-agent-mode.mdx`](../migration-from-agent-mode.mdx) +> for migration guidance. The details below are historical only. + - **Plan**: `docs/plans/2026-03-31-agent-mode-implementation-plan.md` - **Issue**: Not provided (executed from user directive in this session) - **Started**: 2026-03-31 diff --git a/docs/migration-from-agent-mode.mdx b/docs/migration-from-agent-mode.mdx new file mode 100644 index 0000000..4011c5f --- /dev/null +++ b/docs/migration-from-agent-mode.mdx @@ -0,0 +1,51 @@ +--- +title: Migration from agent mode +description: Move integrations from the removed autonomous agent mode to bounded model calls. +--- + +VectorLint no longer supports autonomous workspace-agent mode. The current product is a bounded review harness: callers supply the target and any review context, and VectorLint reviews that supplied content against source-backed rules. + +## What changed + +- `--mode agent` was removed. Use `--model-call single`, `--model-call agent`, or `--model-call auto`. +- Workspace exploration moved out of VectorLint. External agents or scripts gather context before invoking VectorLint. +- Provider-level agent loops were removed. Providers expose structured-output and bounded tool-calling capabilities. +- Model-authored review instructions were removed. Rules remain source-backed. + +## Mapping + +| Before | After | +| --- | --- | +| `vectorlint doc.md --mode agent` | `vectorlint doc.md --model-call agent` for target paging, or `--model-call single` for structured calls | +| `--mode agent --output json` | `--model-call agent --output json` | +| `report_finding` for workspace findings | Move workspace-level findings to the caller; VectorLint reports target findings only | +| `search_content` or `read_file` workspace tools | Caller gathers context and passes it as review context | +| Model-supplied `reviewInstruction` | Edit the source-backed rule file | +| `runAgentToolLoop` on a provider | Implement `ReviewExecutor`, or use `StructuredModelClient` / `ToolCallingModelClient` behind an executor | + +## Why it changed + +The old direction mixed two ownership models: VectorLint as a linter and VectorLint as an autonomous workspace agent. That duplicated result contracts, weakened scope boundaries, and made findings less consistent across output formats. + +The harness model keeps responsibilities clear. The caller owns exploration. VectorLint owns bounded review. + +## What you get + +- Explicit budgets for target size, review context size, chunks per rule, model calls, and elapsed time. +- One finding-processing path for filtering, evidence verification, scoring, diagnostics, and formatting. +- The same result shape across `single`, `agent`, and `auto`. +- A stable contract external agents can call without granting VectorLint workspace exploration authority. + +## Migration steps + +1. Replace `--mode agent` with `--model-call agent` only when target paging is still useful. +2. Use `--model-call single` for normal reviews that do not need target-section paging. +3. Move workspace search, file selection, and context gathering into the caller. +4. Pass any needed context explicitly as review context when calling the review contract. +5. Rewrite subjective rubric rules as objective Via Negativa violation checks. + +## Related + +- [Model calls](/model-calls) +- [How it works](/how-it-works) +- [CLI reference](/cli-reference) diff --git a/docs/model-calls.mdx b/docs/model-calls.mdx new file mode 100644 index 0000000..dbe34c6 --- /dev/null +++ b/docs/model-calls.mdx @@ -0,0 +1,75 @@ +--- +title: Model calls +description: Choose single, agent, or auto model calls for VectorLint reviews. +--- + +VectorLint uses a model-call strategy to decide how the reviewer model sees target content. The strategy changes the invocation shape, not the rule semantics. + +## Choose a strategy + +| Value | Best for | What happens | +| --- | --- | --- | +| `single` | Normal documents and straightforward checks | VectorLint sends structured model calls without tools | +| `agent` | Large or context-heavy targets | VectorLint gives the model one target-only paging tool | +| `auto` | Default CLI behavior | VectorLint chooses `single` or `agent` from target size and rule count | + +Run with the default: + +```bash +vectorlint doc.md +``` + +Force structured calls: + +```bash +vectorlint doc.md --model-call single +``` + +Force target paging: + +```bash +vectorlint doc.md --model-call agent +``` + +## `single` + +`single` is the simplest model call. VectorLint supplies the rule prompt and target content, receives structured candidate findings, and sends them through shared finding processing. + +For large targets, the single path chunks content and merges candidate violations before verification and scoring. + +## `agent` + +`agent` is bounded target paging. The model can call `read_target_section` to read 1-based line ranges from the current target content. + +The agent model call cannot: + +- search the workspace +- read arbitrary files +- read files by URI +- modify content +- override rules +- report findings outside the target + +Use `agent` when a target is large enough that paging through sections helps the reviewer preserve context. + +## `auto` + +`auto` is the CLI default. VectorLint chooses `agent` when the target is larger than `600_000` bytes or when more than five rules apply. Otherwise, it chooses `single`. + +## Boundary + +VectorLint reviews only supplied target content and explicit review context. If an external agent or script wants VectorLint to consider other files, that caller must gather the relevant content and pass it as review context. VectorLint does not discover workspace context on its own. + +## Output + +All model-call strategies return the same review result shape: findings, scores, diagnostics, and usage metadata. Output format selection is separate: + +```bash +vectorlint doc.md --model-call agent --output json +``` + +## Related + +- [How it works](/how-it-works) +- [CLI reference](/cli-reference) +- [Migration from agent mode](/migration-from-agent-mode) diff --git a/docs/rubric-scoring.mdx b/docs/rubric-scoring.mdx index b5c52e1..0e03ce1 100644 --- a/docs/rubric-scoring.mdx +++ b/docs/rubric-scoring.mdx @@ -1,8 +1,21 @@ --- -title: Rubric Scoring -description: This page is coming soon. +title: Rubric scoring +description: Historical note about removed rubric-style judge rules. --- - - This page is coming soon. Check back for updates, or follow the [VectorLint GitHub repository](https://github.com/TRocket-Labs/vectorlint) for the latest news. - +Rubric-style `judge` rules are no longer part of VectorLint's current rule model. VectorLint now uses objective violation checks: rules describe observable conditions, and scoring is based on verified finding count and density. + +## What to use instead + +Write rules as Via Negativa checks. Describe what counts as a violation, require exact evidence from the target content, and provide concrete fix guidance. + +```markdown +Flag a section when it: +1. Makes a factual claim without a source +2. Uses a deprecated command name +3. Repeats the same setup step already shown earlier +``` + +## Migration + +If you have old rubric-style rules, rewrite each criterion as one or more observable violation conditions. See [Migration from agent mode](/migration-from-agent-mode) for related removed-surface guidance and [Defining your style rules](/style-guide) for current rule syntax. diff --git a/docs/specs/2026-07-10-harness-architecture.md b/docs/specs/2026-07-10-harness-architecture.md new file mode 100644 index 0000000..5498b9e --- /dev/null +++ b/docs/specs/2026-07-10-harness-architecture.md @@ -0,0 +1,199 @@ +# VectorLint Harness Architecture + +Status: Current as of the bounded harness refactor. + +Related audit: [`2026-07-10-vectorlint-harness-architecture-audit.md`](../audits/2026-07-10-vectorlint-harness-architecture-audit.md) + +## Purpose And Product Direction + +VectorLint is a bounded content review harness. External callers, such as Codex, Claude Code, CI jobs, scripts, or humans at the CLI, own exploration and decide what content to review. VectorLint owns constrained review of supplied target content against source-backed rules. + +This means VectorLint is not a workspace agent. It does not search arbitrary files, expand scope on its own, rewrite content, or let a model override the rule it is supposed to enforce. It receives target content, rules, optional caller-supplied review context, budgets, and output policy, then returns structured review results. + +Use the shared domain language in [`../../CONTEXT.md`](../../CONTEXT.md) when changing code, docs, tests, or agent handoffs. + +## Review Contract + +The review contract lives in [`src/review/types.ts`](../../src/review/types.ts), with boundary schemas in [`src/review/schemas.ts`](../../src/review/schemas.ts). + +`ReviewRequest` contains: + +- `target`: the target URI, content, content type, and optional byte length. +- `rules`: source-backed rules with `id`, `source`, `body`, optional `name`, optional `severity`, and optional Via Negativa violation conditions. +- `context`: optional caller-supplied review context. +- `budget`: explicit bounds for one review. +- `outputPolicy`: usage and payload telemetry policy. +- `modelCall`: `single`, `agent`, or `auto`. + +Every executor implements: + +```ts +interface ReviewExecutor { + run(request: ReviewRequest): Promise; +} +``` + +`ReviewResult` contains: + +- `findings`: verified findings anchored in target content. +- `scores`: per-rule scores. +- `diagnostics`: operational and finding-processing notes. +- `usage`: optional model-call and token metadata. +- `hadOperationalErrors`: optional flag for partial-result runs with operational errors. + +The contract deliberately does not expose `evaluator`, `judge`, subjective rubric scoring, model-authored rule overrides, or autonomous workspace tools. + +## Rule Model + +VectorLint rules are objective Via Negativa checks. A rule should name observable violation conditions and ask the reviewer to find evidence that those conditions are present. The reviewer should not grade broad alignment against an ideal. + +Good rules answer yes/no questions: + +- Is this sentence using a banned phrase? +- Does this claim omit required evidence? +- Does this section repeat information already stated nearby? + +Future-facing rule authors should avoid subjective judge/rubric designs. Historical rubric-style language remains only where it describes removed behavior or migration guidance. + +## Model Calls + +The model-call strategy is selected with `--model-call single|agent|auto`. The allowed values and default come from [`src/review/executor.ts`](../../src/review/executor.ts), [`src/cli/types.ts`](../../src/cli/types.ts), and [`src/schemas/cli-schemas.ts`](../../src/schemas/cli-schemas.ts). + +### `single` + +`single` uses [`SingleModelCallExecutor`](../../src/executors/single-model-call-executor.ts). It performs one structured model call per rule/chunk through `StructuredModelClient`. For targets above 600 words, it chunks line-numbered content and merges violations before shared finding processing. + +The single executor owns no tool surface. + +### `agent` + +`agent` uses [`AgentModelCallExecutor`](../../src/executors/agent-model-call-executor.ts). It performs one bounded tool-calling run per rule through `ToolCallingModelClient`. + +The only executor-owned tool is `read_target_section`, defined in [`src/executors/target-read-capability-adapter.ts`](../../src/executors/target-read-capability-adapter.ts). It reads 1-based line ranges from in-memory `request.target.content` and returns model-visible errors for invalid ranges. + +The agent model call cannot: + +- search the workspace; +- read arbitrary files; +- read URIs outside the target content; +- rewrite rules; +- create top-level workspace findings; +- override source-backed rule prompts. + +### `auto` + +`auto` resolves through `chooseModelCall` in [`src/review/executor.ts`](../../src/review/executor.ts). It selects `agent` when target content is larger than `AGENT_MODEL_CALL_BYTE_THRESHOLD` (`600_000` bytes) or when more than five rules apply. Otherwise, it selects `single`. + +The CLI default is `auto`. + +## On-Page Boundary + +The on-page boundary is implemented in [`src/review/boundary.ts`](../../src/review/boundary.ts) and enforced by executor design. + +Target content is always in scope. Caller-supplied review context is in scope only because the caller explicitly provided it. Workspace files are out of scope unless the caller passes their content as review context. Rule bodies are source-backed and loaded before the review request is built. + +The target-read adapter performs no filesystem reads. It slices the target content already present in memory. + +## Budgets + +Default budgets live in [`src/review/budget.ts`](../../src/review/budget.ts): + +| Field | Default | Meaning | +| --- | ---: | --- | +| `maxTargetBytes` | `1_000_000` | maximum target content size | +| `maxCallerContextBytes` | `500_000` | maximum caller-supplied context size | +| `maxChunksPerRule` | `20` | maximum chunks/tool steps per rule | +| `maxModelCallsPerReview` | `50` | maximum model calls in one review | +| `maxWallClockMs` | `300_000` | maximum elapsed review time, in milliseconds | + +Budgets limit work, not output. There is intentionally no `maxFindingsPerRule`. There is also no headless retry budget because VectorLint no longer has a headless autonomous executor. + +`maxWallClockMs` is the review timeout. Executors check model-call count and elapsed time before model calls and surface budget exhaustion as diagnostics when partial results can be returned. + +## Finding Processing + +Both model-call strategies use the same finding-processing pipeline in [`src/findings/`](../../src/findings/): + +1. Filter candidate findings through the evidence gate. +2. Verify finding evidence against target content. +3. Deduplicate verified findings. +4. Score by verified finding count and density. +5. Resolve severity. +6. Assemble findings, scores, diagnostics, and operational status. + +Unlocatable quoted evidence becomes a `finding-evidence-not-locatable` diagnostic and emits no finding. VectorLint does not fall back to model-provided line numbers for unverified evidence. + +Diagnostics describe operational or finding-processing conditions such as unlocatable evidence, budget exhaustion, schema parse failures, and provider failures. They are not content findings. + +## Provider And Model Capabilities + +Providers are transport capabilities, not product owners. + +[`StructuredModelClient`](../../src/providers/structured-model-client.ts) performs one structured model call and returns validated output. + +[`ToolCallingModelClient`](../../src/providers/tool-calling-model-client.ts) performs one bounded tool-calling generation using caller-supplied tools and returns structured output. The provider does not define product tools. The executor supplies the tool map and run bounds. + +Neither provider capability is an autonomous agent loop. + +## Architecture Diagram + +```text +CLI / External Caller + | + v +ReviewRequest + - target + - source-backed rules + - caller-supplied context + - budget + - output policy + - modelCall + | + v +chooseModelCall(auto?) ──> single ──> SingleModelCallExecutor + └─> agent ──> AgentModelCallExecutor + | + v + read_target_section + target content only + | + v +Shared Finding Processing + - filter + - verify evidence + - dedupe + - score + - diagnostics + | + v +ReviewResult + | + v +Formatters + - line + - json + - vale-json + - rdjson +``` + +## Removal Notes + +Use `--model-call`, not `--mode agent`. The CLI still rejects `--mode` with a clear migration error so old integrations fail loudly. + +Autonomous workspace-agent behavior is removed. If an integration needs project-wide exploration, it should do that before invoking VectorLint and pass the selected target content and review context explicitly. + +Subjective `judge`/rubric rules are removed from the future-facing architecture rather than migrated. New rules should be written as objective Via Negativa checks with observable violation conditions. + +## References + +- [`docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md`](../audits/2026-07-10-vectorlint-harness-architecture-audit.md) +- [`src/review/types.ts`](../../src/review/types.ts) +- [`src/review/executor.ts`](../../src/review/executor.ts) +- [`src/review/budget.ts`](../../src/review/budget.ts) +- [`src/review/boundary.ts`](../../src/review/boundary.ts) +- [`src/executors/single-model-call-executor.ts`](../../src/executors/single-model-call-executor.ts) +- [`src/executors/agent-model-call-executor.ts`](../../src/executors/agent-model-call-executor.ts) +- [`src/executors/target-read-capability-adapter.ts`](../../src/executors/target-read-capability-adapter.ts) +- [`src/findings/processor.ts`](../../src/findings/processor.ts) +- [`src/providers/structured-model-client.ts`](../../src/providers/structured-model-client.ts) +- [`src/providers/tool-calling-model-client.ts`](../../src/providers/tool-calling-model-client.ts) diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index 677502d..9b30d49 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -156,6 +156,24 @@ Without a search provider, `technical-accuracy` evaluators cannot verify facts a --- +### `--mode agent` no longer works + +**Symptom:** VectorLint exits with an error that `--mode` is no longer supported. + +**Cause:** Autonomous agent mode was removed. VectorLint now uses `--model-call` to choose how the reviewer model is invoked. + +**Fix:** Replace the removed flag with one of the supported model-call values: + +```bash +vectorlint doc.md --model-call single +vectorlint doc.md --model-call agent +vectorlint doc.md --model-call auto +``` + +Use `agent` only when target-section paging helps with large or context-heavy content. It is bounded to the current target and cannot search the workspace. See [Migration from agent mode](/migration-from-agent-mode) for integration guidance. + +--- + ## CI issues ### CI pipeline fails immediately with no findings output diff --git a/docs/use-cases.mdx b/docs/use-cases.mdx index 84bddf7..1d7f7d9 100644 --- a/docs/use-cases.mdx +++ b/docs/use-cases.mdx @@ -11,7 +11,7 @@ VectorLint fits anywhere a team publishes written content at scale and needs con **The path:** Write your standards as plain-language instructions in `VECTORLINT.md`. VectorLint converts them into a "Style Guide Compliance" rule and runs it against every file, every time — no regex, no word lists, no manual review for comma usage or passive voice. -If your team has a detailed guide, see [Defining your style rules](/style-guide) for an extraction prompt that converts it into a `VECTORLINT.md`-optimized file. For more targeted enforcement, write rule pack files that score specific criteria on a weighted rubric. +If your team has a detailed guide, see [Defining your style rules](/style-guide) for an extraction prompt that converts it into a `VECTORLINT.md`-optimized file. For more targeted enforcement, write rule pack files that describe specific, observable violation conditions. **Who does this:** Technical writing teams, content operations teams, developer advocacy teams. @@ -88,4 +88,4 @@ See [Configuring a project](/project-config) for the full configuration referenc - [Quickstart](/quickstart) — run your first evaluation in under ten minutes - [How it works](/how-it-works) — understand the evaluation pipeline, PAT filtering, and scoring -- [Defining your style rules](/style-guide) — create VECTORLINT.md and write custom rules \ No newline at end of file +- [Defining your style rules](/style-guide) — create VECTORLINT.md and write custom rules From 7e32601eb4e71d5dac25c4ee3722b88e164272dc Mon Sep 17 00:00:00 2001 From: Osho Emmanuel Date: Tue, 14 Jul 2026 14:45:11 +0100 Subject: [PATCH 2/4] docs: reframe internal agent-mode cleanup - Remove public migration and deprecation framing for unreleased agent-mode paths. - Drop the migration page, navigation entry, and troubleshooting guidance that implied external users. - Reword historical audit and architecture notes around internal implementation cleanup. --- ...0-vectorlint-harness-architecture-audit.md | 8 +-- docs/docs.json | 1 - ...3-31-agent-mode-implementation-plan.log.md | 4 +- docs/migration-from-agent-mode.mdx | 51 ------------------- docs/model-calls.mdx | 1 - docs/rubric-scoring.mdx | 6 +-- docs/specs/2026-07-10-harness-architecture.md | 10 ++-- docs/troubleshooting.mdx | 18 ------- 8 files changed, 14 insertions(+), 85 deletions(-) delete mode 100644 docs/migration-from-agent-mode.mdx diff --git a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md index 356e150..ab661ec 100644 --- a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md +++ b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md @@ -1,8 +1,10 @@ # VectorLint Harness Architecture Audit > **HISTORICAL AUDIT.** This audit records the July 2026 decision that led to -> the bounded harness refactor. Some "current state" observations below -> describe the pre-refactor code. For the shipped architecture, see +> the bounded harness refactor. Agent mode was an unreleased internal +> implementation path with no public users; this audit records the decision to +> remove it before shipping it as a public contract. Some "current state" +> observations below describe the pre-refactor code. For the shipped architecture, see > [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md). Date: 2026-07-10 @@ -380,6 +382,6 @@ Phase 1 of the recommended refactor sequence ("Stop The Bleeding") landed on bra - `docs/research/**` excluded from linting. - `.github/workflows/typecheck.yml` and `.github/workflows/test.yml` aligned to Node 20. -**Current state.** `npm run verify`, `npm run build`, and built-CLI standard review smoke runs are green. Per Finding 1, the autonomous workspace-agent surface is deprecated at the CLI boundary: `--mode agent` emits a deprecation warning and falls back to standard evaluation. `src/agent/*` and the agent-mode helpers are retained as compile-only quarantine, unreachable from the CLI, pending removal in Phase 4. The deprecation notice and pointer to this audit live in the README `## Agent Mode` section and the `--mode` help text. +**Current state at audit time.** `npm run verify`, `npm run build`, and built-CLI standard review smoke runs are green. Per Finding 1, the unreleased autonomous workspace-agent surface is quarantined at the CLI boundary: `--mode` is blocked so internal agent-mode wiring cannot be reached from the CLI. `src/agent/*` and the agent-mode helpers are retained as compile-only quarantine, unreachable from the CLI, pending removal in Phase 4. A pointer to this audit lives in the README `## Agent Mode` section and the `--mode` help text. This appendix records completion only. Findings 2–9 and the proposed core contract remain owned by Phases 2–5; no spec or architecture document is superseded here. diff --git a/docs/docs.json b/docs/docs.json index f8ee7c1..d88e89c 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -72,7 +72,6 @@ "configuration-schema", "env-variables", "frontmatter-fields", - "migration-from-agent-mode", "troubleshooting" ] } diff --git a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md b/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md index 641d780..233c16b 100644 --- a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md +++ b/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md @@ -4,9 +4,7 @@ > agent-mode implementation. VectorLint is now a bounded review harness with > `single`/`agent`/`auto` model calls. See > [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md) -> for the current architecture and -> [`../migration-from-agent-mode.mdx`](../migration-from-agent-mode.mdx) -> for migration guidance. The details below are historical only. +> for the current architecture. The details below are historical only. - **Plan**: `docs/plans/2026-03-31-agent-mode-implementation-plan.md` - **Issue**: Not provided (executed from user directive in this session) diff --git a/docs/migration-from-agent-mode.mdx b/docs/migration-from-agent-mode.mdx deleted file mode 100644 index 4011c5f..0000000 --- a/docs/migration-from-agent-mode.mdx +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Migration from agent mode -description: Move integrations from the removed autonomous agent mode to bounded model calls. ---- - -VectorLint no longer supports autonomous workspace-agent mode. The current product is a bounded review harness: callers supply the target and any review context, and VectorLint reviews that supplied content against source-backed rules. - -## What changed - -- `--mode agent` was removed. Use `--model-call single`, `--model-call agent`, or `--model-call auto`. -- Workspace exploration moved out of VectorLint. External agents or scripts gather context before invoking VectorLint. -- Provider-level agent loops were removed. Providers expose structured-output and bounded tool-calling capabilities. -- Model-authored review instructions were removed. Rules remain source-backed. - -## Mapping - -| Before | After | -| --- | --- | -| `vectorlint doc.md --mode agent` | `vectorlint doc.md --model-call agent` for target paging, or `--model-call single` for structured calls | -| `--mode agent --output json` | `--model-call agent --output json` | -| `report_finding` for workspace findings | Move workspace-level findings to the caller; VectorLint reports target findings only | -| `search_content` or `read_file` workspace tools | Caller gathers context and passes it as review context | -| Model-supplied `reviewInstruction` | Edit the source-backed rule file | -| `runAgentToolLoop` on a provider | Implement `ReviewExecutor`, or use `StructuredModelClient` / `ToolCallingModelClient` behind an executor | - -## Why it changed - -The old direction mixed two ownership models: VectorLint as a linter and VectorLint as an autonomous workspace agent. That duplicated result contracts, weakened scope boundaries, and made findings less consistent across output formats. - -The harness model keeps responsibilities clear. The caller owns exploration. VectorLint owns bounded review. - -## What you get - -- Explicit budgets for target size, review context size, chunks per rule, model calls, and elapsed time. -- One finding-processing path for filtering, evidence verification, scoring, diagnostics, and formatting. -- The same result shape across `single`, `agent`, and `auto`. -- A stable contract external agents can call without granting VectorLint workspace exploration authority. - -## Migration steps - -1. Replace `--mode agent` with `--model-call agent` only when target paging is still useful. -2. Use `--model-call single` for normal reviews that do not need target-section paging. -3. Move workspace search, file selection, and context gathering into the caller. -4. Pass any needed context explicitly as review context when calling the review contract. -5. Rewrite subjective rubric rules as objective Via Negativa violation checks. - -## Related - -- [Model calls](/model-calls) -- [How it works](/how-it-works) -- [CLI reference](/cli-reference) diff --git a/docs/model-calls.mdx b/docs/model-calls.mdx index dbe34c6..9f52a6b 100644 --- a/docs/model-calls.mdx +++ b/docs/model-calls.mdx @@ -72,4 +72,3 @@ vectorlint doc.md --model-call agent --output json - [How it works](/how-it-works) - [CLI reference](/cli-reference) -- [Migration from agent mode](/migration-from-agent-mode) diff --git a/docs/rubric-scoring.mdx b/docs/rubric-scoring.mdx index 0e03ce1..e6cb0d9 100644 --- a/docs/rubric-scoring.mdx +++ b/docs/rubric-scoring.mdx @@ -12,10 +12,10 @@ Write rules as Via Negativa checks. Describe what counts as a violation, require ```markdown Flag a section when it: 1. Makes a factual claim without a source -2. Uses a deprecated command name +2. Uses an obsolete command name 3. Repeats the same setup step already shown earlier ``` -## Migration +## Updating old rules -If you have old rubric-style rules, rewrite each criterion as one or more observable violation conditions. See [Migration from agent mode](/migration-from-agent-mode) for related removed-surface guidance and [Defining your style rules](/style-guide) for current rule syntax. +If you have old rubric-style rules, rewrite each criterion as one or more observable violation conditions. See [Defining your style rules](/style-guide) for current rule syntax. diff --git a/docs/specs/2026-07-10-harness-architecture.md b/docs/specs/2026-07-10-harness-architecture.md index 5498b9e..52125a3 100644 --- a/docs/specs/2026-07-10-harness-architecture.md +++ b/docs/specs/2026-07-10-harness-architecture.md @@ -53,7 +53,7 @@ Good rules answer yes/no questions: - Does this claim omit required evidence? - Does this section repeat information already stated nearby? -Future-facing rule authors should avoid subjective judge/rubric designs. Historical rubric-style language remains only where it describes removed behavior or migration guidance. +Future-facing rule authors should avoid subjective judge/rubric designs. Historical rubric-style language remains only where it describes removed behavior. ## Model Calls @@ -176,13 +176,13 @@ Formatters - rdjson ``` -## Removal Notes +## Internal Implementation Notes -Use `--model-call`, not `--mode agent`. The CLI still rejects `--mode` with a clear migration error so old integrations fail loudly. +`--model-call` is the documented CLI surface. The internal `--mode` flag is rejected at the CLI boundary so unreleased agent-mode wiring cannot be used accidentally. -Autonomous workspace-agent behavior is removed. If an integration needs project-wide exploration, it should do that before invoking VectorLint and pass the selected target content and review context explicitly. +The unreleased autonomous workspace implementation path is removed. External callers that need project-wide exploration should gather context before invoking VectorLint and pass selected content explicitly. -Subjective `judge`/rubric rules are removed from the future-facing architecture rather than migrated. New rules should be written as objective Via Negativa checks with observable violation conditions. +Subjective `judge`/rubric rules are not part of the future-facing architecture. New rules should be written as objective Via Negativa checks with observable violation conditions. ## References diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index 9b30d49..677502d 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -156,24 +156,6 @@ Without a search provider, `technical-accuracy` evaluators cannot verify facts a --- -### `--mode agent` no longer works - -**Symptom:** VectorLint exits with an error that `--mode` is no longer supported. - -**Cause:** Autonomous agent mode was removed. VectorLint now uses `--model-call` to choose how the reviewer model is invoked. - -**Fix:** Replace the removed flag with one of the supported model-call values: - -```bash -vectorlint doc.md --model-call single -vectorlint doc.md --model-call agent -vectorlint doc.md --model-call auto -``` - -Use `agent` only when target-section paging helps with large or context-heavy content. It is bounded to the current target and cannot search the workspace. See [Migration from agent mode](/migration-from-agent-mode) for integration guidance. - ---- - ## CI issues ### CI pipeline fails immediately with no findings output From 6dde8f597cb53bf0fdf3cd4ba564483365e68bcc Mon Sep 17 00:00:00 2001 From: Osho Emmanuel Date: Tue, 14 Jul 2026 14:47:59 +0100 Subject: [PATCH 3/4] docs: clarify historical audit framing - Replace remaining public transition wording in the historical audit with internal removal language. - Keep the audit historical while aligning it to unreleased agent-mode scope. --- .../2026-07-10-vectorlint-harness-architecture-audit.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md index ab661ec..205e0c6 100644 --- a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md +++ b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md @@ -261,7 +261,7 @@ Future contributors will optimize toward the wrong architecture and trust contra Recommendation: -Mark the agentic spec superseded. Replace it with a harness architecture spec that defines the review request, context boundary, executor interface, output contract, and migration plan. +Mark the agentic spec superseded. Replace it with a harness architecture spec that defines the review request, context boundary, executor interface, output contract, and internal removal plan for unreleased agent-mode paths. ### 9. Secrets And Sensitive Content Need Better Handling @@ -338,7 +338,7 @@ Results: 1. Mark the agentic capabilities spec superseded. 2. Write a new harness architecture spec. 3. Update README and CLI docs. -4. Document migration from autonomous workspace-agent mode to direct/reader/auto execution. +4. Document the bounded harness architecture that replaces the unreleased autonomous workspace-agent implementation path. ## Suggested Target Architecture From ab9b8e1215ff003cb66fd8204a51fb06fcd78a11 Mon Sep 17 00:00:00 2001 From: Osho Emmanuel Date: Tue, 14 Jul 2026 14:54:17 +0100 Subject: [PATCH 4/4] docs: enforce documentation artifact boundaries - Remove tracked audit, run-note, and spec artifacts from product docs. - Replace product-doc links to the removed spec with current usage docs. - Add AGENTS guidance that keeps coordination artifacts in ignored workspace state. --- AGENTS.md | 8 + README.md | 4 +- ...0-vectorlint-harness-architecture-audit.md | 387 ------------------ docs/how-it-works.mdx | 4 - ...3-31-agent-mode-implementation-plan.log.md | 28 -- docs/specs/2026-07-10-harness-architecture.md | 199 --------- 6 files changed, 9 insertions(+), 621 deletions(-) delete mode 100644 docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md delete mode 100644 docs/logs/2026-03-31-agent-mode-implementation-plan.log.md delete mode 100644 docs/specs/2026-07-10-harness-architecture.md diff --git a/AGENTS.md b/AGENTS.md index 0604a5b..9f685e0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,6 +42,14 @@ These guidelines reduce common LLM coding mistakes. They bias toward caution ove - For multi-step tasks, state a brief plan with verification for each step. - Loop until the success criteria are verified or a blocker is clear. +### Documentation Artifact Boundaries + +- Do not commit raw planning or investigation artifacts to the product codebase. +- Keep audits, plans, specs, run notes, and similar coordination artifacts out of tracked repo paths such as `docs/audits/`, `docs/plans/`, `docs/specs/`, `audits/`, `plans/`, and `specs/`. +- Store coordination artifacts in `.agent-runs/` or another ignored workspace location. +- If a durable architectural decision must be committed, write it as an ADR. ADRs are the only allowed committed decision/planning artifacts. +- Product documentation may describe shipped behavior, configuration, and usage, but it must not preserve internal audit/plan/spec documents as reviewed product docs. + ## Project Structure & Module Organization - `src/` diff --git a/README.md b/README.md index 989399b..113378f 100644 --- a/README.md +++ b/README.md @@ -167,9 +167,7 @@ vectorlint doc.md --model-call auto - `auto` is the default. VectorLint chooses `single` for normal inputs and `agent` for large or multi-rule reviews that benefit from target paging. -See the current -[`docs/specs/2026-07-10-harness-architecture.md`](docs/specs/2026-07-10-harness-architecture.md) -for the contract and boundary details. +See [Model calls](docs/model-calls.mdx) for more details. ## Contributing diff --git a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md b/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md deleted file mode 100644 index 205e0c6..0000000 --- a/docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md +++ /dev/null @@ -1,387 +0,0 @@ -# VectorLint Harness Architecture Audit - -> **HISTORICAL AUDIT.** This audit records the July 2026 decision that led to -> the bounded harness refactor. Agent mode was an unreleased internal -> implementation path with no public users; this audit records the decision to -> remove it before shipping it as a public contract. Some "current state" -> observations below describe the pre-refactor code. For the shipped architecture, see -> [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md). - -Date: 2026-07-10 - -Status: Objective system audit plus product-direction audit. Request changes before shipping the current autonomous workspace-agent surface as a public contract. - -## Product Decision - -VectorLint should not keep the current autonomous workspace-agent mode. - -The retained agent-like capability should be a constrained reader executor: a model or headless agent can read the target content in sections when that is useful for context management, but it should not search the workspace, inspect arbitrary files, perform top-level workspace checks, or rewrite the rule being evaluated. - -The intended execution model is: - -- `direct`: send target content and rule in one structured review call. -- `reader`: give the executor a read-section capability over the target content only. -- `auto`: choose `direct` for normal-sized inputs and `reader` for large inputs or explicit context-management needs. - -This is an execution strategy decision, not a return to VectorLint-as-agent. - -## Product Direction Under Review - -VectorLint should become a programmable, bounded, on-page content review harness that another caller can invoke. - -The caller can be Codex, Claude Code, another coding agent, CI, a local CLI wrapper, or a future service. That caller owns exploration, cross-page reasoning, research, and context gathering. VectorLint owns the constrained review of target content against source-backed rules and returns structured findings, scores, diagnostics, and usage metadata. - -In practical terms: - -- VectorLint reviews the target page or explicitly supplied content. -- External context is caller-supplied, not discovered by VectorLint. -- Rules define review behavior, granularity, severity, and scoring constraints. -- Executors can be API models, headless local agents, or constrained reader executors, but they receive the same review request contract. -- VectorLint should not expose a model-controlled workspace tool loop as its core product surface. - -## Objective System Audit Summary - -Separate from product direction, the current codebase has several system-level liabilities: - -- Build health is weak: `npx tsc --noEmit`, `npm run lint`, and parts of `npm run test:run` fail in the current workspace. -- Runtime contracts are duplicated across hand-written TypeScript, JSON-schema objects, Zod schemas, and formatter-specific shapes. -- Standard mode and agent mode project the same underlying findings differently. -- CLI orchestration owns too many responsibilities: config, file matching, evaluation, scoring, output formatting, debug artifacts, and agent routing. -- Evidence verification and counting are inconsistent enough to affect exit behavior. -- Observability and debug paths can record full prompts, content, and outputs. -- Documentation and implementation disagree on the current agent-mode topology and tool contracts. - -## Architecture Verdict - -The current codebase contains two execution models at once: - -1. The older evaluator path: one structured model call per rule or chunk. -2. The newer autonomous workspace-agent path: VectorLint gives a model tools to read/search the workspace and call lint as a nested tool. - -The second path should not continue as-is. The useful piece is not "agent mode" broadly; it is target-aware reading for context management. The current implementation also duplicates result projection, scoring, evidence validation, output formatting, and configuration behavior from the first path. That is why the architecture feels harder to maintain: the code is not just messy; it is serving two different ownership models. - -The highest-leverage change is to define a neutral review-domain contract and make every executor implement that contract. - -## Proposed Core Contract - -The contract should be centered on review, not providers or agents. - -```ts -interface ReviewRequest { - target: ReviewTarget; - rules: ReviewRule[]; - context?: ReviewContext[]; - budget: ReviewBudget; - outputPolicy: ReviewOutputPolicy; -} - -interface ReviewExecutor { - run(request: ReviewRequest): Promise; -} - -interface ReviewResult { - findings: ReviewFinding[]; - scores: ReviewScore[]; - diagnostics: ReviewDiagnostic[]; - usage?: ReviewUsage; -} -``` - -API models, reasoning models, and headless CLI agents become adapter implementations behind this contract. The CLI becomes orchestration around request creation, result formatting, and exit behavior. - -## Major Findings - -### 1. The Current Agent Surface Is Broader Than The Decided Execution Model - -Evidence: - -- `README.md:148` presents `--mode agent` as autonomous cross-file review mode. -- `src/agent/prompt-builder.ts:43` instructs the model to perform workspace-level checks. -- `src/agent/executor.ts:670` exposes `read_file`. -- `src/agent/executor.ts:715` exposes `search_content`. - -Impact: - -VectorLint still behaves like it owns exploration. That puts it in competition with external coding agents instead of becoming a harness those agents can use. It also overshoots the decided reader-executor model, which only needs target-scoped reading for context management. - -Recommendation: - -Remove the current autonomous workspace-agent surface. Preserve only a constrained reader executor if needed: - -- It may read sections of the target content. -- It may not search the workspace. -- It may not read arbitrary files. -- It may not perform top-level workspace findings. -- It may not override source-backed rules. - -### 2. The Provider Interface Mixes Structured Review With Autonomous Tool Loops - -Evidence: - -- `src/providers/llm-provider.ts:28` requires both `runPromptStructured` and `runAgentToolLoop`. -- `src/providers/vercel-ai-provider.ts:136` implements the agent loop on the same provider. -- `src/agent/executor.ts:796` depends on the provider-level agent loop. - -Impact: - -Every future executor has to pretend to be both a structured model provider and an autonomous agent runner. That is the wrong abstraction for headless CLI support and for a constrained reader executor. - -Recommendation: - -Split the contracts: - -- `StructuredModelClient` for API structured output. -- `HeadlessReviewExecutor` for local CLI agents. -- `ReaderReviewExecutor` for target-scoped read-section execution. -- `ReviewExecutor` as the stable domain-level interface. -- `TelemetrySink` or observability decoration as an optional cross-cutting concern. - -### 3. Runtime Type Contracts Are Not Strong Enough For A Harness - -Evidence: - -- `src/providers/vercel-ai-provider.ts:119` returns `output as T`. -- `src/providers/llm-provider.ts:9` defines tool input and output as `unknown`. -- `src/agent/tools-registry.ts:21` exposes tools without typed output contracts. -- `npx tsc --noEmit` currently fails in core files. - -Impact: - -The code presents types as guarantees, but several are only assertions. A programmable harness needs reliable runtime validation because external callers and headless adapters will depend on these contracts. - -Recommendation: - -Tie structured execution to concrete Zod schemas or canonical parsers. Introduce typed tool contracts only if tools remain. Add a `typecheck` script and wire it into CI and build. - -### 4. Standard Mode And Agent Mode Project Results Differently - -Evidence: - -- `src/cli/orchestrator.ts:626` derives standard check severity from scoring. -- `src/agent/executor.ts:431` stamps agent findings with prompt severity. -- `src/agent/executor.ts:598` flattens judge criteria before recording findings. -- `src/cli/orchestrator.ts:1216` builds agent scores only for line output. -- `src/cli/orchestrator.ts:1237` emits structured output without agent scores or diagnostics. - -Impact: - -The same underlying review can produce different severity, rule identity, score data, JSON shape, and exit behavior depending on mode. That makes VectorLint hard to trust as a machine-facing gate. - -Recommendation: - -Extract one shared result projection pipeline: - -```text -PromptEvaluationResult - -> verified findings - -> rule and criterion identity - -> severity and score - -> diagnostics - -> output formatter -``` - -Every executor should return into this pipeline. - -### 5. The On-Page Boundary Is Not Enforced - -Evidence: - -- `src/agent/executor.ts:585` lets `lint` resolve any file inside `workspaceRoot`. -- `src/agent/executor.ts:670` lets `read_file` read any workspace file. -- `src/agent/executor.ts:715` lets `search_content` scan workspace content. -- `src/agent/executor.ts:134` allows model-supplied `reviewInstruction`. -- `src/agent/executor.ts:583` replaces the source-backed prompt body with the model-supplied override. - -Impact: - -Prompt-injected page content can indirectly steer the model to read non-target files or rewrite the rule being evaluated. That breaks deterministic review semantics and creates privacy and security risk. - -Recommendation: - -For the new harness, enforce a target/context allowlist: - -- Target content is always in scope. -- Caller-supplied context is in scope. -- Workspace reads are out of scope unless the caller explicitly includes those files as context. -- Reader execution can page through target content only. -- Rule bodies are source-backed and caller-authored, not model-authored. - -### 6. Evidence Handling Can Misreport Findings - -Evidence: - -- Standard mode skips unverifiable quotes but can still count original surfaced violation totals in some paths. -- `src/agent/executor.ts:415` attempts to locate evidence. -- `src/agent/executor.ts:426` falls back to the model-provided line when location fails. - -Impact: - -Standard mode can fail a run without printing the issue that caused the failure. Agent mode can surface hallucinated evidence as if it were verified. - -Recommendation: - -Use one evidence verifier. Count only emitted, verified findings. If evidence cannot be located, return a diagnostic and either skip the finding or mark the run operationally failed. - -### 7. Cost And Work Are Not Bounded Enough - -Evidence: - -- `src/providers/vercel-ai-provider.ts:161` defaults agent loops to 1000 steps. -- `src/agent/executor.ts:577` allows repeated nested `lint` calls. -- `src/evaluators/base-evaluator.ts:213` runs one model call per chunk. -- `src/agent/executor.ts:680` and `src/agent/executor.ts:715` glob before applying result caps. - -Impact: - -A single review can multiply cost through agent steps, nested lint calls, rules, chunks, and workspace search. Even a constrained reader executor will be slower than a direct call, so `auto` must have clear thresholds and budgets. - -Recommendation: - -Add explicit budgets: - -- Max target bytes. -- Max caller context bytes. -- Max chunks per rule. -- Max model calls per review. -- Max findings per rule. -- Max wall-clock duration. -- Max headless executor retries. - -### 8. Documentation Still Points At The Old Product - -Evidence: - -- `README.md:148` documents autonomous workspace-agent mode. -- `docs/specs/2026-03-17-agentic-capabilities-design.md:24` frames the roadmap around cross-document agent mode. -- The same spec describes implementation details that are not true now: one agent per rule, final `AgentFindingSchema`, AbortSignal propagation, ripgrep JSON, and Vale JSON omission of agent findings. - -Impact: - -Future contributors will optimize toward the wrong architecture and trust contracts that the code does not implement. - -Recommendation: - -Mark the agentic spec superseded. Replace it with a harness architecture spec that defines the review request, context boundary, executor interface, output contract, and internal removal plan for unreleased agent-mode paths. - -### 9. Secrets And Sensitive Content Need Better Handling - -Evidence: - -- `src/config/global-config.ts:13` says the global config stores API keys. -- `src/config/global-config.ts:76` creates the config directory with default permissions. -- `src/config/global-config.ts:81` writes the config file with default permissions. -- `src/observability/langfuse-observability.ts:66` records inputs. -- `src/observability/langfuse-observability.ts:67` records outputs. -- `src/providers/vercel-ai-provider.ts:59` can log full prompts and content. - -Impact: - -Provider keys, unreleased docs, caller context, and model outputs can leak through filesystem permissions, telemetry, debug logs, or persisted artifacts. - -Recommendation: - -Use private file modes for config and review artifacts. Make payload telemetry a separate opt-in from metadata telemetry. Add redaction or safe debug modes. - -## Verification Results - -Commands run from `/Users/klinsmann/Projects/TinyRocketLabs/vectorlint`: - -```bash -npm run lint -npm run test:run -npx tsc --noEmit -``` - -Results: - -- `npm run lint` failed before linting because typed ESLint rules were applied to a `.cjs` research script without parser type information. -- `npm run test:run` passed most tests but failed 4 suites during module resolution for `ora` and `@langfuse/otel`. -- `npx tsc --noEmit` failed with contract errors in agent, orchestrator, observability, logging, and provider modules. - -## Recommended Refactor Sequence - -### Phase 1: Stop The Bleeding - -1. Remove or hide the current autonomous workspace-agent mode. -2. Fix `tsc --noEmit`. -3. Fix test module resolution. -4. Fix lint configuration for non-TypeScript research scripts. -5. Add `typecheck` to CI and build validation. - -### Phase 2: Define The Harness Contract - -1. Create a neutral review-domain module. -2. Define `ReviewRequest`, `ReviewRule`, `ReviewContext`, `ReviewBudget`, `ReviewFinding`, `ReviewScore`, and `ReviewDiagnostic`. -3. Define target/context boundary rules. -4. Define structured output shape once. -5. Define execution strategy: `direct`, `reader`, and `auto`. - -### Phase 3: Share Result Projection - -1. Extract evidence location and verification. -2. Extract filtering. -3. Extract severity and scoring. -4. Extract output formatter inputs. -5. Make standard mode and any executor mode use the same projection path. - -### Phase 4: Replace The Agent Loop With Executors - -1. Keep API model execution as direct execution. -2. Add target-scoped reader execution as another executor. -3. Add headless CLI execution behind the same review contract if still useful. -4. Pass the same review request to each executor. -5. Remove model-controlled rule overrides. -6. Remove workspace read/search tools from core review. - -### Phase 5: Update Documentation - -1. Mark the agentic capabilities spec superseded. -2. Write a new harness architecture spec. -3. Update README and CLI docs. -4. Document the bounded harness architecture that replaces the unreleased autonomous workspace-agent implementation path. - -## Suggested Target Architecture - -```text -CLI / API / External Agent - -> ReviewRequestBuilder - -> ReviewExecutor - -> ApiModelExecutor - -> ReaderExecutor - -> HeadlessCliExecutor - -> ResultProjection - -> EvidenceVerifier - -> ViolationFilter - -> ScoreCalculator - -> Diagnostics - -> OutputFormatter - -> line - -> json - -> rdjson - -> vale-json -``` - -The important inversion is that VectorLint no longer asks, "What tools should this agent use?" It asks, "Given this target, these rules, this context, this execution strategy, and this budget, what findings can be verified?" - -## Final Recommendation - -The decided direction is cleaner than the current architecture. The current pain is not a sign that VectorLint is too complex; it is a sign that the codebase is carrying two ownership models and several duplicated runtime contracts. - -Make VectorLint the harness. Let external agents be agents. Keep reader execution only as a bounded way to manage target-content context. - ---- - -## Appendix: Phase 1 — Completed (2026-07-13) - -Phase 1 of the recommended refactor sequence ("Stop The Bleeding") landed on branch `codex/ci/harness-stop-the-bleeding`. It shifts the verification baseline recorded in "Verification Results" above and addresses Finding 1 at the CLI boundary. - -**Shifted baseline.** The 2026-07-13 pre-work capture differed from the original verification results: `npm run lint` and `npm run test:run` already passed, so the earlier lint failure and the four `ora` / `@langfuse/otel` module-resolution failures were stale. Only `npx tsc --noEmit` was still failing. Phase 1 cleared those remaining type errors (narrowed at boundaries; no strict compiler options relaxed) and added durable gates so the baseline cannot silently regress: - -- `npm run typecheck` (`tsc --noEmit`) and `npm run verify` (typecheck + lint + test:run) scripts. -- `vitest.config.ts` for stable test module resolution. -- `docs/research/**` excluded from linting. -- `.github/workflows/typecheck.yml` and `.github/workflows/test.yml` aligned to Node 20. - -**Current state at audit time.** `npm run verify`, `npm run build`, and built-CLI standard review smoke runs are green. Per Finding 1, the unreleased autonomous workspace-agent surface is quarantined at the CLI boundary: `--mode` is blocked so internal agent-mode wiring cannot be reached from the CLI. `src/agent/*` and the agent-mode helpers are retained as compile-only quarantine, unreachable from the CLI, pending removal in Phase 4. A pointer to this audit lives in the README `## Agent Mode` section and the `--mode` help text. - -This appendix records completion only. Findings 2–9 and the proposed core contract remain owned by Phases 2–5; no spec or architecture document is superseded here. diff --git a/docs/how-it-works.mdx b/docs/how-it-works.mdx index d587087..9bcc1ac 100644 --- a/docs/how-it-works.mdx +++ b/docs/how-it-works.mdx @@ -72,10 +72,6 @@ VectorLint reviews only the target content and explicit review context supplied This makes VectorLint safe to call from external agents and CI systems: the caller owns exploration, and VectorLint owns constrained review. -## More detail - -Read the current [harness architecture spec](https://github.com/TRocket-Labs/vectorlint/blob/main/docs/specs/2026-07-10-harness-architecture.md) for the code-level contract and removed surfaces. - ## Next steps - [Model calls](/model-calls) — choose `single`, `agent`, or `auto` diff --git a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md b/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md deleted file mode 100644 index 233c16b..0000000 --- a/docs/logs/2026-03-31-agent-mode-implementation-plan.log.md +++ /dev/null @@ -1,28 +0,0 @@ -# Execution Log - -> **SUPERSEDED (2026-07-10).** This log describes the removed autonomous -> agent-mode implementation. VectorLint is now a bounded review harness with -> `single`/`agent`/`auto` model calls. See -> [`../specs/2026-07-10-harness-architecture.md`](../specs/2026-07-10-harness-architecture.md) -> for the current architecture. The details below are historical only. - -- **Plan**: `docs/plans/2026-03-31-agent-mode-implementation-plan.md` -- **Issue**: Not provided (executed from user directive in this session) -- **Started**: 2026-03-31 -- **Status**: completed - ---- - -## Tasks - -### Task: Implement agent mode runtime, wiring, and contracts from red tests -- **Status**: completed -- **What was done**: Implemented the provider agent-loop contract, built the new agent runtime modules (types, session store, path safety, progress, executor), wired CLI/orchestrator `--mode agent` and `--print`, and updated README agent-mode documentation. -- **Files changed**: `src/providers/llm-provider.ts`, `src/providers/vercel-ai-provider.ts`, `src/agent/types.ts`, `src/agent/review-session-store.ts`, `src/agent/path-utils.ts`, `src/agent/progress.ts`, `src/agent/executor.ts`, `src/cli/types.ts`, `src/schemas/cli-schemas.ts`, `src/cli/commands.ts`, `src/cli/orchestrator.ts`, `tests/providers/vercel-ai-provider-agent-loop.test.ts`, `README.md` -- **Tried**: Initial agent-mode wiring used `process.cwd()` as repository root, which broke tool-relative file resolution in orchestrator tests; switched to inferred common root across targets for agent execution. - -### Task: Wire lint context, user instructions, and request-failure attribution -- **Status**: completed -- **What was done**: Added per-call lint context prompt augmentation, passed `userInstructionContent` through agent-mode orchestrator into executor system prompt construction, and split request-failure counting from operational finalize/workflow errors in agent mode. -- **Files changed**: `src/agent/executor.ts`, `src/agent/prompt-builder.ts`, `src/cli/orchestrator.ts`, `tests/agent/agent-executor.test.ts`, `tests/orchestrator-agent-output.test.ts`, `tests/agent/prompt-builder.test.ts` -- **Tried**: Initial test assertions for prompt-builder expected older section headings (`Role:`/`Operating Policy`); updated assertions to match current builder copy while preserving behavior checks. diff --git a/docs/specs/2026-07-10-harness-architecture.md b/docs/specs/2026-07-10-harness-architecture.md deleted file mode 100644 index 52125a3..0000000 --- a/docs/specs/2026-07-10-harness-architecture.md +++ /dev/null @@ -1,199 +0,0 @@ -# VectorLint Harness Architecture - -Status: Current as of the bounded harness refactor. - -Related audit: [`2026-07-10-vectorlint-harness-architecture-audit.md`](../audits/2026-07-10-vectorlint-harness-architecture-audit.md) - -## Purpose And Product Direction - -VectorLint is a bounded content review harness. External callers, such as Codex, Claude Code, CI jobs, scripts, or humans at the CLI, own exploration and decide what content to review. VectorLint owns constrained review of supplied target content against source-backed rules. - -This means VectorLint is not a workspace agent. It does not search arbitrary files, expand scope on its own, rewrite content, or let a model override the rule it is supposed to enforce. It receives target content, rules, optional caller-supplied review context, budgets, and output policy, then returns structured review results. - -Use the shared domain language in [`../../CONTEXT.md`](../../CONTEXT.md) when changing code, docs, tests, or agent handoffs. - -## Review Contract - -The review contract lives in [`src/review/types.ts`](../../src/review/types.ts), with boundary schemas in [`src/review/schemas.ts`](../../src/review/schemas.ts). - -`ReviewRequest` contains: - -- `target`: the target URI, content, content type, and optional byte length. -- `rules`: source-backed rules with `id`, `source`, `body`, optional `name`, optional `severity`, and optional Via Negativa violation conditions. -- `context`: optional caller-supplied review context. -- `budget`: explicit bounds for one review. -- `outputPolicy`: usage and payload telemetry policy. -- `modelCall`: `single`, `agent`, or `auto`. - -Every executor implements: - -```ts -interface ReviewExecutor { - run(request: ReviewRequest): Promise; -} -``` - -`ReviewResult` contains: - -- `findings`: verified findings anchored in target content. -- `scores`: per-rule scores. -- `diagnostics`: operational and finding-processing notes. -- `usage`: optional model-call and token metadata. -- `hadOperationalErrors`: optional flag for partial-result runs with operational errors. - -The contract deliberately does not expose `evaluator`, `judge`, subjective rubric scoring, model-authored rule overrides, or autonomous workspace tools. - -## Rule Model - -VectorLint rules are objective Via Negativa checks. A rule should name observable violation conditions and ask the reviewer to find evidence that those conditions are present. The reviewer should not grade broad alignment against an ideal. - -Good rules answer yes/no questions: - -- Is this sentence using a banned phrase? -- Does this claim omit required evidence? -- Does this section repeat information already stated nearby? - -Future-facing rule authors should avoid subjective judge/rubric designs. Historical rubric-style language remains only where it describes removed behavior. - -## Model Calls - -The model-call strategy is selected with `--model-call single|agent|auto`. The allowed values and default come from [`src/review/executor.ts`](../../src/review/executor.ts), [`src/cli/types.ts`](../../src/cli/types.ts), and [`src/schemas/cli-schemas.ts`](../../src/schemas/cli-schemas.ts). - -### `single` - -`single` uses [`SingleModelCallExecutor`](../../src/executors/single-model-call-executor.ts). It performs one structured model call per rule/chunk through `StructuredModelClient`. For targets above 600 words, it chunks line-numbered content and merges violations before shared finding processing. - -The single executor owns no tool surface. - -### `agent` - -`agent` uses [`AgentModelCallExecutor`](../../src/executors/agent-model-call-executor.ts). It performs one bounded tool-calling run per rule through `ToolCallingModelClient`. - -The only executor-owned tool is `read_target_section`, defined in [`src/executors/target-read-capability-adapter.ts`](../../src/executors/target-read-capability-adapter.ts). It reads 1-based line ranges from in-memory `request.target.content` and returns model-visible errors for invalid ranges. - -The agent model call cannot: - -- search the workspace; -- read arbitrary files; -- read URIs outside the target content; -- rewrite rules; -- create top-level workspace findings; -- override source-backed rule prompts. - -### `auto` - -`auto` resolves through `chooseModelCall` in [`src/review/executor.ts`](../../src/review/executor.ts). It selects `agent` when target content is larger than `AGENT_MODEL_CALL_BYTE_THRESHOLD` (`600_000` bytes) or when more than five rules apply. Otherwise, it selects `single`. - -The CLI default is `auto`. - -## On-Page Boundary - -The on-page boundary is implemented in [`src/review/boundary.ts`](../../src/review/boundary.ts) and enforced by executor design. - -Target content is always in scope. Caller-supplied review context is in scope only because the caller explicitly provided it. Workspace files are out of scope unless the caller passes their content as review context. Rule bodies are source-backed and loaded before the review request is built. - -The target-read adapter performs no filesystem reads. It slices the target content already present in memory. - -## Budgets - -Default budgets live in [`src/review/budget.ts`](../../src/review/budget.ts): - -| Field | Default | Meaning | -| --- | ---: | --- | -| `maxTargetBytes` | `1_000_000` | maximum target content size | -| `maxCallerContextBytes` | `500_000` | maximum caller-supplied context size | -| `maxChunksPerRule` | `20` | maximum chunks/tool steps per rule | -| `maxModelCallsPerReview` | `50` | maximum model calls in one review | -| `maxWallClockMs` | `300_000` | maximum elapsed review time, in milliseconds | - -Budgets limit work, not output. There is intentionally no `maxFindingsPerRule`. There is also no headless retry budget because VectorLint no longer has a headless autonomous executor. - -`maxWallClockMs` is the review timeout. Executors check model-call count and elapsed time before model calls and surface budget exhaustion as diagnostics when partial results can be returned. - -## Finding Processing - -Both model-call strategies use the same finding-processing pipeline in [`src/findings/`](../../src/findings/): - -1. Filter candidate findings through the evidence gate. -2. Verify finding evidence against target content. -3. Deduplicate verified findings. -4. Score by verified finding count and density. -5. Resolve severity. -6. Assemble findings, scores, diagnostics, and operational status. - -Unlocatable quoted evidence becomes a `finding-evidence-not-locatable` diagnostic and emits no finding. VectorLint does not fall back to model-provided line numbers for unverified evidence. - -Diagnostics describe operational or finding-processing conditions such as unlocatable evidence, budget exhaustion, schema parse failures, and provider failures. They are not content findings. - -## Provider And Model Capabilities - -Providers are transport capabilities, not product owners. - -[`StructuredModelClient`](../../src/providers/structured-model-client.ts) performs one structured model call and returns validated output. - -[`ToolCallingModelClient`](../../src/providers/tool-calling-model-client.ts) performs one bounded tool-calling generation using caller-supplied tools and returns structured output. The provider does not define product tools. The executor supplies the tool map and run bounds. - -Neither provider capability is an autonomous agent loop. - -## Architecture Diagram - -```text -CLI / External Caller - | - v -ReviewRequest - - target - - source-backed rules - - caller-supplied context - - budget - - output policy - - modelCall - | - v -chooseModelCall(auto?) ──> single ──> SingleModelCallExecutor - └─> agent ──> AgentModelCallExecutor - | - v - read_target_section - target content only - | - v -Shared Finding Processing - - filter - - verify evidence - - dedupe - - score - - diagnostics - | - v -ReviewResult - | - v -Formatters - - line - - json - - vale-json - - rdjson -``` - -## Internal Implementation Notes - -`--model-call` is the documented CLI surface. The internal `--mode` flag is rejected at the CLI boundary so unreleased agent-mode wiring cannot be used accidentally. - -The unreleased autonomous workspace implementation path is removed. External callers that need project-wide exploration should gather context before invoking VectorLint and pass selected content explicitly. - -Subjective `judge`/rubric rules are not part of the future-facing architecture. New rules should be written as objective Via Negativa checks with observable violation conditions. - -## References - -- [`docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md`](../audits/2026-07-10-vectorlint-harness-architecture-audit.md) -- [`src/review/types.ts`](../../src/review/types.ts) -- [`src/review/executor.ts`](../../src/review/executor.ts) -- [`src/review/budget.ts`](../../src/review/budget.ts) -- [`src/review/boundary.ts`](../../src/review/boundary.ts) -- [`src/executors/single-model-call-executor.ts`](../../src/executors/single-model-call-executor.ts) -- [`src/executors/agent-model-call-executor.ts`](../../src/executors/agent-model-call-executor.ts) -- [`src/executors/target-read-capability-adapter.ts`](../../src/executors/target-read-capability-adapter.ts) -- [`src/findings/processor.ts`](../../src/findings/processor.ts) -- [`src/providers/structured-model-client.ts`](../../src/providers/structured-model-client.ts) -- [`src/providers/tool-calling-model-client.ts`](../../src/providers/tool-calling-model-client.ts)