Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 58 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,54 @@
# Repository Guidelines

This repository implements VectorLint — a prompt‑driven, structured‑output content evaluator. Use this guide to navigate the codebase, run it locally, and contribute safely.
This repository implements VectorLint — a prompt‑driven, structured‑output content review harness. Use this guide to navigate the codebase, run it locally, and contribute safely.

Use [`CONTEXT.md`](./CONTEXT.md) as the shared VectorLint domain language across code, docs, tests, and agent work. Prefer its terms when naming modules, writing tests, drafting docs, and describing architecture.

## Agent Behavior Guidelines

These guidelines reduce common LLM coding mistakes. They bias toward caution over speed; use judgment for trivial tasks.

### Think Before Coding

- State assumptions explicitly before implementing.
- If multiple interpretations exist, present them instead of picking silently.
- If a simpler approach exists, say so and push back when warranted.
- If something is unclear, stop, name the confusion, and ask.

### Simplicity First

- Write the minimum code that solves the problem.
- Do not add features beyond what was asked.
- Do not introduce abstractions for single-use code.
- Do not add flexibility or configurability that was not requested.
- Do not add error handling for impossible scenarios.
- If a change becomes larger than needed, simplify before finishing.

### Surgical Changes

- Touch only the files and lines required by the request.
- Do not improve adjacent code, comments, or formatting unless required.
- Match existing style, even when another style is tempting.
- If you notice unrelated dead code, mention it instead of deleting it.
- Remove imports, variables, functions, or files made unused by your own changes.
- Do not remove pre-existing dead code unless asked.

### Goal-Driven Execution

- Turn requests into verifiable goals before implementing.
- For bug fixes, write or identify a reproduction, then make it pass.
- For validation changes, cover invalid inputs, then make the checks pass.
- For refactors, verify behavior before and after the change.
- For multi-step tasks, state a brief plan with verification for each step.
- Loop until the success criteria are verified or a blocker is clear.

### Documentation Artifact Boundaries

- Do not commit raw planning or investigation artifacts to the product codebase.
- Keep audits, plans, specs, run notes, and similar coordination artifacts out of tracked repo paths such as `docs/audits/`, `docs/plans/`, `docs/specs/`, `audits/`, `plans/`, and `specs/`.
- Store coordination artifacts in `.agent-runs/` or another ignored workspace location.
- If a durable architectural decision must be committed, write it as an ADR. ADRs are the only allowed committed decision/planning artifacts.
- Product documentation may describe shipped behavior, configuration, and usage, but it must not preserve internal audit/plan/spec documents as reviewed product docs.

## Project Structure & Module Organization

Expand All @@ -15,9 +63,12 @@ This repository implements VectorLint — a prompt‑driven, structured‑output
- `output/` — TTY formatting (reporter, evidence location, line numbering)
- `prompts/` — YAML frontmatter parsing, schema validation, directive loading
- `providers/` — LLM abstractions (OpenAI, Anthropic, Azure, Gemini), request builder, provider factory
- `review/` — neutral review contract, boundary, budget, schemas, and model-call selection
- `executors/` — bounded `single` and `agent` model-call executors behind `ReviewExecutor`
- `findings/` — shared finding verification, filtering, scoring, diagnostics, and result assembly
- `scan/` — file discovery (fast‑glob) honoring config and exclusions
- `schemas/` — Zod schemas for all external data (API responses, config, CLI, env)
- `scoring/` — score calculation (density-based for check, rubric-based for judge)
- `scoring/` — score calculation for objective violation checks
- `types/` — TypeScript type definitions
- `presets/` — bundled rule packs (e.g., `VectorLint/`)
- `tests/` — Vitest specs for config, scanning, evaluation, providers
Expand Down Expand Up @@ -126,12 +177,14 @@ For documents >600 words, VectorLint automatically chunks content:
## Architecture & Principles

- Boundary validation: all external data (files, CLI, env, APIs) validated at system boundaries using Zod schemas
- Bounded harness model: callers own exploration and context gathering; VectorLint owns constrained review through `single`/`agent`/`auto` model calls behind the `ReviewExecutor` contract
- On-page boundary: executors review target content plus explicit review context only; do not add workspace search, arbitrary file reads, model-authored rule overrides, or autonomous agent behavior
- Type safety: strict TypeScript with no `any`; use `unknown` + schema validation for external data
- Dependency inversion: depend on `LLMProvider` and `SearchProvider` interfaces; keep providers thin (transport only)
- Dependency inversion: depend on `StructuredModelClient`, `ToolCallingModelClient`, and `SearchProvider` interfaces; keep providers thin (transport only)
- Dependency injection: inject `RequestBuilder` via provider constructor to avoid coupling
- Separation of concerns: rules define rubric; schemas enforce structure; CLI orchestrates; evaluators process; reporters format
- Separation of concerns: rules define observable violation checks; schemas enforce structure; CLI orchestrates; executors run model calls; findings process results; reporters format output
- Separation of concerns: when a file starts combining contracts, orchestration, and utility logic, extract shared helpers and types into focused modules
- Extensibility: add providers by implementing `LLMProvider` or `SearchProvider`; add evaluators via registry pattern
- Extensibility: add model providers by implementing the structured/tool-calling model client interfaces or search by implementing `SearchProvider`
- Error handling: prefer the repository's custom error hierarchy over native `Error`; catch blocks use `unknown` type and extend existing custom error types before introducing raw exceptions
- Shared domain constants: avoid magic strings for core runtime concepts; define shared constants, enums, or types and import them where needed
- Naming: choose domain-accurate names that reflect the real abstraction level; avoid use-case-specific terminology in shared runtime code
Expand Down
151 changes: 151 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# VectorLint

VectorLint is a content review harness. This language file defines the terms used to describe reviews, rules, findings, scoring, and execution so product, documentation, tests, and code can share the same vocabulary.

## Language

### Review Model

**Content Review Harness**:
A system that reviews supplied target content against source-backed rules and returns structured review results. The harness does not own exploration, research, workspace search, or content rewriting.
_Avoid_: Autonomous agent, workspace agent, agent mode

**Review**:
One evaluation of target content against one or more rules. A review produces findings, scores, diagnostics, usage metadata, and output.
_Avoid_: Lint run when referring to the domain process, agent session

### Review Inputs

**Caller**:
The person or system that invokes VectorLint and supplies the target, target content, and any allowed review context. A caller does not define rules or configuration as part of the review invocation.
_Avoid_: Agent, user agent, workspace owner

**Target**:
The subject of a review. A target identifies what VectorLint is reviewing, regardless of whether it came from a file, memory, or another caller-provided source.
_Avoid_: File when the review subject may not be a file, page when the content may not be documentation

**Target File**:
A file used as the source of a target. A target file is one way to provide target content, but not every target must come from a file.
_Avoid_: Target when the distinction between the file and the review subject matters

**Target Content**:
The actual text or content body reviewed for a target. Findings and finding evidence are grounded in target content.
_Avoid_: Target file, page content

**Review Context**:
Additional content explicitly supplied to VectorLint for a review. Review context is in scope only because it was allowed as part of the review input.
_Avoid_: Caller context, workspace context, discovered context, ambient context

**On-Page Boundary**:
The rule that VectorLint reviews only target content and review context. VectorLint must not discover arbitrary workspace files or expand the review scope on its own.
_Avoid_: Workspace scope, project-wide scope, cross-file scope

### Rules

**Rule**:
A source-backed instruction that defines observable violation conditions VectorLint should look for. Rules exist before a review is invoked.
_Avoid_: Prompt when referring to the domain concept, model instruction when implying the model authored it

**Via Negativa Review**:
A review approach that looks for evidence a rule was violated rather than evidence the content aligns with an ideal. VectorLint rules should be written so a model can answer whether an observable violation condition is present.
_Avoid_: Alignment review, subjective assessment, rubric judgment

**Violation Condition**:
An observable yes/no condition that counts as a rule violation when present in target content. A violation condition should be specific enough to ground a finding in finding evidence.
_Avoid_: Criterion when it implies subjective grading, preference, guideline

**Rule Pack**:
A named collection of related rules that can be applied together.
_Avoid_: Preset when describing the domain concept, folder

**Review Configuration**:
Pre-existing settings that determine how VectorLint runs reviews, selects rules, and formats behavior. Configuration is not the same as target content or review context.
_Avoid_: Caller input when referring to review invocation, target configuration

**Check Rule**:
A rule expressed through observable violation conditions. This is the only future-facing rule style in VectorLint.
_Avoid_: Direct rule, standard rule

### Execution

**Model Call**:
The call shape VectorLint uses to run a reviewer model against target content during a review.
_Avoid_: Execution strategy, content access, process mode, evaluator type, rule type, mode

**Single Model Call**:
A model call where VectorLint supplies the review request and target content without giving the model a read tool. If VectorLint chunks a large target, each chunk is still reviewed through a single model call.
_Avoid_: Direct strategy, standard mode, check path

**Agent Model Call**:
A bounded model call where the model may request sections of the target content through a single target-scoped read capability. This is for context management during large or context-sensitive reviews; it is not workspace exploration.
_Avoid_: Autonomous agent mode, workspace agent, file reader

**Auto Model Call**:
The model call value where VectorLint deterministically chooses between single and agent.
_Avoid_: Smart mode, agent fallback

**Target Read Capability**:
The bounded ability to read line ranges from target content. It cannot read arbitrary files, search the workspace, rewrite rules, or create top-level workspace findings.
_Avoid_: Tool suite, workspace tools, read_file

### Findings

**Candidate Finding**:
A potential issue raised before VectorLint has decided whether it should be reported. Candidate findings may be filtered out or become diagnostics.
_Avoid_: Violation when the issue has not been accepted, result

**Verified Finding**:
A finding that VectorLint has accepted for reporting because it is grounded in the target content and passes review filters.
_Avoid_: Issue when finding-evidence status matters, raw finding

**Finding**:
A reported content issue produced by a review. When precision matters, use candidate finding or verified finding.
_Avoid_: Violation as the default user-facing term, problem

**Finding Evidence**:
The exact target-content text or surrounding context that supports a finding.
_Avoid_: Quote when the text may include surrounding context, match when referring to the concept rather than a located span

**Finding Evidence Verification**:
The act of confirming that the finding evidence for a candidate finding can be located in the target content. Unverified finding evidence should not become a verified finding.
_Avoid_: Line fallback, model-provided location

**Finding Processing**:
The review step that turns candidate findings into verified findings, diagnostics, scores, and final review output.
_Avoid_: Projection, result processing, output processing

**Diagnostic**:
A structured note about review execution or finding processing, especially when something affects trust, completeness, or interpretation but is not itself a content finding.
_Avoid_: Finding, warning when referring to the structured domain object

### Scoring And Output

**Severity**:
The impact level attached to a finding or rule outcome.
_Avoid_: Priority, importance

**Score**:
A normalized quality measurement for a rule or review. Scores come from verified finding count or density for objective violation checks.
_Avoid_: Grade when referring to the domain object, rating

**Review Result**:
The structured outcome of a review: verified findings, scores, diagnostics, usage metadata, and operational status.
_Avoid_: Projection result, formatter result, raw model output

**Output Format**:
The representation used to present a review result to a human or machine, such as line output or JSON.
_Avoid_: Review result, report type

**Review Budget**:
The explicit limits for a review, such as model calls, target size, review context size, chunks per rule, and duration. A review budget limits work, not the number of verified findings emitted.
_Avoid_: Rate limit, timeout when referring to the full budget concept

### Historical Terms

**Autonomous Agent Mode**:
The old VectorLint direction where VectorLint exposed workspace tools to a model and let it explore beyond the target content. This is historical language only; current domain language should use single model call, agent model call, caller, and content review harness.
_Avoid_: Agent mode as a current feature, workspace-agent review

**Judge Rule**:
The historical rubric-style assessment rule where a model graded content against criteria. This rule style is removed from the future-facing product model.
_Avoid_: Rubric mode, subjective check, evaluator kind
32 changes: 23 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# VectorLint: Prompt it, Lint it! [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

VectorLint is a command-line tool that evaluates and scores content using LLMs. It uses [LLM-as-a-Judge](https://en.wikipedia.org/wiki/LLM-as-a-Judge) to catch terminology, technical accuracy, and style issues that require contextual understanding.
VectorLint is a command-line content review harness that uses LLMs to find observable terminology, technical accuracy, and style violations that require contextual understanding.

![VectorLint Screenshot](./assets/VectorLint_screenshot.jpeg)

Expand Down Expand Up @@ -144,16 +144,30 @@ Notes:
- Prompts and outputs are recorded when Langfuse observability is enabled.
- Do not send secrets, credentials, or PII unless your policy explicitly allows observability tooling to access that data.

## Agent Mode (under review)
## How VectorLint Runs Reviews

The `--mode agent` flag is under active rework. It currently enables an
autonomous cross-file review mode that is being **removed** in favor of a
bounded harness model. See
[`docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md`](docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md)
for the decision and the refactor plan.
VectorLint is a bounded review harness, not a workspace agent. Callers choose
the target content and any review context; VectorLint reviews that supplied
content against source-backed rules and returns findings, scores, diagnostics,
and usage metadata.

Until the refactor lands, `--mode agent` prints a deprecation warning and
falls back to standard mode. Do not build integrations against it.
Use `--model-call` to choose how the reviewer model is invoked:

```bash
vectorlint doc.md --model-call single
vectorlint doc.md --model-call agent
vectorlint doc.md --model-call auto
```

- `single` runs structured model calls without tools. This is best for normal
documents and straightforward rule checks.
- `agent` runs a bounded target-only model call that can page through the
current target with `read_target_section`. It cannot search the workspace,
read arbitrary files, or override rules.
- `auto` is the default. VectorLint chooses `single` for normal inputs and
`agent` for large or multi-rule reviews that benefit from target paging.

See [Model calls](docs/model-calls.mdx) for more details.

## Contributing

Expand Down
Loading