Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
3c04c62
Add typecheck script and clear tsc --noEmit baseline
oshorefueled Jul 13, 2026
e319bba
Exclude docs/research from eslint
oshorefueled Jul 13, 2026
20e0acc
Add vitest.config.ts for stable test resolution
oshorefueled Jul 13, 2026
3580e49
Add npm run verify aggregate gate
oshorefueled Jul 13, 2026
0fe8700
Add typecheck CI job and align test workflow Node version
oshorefueled Jul 13, 2026
6c271f1
Deprecate --mode agent and fall back to standard
oshorefueled Jul 13, 2026
a2e9f25
Append Phase 1 completion note to harness audit
oshorefueled Jul 13, 2026
888f027
Add root .vectorlint.ini for reproducible smoke validation
oshorefueled Jul 13, 2026
99f73c2
Add neutral review-domain contract module
oshorefueled Jul 13, 2026
e4259e7
Extract single finding evidence verifier
oshorefueled Jul 14, 2026
25ec49a
Add shared finding severity and scoring helpers
oshorefueled Jul 14, 2026
20d42ff
Implement shared finding processor contract
oshorefueled Jul 14, 2026
671da9c
Route standard check evaluation through shared processor
oshorefueled Jul 14, 2026
9fe6440
Reject judge prompt type at the loader boundary
oshorefueled Jul 14, 2026
d36d06d
Remove legacy judge review path
oshorefueled Jul 14, 2026
d4ea9a7
Remove orphaned orchestrator param types from CLI contract
oshorefueled Jul 14, 2026
15ec7a3
Document finding-processing behavior and clean stale judge docs
oshorefueled Jul 14, 2026
4fa0701
Sanitize prompt criteria at the findings boundary
oshorefueled Jul 14, 2026
699f355
Define structured and tool-calling model clients
oshorefueled Jul 14, 2026
74957bc
Implement SingleModelCallExecutor for single modelCall
oshorefueled Jul 14, 2026
8bf8e86
Implement AgentModelCallExecutor with target-only read tool
oshorefueled Jul 14, 2026
8bfb47d
Wire --model-call executors into the CLI and remove the agent surface…
oshorefueled Jul 14, 2026
6e2a04f
refactor(cli): reject legacy mode flag
oshorefueled Jul 14, 2026
64e434d
refactor(cli): harden config and payload telemetry
oshorefueled Jul 14, 2026
0c18c2d
refactor(observability): remove legacy agent telemetry operation
oshorefueled Jul 14, 2026
9c122f8
refactor(errors): drop orphaned AgentToolError class
oshorefueled Jul 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '18'
node-version: '20'
cache: 'npm'

- name: Install dependencies
Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/typecheck.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Typecheck

on:
push:
branches: [main, release-docs]
pull_request:
branches: [main, release-docs]

jobs:
typecheck:
name: tsc --noEmit
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Run typecheck
run: npm run typecheck
9 changes: 9 additions & 0 deletions .vectorlint.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# VectorLint Configuration
# Global settings
RulesPath=
Concurrency=4
DefaultSeverity=warning

# Default rules for all markdown files
[**/*.md]
RunRules=VectorLint
197 changes: 13 additions & 184 deletions CREATING_RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ A comprehensive guide to creating powerful, reusable content evaluations using V
- [Rule Anatomy](#rule-anatomy)
- [Evaluation Modes](#evaluation-modes)
- [Check Rules](#check-rules)
- [Judge Rules](#judge-rules)
- [Target Specification](#target-specification)
- [Configuration Reference](#configuration-reference)
- [Best Practices](#best-practices)
Expand All @@ -25,7 +24,7 @@ VectorLint rules are Markdown files with YAML frontmatter that define how your c
- **Rule = Prompt file** (`.md` file organized in rule packs)
- **Pack** = Subdirectory containing related rules (typically named after a company/style guide)
- **Criteria** = Individual quality checks within a rule
- **Score** = LLM-assigned rating (1-4 scale for judge, density-based for check)
- **Score** = Density-based quality score derived from violation count

- **Severity** = How failures are reported (`error` or `warning`)

Expand Down Expand Up @@ -74,27 +73,20 @@ project/

## Evaluation Modes

VectorLint uses a single **Base Evaluator** (`evaluator: base`) that operates in two distinct modes, determined by the `type` field:
VectorLint uses a single **Base Evaluator** (`evaluator: base`) that operates in one mode, determined by the `type` field:

| Mode | `type` | Use Case | Scoring | Output |
| ---------- | ------- | ------------------------------------- | --------------------------- | ----------------------- |
| **Check** | `check` | Pass/fail checks, counting violations | 10 points - 1 per violation | List of specific issues |
| **Judge** | `judge` | Multi-dimensional quality scoring | 0-4 scale per criterion | Weighted average score |

### When to Use Each
The deprecated `type: judge` (alias `subjective`) is no longer supported; a rule using it is rejected at load.

**Use Check when:**
### When to Use Check

- You need to find specific errors (e.g., "Find all grammar mistakes")
- The check is binary (Pass/Fail) for each item
- You want a list of specific violations to fix

**Use Judge when:**

- You're measuring quality on a spectrum (e.g., "How engaging is this?")
- You have multiple dimensions (Clarity, Tone, Depth)
- You need weighted importance (some criteria matter more)

---

## Check Rules
Expand Down Expand Up @@ -141,71 +133,6 @@ Check this content for grammar issues, spelling errors, and punctuation mistakes

---

## Judge Rules

Judge rules use weighted criteria and a 1-4 rubric for sophisticated quality measurement.

### Structure

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: HeadlineEvaluator
name: Headline Evaluator

severity: error
criteria:
- name: Value Communication
id: ValueCommunication
weight: 12
- name: Curiosity Gap
id: CuriosityGap
weight: 2
---

You are a headline evaluator... [Your detailed instructions]

## RUBRIC

# Value Communication <weight=12>

### Excellent <score=4>

Specific, immediately appealing benefit

### Good <score=3>

Clear benefit but less specific impact

...
```

### The 1-4 Scoring Scale

VectorLint uses a **1-4 scale** for all judge criteria, which is then normalized to a 1-10 scale:

| LLM Rating | Meaning | Normalized Score |
| :--------- | :-------- | :--------------- |
| **4** | Excellent | **10.0** |
| **3** | Good | **7.0** |
| **2** | Fair | **4.0** |
| **1** | Poor | **1.0** |

### Score Calculation

1. **Normalization**: We map the 1-4 rating to a 1-10 score using the formula: `1 + ((Rating - 1) / 3) * 9`.
2. **Weighted Average**: The final score is the weighted average of all normalized criterion scores.

**Example:**

- Criterion: "Value Communication" (weight=12)
- Rating: 3 (Good) -> Normalized: 7.0
- Weighted Points: 7.0 \* 12 = 84 points

---

## Target Specification

The `target` field allows you to:
Expand Down Expand Up @@ -243,27 +170,17 @@ target:

### Frontmatter Fields

| Field | Type | Required | Description |
| ------------- | ------------- | -------- | ------------------------------------------------------------------ |
| `specVersion` | string/number | No | Rule specification version (use `1.0.0`) |
| `evaluator` | string | No | Evaluator type: `base`, `technical-accuracy` (default: `base`) |
| `type` | string | No | Mode: `judge` or `check` (default: `check`) |
| `id` | string | **Yes** | Unique identifier (used in error reporting) |
| `name` | string | **Yes** | Human-readable name |
| Field | Type | Required | Description |
| ------------- | ------------- | -------- | -------------------------------------------------------------- |
| `specVersion` | string/number | No | Rule specification version (use `1.0.0`) |
| `evaluator` | string | No | Evaluator type: `base`, `technical-accuracy` (default: `base`) |
| `type` | string | No | Mode: `check` (default: `check`). `judge`/`subjective` rejected at load |
| `id` | string | **Yes** | Unique identifier (used in error reporting) |
| `name` | string | **Yes** | Human-readable name |

| `severity` | string | No | `error` or `warning` (default: `warning`) |
| `evaluateAs` | string | No | `document` or `chunk` - whether to evaluate content as a whole or in chunks (default: `chunk`) |
| `target` | object | No | Content matching specification |
| `criteria` | array | **Yes\*** | List of evaluation criteria (\*required for judge) |

### Criterion Fields

| Field | Type | Required | Description |
| -------- | ------ | -------- | ------------------------------------------ |
| `name` | string | **Yes** | Human-readable criterion name |
| `id` | string | **Yes** | Unique identifier (PascalCase recommended) |
| `weight` | number | No | Importance weight (default: 1) |
| `target` | object | No | Criterion-specific content matching |

---

Expand All @@ -288,25 +205,10 @@ You are a headline evaluator for developer blog posts. Assess whether the headli
2. Uses natural, conversational language (avoid buzzwords)
3. Creates curiosity without being clickbait

For each criterion, provide a score (0-4) and specific examples from the text.
```

### 2. **Use Meaningful Weights (Subjective)**

Scale weights to reflect real-world importance:

```yaml
criteria:
# Technical accuracy is critical
- name: Technical Accuracy
weight: 40

# Readability is important
- name: Readability
weight: 30
For each violation, quote the exact text and suggest a concrete fix.
```

### 3. **Provide Context in Prompts**
### 2. **Provide Context in Prompts**

Help the LLM understand your domain:

Expand Down Expand Up @@ -338,79 +240,6 @@ Check this content for grammar issues, spelling errors, and punctuation mistakes
Report any errors found with specific examples.
```

### Example 2: Headline Evaluator (Judge)

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: Headline
name: Headline Evaluator

severity: error
target:
regex: '^#\s+(.+)$'
flags: "mu"
group: 1
required: true
suggestion: Add an H1 headline for the article.
criteria:
- name: Value Communication
id: ValueCommunication
weight: 10
- name: Language Authenticity
id: LanguageAuthenticity
weight: 5
---

You are a headline evaluator. Assess the H1 headline for:

1. **Value Communication** (10 points): Does it clearly state what the reader gains?
2. **Language Authenticity** (5 points): Does it use natural, conversational language?

## RUBRIC

# Value Communication <weight=10>

### Excellent <score=4>

Specific, immediately appealing benefit clearly stated

...
```

### Example 3: AI Pattern Detector (Judge)

```markdown
---
specVersion: 1.0.0
evaluator: base
type: judge
id: AIPatterns
name: AI Pattern Detector

severity: warning
criteria:
- name: Language Authenticity
id: LanguageAuthenticity
weight: 40
- name: Structural Naturalness
id: StructuralNaturalness
weight: 30
---

Detect AI-generated writing patterns in this content.

## INSTRUCTION

Scan for common AI patterns:

1. **Buzzwords**: leverage, synergy, elevate
2. **Formulaic transitions**: Moreover, Furthermore
...
```

## Resources

- [VectorLint README](./README.md) - Installation and basic usage
Expand Down
32 changes: 9 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,9 @@ If you can write a prompt for it, you can lint it with VectorLint.

## Quality Scores

VectorLint scores your content using error density and a rubric-based system, enabling you to measure quality across documentation. This gives your team a shared understanding of which content needs attention and helps track improvements over time.
VectorLint scores your content using error density, enabling you to measure quality across documentation. This gives your team a shared understanding of which content needs attention and helps track improvements over time.

- **Density-Based Scoring:** For errors that can be counted, scores are calculated based on **error density (errors per 100 words)**, making quality assessment fair across documents of any length.
- **Rubric-Based Scoring:** For more nuanced quality standards, like flow and completeness, scores are graded on a 1-4 rubric system and then normalized to a **1-10 scale**.

## How VectorLint Reduces False Positives

Expand Down Expand Up @@ -145,29 +144,16 @@ Notes:
- Prompts and outputs are recorded when Langfuse observability is enabled.
- Do not send secrets, credentials, or PII unless your policy explicitly allows observability tooling to access that data.

## Agent Mode
## Agent Mode (under review)

Agent mode is for reviews that need context from several files such as
documentation drift and cross-file accuracy.

Run VectorLint in autonomous agent mode:

```bash
vectorlint doc.md --mode agent
```

For machine-parseable output:

```bash
vectorlint doc.md --mode agent --output json
```

To suppress interactive progress in line output:

```bash
vectorlint doc.md --mode agent --print
```
The `--mode agent` flag is under active rework. It currently enables an
autonomous cross-file review mode that is being **removed** in favor of a
bounded harness model. See
[`docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md`](docs/audits/2026-07-10-vectorlint-harness-architecture-audit.md)
for the decision and the refactor plan.

Until the refactor lands, `--mode agent` prints a deprecation warning and
falls back to standard mode. Do not build integrations against it.

## Contributing

Expand Down
Loading