diff --git a/TODO.md b/TODO.md index 02d00ff..cede216 100644 --- a/TODO.md +++ b/TODO.md @@ -1,15 +1,6 @@ # Spectra Roadmap -## 1. Publish `@mohanscodex/spectra-code` to npm - -Make the Spectra Code package installable via npm (or other package managers) so users can add it as a dependency rather than needing the full monorepo. - -- [x] Ensure `packages/code/package.json` has correct `name`, `version`, `exports`, `files`, `bin` entries -- [x] Verify `@mohanscodex/spectra-code` resolves and imports correctly in an isolated project (extend `test:import` to include it) -- [x] Set up changeset release workflow to include the code package -- [x] Document install + usage instructions for npm consumers - -## 2. Custom states from tool calls (generative UI support) +## 1. Custom states from tool calls (generative UI support) Tool calls should be capable of emitting custom states during execution. This enables generative UI patterns where the tool can surface intermediate progress, status changes, or arbitrary state updates to the caller — useful for showing loading states, progress bars, or dynamic UI transitions. @@ -18,7 +9,7 @@ Tool calls should be capable of emitting custom states during execution. This en - [ ] Surface these states through the agent event stream so callers can render them reactively - [ ] Ensure both TypeScript (`packages/agent`) and Rust (`crates/spectra-rs`) support this -## 3. Real-time tool content streaming +## 2. Real-time tool content streaming Ability to stream a tool's content in real time as it is produced, rather than waiting for the full `ToolResult` before surfacing anything. This builds on the current tool call implementation to support progressive output. @@ -27,48 +18,32 @@ Ability to stream a tool's content in real time as it is produced, rather than w - [ ] Ensure backward compatibility — non-streaming tools (current `execute` returning `ToolResult`) still work - [ ] Implement in both TypeScript (`packages/agent`) and Rust (`crates/spectra-rs`) -## 4. Context compaction +## 3. Context compaction Automatic context management that summarizes old conversation history when approaching token limits, preserving recent turns verbatim. -- [x] Implement overflow detection: trigger when total tokens >= `usable capacity - 20K buffer` -- [x] Head/tail split: summarize older context, preserve last 2-4 turns verbatim (25% of usable tokens, clamped 2K-8K) -- [x] Structured summary template (Goal, Constraints, Progress, Key Decisions, Next Steps, Critical Context, Relevant Files) - [ ] Anchored summaries: incrementally update previous summary instead of rebuilding from scratch -- [x] Tool output truncation during compaction (2K chars max, media stripped) - [ ] Async pruning pass: mark old tool outputs as compacted, protect recent 40K, never prune `skill` tool outputs - [ ] Auto-continue after compaction ("Continue if you have next steps...") -- [x] Configurable: `compaction.auto` toggle, `compaction.reserved` buffer, `compaction.preserve_recent_tokens` -- [x] Implement in TypeScript (`packages/agent`) for Spectra Code, with Rust SDK following later -## 5. Agent loop safety guards +## 4. Agent loop safety guards Defensive mechanisms in the agent loop to prevent common failure modes. Based on patterns from OwlCoda's conversation engine. -- [x] Tool loop detection: track consecutive identical tool calls, hard-stop after threshold (e.g., 5 identical calls) - [ ] Narration loop detection: detect repetitive output patterns (e.g., same sentence repeated 3+ times), interrupt and prompt user - [ ] Output bloat detection: warn when single tool output exceeds reasonable size (e.g., 50K chars), offer to truncate - [ ] Task no-progress hard stop: if agent makes N turns with no file writes or meaningful progress, pause and ask for direction - [ ] Convergence state machine: track whether agent is making forward progress or cycling between states - [ ] Surface safety events (interrupt, warning, hard-stop) through the agent event stream so TUI can render them -## 6. Expanded slash commands +## 5. Expanded slash commands Broaden the slash command surface to cover observability, session management, git workflows, and configuration — matching capabilities found in OwlCoda (70+ commands). -**Observability:** -- [x] `/cost` — show estimated cost for current session (opens cost dialog with detailed breakdown) -- [x] `/tokens` — show token usage breakdown (input, output, context window %) -- [x] `/stats` — session statistics (model, provider, turns, duration, tok/s, cost) -- [x] `/context` — show context window usage and remaining capacity -- [x] `/status` — system status (model, provider, MCPs, agent, tokens, cost) - **Session:** - [ ] `/save` — explicitly save current session -- [x] `/search` — search sessions (opens session list dialog) - [ ] `/export` — export session to JSON/Markdown - [ ] `/history` — show conversation turn history -- [x] `/compress` — manually trigger context compaction **Git:** - [ ] `/commit` — stage and commit changes with AI-generated message (requires template prompt system) @@ -79,7 +54,7 @@ Broaden the slash command surface to cover observability, session management, gi - [ ] `/permissions` — view/edit tool permission settings (command registered but no dialog renders — broken) - [ ] `/settings` — open settings panel (command registered but no dialog renders — broken) -## 7. Plugin system +## 6. Plugin system Dynamic, hook-based plugin system. Plugins extend behavior without modifying core code. @@ -115,31 +90,25 @@ interface SpectraPlugin { - [ ] Plugin discovery CLI: `spectra plugin list`, `spectra plugin install` - [ ] Integrate with existing Extension trait in Rust SDK (parallel implementation) -## 8. Observability middleware +## 7. Observability middleware Per-request metrics, cost tracking, and runtime health visibility at the HTTP proxy/transport layer. -- [x] Per-model cost tracking: token counts × configured pricing, cumulative per session - [ ] Request latency metrics: time-to-first-token, total duration, p50/p95/p99 aggregates - [ ] Rate limit headers parsing: extract `X-RateLimit-*` headers from provider responses, surface to UI -- [x] Circuit breaker state exposure: expose open/half-open/closed state per model via `/status` or event stream -- [x] Token usage breakdown: input, output, cache read, cache write tokens per request - [ ] Expose metrics through agent event stream so TUI can render `/cost`, `/tokens`, `/stats` commands -## 9. Session handling overhaul +## 8. Session handling overhaul Spectra's session system is fundamentally weaker than OpenCode's. The current JSON-per-file storage, shallow fork, and missing compaction will not scale. **Storage:** -- [x] Migrate from JSON files to SQLite (indexed queries, pagination, cascade deletes) - [ ] Schema: sessions table, messages table, parts table with foreign keys -- [x] Indexed columns: project_id, parent_id, time_created, time_updated - [ ] Cursor-based pagination for session listing **Fork & Branch:** - [ ] Deep copy with ID remapping (prevent collisions between forked sessions) - [ ] Fork from specific message point (not just entire session) -- [x] Parent-child relationship tracking via parent_id - [ ] Fork count in title (e.g. "Title (fork #1)") **Search & Filtering:** @@ -159,7 +128,7 @@ Spectra's session system is fundamentally weaker than OpenCode's. The current JS - [ ] Sort by updated/created - [ ] Show token count and cost per session -## 10. Skills system +## 9. Skills system Learned, reusable skill files that provide specialized workflows and context for specific tasks. Skills are markdown-based instruction files with YAML frontmatter, following the Claude Code format (compatible with OwlCoda and OpenCode). @@ -193,15 +162,6 @@ skill-name/ 3. Config-defined: custom paths + URLs in config file **Implementation:** -- [x] Skill loader: scan directories, parse YAML frontmatter, validate `name` field -- [x] Auto-tag extraction from directory category, name segments, section headers, description keywords -- [x] TF-IDF index with cosine similarity matching (zero-dependency, cached 60s TTL) -- [x] `find_skills` tool: query mode (scored results) + `all: true` fallback (full catalog) -- [x] `skill` tool: load full SKILL.md by name with `$ARGUMENTS` substitution -- [x] String substitutions: `$ARGUMENTS`, `$0`, `${SPECTRA_SKILL_DIR}` -- [x] Bundled skills: ship 65 skills with Spectra Code, resolved via `import.meta.url` -- [x] Three-layer precedence: user-defined (`~/.claude/skills/`) > project (`.claude/skills/`) > bundled -- [x] Skills hint in system prompt: "Use find_skills to discover skills" - [ ] Dynamic context injection: `` !`command` `` syntax to run shell commands before injection - [ ] Permission system: per-skill allow/deny/ask via config @@ -315,7 +275,7 @@ skill-name/ *Meta / Using Skills (1):* - `using-skills` — mandatory workflows for how to find, read, and use skills -## 11. Template prompt system +## 10. Template prompt system Commands like `/commit` and `/review` need structured prompts that are loaded from files, not hardcoded in JS. This enables maintainable, user-overridable command behavior. @@ -329,22 +289,19 @@ Commands like `/commit` and `/review` need structured prompts that are loaded fr - [ ] `commit.txt` — git commit protocol (run git status/diff/log, analyze staged changes, draft message, commit) - [ ] `review.txt` — code review template (determine review type, gather context, check bugs/structure/performance) -## 12. Subagent spawning from commands +## 11. Subagent spawning from commands Commands like `/review` need to spawn a child agent session with restricted tools (read-only for review, full access for commit). **Design:** -- [x] `subtask: true` flag on command definition — spawns a child session (implemented as `mode: 'subagent'` on AgentDefinition) - [ ] Child session inherits permission rules from parent (external_directory, deny rules) -- [x] Tool restrictions: `/review` gets read-only tools (read, glob, grep, bash for git), no write/edit -- [x] After subtask completes, inject result into parent session as context - [ ] Child session title: `"${description} (@${agent} subagent)"` **Commands that need this:** - [ ] `/review` — spawns read-only subagent with review template - [ ] `/commit` — can run inline (main agent) or spawn subagent with commit template -## 13. Commit protocol in bash tool +## 12. Commit protocol in bash tool Embed git commit instructions directly in the bash tool's system prompt, so the agent knows the correct commit workflow without needing a dedicated command. @@ -355,47 +312,7 @@ Embed git commit instructions directly in the bash tool's system prompt, so the - [ ] Secret detection: refuse to commit files likely containing secrets (.env, credentials.json) - [ ] Style matching: read recent commit messages to match tone/format -## 14. Evolving skills (self-learning system) - -Skills that are automatically synthesized from past sessions, creating a self-improving agent that learns from successful interactions. - -**Three-tier skill system (highest precedence wins):** -1. Bundled — read-only defaults from the package (lowest) -2. Evolving — auto-synthesized from sessions, stored in `~/.spectra/skills/` (middle) -3. User-defined — manually created in project/user dirs (highest) - -**Storage:** -- `~/.spectra/skills//metadata.json` — full skill document with useCount, version, parentId -- `~/.spectra/skills//SKILL.md` — rendered markdown (upstream-compatible) - -**Synthesis flow:** -- After session ends, analyze trace: tools called, success/failure, complexity score -- If complexity >= threshold (min 3 tool calls, min 6 messages), trigger synthesis -- LLM generates SKILL.md from session trace (name, description, when_to_use, procedure, pitfalls) -- Before saving, check for duplicates via TF-IDF similarity (threshold 0.7) -- If similar skill exists: evolve (version bump) or fork (new ID with parentId linkage) -- If no duplicate: save as new skill - -**Evolution:** -- Version bump: update existing skill in-place (increment version) -- Fork: create new skill with different ID, link via parentId -- `getSkillLineage()`: walk parentId chain for version history - -**useCount tracking:** -- Increment useCount when skill is loaded via `skill` tool -- Update updatedAt timestamp -- Boost score in TF-IDF matching: `score * (1 + min(0.1 * log(1 + useCount), 0.5))` - -**Implementation:** -- [x] Skill storage: save/load evolving skills from `~/.spectra/skills/` -- [x] Session trace extraction: summarize tools called, outcomes, patterns -- [x] Skill synthesis prompt: generate SKILL.md from trace -- [x] Duplicate detection: TF-IDF similarity check before saving -- [x] Evolution/forking: version bump or parentId-linked fork -- [x] useCount tracking: increment on load, boost in matching -- [x] Three-tier merge: bundled → evolving → user in `discoverAndCreateSkillTools()` - -## 15. Coding plan provider integrations +## 13. Coding plan provider integrations Support bundled access to popular AI coding subscription plans — giving users affordable, multi-model access without managing individual provider API keys. These plans are OpenAI/Anthropic-compatible and work with any agent that speaks those protocols. @@ -424,6 +341,12 @@ Support bundled access to popular AI coding subscription plans — giving users - [ ] Model capability registry: map each plan's models to their context windows, strengths, and benchmarks - [ ] Integration with existing provider system: plans register as providers in `packages/ai` registry +## 14. TUI / UX fixes + +- [x] Fix scroll issue in the slash (`/`) command menu in prompt input +- [ ] Double Esc press to interrupt an active stream +- [ ] Clearer status reporting for subagents and other areas when they encounter an error + ### Future (deferred) The following are deferred until the core system is stable and functional: diff --git a/packages/code/src/__tests__/edit-match.test.ts b/packages/code/src/__tests__/edit-match.test.ts new file mode 100644 index 0000000..5454ea9 --- /dev/null +++ b/packages/code/src/__tests__/edit-match.test.ts @@ -0,0 +1,268 @@ +import { describe, it, expect } from 'vitest'; +import { + findEditMatch, + applyEdit, + replaceOnce, + normalizeLineEndings, + detectLineEnding, + convertToLineEnding, +} from '../tools/edit-match.js'; + +describe('edit-match: line-ending helpers', () => { + it('detects CRLF vs LF by first occurrence', () => { + expect(detectLineEnding('a\nb')).toBe('\n'); + expect(detectLineEnding('a\r\nb')).toBe('\r\n'); + // Mixed: first occurrence wins. + expect(detectLineEnding('a\r\nb\nc')).toBe('\r\n'); + expect(detectLineEnding('a\nb\r\nc')).toBe('\n'); + }); + + it('normalises CRLF and lone CR to LF', () => { + expect(normalizeLineEndings('a\r\nb')).toBe('a\nb'); + expect(normalizeLineEndings('a\rb')).toBe('a\nb'); + expect(normalizeLineEndings('a\nb')).toBe('a\nb'); + }); + + it('converts LF text to the requested ending', () => { + expect(convertToLineEnding('a\nb', '\n')).toBe('a\nb'); + expect(convertToLineEnding('a\nb', '\r\n')).toBe('a\r\nb'); + }); +}); + +describe('edit-match: exact match', () => { + it('finds a unique exact substring', () => { + const content = 'function foo() {\n\treturn 1;\n}\n'; + const m = findEditMatch(content, '\treturn 1;'); + expect(m.ok).toBe(true); + expect(m.index).toBe(content.indexOf('\treturn 1;')); + expect(m.length).toBe('\treturn 1;'.length); + }); + + it('reports not-found for absent text', () => { + const m = findEditMatch('hello world', 'goodbye'); + expect(m.ok).toBe(false); + expect(m.error).toMatch(/Could not find/i); + }); + + it('reports ambiguous for multiple exact matches', () => { + const content = 'const x = 1;\nconst y = 2;\nconst x = 1;\n'; + const m = findEditMatch(content, 'const x = 1;'); + expect(m.ok).toBe(false); + expect(m.error).toMatch(/multiple matches/i); + }); +}); + +describe('edit-match: CRLF tolerance', () => { + // CRLF/LF tolerance is exercised through applyEdit, which normalises the + // oldString to the file's detected line ending BEFORE running the match + // pipeline. That lets SimpleReplacer (exact) handle it cleanly, avoiding + // the trailing-\r artifact that LineTrimmedReplacer would otherwise yield + // when splitting CRLF content on \n. This mirrors opencode's flow. + + it('applyEdit matches an LF oldString against a CRLF file', () => { + const content = 'line one\r\nline two\r\nline three\r\n'; + const { content: next, error } = applyEdit(content, 'line one\nline two\nline three', 'A\r\nB\r\nC'); + expect(error).toBeUndefined(); + expect(next).toBe('A\r\nB\r\nC\r\n'); + }); + + it('applyEdit matches a CRLF oldString against an LF file', () => { + const content = 'line one\nline two\nline three\n'; + const { content: next, error } = applyEdit(content, 'line one\r\nline two\r\nline three', 'A\nB\nC'); + expect(error).toBeUndefined(); + expect(next).toBe('A\nB\nC\n'); + }); + + it('applyEdit matches a multi-line LF block against a CRLF file (the reported message.tsx case)', () => { + const content = 'export function Foo() {\r\n\treturn
;\r\n}\r\n'; + const { content: next, error } = applyEdit( + content, + 'export function Foo() {\n\treturn
;\n}', + 'export function Bar() {\n\treturn ;\n}', + ); + expect(error).toBeUndefined(); + expect(next).toBe('export function Bar() {\r\n\treturn ;\r\n}\r\n'); + }); + + it('applyEdit preserves CRLF in the file when newString uses LF', () => { + const content = 'foo\r\nbar\r\nbaz\r\n'; + const { content: next, error } = applyEdit(content, 'bar', 'BAR'); + expect(error).toBeUndefined(); + expect(next).toBe('foo\r\nBAR\r\nbaz\r\n'); + }); + + it('applyEdit preserves LF in the file when newString uses CRLF', () => { + const content = 'foo\nbar\nbaz\n'; + const { content: next, error } = applyEdit(content, 'bar\r\n', 'QUUX\r\n'); + expect(error).toBeUndefined(); + // Detected ending is LF; newString CRLFs are normalised down to LF. + expect(next).toBe('foo\nQUUX\nbaz\n'); + }); +}); + +describe('edit-match: tab vs space tolerance (reported bug)', () => { + // The reported failure: a deeply tab-indented .tsx file, agent's oldString + // arrives with tabs turned into spaces somewhere in the parameter pipeline. + it('matches space-indented oldString against tab-indented file (LineTrimmed)', () => { + const content = 'export function Foo() {\n\treturn (\n\t\t
\n\t\t\thi\n\t\t
\n\t);\n}\n'; + // oldString uses 4-space indentation per level instead of tabs. + const oldString = ' hi'; + const m = findEditMatch(content, oldString); + expect(m.ok).toBe(true); + // The matched span is the file's real tab bytes. + expect(content.slice(m.index!, m.index! + m.length!)).toBe('\t\t\thi'); + }); + + it('matches tab-indented oldString against space-indented file (LineTrimmed)', () => { + const content = ' foo()\n bar()\n'; + const m = findEditMatch(content, '\tfoo()'); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe(' foo()'); + }); + + it('matches a multi-line tab-indented block against space-indented oldString', () => { + const content = 'if (x) {\n\t\tdoA();\n\t\tdoB();\n\t}\n'; + const oldString = ' doA();\n doB();'; + const m = findEditMatch(content, oldString); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe('\t\tdoA();\n\t\tdoB();'); + }); +}); + +describe('edit-match: whitespace-normalised strategy', () => { + it('collapses internal whitespace runs when matching a single line', () => { + const content = 'const x = 1;\n'; + const m = findEditMatch(content, 'const x = 1;'); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe('const x = 1;'); + }); + + it('matches multi-line block ignoring inter-line whitespace differences', () => { + // WhitespaceNormalizedReplacer's multi-line branch yields whole file + // lines (including any leading indent), so the span is the full matched + // block, not a substring trimmed to the search's start column. + const content = ' a b\n c d\n'; + const m = findEditMatch(content, 'a b\nc d'); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe(' a b\n c d'); + }); +}); + +describe('edit-match: indentation-flexible strategy', () => { + it('matches a block shifted by a different common indent', () => { + const content = 'function outer() {\n\t\tfunction inner() {\n\t\t\treturn 0;\n\t\t}\n\t}\n'; + const oldString = 'function inner() {\n\t\treturn 0;\n\t}'; + // oldString has 1-level indent; file has 2-level. Strip common indent + // from both and the de-indented bodies match. + const m = findEditMatch(content, oldString); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe( + '\t\tfunction inner() {\n\t\t\treturn 0;\n\t\t}', + ); + }); +}); + +describe('edit-match: escape-normalised strategy', () => { + it('unescapes literal \\n / \\t sequences emitted by the model', () => { + const content = 'console.log("a\tb");\n'; + // Model emitted literal backslash-t instead of a tab character. + const m = findEditMatch(content, 'console.log("a\\tb");'); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe('console.log("a\tb");'); + }); +}); + +describe('edit-match: trimmed-boundary strategy', () => { + it('matches a search with surrounding whitespace against trimmed core', () => { + const content = 'function foo() { return 1; }\n'; + const m = findEditMatch(content, ' foo() { return 1; } '); + expect(m.ok).toBe(true); + expect(content.slice(m.index!, m.index! + m.length!)).toBe('foo() { return 1; }'); + }); +}); + +describe('edit-match: block-anchor fuzzy strategy', () => { + it('locates a 3+ line block via first/last anchors with garbled middle', () => { + const content = [ + 'export function foo() {', + ' const result = computeSomethingVerySpecific(1, 2, 3);', + ' return result;', + '}', + ].join('\n'); + // Search with a typo in the middle line; anchors at top and bottom match. + const oldString = [ + 'export function foo() {', + ' const result = computeSomethingSlightlyDifferent();', + ' return result;', + '}', + ].join('\n'); + const m = findEditMatch(content, oldString); + expect(m.ok).toBe(true); + }); + + it('skips block-anchor when there are fewer than 3 search lines', () => { + // A 2-line search cannot use the anchor strategy; it should fail cleanly + // rather than crash, since no other strategy can match a typo. + const content = 'first line\nsecond line\n'; + const m = findEditMatch(content, 'first line\ntotally different'); + expect(m.ok).toBe(false); + }); +}); + +describe('edit-match: context-aware strategy', () => { + it('matches when >=50% of non-empty middle lines agree', () => { + const content = [ + 'def hello():', + ' print("hi")', + ' x = 1', + ' y = 2', + ' return x + y', + ].join('\n'); + const oldString = [ + 'def hello():', + ' print("hi")', + ' x = 1', + ' z = 99', + ' return x + y', + ].join('\n'); + const m = findEditMatch(content, oldString); + expect(m.ok).toBe(true); + }); +}); + +describe('edit-match: multi-occurrence (replaceOnce semantics)', () => { + it('findEditMatch is ambiguous for a repeated substring', () => { + const content = 'TODO: fix\nTODO: fix\n'; + const m = findEditMatch(content, 'TODO: fix'); + // Without a uniqueness guarantee, the pipeline reports multiple matches. + expect(m.ok).toBe(false); + expect(m.error).toMatch(/multiple matches/i); + }); + + it('replaceOnce throws on ambiguous match', () => { + const content = 'dup\ndup\n'; + expect(() => replaceOnce(content, 'dup', 'x')).toThrow(/multiple matches/i); + }); +}); + +describe('edit-match: applyEdit edge cases', () => { + it('returns a no-change error when newString equals the matched text', () => { + const content = 'hello world\n'; + const { error, content: next } = applyEdit(content, 'hello', 'hello'); + expect(error).toMatch(/no changes/i); + expect(next).toBeUndefined(); + }); + + it('errors on not-found', () => { + const { error, content: next } = applyEdit('foo\n', 'bar', 'baz'); + expect(error).toMatch(/could not find/i); + expect(next).toBeUndefined(); + }); + + it('applies a clean edit and returns the spliced content', () => { + const content = 'alpha\nbeta\ngamma\n'; + const { content: next, error } = applyEdit(content, 'beta', 'BETA'); + expect(error).toBeUndefined(); + expect(next).toBe('alpha\nBETA\ngamma\n'); + }); +}); diff --git a/packages/code/src/tools/edit-match.ts b/packages/code/src/tools/edit-match.ts new file mode 100644 index 0000000..0a2477d --- /dev/null +++ b/packages/code/src/tools/edit-match.ts @@ -0,0 +1,525 @@ +/** + * Whitespace-tolerant find-and-replace matching for the edit tool. + * + * The edit tool's oldString parameter can arrive with whitespace that doesn't + * exactly match the file's bytes — common cases are tabs normalised to spaces + * and \r\n normalised to \n somewhere in the parameter pipeline. Exact substring + * matching then fails even though the visible content is identical. + * + * The approach runs a pipeline of replacers, each yielding candidate substrings + * that are guaranteed to exist verbatim in the file. The driver re-locates each + * candidate in the original content and splices the file's actual bytes — never + * the caller's oldString — so the file's real indentation and line endings are + * what get removed. + */ +export type LineEnding = '\n' | '\r\n'; + +/** Normalize CRLF (and lone CR) down to LF. */ +export function normalizeLineEndings(text: string): string { + return text.replace(/\r\n/g, '\n').replace(/\r/g, '\n'); +} + +/** Detect a file's dominant line ending by first occurrence. */ +export function detectLineEnding(text: string): LineEnding { + const crlfIdx = text.indexOf('\r\n'); + const lfIdx = text.indexOf('\n'); + if (lfIdx === -1) return '\n'; + if (crlfIdx === -1) return '\n'; + return crlfIdx < lfIdx ? '\r\n' : '\n'; +} + +/** Convert a (LF-normalised) string to the given line ending. */ +export function convertToLineEnding(text: string, ending: LineEnding): string { + if (ending === '\n') return text; + return text.replaceAll('\n', '\r\n'); +} + +export interface MatchResult { + /** True when a unique match was located. */ + ok: boolean; + /** Byte offset of the matched span in the ORIGINAL file content. */ + index?: number; + /** Length of the ACTUAL matched span in the file (not the caller's oldString). */ + length?: number; + /** Human-readable reason when ok is false. */ + error?: string; +} + +/** + * A replacer yields candidate substrings that are guaranteed to exist verbatim + * in `content`. Each candidate is re-located by the driver with indexOf so the + * spliced span is always the file's real bytes. + */ +type Replacer = (content: string, find: string) => Generator; + +/** Similarity threshold for block-anchor matching when only one candidate is found. */ +const SINGLE_CANDIDATE_SIMILARITY_THRESHOLD = 0.0; +/** Similarity threshold for block-anchor matching when multiple candidates are found. */ +const MULTIPLE_CANDIDATES_SIMILARITY_THRESHOLD = 0.3; + +/** Standard Levenshtein edit distance. */ +function levenshtein(a: string, b: string): number { + if (a === '' || b === '') { + return Math.max(a.length, b.length); + } + const matrix = Array.from({ length: a.length + 1 }, (_, i) => + Array.from({ length: b.length + 1 }, (_, j) => (i === 0 ? j : j === 0 ? i : 0)), + ); + + for (let i = 1; i <= a.length; i++) { + for (let j = 1; j <= b.length; j++) { + const cost = a[i - 1] === b[j - 1] ? 0 : 1; + matrix[i][j] = Math.min( + matrix[i - 1][j] + 1, + matrix[i][j - 1] + 1, + matrix[i - 1][j - 1] + cost, + ); + } + } + return matrix[a.length][b.length]; +} + +/** Sum the byte offset of the start of line `lineIndex` (newline = 1 char). */ +function lineStartOffset(lines: string[], lineIndex: number): number { + let offset = 0; + for (let k = 0; k < lineIndex; k++) { + offset += lines[k].length + 1; + } + return offset; +} + +/** Span (start, end) for lines [startLine..endLine] inclusive, joined by \n. */ +function lineSpanOffsets(lines: string[], startLine: number, endLine: number): { start: number; end: number } { + const start = lineStartOffset(lines, startLine); + let end = start; + for (let k = startLine; k <= endLine; k++) { + end += lines[k].length; + if (k < endLine) end += 1; + } + return { start, end }; +} + +/** Strategy 0: exact substring. Yields the search string back verbatim. */ +const SimpleReplacer: Replacer = function* (_content, find) { + yield find; +}; + +/** + * Strategy 1: compare line-by-line by trimmed content. Tolerates tabs-vs-spaces + * and mixed indentation. Yields the file's actual span (original bytes). + */ +const LineTrimmedReplacer: Replacer = function* (content, find) { + const originalLines = content.split('\n'); + const searchLines = find.split('\n'); + + if (searchLines[searchLines.length - 1] === '') { + searchLines.pop(); + } + + for (let i = 0; i <= originalLines.length - searchLines.length; i++) { + let matches = true; + for (let j = 0; j < searchLines.length; j++) { + if (originalLines[i + j].trim() !== searchLines[j].trim()) { + matches = false; + break; + } + } + + if (matches) { + const { start, end } = lineSpanOffsets(originalLines, i, i + searchLines.length - 1); + yield content.substring(start, end); + } + } +}; + +/** + * Strategy 2: anchor on the first and last trimmed line, then score middle lines + * by Levenshtein similarity. Rescues matches where the model garbled inner lines. + * Only applies to searches of 3+ lines. + */ +const BlockAnchorReplacer: Replacer = function* (content, find) { + const originalLines = content.split('\n'); + const searchLines = find.split('\n'); + + if (searchLines.length < 3) return; + + if (searchLines[searchLines.length - 1] === '') { + searchLines.pop(); + } + + const firstLineSearch = searchLines[0].trim(); + const lastLineSearch = searchLines[searchLines.length - 1].trim(); + const searchBlockSize = searchLines.length; + + // Collect all candidate positions where both anchors match. + const candidates: Array<{ startLine: number; endLine: number }> = []; + for (let i = 0; i < originalLines.length; i++) { + if (originalLines[i].trim() !== firstLineSearch) continue; + for (let j = i + 2; j < originalLines.length; j++) { + if (originalLines[j].trim() === lastLineSearch) { + candidates.push({ startLine: i, endLine: j }); + break; + } + } + } + + if (candidates.length === 0) return; + + const scoreMiddle = (startLine: number, endLine: number): number => { + const actualBlockSize = endLine - startLine + 1; + const linesToCheck = Math.min(searchBlockSize - 2, actualBlockSize - 2); + if (linesToCheck <= 0) return 1.0; + + let similarity = 0; + for (let j = 1; j < searchBlockSize - 1 && j < actualBlockSize - 1; j++) { + const originalLine = originalLines[startLine + j].trim(); + const searchLine = searchLines[j].trim(); + const maxLen = Math.max(originalLine.length, searchLine.length); + if (maxLen === 0) continue; + similarity += 1 - levenshtein(originalLine, searchLine) / maxLen; + } + return similarity / linesToCheck; + }; + + // Single candidate: relaxed threshold (any non-negative similarity). + if (candidates.length === 1) { + const { startLine, endLine } = candidates[0]; + let similarity = 0; + const actualBlockSize = endLine - startLine + 1; + const linesToCheck = Math.min(searchBlockSize - 2, actualBlockSize - 2); + if (linesToCheck > 0) { + for (let j = 1; j < searchBlockSize - 1 && j < actualBlockSize - 1; j++) { + const originalLine = originalLines[startLine + j].trim(); + const searchLine = searchLines[j].trim(); + const maxLen = Math.max(originalLine.length, searchLine.length); + if (maxLen === 0) continue; + similarity += (1 - levenshtein(originalLine, searchLine) / maxLen) / linesToCheck; + if (similarity >= SINGLE_CANDIDATE_SIMILARITY_THRESHOLD) break; + } + } else { + similarity = 1.0; + } + + if (similarity >= SINGLE_CANDIDATE_SIMILARITY_THRESHOLD) { + const { start, end } = lineSpanOffsets(originalLines, startLine, endLine); + yield content.substring(start, end); + } + return; + } + + // Multiple candidates: pick the highest-scoring one if it clears the bar. + let bestMatch: { startLine: number; endLine: number } | null = null; + let maxSimilarity = -1; + for (const candidate of candidates) { + const similarity = scoreMiddle(candidate.startLine, candidate.endLine); + if (similarity > maxSimilarity) { + maxSimilarity = similarity; + bestMatch = candidate; + } + } + + if (maxSimilarity >= MULTIPLE_CANDIDATES_SIMILARITY_THRESHOLD && bestMatch) { + const { start, end } = lineSpanOffsets(originalLines, bestMatch.startLine, bestMatch.endLine); + yield content.substring(start, end); + } +}; + +/** + * Strategy 3: collapse all whitespace runs to single spaces, then match. Handles + * intra-line whitespace differences for single lines and multi-line blocks. + */ +const WhitespaceNormalizedReplacer: Replacer = function* (content, find) { + const normalizeWhitespace = (text: string) => text.replace(/\s+/g, ' ').trim(); + const normalizedFind = normalizeWhitespace(find); + + const lines = content.split('\n'); + for (let i = 0; i < lines.length; i++) { + const line = lines[i]; + if (normalizeWhitespace(line) === normalizedFind) { + yield line; + } else { + const normalizedLine = normalizeWhitespace(line); + if (normalizedLine.includes(normalizedFind)) { + const words = find.trim().split(/\s+/); + if (words.length > 0) { + const pattern = words.map((word) => word.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('\\s+'); + try { + const regex = new RegExp(pattern); + const match = line.match(regex); + if (match) yield match[0]; + } catch { + // Invalid regex pattern, skip. + } + } + } + } + } + + const findLines = find.split('\n'); + if (findLines.length > 1) { + for (let i = 0; i <= lines.length - findLines.length; i++) { + const block = lines.slice(i, i + findLines.length); + if (normalizeWhitespace(block.join('\n')) === normalizedFind) { + yield block.join('\n'); + } + } + } +}; + +/** + * Strategy 4: strip the minimum common leading indent from both sides, then match. + * Lets a search block match regardless of how far it was shifted left or right. + */ +const IndentationFlexibleReplacer: Replacer = function* (content, find) { + const removeIndentation = (text: string) => { + const lines = text.split('\n'); + const nonEmptyLines = lines.filter((line) => line.trim().length > 0); + if (nonEmptyLines.length === 0) return text; + + const minIndent = Math.min( + ...nonEmptyLines.map((line) => { + const match = line.match(/^(\s*)/); + return match ? match[1].length : 0; + }), + ); + + return lines.map((line) => (line.trim().length === 0 ? line : line.slice(minIndent))).join('\n'); + }; + + const normalizedFind = removeIndentation(find); + const contentLines = content.split('\n'); + const findLines = find.split('\n'); + + for (let i = 0; i <= contentLines.length - findLines.length; i++) { + const block = contentLines.slice(i, i + findLines.length).join('\n'); + if (removeIndentation(block) === normalizedFind) { + yield block; + } + } +}; + +/** + * Strategy 5: unescape literal \n / \t / \\ sequences a model may have emitted, + * then match. Handles the case where escapes weren't interpreted by the pipeline. + */ +const EscapeNormalizedReplacer: Replacer = function* (content, find) { + const unescapeString = (str: string): string => { + return str.replace(/\\(n|t|r|'|"|`|\\|\n|\$)/g, (match, capturedChar: string) => { + switch (capturedChar) { + case 'n': + return '\n'; + case 't': + return '\t'; + case 'r': + return '\r'; + case "'": + return "'"; + case '"': + return '"'; + case '`': + return '`'; + case '\\': + return '\\'; + case '\n': + return '\n'; + case '$': + return '$'; + default: + return match; + } + }); + }; + + const unescapedFind = unescapeString(find); + + if (content.includes(unescapedFind)) { + yield unescapedFind; + } + + const lines = content.split('\n'); + const findLines = unescapedFind.split('\n'); + + for (let i = 0; i <= lines.length - findLines.length; i++) { + const block = lines.slice(i, i + findLines.length).join('\n'); + if (unescapeString(block) === unescapedFind) { + yield block; + } + } +}; + +/** + * Strategy 6: if the search has surrounding whitespace, try matching just the + * trimmed core, or a file block whose trim() equals the trimmed search. + */ +const TrimmedBoundaryReplacer: Replacer = function* (content, find) { + const trimmedFind = find.trim(); + if (trimmedFind === find) return; + + if (content.includes(trimmedFind)) { + yield trimmedFind; + } + + const lines = content.split('\n'); + const findLines = find.split('\n'); + for (let i = 0; i <= lines.length - findLines.length; i++) { + const block = lines.slice(i, i + findLines.length).join('\n'); + if (block.trim() === trimmedFind) { + yield block; + } + } +}; + +/** + * Strategy 7: anchor on the first and last trimmed line of a 3+ line search, + * then accept if >=50% of non-empty middle lines match when trimmed. + */ +const ContextAwareReplacer: Replacer = function* (content, find) { + const findLines = find.split('\n'); + if (findLines.length < 3) return; + + if (findLines[findLines.length - 1] === '') { + findLines.pop(); + } + + const contentLines = content.split('\n'); + const firstLine = findLines[0].trim(); + const lastLine = findLines[findLines.length - 1].trim(); + + for (let i = 0; i < contentLines.length; i++) { + if (contentLines[i].trim() !== firstLine) continue; + + for (let j = i + 2; j < contentLines.length; j++) { + if (contentLines[j].trim() === lastLine) { + const blockLines = contentLines.slice(i, j + 1); + const block = blockLines.join('\n'); + + if (blockLines.length === findLines.length) { + let matchingLines = 0; + let totalNonEmptyLines = 0; + + for (let k = 1; k < blockLines.length - 1; k++) { + const blockLine = blockLines[k].trim(); + const searchLine = findLines[k].trim(); + + if (blockLine.length > 0 || searchLine.length > 0) { + totalNonEmptyLines++; + if (blockLine === searchLine) { + matchingLines++; + } + } + } + + if (totalNonEmptyLines === 0 || matchingLines / totalNonEmptyLines >= 0.5) { + yield block; + break; + } + } + break; + } + } + } +}; + +/** Strategy 8: yields every exact occurrence so the driver can replace them all. */ +const MultiOccurrenceReplacer: Replacer = function* (content, find) { + let startIndex = 0; + while (true) { + const index = content.indexOf(find, startIndex); + if (index === -1) break; + yield find; + startIndex = index + find.length; + } +}; + +const REPLACERS: Replacer[] = [ + SimpleReplacer, + LineTrimmedReplacer, + BlockAnchorReplacer, + WhitespaceNormalizedReplacer, + IndentationFlexibleReplacer, + EscapeNormalizedReplacer, + TrimmedBoundaryReplacer, + ContextAwareReplacer, + MultiOccurrenceReplacer, +]; + +/** + * Locate the byte span of `oldString` in `content` using a tolerant pipeline. + * + * The first strategy that yields a UNIQUE candidate wins. If a strategy yields + * only ambiguous candidates, later strategies are tried. Returns offsets into + * the ORIGINAL content (never the caller's oldString), so splicing removes the + * file's actual bytes. + */ +export function findEditMatch(content: string, oldString: string): MatchResult { + let notFound = true; + + for (const replacer of REPLACERS) { + for (const search of replacer(content, oldString)) { + const index = content.indexOf(search); + if (index === -1) continue; + notFound = false; + const lastIndex = content.lastIndexOf(search); + if (index !== lastIndex) continue; + return { ok: true, index, length: search.length }; + } + } + + if (notFound) { + return { + ok: false, + error: + 'Could not find the specified text in the file. It must match including whitespace and indentation. Try reading the file first and copying the exact bytes.', + }; + } + return { + ok: false, + error: 'Found multiple matches for the specified text. Include more surrounding context to make the match unique.', + }; +} + +/** + * Convenience: locate the match and return the spliced result. + * + * Throws on failure (no match / ambiguous). The caller is expected to normalise + * line endings on `newString` before calling — or use {@link applyEdit} which + * does it for you. + */ +export function replaceOnce(content: string, oldString: string, newString: string): string { + const match = findEditMatch(content, oldString); + if (!match.ok || match.index === undefined || match.length === undefined) { + throw new Error(match.error ?? 'edit match failed'); + } + return content.slice(0, match.index) + newString + content.slice(match.index + match.length); +} + +/** + * Locate the match, splice the replacement, and preserve the file's line ending + * in the inserted text. This is the high-level helper the edit tool uses. + * + * Returns `{ content: newFileContents, error?: string }`. On failure `content` + * is undefined and `error` describes why. + */ +export function applyEdit( + content: string, + oldString: string, + newString: string, +): { content?: string; error?: string } { + const ending = detectLineEnding(content); + const normalizedOld = convertToLineEnding(normalizeLineEndings(oldString), ending); + const normalizedNew = convertToLineEnding(normalizeLineEndings(newString), ending); + + const match = findEditMatch(content, normalizedOld); + if (!match.ok || match.index === undefined || match.length === undefined) { + return { error: match.error }; + } + + const next = + content.slice(0, match.index) + normalizedNew + content.slice(match.index + match.length); + + if (next === content) { + return { error: "No changes made - the replacement didn't modify the file." }; + } + + return { content: next }; +} diff --git a/packages/code/src/tools/edit.ts b/packages/code/src/tools/edit.ts index d952fd1..a589632 100644 --- a/packages/code/src/tools/edit.ts +++ b/packages/code/src/tools/edit.ts @@ -2,6 +2,7 @@ import { z } from 'zod'; import { createTwoFilesPatch } from 'diff'; import type { SpectraTool } from './types.js'; import { errorResult, textResult } from './utils.js'; +import { applyEdit } from './edit-match.js'; import { readFileSync, writeFileSync, existsSync } from 'fs'; import { resolve, relative, basename } from 'path'; @@ -9,10 +10,10 @@ export const editTool: SpectraTool = { name: 'edit', capabilities: { reads: false, writes: true }, description: `Edit a file by finding and replacing text. -The tool finds the exact old string in the file and replaces it with the new string. +The tool finds oldString in the file and replaces it with newString. +Matching is whitespace-tolerant: tabs/spaces and \\r\\n/\\n differences are accepted. For best results: - Include enough surrounding context in the old string for a unique match -- Use exact text including whitespace - If the old string appears multiple times, include more context to disambiguate Prefer the write tool for large or new files.`, displayName: (args: { path: string }) => relative(process.cwd(), resolve(args.path)), @@ -28,15 +29,17 @@ Prefer the write tool for large or new files.`, } const content = readFileSync(resolved, 'utf-8'); - if (!content.includes(oldString)) { - return errorResult(`Could not find the specified text in ${relative(process.cwd(), resolved)}. -The text may have different whitespace or formatting. Try reading the file first.`); - } - const newContent = content.replace(oldString, newString); - if (newContent === content) { - return errorResult("No changes made - the replacement didn't modify the file."); + // applyEdit runs a tolerant match pipeline (exact -> line-trimmed -> + // whitespace-normalised -> indentation-flexible -> fuzzy anchors -> ...) + // and splices the file's actual bytes, so tab/CRLF normalisation in the + // oldString parameter no longer causes false "not found" failures. It + // also normalises newString to the file's detected line ending. + const result = applyEdit(content, oldString, newString); + if (result.error) { + return errorResult(`${result.error} [${relative(process.cwd(), resolved)}]`); } + const newContent = result.content!; writeFileSync(resolved, newContent, 'utf-8'); diff --git a/packages/code/src/tui/components/slash-autocomplete.tsx b/packages/code/src/tui/components/slash-autocomplete.tsx index dc9ce2c..a808534 100644 --- a/packages/code/src/tui/components/slash-autocomplete.tsx +++ b/packages/code/src/tui/components/slash-autocomplete.tsx @@ -1,4 +1,4 @@ -import { useMemo } from 'react'; +import { useRef, useEffect, useMemo } from 'react'; import { c } from '../theme.js'; import type { CmdItem } from './command-palette.js'; @@ -14,32 +14,41 @@ export interface SlashAutocompleteProps { promptWidth?: number; } -const MAX_ITEMS = 8; +// Visible-row budget for the list window. Unlike the command palette (a centered +// modal with lots of vertical room), this menu floats directly above the prompt +// and grows upward — so the window must be clamped to the space actually +// available above the prompt, not a flat cap. The scrollbox then handles any +// overflow beyond the window. +const MAX_LIST_ROWS = 8; +const MIN_LIST_ROWS = 3; +const MENU_CHROME = 3; // header + divider + footer rows surrounding the list export function SlashAutocomplete(props: SlashAutocompleteProps) { const { query, selected, items, termWidth, termHeight, route, promptTop, promptLeft, promptWidth } = props; - - const count = Math.min(items.length, MAX_ITEMS); - if (count === 0) return null; - - const mw = Math.min(50, termWidth - 8); - const mh = count + 4; + const scrollRef = useRef(null); const isChat = route === 'chat'; - const menuLeft = promptLeft ?? 3; - const menuWidth = promptWidth ?? mw; + // Vertical space above the prompt that the menu is allowed to occupy. + // On chat route the menu is anchored above the prompt; on home route there is + // no prompt so we fall back to the full terminal height. + const spaceAbove = isChat ? Math.max(0, (promptTop ?? termHeight) - MENU_CHROME - 1) : termHeight; + const listH = Math.max(MIN_LIST_ROWS, Math.min(MAX_LIST_ROWS, items.length, spaceAbove)); + const mh = listH + MENU_CHROME; + const menuLeft = promptLeft ?? 3; + const menuWidth = promptWidth ?? Math.min(50, termWidth - 8); const menuTop = isChat ? (promptTop ?? termHeight) - mh - 1 : Math.floor(termHeight / 2) - mh - 2; const rows = useMemo(() => { const r: any[] = []; - for (let i = 0; i < count; i++) { + for (let i = 0; i < items.length; i++) { const item = items[i]; const isSel = i === selected; r.push( { + if (!scrollRef.current) return; + const sel = items[selected]; + if (!sel) return; + const el = scrollRef.current; + if (typeof el.scrollChildIntoView === 'function') { + el.scrollChildIntoView(sel.id); + return; + } + const child = el.getChildren?.()?.find?.((ch: any) => ch.id === sel.id); + if (child) { + const y = child.y - (el.y || 0); + if (y >= (el.height || listH)) el.scrollBy?.(y - (el.height || listH) + 1); + if (y < 0) el.scrollBy?.(y); + } + }, [selected, items, listH]); return ( {'─'.repeat(menuWidth - 2)} - {rows} - {items.length > MAX_ITEMS && ( - - ...{items.length - MAX_ITEMS} more - - )} + { + scrollRef.current = r; + }} + maxHeight={listH} + scrollY={true} + scrollbarOptions={{ visible: false }} + > + {rows} + {'\u2191\u2193'} navigate esc dismiss