feat: VS Code extension — Mnemonic for VS Code

[jkbennitt]

Overview

Build a VS Code extension that surfaces mnemonic's semantic memory system directly in the editor. The daemon and REST API already exist -- the extension is a smart client that connects to localhost:9999.

Goal: Developers should not have to leave the editor or open the dashboard. The right memory should appear at the right moment, automatically.

---

Architecture

VS Code Extension (TypeScript)
    | REST API + WebSocket (localhost:9999)
Mnemonic Daemon (already running)
    |
SQLite + LLM

Zero new infrastructure. Extension is a pure REST/WS client against the existing API.

---

Prerequisite: Daemon API Work

Before the extension MVP ships, the REST API needs parity with what MCP tools already expose.

Must-have for Phase 1 (~2.5 hours)

Endpoint	What	Why	Effort
Wire project, source, type, min_salience through POST /api/v1/query	Retrieval agent already supports these; REST handler does not expose them	Sidebar needs project filtering, CodeLens needs source filtering	~30 min
GET /api/v1/memories/by-file?path=...	Extract concepts from path server-side, return relevant memories	Core Related Memories sidebar feature -- cleaner than client-side concept extraction	~1 hour
POST /api/v1/query/batch	Array of queries in, array of results out	CodeLens performance -- one request per file, not per symbol	~1 hour

Must-have for Phase 2 (~3 hours)

Endpoint	What	Why
POST /api/v1/memories/{id}/amend	Update memory content in place	Memory editing in sidebar/WebView
POST /api/v1/memories/{id}/archive	Archive (forget) a memory	Memory management
GET /api/v1/sessions/{id}/summary	Session summary	Welcome back notification
REST equivalents for recall_project, get_context, session_summary	Expose MCP-only tools via REST	Sidebar project context, proactive suggestions

Existing API assets the extension can use today

• POST /api/v1/query -- semantic search (limited params, fixed in prereq)
• POST /api/v1/remember -- create memories
• POST /api/v1/feedback -- rate recall quality
• GET /api/v1/health -- daemon status + version + memory count
• GET /api/v1/memories/{id}/context -- rich memory detail
• GET /api/v1/activity -- daemon recent concept activity tracker
• GET /api/v1/system/update-check -- version check
• WS /ws -- real-time events (16 event types)
• SearchByEntity(name, entityType, limit) -- store method for symbol-level CodeLens

---

Features (prioritized)

Phase 1 -- MVP (ship first)

1. Memory Sidebar

TreeView panel in the Explorer sidebar:

• Related Memories -- updates based on the active file via GET /api/v1/memories/by-file?path=... (server-side concept extraction)
• Project Context -- pinned section showing active patterns, recent decisions/insights, current episodes
• Powered by project-filtered queries once prereq wiring is done

2. Quick Recall (Cmd+Shift+M)

QuickPick dialog for instant semantic search:

• Natural language input via POST /api/v1/query (with project filtering)
• Select a result to copy to clipboard or insert as comment

3. Status Bar

Minimal status bar item showing memory count and connection status.

• Click to open sidebar
• Daemon connection status via GET /api/v1/health
• Graceful degradation: empty/offline states, stale cache where possible

---

Phase 2 -- Real-time + Terminal

4. WebSocket Integration

Subscribe to /ws for real-time events:

• memory_encoded -> invalidate sidebar cache for affected file/concepts
• consolidation_completed -> update project context section
• pattern_discovered -> notify user of new patterns
• Foundation for session awareness and CodeLens cache invalidation

5. Session Awareness

Driven by WebSocket events (not polling). Uses consolidation_completed events + GET /api/v1/sessions/{id}/summary.

6. Terminal Error Memory

Pattern-match terminal errors against mnemonic error-type memories via POST /api/v1/query with type: error filtering. High value, straightforward once query params are wired.

7. Remember from Editor

Right-click context menu on selected code:

• Remember This Decision -> POST /api/v1/remember with type: decision
• Remember This Error -> type: error
• What Do I Know About This? -> recall scoped to selection
• Remember This Insight -> type: insight

---

Phase 3 -- CodeLens + Detail Views

8. Inline Memory Hints (CodeLens)

File-level batch query architecture, not per-symbol:

1. On file open, extract file path + top-level symbols via DocumentSymbolProvider
2. Single POST /api/v1/query/batch request with all symbols
3. Client-side distribute results to CodeLens positions
4. Cache invalidated via WebSocket memory_encoded events -- no polling

9. Memory Detail WebView

Click a memory in the sidebar to open a rich WebView panel showing full context via GET /api/v1/memories/{id}/context, associated memories, concepts, tags, salience score, and edit/forget controls.

10. Settings UI

• mnemonic.endpoint -- daemon URL (default: http://127.0.0.1:9999)
• mnemonic.token -- API auth token
• mnemonic.inlineHints.enabled -- toggle CodeLens
• mnemonic.inlineHints.minSalience -- threshold (0.0-1.0)
• mnemonic.sidebar.autoRefresh -- refresh interval (default: 30s)
• mnemonic.notifications.sessionStart -- toggle welcome-back notifications

---

What NOT to build

• Do not duplicate the dashboard -- web UI exists for deep exploration
• Do not add a chat interface -- Claude Code + MCP tools handles this
• Do not require auth for localhost
• Do not poll for real-time data -- use the WebSocket

Concept Extraction Strategy

Server-side via GET /api/v1/memories/by-file -- daemon has concepts.FromPath() in Go. Extension sends path, server extracts concepts and queries.

Error States / Graceful Degradation

• Daemon not running: status bar offline, sidebar/CodeLens empty with Start daemon action
• LLM unavailable: features still work (FTS search, cached embeddings)
• Slow responses: sidebar loading state, CodeLens uses stale cache

Repo Structure

TBD -- monorepo (vscode/ dir) or separate repo. Separate repo likely better for independent release cadence.

Tech Stack

• VS Code Extension API (TypeScript)
• REST client + WebSocket against localhost:9999
• TreeView for sidebar, CodeLens for inline hints, QuickPick for search
• WebView only for rich memory detail (Phase 3)

Publishing

• Publisher: AppSprout (appsprout or appsprout-dev)
• Marketplace: https://marketplace.visualstudio.com
• Requires Azure DevOps PAT with Marketplace Publish scope

Monetization Notes

The extension is free -- it drives adoption of the core product. Pro features (cloud sync, team memory) are what we charge for. The extension makes the free tier stickier by reducing friction to zero.

[CalebisGross]

Deep Review

@jkbennitt — the feature design is strong. Phased approach, "what not to build" discipline, focus on surfacing memory at the right moment. But there's a meaningful gap between what the issue assumes the REST API exposes and what it actually does today. Here's the full audit.

---

API Gaps That Block the Extension

POST /api/v1/query is weaker than the MCP recall tool. The REST handler only accepts 4 fields:

{ "query": string, "limit": int, "synthesize": bool, "include_reasoning": bool }

But the retrieval agent internally supports a much richer QueryRequest — Project, Source, State, Type, MinSalience, TimeFrom/To, ExcludeConcepts. The extension needs Project filtering at minimum (sidebar: "memories for this project"), and Source filtering for CodeLens (show only filesystem-origin memories). These fields exist in the retrieval agent — they're just not wired through the REST handler.

10 MCP tools have no REST equivalent:

Tool	Extension needs it for	Add difficulty
recall_project	Sidebar "Project Context" section	Medium
session_summary	Phase 2 "Welcome back" notification	Medium
get_context	Proactive suggestions from activity	Medium
batch_recall	Performance — sidebar + CodeLens need multiple queries	Low
amend	Phase 3 memory editing in WebView	Low
recall_session	Session awareness	Low
recall_timeline	Time-based browsing	Low
forget	Phase 3 memory management	Low
exclude_path / list_exclusions	Settings integration	Low

Phase 1 MVP can work with what exists today if we wire project/source/type filtering through /api/v1/query. Phase 2 and 3 require 5-6 new REST routes.

---

The "Related Memories for This File" Problem

This is the killer feature and the hardest to get right. How it actually has to work today:

1. Extract concepts from file path — e.g., /src/auth/token.go → ["auth", "token"]
2. Send as query string to POST /api/v1/query
3. Retrieval agent does FTS + embedding search + spread activation
4. Return ranked results

This works but it's indirect. Raw memories store file paths in metadata.path, but no REST route queries by path. A dedicated GET /api/v1/memories/by-file?path=/src/auth/token.go that does concept extraction server-side would be cleaner and more accurate.

Also worth noting: SearchByEntity(name, entityType, limit) exists in the store and searches structured concept sets for entities by name. If the extension extracts function names via VS Code's symbol provider, it could find memories referencing specific functions. Not mentioned in the issue but would make CodeLens dramatically better.

---

CodeLens Performance

"Query per top-level symbol" is the wrong architecture. A file with 20 exported functions = 20 HTTP requests on file open. Better approaches:

1. File-level batch query — one request with the file path + extracted symbols, return all relevant memories, client-side distribute to CodeLens positions
2. WebSocket subscription — subscribe to memory_encoded events, invalidate cache only when new memories match current file concepts. No polling.
3. batch_recall REST equivalent — send all symbol queries in one request

---

The WebSocket Is Underutilized

The issue doesn't mention the WebSocket at /ws. It broadcasts 16 event types including:

• memory_encoded — new memory ready
• consolidation_completed — includes pattern/abstraction counts
• pattern_discovered — new pattern detected
• watcher_event — real-time file activity

The "Session Awareness" feature (Phase 2) proposes polling for session summaries. The WebSocket already pushes consolidation_completed with exactly the data that notification needs.

---

Missing from the Issue

1. Auth token — config.yaml has an optional token field for API auth. Extension settings should include mnemonic.token.

2. Repo structure — monorepo (vscode/ dir) or separate repo? Affects CI, release cadence, version coupling.

3. GET /api/v1/memories/{id}/context — returns rich context (memory + resolution + concept set + attributes + episode). Perfect for Phase 3 "Memory Detail WebView" but not mentioned.

4. GET /api/v1/activity — returns daemon's recent concept activity tracker. Extension could highlight which files/concepts are "hot" right now.

5. Concept extraction — extension needs to extract concepts from file paths. Daemon has concepts.FromPath() in Go. Should the extension reimplement in TypeScript, or should the API accept a path and extract server-side?

6. Error states — what happens when daemon isn't running? Status bar shows "offline" but sidebar, CodeLens, and quick recall all need graceful degradation strategy. Stale cache? Empty state? "Start daemon" action?

---

Phase Ordering Suggestions

Phase 1 is right. Sidebar + Quick Recall + Status Bar. Ship it.

Phase 2 should lead with the WebSocket, not CodeLens. CodeLens is flashiest but has the hardest performance/relevance problems. Session awareness via WebSocket events is easier, higher confidence, and validates the real-time connection pattern CodeLens will need.

"Terminal Error Memory" (Phase 3) is undervalued. Consider moving to Phase 2. VS Code has terminal output access. Pattern-matching errors against mnemonic's error-type memories is high value — just a POST /api/v1/query with type filtering once we wire it through.

---

Prerequisite Work on Daemon Side

Before the extension MVP ships:

1. Wire project, source, type, min_salience through POST /api/v1/query — retrieval agent already supports these, handler just doesn't expose them. ~30 min.
2. Add GET /api/v1/memories/by-file?path=... — extract concepts from path server-side, return relevant memories. ~1 hour.
3. Add POST /api/v1/query/batch — array of queries in, array of results out. Needed for CodeLens. ~1 hour.

For Phase 2, add:
4. POST /api/v1/memories/{id}/amend
5. POST /api/v1/memories/{id}/archive (forget)
6. GET /api/v1/sessions/{id}/summary

[jkbennitt]

Updated the issue body to incorporate your feedback — added the prerequisite API work section (query param wiring, by-file endpoint, batch query), WebSocket-first architecture for Phase 2, reordered phases (Terminal Error Memory moved up, CodeLens moved to Phase 3 with batch architecture), added auth token to settings, concept extraction strategy, error states/graceful degradation, and documented all existing API assets the extension can use today.

[CalebisGross]

@jkbennitt updated spec looks great. A few things to tighten up:

1. Wrong endpoint name — "Remember from Editor" (Phase 2, item 7) references POST /api/v1/remember. That doesn't exist — it's POST /api/v1/memories. Someone will copy-paste from the spec.

2. by-file endpoint should accept symbols too. If the extension is already extracting top-level symbols via DocumentSymbolProvider for CodeLens, send them in the same request. One endpoint that takes path + optional symbols[] and returns memories grouped by match type (file-level vs symbol-level) serves both sidebar and CodeLens from a single call.

3. No debounce strategy for file switching. onDidChangeActiveTextEditor fires on every tab switch. Without a 300-500ms debounce, the sidebar hammers the API during rapid tab navigation.

4. WebSocket reconnection strategy. Laptop sleep, daemon restart, network hiccup — the WS will drop. Need exponential backoff with jitter, degrade to polling /api/v1/health until reconnected. Important since Phase 2 makes WS the backbone.

5. No testing strategy. How do you test an extension that depends on a running daemon? Mock API? Integration tests against a real daemon? Matters for CI — can't easily run the daemon in GitHub Actions.

6. Cache specifics are vague. "Stale cache where possible" — but what eviction? LRU per file? TTL? 50 open files = 50 cached results. Probably fine, worth a sentence on bounds.

None of these are blockers. Spec is solid.

[CalebisGross]

@jkbennitt — spec is solid and the API prerequisites are mapped. When you're ready to pick this up, the daemon side is in good shape:

• REST API is stable at /api/v1/ (memories, recall, status, activity, analytics)
• MCP server has 24 tools — the extension could potentially use MCP directly instead of REST for richer integration
• New as of today: every encoding now has EPR/FR/TED quality metrics (Continuous learning: encoding model that improves from operational experience #391 Phase A) — the extension could surface encoding quality in the sidebar or diagnostics panel

No rush, just flagging that the backend is ready when you are.

[jkbennitt]

Phase 1 MVP — Implementation Complete

Branch: feat/vscode-extension (4659a11)

Daemon API Prerequisites (Go)

All three prereqs from the spec are done:

Endpoint	What	Status
POST /api/v1/query	Added project, source, type/types, state, min_salience, include_patterns, include_abstractions, time_from, time_to — all wired through to the retrieval agent's existing QueryRequest fields	✅
GET /api/v1/memories/by-file	Accepts path + optional symbols[]. Server-side concept extraction via concepts.FromPath(), symbol search via SearchByEntity(), deduplication between file/symbol results	✅
POST /api/v1/query/batch	Max 10 queries, parallel goroutine execution, per-query error handling, event publishing + feedback loop	✅

VS Code Extension (vscode/)

Phase 1 features:

• Memory Sidebar — Two TreeViews: "Related Memories" (file-scoped via by-file endpoint, LRU cache 50 entries/60s TTL, 400ms debounce on tab switch, grouped by memory type) + "Project Context" (workspace-scoped, auto-refresh every 60s)
• Quick Recall (Ctrl+Shift+M) — QuickPick with 200ms debounced search, copy/insert/detail actions, automatic feedback submission
• Status Bar — Shows $(brain) 1,234 connected / $(brain) Offline disconnected, click to focus sidebar
• Connection Monitor — Health polling with exponential backoff (30s → 120s cap), state machine with event emitter
• Settings — mnemonic.endpoint, mnemonic.token, mnemonic.healthPollIntervalMs, mnemonic.fileChangeDebounceMs, mnemonic.maxRelatedMemories, hot-reload on config change
• Error states — Graceful degradation in sidebar/status bar/quick recall when daemon is offline

Review feedback addressed

• ✅ by-file accepts symbols (Caleb's comment)
• ✅ 400ms debounce for file switching (within 300-500ms spec)
• ✅ LRU cache with TTL and size bounds documented
• ✅ Auth token in settings
• ✅ Correct endpoint: POST /api/v1/memories (not /api/v1/remember)

Build verification

• go vet ./... — clean
• go test ./internal/api/... — all pass
• npx tsc --noEmit — zero errors
• npm run build — clean esbuild bundle

Not included (future phases)

• WebSocket integration (Phase 2)
• Terminal error memory (Phase 2)
• Remember from Editor context menu (Phase 2)
• CodeLens (Phase 3)
• Memory detail WebView (Phase 3)
• POST /api/v1/memories/{id}/amend, POST /api/v1/memories/{id}/archive, GET /api/v1/sessions/{id}/summary (Phase 2 API prereqs)

[jkbennitt]

Phase 2 — Real-time + Terminal: Implementation Complete

Commit: e067c2e

Daemon API Endpoints (Go)

Endpoint	What	Status
POST /api/v1/memories/{id}/amend	Update memory content in place, publish MemoryAmended event, preserve associations	✅
POST /api/v1/memories/{id}/archive	Set memory state to "archived" (non-destructive)	✅
GET /api/v1/sessions/{id}/summary	Session breakdown by type, recent items, episode context. "current" resolves to latest session	✅

VS Code Extension Features

• WebSocket client — persistent connection to ws://localhost:9999/ws with exponential backoff + jitter (1s base → 60s cap), coordinated with ConnectionMonitor lifecycle, 45s stale-connection detection
• Real-time cache invalidation — memory_encoded / memory_amended → clear sidebar cache + re-fetch; consolidation_completed → refresh project context; pattern_discovered → user notification
• Session awareness — "Welcome back" notification on activate showing decision/error/insight breakdown
• Terminal error monitor — pattern-matches terminal output against common error signatures, queries Mnemonic for related error memories, shows "Found N related memories" notification. Runtime-detects onDidWriteTerminalData (VS Code 1.93+), gracefully disabled on older versions
• Remember from Editor — right-click context submenu: "Remember This Decision/Error/Insight" + "What Do I Know About This?" Selected text + file context sent to POST /api/v1/memories with source: "vscode"
• Status bar live indicator — $(brain) 1,234 $(pulse) when WebSocket connected, $(brain) 1,234 when polling only
• New settings — mnemonic.websocketEnabled, mnemonic.terminalErrorMonitoring

Build verification

• go build / go vet ./... / go test ./internal/api/... — all clean
• npx tsc --noEmit — zero errors
• npm run build — clean esbuild bundle

Remaining: Phase 3 (CodeLens + Detail Views)

• Inline memory hints via CodeLens with batch query
• Memory detail WebView (rich rendered panel vs current markdown doc)
• Full settings UI (inlineHints.enabled, inlineHints.minSalience, sidebar.autoRefresh, notifications.sessionStart)

[jkbennitt]

Correction: The issue body references POST /api/v1/remember in the "Remember from Editor" section (Phase 2, item 7) and in "Existing API assets." The correct endpoint is POST /api/v1/memories. The extension implementation uses the correct endpoint. Flagged by Caleb in his second review comment.

[jkbennitt]

Phase 3 Plan — CodeLens + Detail Views

@CalebisGross — ready for review before implementing the final phase.

Feature 8: Inline Memory Hints (CodeLens)

Architecture: Uses getMemoriesByFile(path, symbols) — one request per file, not per symbol. Server does concept extraction + symbol matching, returns file_results + symbol_results grouped by symbol name. Avoids the 10-query batch limit.

Flow:

1. provideCodeLenses(document) extracts symbols via vscode.executeDocumentSymbolProvider (Function, Method, Class, Interface)
2. Single getMemoriesByFile(filePath, symbolNames) call (LRU cached)
3. For each symbol with matches: $(brain) N memories CodeLens at symbol's line
4. File-level results get a CodeLens at line 0
5. No CodeLens for symbols with zero matches
6. Click → QuickPick of matched memories → select opens detail

Cache invalidation: WebSocket memory_encoded event → clear cache + fire _onDidChangeCodeLenses. Same pattern as sidebar.

Settings: mnemonic.inlineHints.enabled (bool, default true), mnemonic.inlineHints.minSalience (float, default 0.0)

Feature 9: Memory Detail WebView

Replaces the current TextDocumentContentProvider (markdown) with an interactive WebView panel.

Singleton panel — reuses existing panel when already open. HTML rendered with VS Code theme CSS variables for automatic light/dark support. Nonce-based CSP.

Sections: Header (type/source/salience/state), Content, Resolution (gist + narrative), Concepts (pill badges), Episode info, Action bar: Edit Content, Archive, Rate Helpful/Irrelevant

Actions via VS Code webview messaging:

• Edit → showInputBox() → amendMemory() → panel refresh
• Archive → confirmation dialog → archiveMemory()
• Feedback → submitFeedback() with query_id (when available)

Feature 10: Settings

Setting	Type	Default
mnemonic.inlineHints.enabled	boolean	true
mnemonic.inlineHints.minSalience	number	0.0
mnemonic.sidebar.autoRefresh	number (sec)	30
mnemonic.notifications.sessionStart	boolean	true

Wired into existing ProjectContextProvider (refresh interval) and SessionAwareness (notification toggle).

Files

File	Action
vscode/src/providers/memoryCodeLensProvider.ts	Create
vscode/src/providers/memoryDetailPanel.ts	Create
vscode/src/commands/openMemoryDetail.ts	Rewrite (TextProvider → WebView)
vscode/src/extension.ts	Modify (wire CodeLens + settings)
vscode/package.json	Modify (4 settings + 1 command)
vscode/src/components/sessionAwareness.ts	Modify (notification toggle)

No new daemon work needed

All APIs exist: getMemoriesByFile, getMemoryContext, amendMemory, archiveMemory, submitFeedback, batchQuery.

Open questions

1. Should CodeLens register for all file types ({ scheme: 'file' }) or a configurable language list setting?
2. Should the "Edit Content" action use showInputBox (single-line, simpler) or a textarea in the WebView itself (multiline, more complex)?
3. Should feedback buttons be visible when query_id isn't available (memory opened from sidebar, not from a recall)?