dev: wait for port free in dev-reload; http: append startup audit line when writing pid#30
Open
mchzimm wants to merge 29 commits into
Open
dev: wait for port free in dev-reload; http: append startup audit line when writing pid#30mchzimm wants to merge 29 commits into
mchzimm wants to merge 29 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…project-scoped nodes/queries (project_root, project_branch, memory_scope)
…mory-scope recording and retrieval
…s, providers); add graph stop handle to avoid leaks
…non-blocking DB saves; disable MEMORY.md auto writes via env guard
…t-tool, and query; add auto-memory module
…s and files on learn
… + merge-upsert; migrate canonical_id and merge duplicates; expose /api/graph/edges; fallback heatmap/hook-log to graph/session_log; background init on server start
…ram ui (PID-safe)
…rver via engram ui
… and explicit 'remember' auto-learn in UserPromptSubmit
…; inject SessionStart on resume; persist explicit remembers
…e when writing pid
5 tasks
There was a problem hiding this comment.
Pull request overview
This PR substantially expands engramx’s runtime/data model to support a single global DB with per-project scoping and “memory scopes” (project/global/entity), adds aggressive auto-memory ingestion paths, updates the dashboard to switch between scopes/projects, and also introduces a dev-reload helper script plus server startup auditing.
Changes:
- Add global DB + project scoping/memory scopes, plus canonical-id support for deterministic dedupe/merging.
- Add auto-memory ingestion (SessionStart, PostToolUse, explicit “remember…” prompts, query results) and new hook surface(s).
- Improve dev workflow (nodemon-based dev reload script) and expand the dashboard (scope/project selector, graph edges, token history).
Reviewed changes
Copilot reviewed 60 out of 61 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/memory-scope.test.ts | Adds coverage for storing/retrieving memories across scopes. |
| tests/intercept/spike-regression.test.ts | Updates expected intercept text prefixes to engramx. |
| tests/intercept/render-file-structure.test.ts | Updates expected structure summary header prefix. |
| tests/intercept/handlers/user-prompt.test.ts | Updates expected UserPromptSubmit context prefix. |
| tests/intercept/handlers/session-start.test.ts | Updates expected SessionStart brief prefix. |
| tests/intercept/handlers/read.test.ts | Updates expected Read deny reason prefix. |
| tests/intercept/handlers/pre-compact.test.ts | Updates expected PreCompact brief prefix. |
| tests/intercept/handlers/cwd-changed.test.ts | Updates expected cwd-changed context prefix. |
| tests/intercept/handlers/bash.test.ts | Updates expected bash deny reason prefix. |
| tests/intercept/formatter.test.ts | Updates expected formatter headers/prefixes. |
| tests/intercept/cli-intercept.test.ts | Updates expected CLI intercept output prefixes. |
| tests/fixtures/hook-payloads/README.md | Updates documented hook payload examples to engramx. |
| src/watcher.ts | Passes projectRoot/memoryScope through store operations; updates user-facing log strings. |
| src/tuner/index.ts | Renames built-in provider ids to engramx:*. |
| src/tuner/config.ts | Adds auto-memory config fields + defaults. |
| src/setup/wizard.ts | Updates IDE adapter instructions for engramx naming. |
| src/server/ui.ts | Adds dashboard scope/project selector, graph legend, tokens refresh + time-series, scoped API calls, SSE throttling. |
| src/server/ui-graph.ts | Adds edge rendering, scope-based node styling/shaping, consolidation logic, keyboard zoom, and stop-handle. |
| src/server/learn-ws.ts | Adds a minimal WebSocket ingestion protocol for learn(). |
| src/server/index.ts | Updates server startup stderr messages to engramx. |
| src/server/http.ts | Adds dashboard scope API, graph edges API, websocket upgrade path, and other scope-aware dashboard behaviors. |
| src/providers/types.ts | Updates provider docstrings and priority list to engramx:*. |
| src/providers/resolver.ts | Updates headers/provider ids and makes cache warming project-scoped. |
| src/providers/lsp.ts | Renames provider id to engramx:lsp. |
| src/providers/engram-structure.ts | Renames provider id/header docs to engramx:structure. |
| src/providers/engram-mistakes.ts | Renames provider id and scopes getNodesByFile calls. |
| src/providers/engram-git.ts | Renames provider id to engramx:git. |
| src/providers/ast.ts | Renames provider id/docs to engramx:ast. |
| src/miners/session-miner.ts | Switches session-mined node ids to canonical ids. |
| src/miners/pdf-miner.ts | Adds best-effort PDF text extraction helper (pdf-parse/pdftotext). |
| src/miners/linking-helpers.ts | Adds keyword/path/command extraction to infer linking candidates from free text. |
| src/miners/git-miner.ts | Updates timeout log prefix to engramx. |
| src/miners/conclusions-miner.ts | Adds conclusion + fragment node generation and rationale edges using canonical ids. |
| src/intercept/memory-md.ts | Renames markers to engramx:* and adds write-to-arbitrary-path helper + opt-out env var. |
| src/intercept/handlers/user-prompt.ts | Adds explicit remember→learn path and updates injected header prefix. |
| src/intercept/handlers/session-start.ts | Injects brief on resume; triggers auto-memory on session start; updates header prefix. |
| src/intercept/handlers/read.ts | Updates provider ids and makes node/edge queries project-scoped. |
| src/intercept/handlers/pre-compact.ts | Updates brief header prefix. |
| src/intercept/handlers/post-tool.ts | Triggers auto-memory ingestion on Read/Edit/Write tool events. |
| src/intercept/handlers/mistake-guard.ts | Makes mistake queries project-scoped; updates provider reference string. |
| src/intercept/handlers/cwd-changed.ts | Updates injected header prefix. |
| src/intercept/handlers/assistant-message.ts | Adds opt-in hook to persist assistant content as memory (fire-and-forget). |
| src/intercept/dispatch.ts | Adds AssistantMessage dispatch; improves cost path resolution; records session token stats. |
| src/intercept/auto-memory.ts | Adds aggressive auto-memory ingestion + per-scope dedupe hash stats. |
| src/intelligence/token-tracker.ts | Adds project-scoped token stats and session time-series log. |
| src/integrations/pi.ts | Adds a minimal Node 20+ HTTP client for /hook, /learn, /query. |
| src/hooks.ts | Updates hook output prefix to engramx. |
| src/graph/store.ts | Adds project scoping columns, canonical-id merge upsert, async save, and scoped query methods. |
| src/graph/query.ts | Adds projectRoot scoping to graph traversal + updates structure summary header prefix. |
| src/graph/canonical.ts | Adds deterministic canonical id generation helper. |
| src/doctor/report.ts | Updates AST reinstall guidance to engramx. |
| src/db/migrate.ts | Bumps schema version + adds migrations for project scoping and canonical ids. |
| src/core.ts | Switches to a single global DB; adds project stat key helper; adds scoped query; expands learn() with conclusions/linking; adds token instrumentation. |
| src/cli.ts | Adds learn --scope; expands memory-sync to support scopes and new output locations; updates error prefixes. |
| src/autogen.ts | Renames autogen markers and error prefix to engramx. |
| scripts/dev-reload.sh | Adds dev reload helper that kills listeners, rebuilds, and waits for port to free before restart. |
| package.json | Adds dev:reload and nodemon devDependency. |
| package-lock.json | Locks nodemon and transitive deps. |
| examples/pi-client.js | Adds an example client for the HTTP server /hook and /learn. |
| docs/operations/dev-reload-port-audit.md | Documents dev-reload port waiting and startup audit behavior + test steps. |
| docs/integrations/pi.md | Documents external integration via the local HTTP server and example client. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+114
to
+121
| // If export fails for any reason, fall back to a best-effort | ||
| // synchronous write attempt (rare). Swallow errors to avoid | ||
| // crashing callers. | ||
| try { | ||
| writeFileSync(this.dbPath, Buffer.from([])); | ||
| } catch { | ||
| /* swallow */ | ||
| } |
Comment on lines
175
to
200
| @@ -121,14 +193,134 @@ export class GraphStore { | |||
| JSON.stringify(node.metadata), | |||
| node.validUntil ?? null, | |||
| node.invalidatedByCommit ?? null, | |||
| projectRoot, | |||
| projectBranch, | |||
| memoryScope, | |||
| ] | |||
| ); | |||
| const store = await getStore(root); | ||
| try { | ||
| const allNodes = store.getAllNodes(); | ||
| const allNodes = store.getAllNodes(projectRoot); |
Comment on lines
575
to
+600
| export async function learn( | ||
| projectRoot: string, | ||
| text: string, | ||
| sourceLabel = "manual" | ||
| sourceLabel = "manual", | ||
| memoryScope: string = "project" | ||
| ): Promise<{ nodesAdded: number }> { | ||
| const { nodes, edges } = learnFromSession(text, sourceLabel); | ||
| if (nodes.length === 0 && edges.length === 0) return { nodesAdded: 0 }; | ||
| // Primary session mining (decisions/mistakes/patterns) | ||
| const sessionResult = learnFromSession(text, sourceLabel); | ||
| const conclusionResult = generateConclusionNodes(text, sourceLabel); | ||
|
|
||
| const combinedNodes = [...sessionResult.nodes, ...conclusionResult.nodes]; | ||
| const combinedEdges = [...sessionResult.edges, ...conclusionResult.edges]; | ||
|
|
||
| if (combinedNodes.length === 0 && combinedEdges.length === 0) return { nodesAdded: 0 }; | ||
|
|
||
| const store = await getStore(projectRoot); | ||
| try { | ||
| store.bulkUpsert(nodes, edges); | ||
| // Bulk upsert nodes + edges (project-scoped) | ||
| store.bulkUpsert(combinedNodes, combinedEdges, projectRoot, undefined, memoryScope); | ||
|
|
||
| // Post-insert: create linking edges from conclusion nodes to existing | ||
| // graph nodes by simple keyword overlap. This helps surface relations | ||
| // between learned conclusions/fragments and code entities/files. | ||
| const now = Date.now(); | ||
| const allNodes = store.getAllNodes(projectRoot); | ||
|
|
Comment on lines
280
to
286
| const root = pathResolve(opts.project); | ||
| if (!existsSync(join(root, ".engram", "graph.db"))) { | ||
| console.error( | ||
| `engram: no graph found at ${root}. Run 'engram init' first.` | ||
| `engramx: no graph found at ${root}. Run 'engram init' first.` | ||
| ); | ||
| process.exit(1); | ||
| } |
Comment on lines
+166
to
+174
| // Explicit memory requests: if the user explicitly asks the agent to | ||
| // remember/save/store this prompt, persist it to the project's memory. | ||
| // Fire-and-forget so we don't delay the hook response. We bypass the | ||
| // autoMemoryEnabled flag for explicit user requests. | ||
| try { | ||
| const explicitRemember = /^\s*(?:remember|save|store|dont forget|don't forget|note)\b/i; | ||
| if (explicitRemember.test(prompt)) { | ||
| void learn(projectRoot, prompt, "user:remember", "project").catch(() => {}); | ||
| } |
Comment on lines
+741
to
+747
| window.addEventListener('keydown', (ev) => { | ||
| if (ev.key === '-' || ev.key === '_') { | ||
| keyboardZoom(0.9); | ||
| } else if (ev.key === '=' || ev.key === '+' ) { | ||
| keyboardZoom(1.1); | ||
| } | ||
| }); |
Comment on lines
+213
to
+218
| // Compute canonical ids for existing nodes and backfill | ||
| try { | ||
| // Import computeCanonicalId dynamically so migrations are self-contained | ||
| const { computeCanonicalId } = require("../graph/canonical.js"); | ||
| const res = db.exec("SELECT id, label, kind, memory_scope, project_root FROM nodes"); | ||
| const rows = (res[0] && res[0].values) ? res[0].values : []; |
Comment on lines
+23
to
34
| // Global DB config: single database for all projects | ||
| const GLOBAL_DB_DIR = process.env.ENGRAM_GLOBAL_DB_DIR || join(homedir(), ".engramx"); | ||
| const GLOBAL_DB_FILE = process.env.ENGRAM_GLOBAL_DB_FILE || "memory.db"; | ||
|
|
||
| export function getGlobalDbPath(): string { | ||
| return process.env.ENGRAM_GLOBAL_DB_PATH || join(GLOBAL_DB_DIR, GLOBAL_DB_FILE); | ||
| } | ||
|
|
||
| export function getDbPath(_projectRoot: string): string { | ||
| // Backwards-compatible alias: always use the single global DB. | ||
| return getGlobalDbPath(); | ||
| } |
Comment on lines
+738
to
+742
| const projects = await listKnownProjects(store); | ||
| const scopes = [ | ||
| { id: "accumulative", label: "ACCUMMULATIVE MEMORIES" }, | ||
| { id: "global", label: "GLOBAL MEMORIES" }, | ||
| { id: "personal", label: "PERSONAL MEMORIES" }, |
added 2 commits
May 8, 2026 04:07
…pear in dashboard
…ter learn to populate file nodes
Author
|
Additional fixes included in this branch:
I also retroactively added missing project_root entries for existing projects in your local DB (e.g., /Users/max/prjs/worlds.ai.rust) so the dashboard now lists them. If you prefer a different approach (no background init), I can revert that part and instead expose a UI hint to run 🔍 Scanning codebase... 📊 Token savings: 111.6x fewer tokens vs relevant files (211.2x vs full corpus) ✅ Ready. Your AI now has persistent memory. 💡 Next step: engram install-hook — enables automatic Read interception (82% token savings) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Files changed:
Testing steps are in docs/operations/dev-reload-port-audit.md.
Follow-ups / todos are listed in the docs file and tracked in issue #31.