feat(smc): add Structured Memory Compression engine (Phase 1)

[pythondatascrape]

Summary

• Adds internal/smc/ package with core SMC engine: category schema, k-parameter controller, conversation matrix, rule-based decomposer, and preference signal types
• Integrates SMC compression into the proxy handler as an alternative to the existing sliding-window compressor, enabled via config
• Wires SMC configuration through config.yaml and adds --k CLI flag to engram serve

Details

New types (internal/smc/):

• CategorySchema — defines named content categories (e.g., facts, preferences, instructions) with per-category k-parameter overrides
• KController — manages compression ratio (k) globally and per-category
• ConversationMatrix — stores decomposed conversation rows and serializes back to provider.Message format
• RuleDecomposer — heuristic keyword-based decomposer that extracts category-relevant content from messages
• PreferenceSignal — correction tracking type for future adaptive compression

Integration:

• proxy.Handler.EnableSMC() activates matrix-based compression; falls back to sliding-window when SMC is disabled
• SMCConfig added to ProxyConfig with sensible defaults (k=0.5, default 3-category schema)
• E2E integration test demonstrates ~57% token reduction on multi-turn conversations

16 files changed, 1060 insertions(+), 3 deletions(-)

Test plan

• All unit tests pass (internal/smc/, internal/config/, internal/proxy/)
• E2E integration test verifies multi-turn compression and custom schemas
• Race detector clean (go test -race)
• go vet clean
• Binary builds successfully

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces a Phase 1 Structured Matrix Compression (SMC) engine and wires it into the proxy/serve path, alongside codebook/token-accounting optimizations to reduce context overhead and improve reporting accuracy.

Changes:

• Added internal/smc package (schema, k-controller, conversation matrix, rule-based decomposer, preference signal) with unit + e2e tests.
• Integrated SMC into the proxy (optional compression path), improved system-prompt parsing/token accounting, and added SMC wiring via engram serve flags/config.
• Optimized codebook serialization by omitting default enum values and marking defaults in definitions; adjusted related tests and statusline percent computation.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
internal/smc/category.go	Adds configurable category schema, defaults, and validation.
internal/smc/category_test.go	Tests schema defaults/validation.
internal/smc/k.go	Adds k-controller for global/per-category k.
internal/smc/k_test.go	Tests k-controller behaviors.
internal/smc/matrix.go	Implements conversation matrix and serialization to provider messages.
internal/smc/matrix_test.go	Tests matrix append/serialization/token counting.
internal/smc/decompose.go	Implements Phase 1 rule-based decomposer.
internal/smc/decompose_test.go	Tests rule-based decomposition and k impact.
internal/smc/preference.go	Adds preference/correction signal type.
internal/smc/integration_test.go	Adds end-to-end SMC tests (multi-turn + custom schema).
internal/proxy/handler.go	Adds SMC mode, system prompt extraction, and token accounting updates.
internal/proxy/handler_test.go	Adds tests for system prompt token accounting, array-form system parsing, and SMC path.
internal/proxy/compressor.go	Treats window sizes <2 as “no compression”.
internal/proxy/proxy.go	Adds server helper to enable SMC on the underlying handler.
cmd/engram/serve.go	Adds --window and --k, wires SMC config/enablement into proxy startup.
internal/server/handler.go	Adds windowSize and periodic codebook-def injection gating.
internal/server/handler_test.go	Updates constructors and adds window-based reinjection tests.
internal/server/handler_bench_test.go	Updates benchmark to new handler constructor signature.
internal/config/config.go	Adds SMC config under proxy config with defaults.
internal/config/config_test.go	Tests SMC config defaults and custom category loading.
internal/context/codebook.go	Adds enum defaults, omits default enum fields in serialization, marks defaults in definitions.
internal/context/codebook_test.go	Updates/extends tests for new default omission/definition behavior.
internal/context/response_codebook_test.go	Updates expectations to reflect default omission behavior.
internal/context/history_test.go	Updates history test expectation after default omission changes.
internal/optimizer/format.go	Uses shared percent calculation for context saved %.
internal/optimizer/format_test.go	Adds test ensuring percent uses raw context values.
docs/superpowers/specs/*	Adds design specs for SMC + codebook optimizations.
docs/superpowers/plans/*	Adds implementation plans for SMC Phase 1 + codebook optimizations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In smcCompress, the loop decomposes all complete user+assistant pairs (i < pairs), but later the function also appends the last pair raw (messages[lastPairStart:]). This duplicates the last completed exchange (once in the matrix, once raw) on the first call, and makes the “except the last pair” comment inaccurate. Consider decomposing only up to pairs-1 (or otherwise excluding the last complete pair) when you intend to keep that pair raw for recency.

[Copilot]

smcCompress mutates the per-session ConversationMatrix (matrix.Append) without any synchronization. Since ServeHTTP can be called concurrently for the same session ID, this can race on the matrix’s underlying slice and corrupt history. Consider guarding per-session matrix mutation with a mutex (per matrix or held under smcMu) or otherwise ensuring requests for a session are serialized.

[Copilot]

smcCompress calls Decompose with context.Background(), so request cancellation/timeouts from the inbound HTTP request won’t propagate. Passing r.Context() through (e.g., add a ctx parameter to smcCompress) will make future decomposers (LLM-based / slower) behave correctly under client disconnects.

Suggested change

	row, err := h.smcDecomposer.Decompose(context.Background(), exchange, h.smcSchema)
	row, err := h.smcDecomposer.Decompose(ctx, exchange, h.smcSchema)

[Copilot]

smcMatrices is an unbounded in-memory map keyed by sessionID with no eviction/cleanup path. In long-running proxy processes (or if session IDs are derived from fingerprints), this can grow without bound and increase memory usage over time. Consider adding a cleanup strategy (TTL/LRU) or tying lifecycle to the existing session stats/retention behavior.

[Copilot]

EnableSMC mutates multiple Handler fields (smcEnabled/schema/k/decomposer/map) without any locking. If this is ever called after the proxy has started serving, it can race with ServeHTTP and/or drop existing matrices. Consider either (1) making EnableSMC safe for concurrent use (mutex/atomic + preserve existing matrices), or (2) documenting/enforcing that it must be called only during initialization before Start().

[Copilot]

anthropicRequest.RawSystem is a json.RawMessage without omitempty. If the incoming request omits system, RawSystem will remain nil and re-marshaling will emit "system": null, changing the wire payload. Consider adding omitempty (and/or only re-marshaling when you actually modify the request) to preserve the original shape.

Suggested change

	RawSystem json.RawMessage `json:"system"`
	RawSystem json.RawMessage `json:"system,omitempty"`

[Copilot]

SMC schema values are converted from config and used without validation. Since CategorySchema.Validate exists, it would be safer to validate the schema (non-empty categories, no duplicates, no blank names) before enabling SMC and fail fast with a clear error if the config is invalid.

Suggested change

	smcSchema := configToSMCSchema(smcCfg)
	smcSchema := configToSMCSchema(smcCfg)
	if err := smcSchema.Validate(); err != nil {
	return fmt.Errorf("invalid proxy SMC schema: %w", err)
	}

[Copilot]

TestHandler_SMCCompression only asserts the handler returns 200, but doesn’t assert that SMC compression actually changed the outbound messages payload (e.g., presence of intent= matrix rows and absence of [CONTEXT_SUMMARY], or that the message count shrinks/grows as expected). Since the upstream server can capture the request body, consider asserting on the received JSON to ensure the SMC path is really exercised.

[Copilot]

KController.CompressionRatio’s formula and doc appear inverted relative to the rest of SMC: RuleDecomposer.compress treats higher k as less compression (keeps more text), but CompressionRatio currently decreases as k increases (k=0→1.0, k=1→minRatio) while the comments call k=0 “maximum compression”. Consider aligning this method with the compress() mapping (e.g., ratio = minRatio + (1-minRatio)*k) or renaming/re-documenting it so k has a consistent meaning across the package.

Suggested change

	// CompressionRatio returns the target compression ratio for a category.
	// Formula: compression_ratio = 1 - (1 - minRatio) * k
	// At k=0: returns 1.0 (maximum compression).
	// At k=1: returns minRatio (minimum compression, maximum fidelity).
	func (kc KController) CompressionRatio(category string, minRatio float64) float64 {
	k := kc.EffectiveK(category)
	return 1 - (1-minRatio)*k
	// CompressionRatio returns the target retained-text ratio for a category.
	// Formula: compression_ratio = minRatio + (1 - minRatio) * k
	// At k=0: returns minRatio (maximum compression).
	// At k=1: returns 1.0 (minimum compression, maximum fidelity).
	func (kc KController) CompressionRatio(category string, minRatio float64) float64 {
	k := kc.EffectiveK(category)
	return minRatio + (1-minRatio)*k

[Copilot]

RuleDecomposer.compress converts the input to []rune twice (len([]rune(text)) and then runes := []rune(text)), which doubles allocations for non-empty text. Consider converting once and reusing the slice for both length calculation and truncation.