Budget-aware retrieval: oversized filtering, MMR budgeting, chain output budgeting

[gvonness-apolitical]

Summary

• Oversized chunk filtering: Chunks individually larger than the query's maxTokens are excluded a priori — filtered before MMR in search, and passed through (but not scored) in chain walking so the chain doesn't break at large nodes
• Budget-aware MMR: The greedy selection loop tracks remaining token budget and skips candidates that exceed it, preventing large chunks from consuming diversity slots they can't fill. Applies below MMR threshold too
• Budget-aware chain formatting: Chain output assembly iterates through chunks in order and only includes those that fit within the remaining budget — no partial chunks returned
• Documentation: Updated CHANGELOG (v0.9.2), traversal algorithm reference (pseudocode, design decisions, pipeline diagram), how-it-works guide (search/chain pipeline steps, MMR section), and MCP tools reference (token limits section)

Test plan

• npm run build && npm test — 2038 tests pass
• 11 new tests: 4 chain-walker oversized passthrough, 4 MMR budget-aware selection, 3 chain-assembler budget-aware formatting
• search max_tokens:500 with large chunks in memory → no chunk exceeding 500 tokens in response
• recall through a chain containing an oversized node → chain continues past it, node excluded from output