Protect Skippy stage decode from KV pressure

[IvGolovach]

Summary

This protects Skippy stage decode/replay from resident-prefix KV pressure before the binary transport asks the native runtime to decode.

When a stage is about to decode with a resident prefix already occupying KV space, the server now computes a bounded resident-prefix eviction plan first. That gives the Rust-side cache a chance to free stale prefix records before decode work hits native KV admission, while keeping the newly landed native KV compaction from #764 as the lower-level fragmentation recovery path.

Why

Issue #652 is not only one allocator failure mode. #764 addresses native unified-KV fragmentation. This PR covers the adjacent Skippy server path where resident-prefix cache entries can still consume KV budget before binary stage decode/replay work starts.

The intent is to reduce avoidable llama_decode / slot-pressure failures under long Goose/Pi-style tool loops without changing protocol, schema, or the Skippy ABI.

Diff Scope

• Adds binary_transport::kv_eviction to compute bounded resident-prefix eviction decisions before decode/replay work.
• Wires the eviction decision into binary stage decode admission.
• Rehomes proactive-eviction telemetry helpers from frontend utility code into kv_integration.
• Adds regression coverage for binary decode eviction admission and bounded telemetry attributes.
• Keeps mesh protocol, gossip, API schema, and Skippy ABI unchanged.

Branch Integrity

• Base branch: main
• Validated base: f9bd75a97378be014f52c4e68c13e81d3399e65e
• Ahead/behind: 0 behind / 1 ahead
• Merge base: f9bd75a97378be014f52c4e68c13e81d3399e65e
• Introduced commit: dcbfecfa14690fa6f250194cf239add52d2a022a Evict binary stage KV before decode

Diff Hygiene

Changed files:

• crates/skippy-server/src/binary_transport.rs
• crates/skippy-server/src/binary_transport/kv_eviction.rs
• crates/skippy-server/src/binary_transport/tests.rs
• crates/skippy-server/src/frontend.rs
• crates/skippy-server/src/frontend/util.rs
• crates/skippy-server/src/kv_integration/mod.rs

Proof:

• git diff --check origin/main...HEAD: PASS, no output.
• git diff --check: PASS, no output.
• git diff --cached --check: PASS, no output.

Validation

Validation tier: Tier 3 - shared Skippy binary stage runtime path refreshed onto current main after PR #764; binary decode/replay now reserves resident-prefix KV capacity before decode work, complementing native KV compaction without protocol/schema/ABI changes.

• git fetch --no-tags origin main:refs/remotes/origin/main codex/kv-tool-loop-runtime-closure:refs/remotes/origin/codex/kv-tool-loop-runtime-closure: PASS, origin/main at f9bd75a9.
• git rebase origin/main: PASS, no conflicts.
• git diff --check origin/main...HEAD: PASS, no output.
• git diff --check: PASS, no output.
• git diff --cached --check: PASS, no output.
• scripts/prepare-llama.sh pinned: PASS, 82 patches applied; upstream 22cadc1944f4658214aee03abd08240358840a95, patched c9b0a02726a0608efa351bf648de9eef6909a565.
• scripts/build-llama.sh: PASS, patched CPU stage ABI libraries built.
• cargo fmt --all -- --check: PASS.
• LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p skippy-server binary_transport --lib -- --test-threads=1: PASS, 6 passed.
• LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p skippy-server proactive_eviction --lib -- --test-threads=1: PASS, 1 passed.
• cargo test -p skippy-cache evict_lru_until_tokens --lib -- --test-threads=1: PASS, 2 passed.
• LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo test -p skippy-server --lib -- --test-threads=1: PASS, 107 passed.
• LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo check -p mesh-llm: PASS.
• LLAMA_STAGE_BUILD_DIR=<stage-build-dir> cargo clippy -p skippy-server --all-targets -- -D warnings: PASS.
• python3 -m unittest scripts.tests.test_qa_kv_tool_loop_stability: PASS, 18 passed.

Required remote gates: PASS on refreshed head dcbfecfa14690fa6f250194cf239add52d2a022a; PR Builds and PR Quality Checks completed successfully.

Ledger: not applicable - not required for selected validation tier/change family.

Version: not applicable - no release/version sync required for this non-release Skippy runtime behavior change.

Not Run

• Live overlapping Goose/Pi tool-loop smoke: no local direct-model Skippy endpoint was available. Deterministic binary transport, frontend telemetry, resident-prefix eviction, cache eviction, and QA harness unit tests cover changed branches.
• just build: not required for selected validation tier; Rust-only server path changed and patched stage ABI plus shipped mesh cargo check cover changed runtime linkage.

Runtime Safety

• No new blocking locks.
• No new unbounded queues.
• No mesh protocol or gossip invariant changed.
• Proactive eviction is bounded by the resident-prefix planner and runs before binary decode/replay work, not inside the native decode loop.
• No invariant regression introduced.

Rollback Plan

Rollback: revert this PR.

git revert <post_merge_commit_sha>

DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.

Known Residual Risks

The remaining issue-level proof is a live #652-style direct-model Goose/Pi overlap certification on a loaded Skippy endpoint. This PR is merge-ready as a deterministic runtime hardening step, while that live certification remains the final behavioral confirmation once suitable local or remote model hardware is available.

[i386]

@michaelneale would be good for you to look at this one too

[i386]

Approved but would like @michaelneale or @ndizazzo to double check as I have not done much with KV recently

[ndizazzo]

@IvGolovach Oh hmm.. I think there’s a small hole here: on a one-chunk PrefillFinalEmbd, we may not have a runtime session yet when this eviction runs.

Since the batch size check needs an active session, that can turn into session … is not active, get logged, and then we carry on without actually evicting anything. So the new guard may not kick in for a valid final-prefill.

Might be worth either moving this until after the session is created for PrefillFinalEmbd, or getting the batch target from somewhere that doesn’t require an active session. I’d also add a regression for the one-chunk final-prefill case so this doesn’t quietly slip back in.

Other than that, this could use a rebase to pick up CI runner changes because I removed Blacksmith... sorry!

[michaelneale]

conceptually I think ok - but really does need a live test somehow, should be able to fire it up even with a small model and slam it with goose or pi.
Only concern I have is if it goes too hard and cache ends up not really doing anything.

that linked issue - I thought that was solved a few weeks ago, so how confident are we in the test coverage that this is fixing what it thinks it is?

[IvGolovach]

Thanks, this was the right thing to force with a live run.

I refreshed the branch on current main and added a second guard around the local OpenAI path: after tokenization, local/embedded-stage0 generation now reserves an active KV token budget before prefill/decode. That means overlapping long prompts queue against the context-sized KV footprint instead of all entering the native runtime and competing for the same unified KV cells.

I also kept the one-chunk PrefillFinalEmbd regression in the test surface; the full skippy-server lib suite is green, including the one-chunk final-prefill case.

Live evidence on the final dynamic-native-runtime build:

• ctx=8192 + resident KV + SmolLM2 + 3-way overlap pressure: native log scan is clean; no failed memory-slot / llama_decode failure reproduced. The full harness is still red because SmolLM2 does not reliably emit tool calls and the tiny ctx evicts cache to preserve decode room.
• ctx=32768 + the same setup: exact-prefix cache is still alive, with prompt_tokens=4506 and cached_tokens=4505. So the fix is not just disabling cache; it preserves cache when the context has room and protects decode when it does not.

Goose/Pi were not available locally, so I used the repo qa-kv-tool-loop-stability harness against the release binary. That gives the same OpenAI overlap pressure shape, but I would still welcome a real Goose/Pi pass from a machine with a stronger tool-capable model.