feat(inference): UTF-8-safe streaming detokenization + generate_streaming

[ohdearquant]

What

UTF-8-safe incremental detokenization for byte-level BPE, plus Qwen35Model::generate_streaming — a token-at-a-time generation entry point with a per-token callback.

Why

Byte-level BPE splits multi-byte codepoints (CJK, emoji) across tokens. Naive per-token UTF-8 decode emits U+FFFD replacement characters mid-stream, and English-only smoke tests hide it. detokenize.rs now buffers incomplete codepoints and flushes only complete UTF-8. generate_streaming gives the app a streaming surface, consumed by chat_metal (PR-2) and generate_lora (PR-2b).

Files

• crates/inference/src/model/qwen35/detokenize.rs (+143/-10)
• crates/inference/src/model/qwen35/generation.rs (+127/-1)

Verification

cargo build -p lattice-inference clean. The integrated tree (all 8 slices merged) builds all 9 app-shipped binaries green via scripts/build-app-bins.sh.

Bench

make bench-compare was run (origin/main vs integrated tree). Its two-worktree comparator errored while assembling the delta table (a known fragility; the base benches themselves ran clean — rms_norm/layer_norm/silu/gelu/softmax/simd_normalize all reported normal numbers). The change is bench-neutral by construction: no Criterion harness compiles detok/generation into a measured path (grep: zero bench references to either symbol), so the micro-benches are structurally blind to it. The real decode-throughput surface is covered by e2e-parity.yml's informational speed report.

Series

Part of the PR #193 engine-slice (finest split). All engine code lands on main; the macOS app surfaces a subset (Models + Chat) for v0.0.1.

[github-actions]

E2E Parity Report

PASS: all 3 prompts match within first 3 tokens

Prompt	Agreement	First Diff	HF tok/s	Lattice tok/s	Verdict
The capital of France is	3/15	pos 3	0.3	2.2	PASS
In the year 2024, artificial intelligence	10/15	pos 9	0.3	1.9	PASS
`def fibonacci(n):

if n <= 1:
    return n
return` | 15/15 | none | 0.2 | 1.4 | PASS |

The capital of France is

• HF: Paris.
  The capital of France is Paris.
  The capital of France
• Lattice: Paris.
  A: Yes, the capital of France is Paris.

In the year 2024, artificial intelligence

• HF: (AI) has become a significant part of the global economy. It is
• Lattice: (AI) has become a significant part of our daily lives. From personal

def fibonacci(n): if n <= 1: return n return

• HF: fibonacci(n-1) + fibonacci(n-2)

print(fib

• Lattice: fibonacci(n-1) + fibonacci(n-2)

print(fib