feat(boxen): C M2 #810 -- input_decoder CSI cursor keys + modifier params + ESC disambiguation by jsavin · Pull Request #817 · jsavin/Frontier

jsavin · 2026-06-30T02:33:57Z

Summary

Phase C input decoder milestone M2 -- the first milestone that emits real boxen_event_t values from the buffered byte stream. M1 (#816) landed the harness + stub; this PR implements the CSI cursor-key state machine, modifier-param decoding (the #805 root cause), SS3 cursor/nav/F-keys, ESC-prefix Meta, ESC-alone disambiguation, control characters, and UTF-8 multi-byte assembly.

Plan: planning/phase_c/INPUT_DECODER_PLAN.md sections 3.1-3.5, 3.10-3.12, 4 (state machine), 7 (M2 scope).

What landed

State machine (per plan section 4): GROUND, ESC_RECEIVED, CSI_COLLECTING, SS3_RECEIVED, UTF8_CONT. Parser state is persistent across input_decoder_poll calls so partial sequences split across inject_bytes boundaries (M6: read(2) boundaries) survive cleanly.

Modifier-param decoder (modifier_from_param, plan section 4.3): the 16-entry CSI modifier table. N=2..16 -> SHIFT|ALT|CTRL|META combinations. N=9 (META) is the bit termbox2 silently dropped on Terminal.app -- the #805 root cause is now decoded correctly.

CSI dispatch: A/B/C/D (arrows), H/F (Home/End), P/Q/R/S (F1-F4 with modifier), ~-form for Ins/Del/PgUp/PgDn and F-keys (F1-legacy \e[11~, F5..F12). All \e[1;N<letter> and \e[N;M~ forms decode modifier properly.

SS3 dispatch: A/B/C/D, H/F, P/Q/R/S without modifiers (VT100 standard).

ESC handling:

\ea / \eb / \ef and any ESC+printable -> ch=letter, mod=ALT (plan section 3.5 policy: keeps Alt-letter distinct from \e[1;3D word-motion so the REPL keymap can bind both independently).
Lone ESC at poll boundary -> KEY_ESCAPE (plan section 4.4; test-seam equivalent of the production non-blocking drain).
Double ESC -> two ESCAPEs across two polls.

Control characters (plan section 3.11): 0x01..0x1A -> CTRL_A..CTRL_Z; 0x09=TAB; 0x0D=ENTER; 0x08 and 0x7F both -> BACKSPACE (macOS Terminal.app sends DEL for Backspace); 0x00=CTRL_SPACE; 0x1C=CTRL_BACKSLASH.

UTF-8 assembly (plan section 3.10): 2/3/4-byte codepoints accumulated through UTF8_CONT state; emit on completion; continuation bytes split across inject calls survive correctly.

Buffer mechanics: added head cursor to the existing fill buffer. compact_buffer() reclaims the drained prefix lazily (on next inject) so partial-sequence state preservation costs only O(live tail) on memmove. CSI parameter buffer capped at 64 bytes to bound DoS risk from a runaway terminal stream.

Scope guardrails (per plan + task hard rules)

No changes to backend_tb2.c (M6 cutover).
No changes to ws_server.c, ws_server.h, databases/Virgin.root, or any rendering path verb (tb_set_cell, tb_present, tb_clear, tb_width, tb_height).
No tb_set_input_mode(TB_INPUT_MOUSE) calls anywhere.
Mouse parsing (\e[<...M/m), bracketed paste, Kitty \e[...u all deferred to M3/M4/M5 (their tests remain SKIP).

Tests

Added behavioral tests (un-skipped from the M1 stubs and made into real assert(ev.type == ... && ev.key.key == ... && ev.key.mod == ...) checks):

CSI plain arrows (A/B/C/D), SS3 arrows + Home/End
Modifier params 2..16 for all four arrow directions
Named boxen REPL: mousewheel = history, option-arrows insert b/f literals #805 spot-tests for \e[1;3C and \e[1;3D (Option-Right / Option-Left on Terminal.app)
CSI nav (Home/End/Ins/Del/PgUp/PgDn) plain and with modifiers
F1-F4 SS3; F1-shift and F4-alt CSI letter form; F1-legacy \e[11~; F5..F12 ~-form; F5 with CTRL
ESC-prefix Meta (Alt-a, Alt-b, Alt-f); ESC-alone; double ESC
Partial-sequence splits (CSI \e[1; then 3D; SS3 \eO then A; \e[ then A)
Control chars (^A, ^C, ^E, ^Z, TAB, ENTER, ^H, DEL) + printable ASCII
UTF-8 two/three/four-byte codepoints + cross-inject continuation split

Manual smoke deferred to M6 cutover -- decoder is not yet wired into backend_tb2.c, so Option-arrows in a live REPL session don't work yet by design.

Test results

Layer	Before (M1 head)	After (M2 head)
`input_decoder_tests`	62/62 with 49 SKIP	94/94 with 18 SKIP (M3/M4/M5 only)
Full unit suite	883 passed	915 passed, 0 failed
Integration suite	2186 / 21 / 207 (pass/fail/skip)	2186 / 21 / 207 -- exact baseline match
`make -C frontier-cli`	clean	clean (no new warnings)

The 21 integration failures are the known pre-existing baseline (html publishing pipeline, sys.countapps, TCP/startup networking) -- none touch boxen input.

Test plan

make -C tests input_decoder_tests && ./tests/input_decoder_tests -- 94/94
./tools/run_headless_tests.sh -- 915/915
cd tests && make test-integration -- 2186 passed / 21 failed (baseline) / 207 skipped
make -C frontier-cli -- clean build
/gate -- pending (run as next step)
Manual Option-arrow smoke -- deferred to M6 (decoder is not yet wired into backend_tb2.c)

Closes #810

🤖 Generated with Claude Code /auto

…rams + ESC disambiguation Implements milestone M2 of the Option C input decoder per planning/phase_c/INPUT_DECODER_PLAN.md section 7. This is the first milestone that emits real boxen_event_t values from the buffered byte stream; M1 was harness-only. Decoder state machine - GROUND, ESC_RECEIVED, CSI_COLLECTING, SS3_RECEIVED, UTF8_CONT - modifier_from_param(N) maps the 16-entry CSI modifier table (N=2..16) to BOXEN_MOD_SHIFT|ALT|CTRL|META. N=9 (META) is the bit termbox2 silently dropped on Terminal.app -- the #805 root cause. - CSI dispatch: A/B/C/D (arrows), H/F (Home/End), P/Q/R/S (F1-F4 with mod), ~-form (Ins/Del/PgUp/PgDn, F1-legacy, F5..F12 with mod). - SS3 dispatch: A/B/C/D, H/F, P/Q/R/S (no modifier per VT100 standard). - ESC-prefix Meta: \ea / \eb / \ef and any printable ESC-prefix emit ch=letter, mod=ALT (plan section 3.5 policy; decoder keeps Alt-letter distinct from \e[1;3D word-motion so REPL keymap can choose). - ESC-alone: emit KEY_ESCAPE when poll boundary follows ESC with no follow-up byte (plan section 4.4; test-seam equivalent of the production non-blocking-drain rule). - Double ESC: first ESCAPE emits immediately, second ESC re-enters ESC_RECEIVED for next poll. - Control chars 0x01..0x1A -> CTRL_A..CTRL_Z; 0x09=TAB, 0x0D=ENTER, 0x08/0x7F=BACKSPACE, 0x00=CTRL_SPACE, 0x1C=CTRL_BACKSLASH. - UTF-8 multi-byte assembly (2/3/4-byte) accumulated across inject calls and emitted on completion. Buffer + partial-sequence robustness - Added head cursor to the existing fill_len buffer so parsing state survives across poll calls. compact_buffer() reclaims drained prefix; called from inject_bytes before measuring available space. - Partial sequences in any state (ESC + nothing in CSI_COLLECTING, \eO with no follow-up, UTF-8 lead with missing continuation, etc.) hold their state until the next inject + poll completes them. BOXEN_ERR_TIMEOUT returns with *out zeroed. No fall-through to printable. Plan section 3.12. - CSI parameter buffer capped at 64 bytes; overflow resets to GROUND to bound DoS from runaway parameter streams. Tests (tests/input_decoder_tests.c) - All M2-scope tests un-skipped and made behavioral (assert on ev.type, ev.key.key, ev.key.ch, ev.key.mod): - CSI plain arrows + SS3 arrows + SS3 Home/End - Modifier params 2..16 for all four arrow directions, plus named spot-tests for the #805 \e[1;3C/D cases - CSI nav (Home/End/Ins/Del/PgUp/PgDn) plain and with modifiers - F1-F4 SS3 form; F1-shift / F4-alt CSI letter form; F1-legacy (\e[11~) and F5..F12 ~-form; F5 with CTRL - ESC-prefix Meta (Alt-a, Alt-b, Alt-f) and ESC-alone, ESC+ESC - Partial-sequence splits (CSI \e[1; then 3D; SS3 \eO then A; \e[ then A) - Control chars (^A, ^C, ^E, ^Z, TAB, ENTER, ^H, DEL) + printable ASCII baseline - UTF-8 two/three/four-byte codepoints + cross-inject continuation split for the em-dash - Counts: 94/94 green (input_decoder_tests), 18 SKIP stubs remain for M3 mouse / M4 paste / M5 Kitty. Pre-M2: 62/62 with 49 SKIPs. Scope guardrails (per plan section 7 + task hard rules) - No changes to backend_tb2.c (M6 cutover). - No changes to ws_server.c, ws_server.h, Virgin.root, or any rendering path verb (tb_set_cell, tb_present, tb_clear, tb_width, tb_height). - No tb_set_input_mode(TB_INPUT_MOUSE) calls anywhere. Test results - input_decoder_tests: 94/94 (red 13/62 + 49 SKIP at M1 head -> green 94/94 with 18 remaining SKIP) - Full unit suite: 915/915 passed, 0 failed (up from 883/883 at M1 head; +32 = new M2 tests minus 1 reverted skip-count delta) - Integration suite: 2186 passed / 207 skipped / 21 failed (exact match to pre-existing baseline; 21 failures are html.publishing-pipeline, sys.countapps, TCP/startup network -- all unrelated to boxen input) - frontier-cli build: clean (no new warnings) Closes #810

- Add input_decoder_parse_errors() observability counter for silent rejection paths (CSI overflow, unknown final bytes, UTF-8 errors) - Harden state machine: CSI_SWALLOW state for overflow recovery, validate UTF-8 codepoints (reject surrogates, overlong, > U+10FFFF) - 20 additional behavioral tests covering the hardening paths Test counts: 915 -> 935 unit (all green), integration baseline match.

jsavin added 2 commits June 29, 2026 19:32

jsavin merged commit 4ef9487 into develop Jun 30, 2026
1 check passed

jsavin deleted the phase-c-m2-csi-decoder branch June 30, 2026 02:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(boxen): C M2 #810 -- input_decoder CSI cursor keys + modifier params + ESC disambiguation#817

feat(boxen): C M2 #810 -- input_decoder CSI cursor keys + modifier params + ESC disambiguation#817
jsavin merged 2 commits into
developfrom
phase-c-m2-csi-decoder

jsavin commented Jun 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jsavin commented Jun 30, 2026

Summary

What landed

Scope guardrails (per plan + task hard rules)

Tests

Test results

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant