feat(bench): fire axis, fullscreen measurement, 5 more terminals, and HTML reports#23
Merged
Conversation
Seventh harness axis: bench/tools/fire.c renders the DOOM fire effect as truecolor half-blocks into a fixed 80x24 region (byte-identical stream for every terminal; deterministic seed) and reports producer-side fps under pty flow control. Median of 3x10s reps, ranked higher-is-better. A manual full-window mode (fire ... full) approximates the upstream full-window condition and never enters the scored axis.
Manual in-terminal self-check now runs any subset of the suite's workloads (ascii/unicode/scroll/fire/latency), builds only the tools it needs, resets terminal state after cat floods (scroll_stress leaves a DECSTBM region and a homed cursor that overlapped the results), and reports MiB/s via nowns. The fire test renders to the live window size by default (upstream DOOM-fire condition); FIRE_FIXED=1 restores the fixed 80x24 region.
Entries activate only when installed (skip, never faked). Verified on this machine: Terminal.app runs as our own second instance with the wrapper as document argument (real tty, env inherited via login -p, user instance untouched); iTerm2 ignores $SHELL, so its pty child is created via AppleScript with a generated NOA_* env-wrapper, skipped entirely when a user instance is running; Rio ignores XDG_CONFIG_HOME but honors RIO_CONFIG_HOME (isolated there, confirm-before-quit=false pinned). Alacritty uses -e with an empty iso config so ~/.alacritty.toml is never consulted. Warp's $SHELL launch is unverified and it carries account state, so it is opt-in only (--only ...,warp), excluded from the default set. iTerm2/Warp/Terminal prefs are not isolable - disclosed per run in results.json.isolation.
throughput/scroll/fire/latency/load-active now measure every terminal in
native macOS fullscreen: identical physical geometry for all terminals,
neutralizing the grid-pinning gaps (Termy/iTerm2/Warp/Terminal expose no
size control). The wrapper gains a NOA_GO gate so no bytes are consumed at
pre-fullscreen geometry, and load-active snapshots its before-CPU after the
fullscreen settle.
Mechanics, all verified on this machine: kitty --start-as=fullscreen and
alacritty window.startup_mode are native-flagged (an AX verify can
false-negative them); ghostty is NOT - its --fullscreen suppresses the -e
command entirely, so it takes the AXFullScreen path and keeps
--window-position-x/y for display pinning. Every measured window is pinned
to the display under the cursor at run start (bench/tools/dispinfo,
recorded as target_display). Accessibility is probed once up front with a
real AX attribute read ('UI elements enabled' can return true while access
is denied; name listing passes without the permission) - when denied, the
harness triggers the OS permission prompt, prints System Settings steps,
and measures ALL terminals windowed so a run never mixes geometries.
startup stays ungated, memory/load-idle stay windowed (multitab's wincount
cannot see fullscreen Spaces), --equalize disables fullscreen.
…ion) On fullscreen runs the fire axis now fills the live window — upstream DOOM-fire-zig's official condition; the fullscreen gate guarantees the winsize read happens at final geometry, and the per-terminal cell count (each terminal's font defaults, as in upstream comparisons) is disclosed per rep (region in raw.tsv, parenthesized in table.md) plus a per-run fire_condition meta. Windowed fallback runs keep the fixed 80x24 byte-identical region — full-window fps on unequal window geometry would measure the geometry lottery. The two conditions are marked mutually incomparable. Also replaces the Accessibility probe with a real AX attribute read: 'UI elements enabled' can return true while attribute access still fails -1719 (global flag, not this process's TCC grant — verified), which made a denied run mislabel its fire condition as fullscreen.
A subset run (--axes fire --only kitty) printed the unselected axes as all-UNMEASURED and an all-n/a noa-rank section, both reading as failed measurements. Sections and rank rows now appear only for axes with data, and the rank section only when noa was measured.
Renders a results dir into a self-contained report.html (stdlib-only, inline CSS/SVG, works offline): one bar-chart section per axis with data, value labels, higher/lower-better notes, noa highlighted, UNMEASURED listed instead of drawn as zero bars, and run meta (fullscreen, fire_condition, target_display, quiescence) in the header. Latency bars are log-scaled (frame-paced responders are ~1000x). Multiple dirs append a noa-trend section. run_all.sh now generates it after aggregate (non-fatal) and prints the report path.
simota
marked this pull request as ready for review
July 17, 2026 05:22
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7df37d417d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+151
to
+153
| - Window-size-scaled numbers in the scored axis. A manual full-window mode | ||
| exists for upstream-style anchor runs (`fire <secs> <result> full`, or | ||
| `FIRE_FULL=1 bench/run_benchmark.sh fire`) — fps scales ~1/cell-count |
There was a problem hiding this comment.
For default runs FULLSCREEN=1, bench/run_all.sh exports NOA_FIRE_ARG=full and includes that fire fps in table.md/rankings, so this statement that window-size-scaled numbers never enter the scored axis is no longer true. This can make future benchmark audits compare the wrong condition or use the obsolete FIRE_FULL knob while bench/run_benchmark.sh now uses default-full/FIRE_FIXED, so the spec should match the implemented full-window default.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Positions noa as the fastest terminal at swallowing output floods (docs/positioning.md + README) and extends the benchmark harness to defend that claim:
fireaxis — DOOM-fire IO stress (truecolor half-block full-region repaint, producer-side fps under pty flow control). Full-window on fullscreen runs (upstream DOOM-fire condition), fixed 80×24 byte-identical region on windowed fallback; the two are marked mutually incomparable.bench/tools/dispinfo+NOA_GOworkload gate). All-or-nothing Accessibility probe: a denied run measures everything windowed, never mixed. Known upstream quirk handled: ghostty--fullscreensuppresses-e, so ghostty takes the AX path.$SHELL). Entries activate only when installed.bench/visualize.py— self-contained HTML report (inline SVG, stdlib-only) generated per run;run_benchmark.shreworked for per-test selection.Measurement-fairness decisions (region pinning, isolation mechanisms, probe pitfalls) are documented in
bench/METHODOLOGY.mdanddocs/specs/bench-doom-fire.md.Test plan
fire.c: deterministic byte stream verified (cmpon common prefix); built with-Wall -Wextraclean; smoke-measured on all installed terminals (zero UNMEASURED, zero leftover processes).bash -n/sh -n/py_compileon all touched scripts;visualize.pyexercised on real results dirs (single + multi), no external resource references.cargochecks not applicable.