diff --git a/.gitignore b/.gitignore index 08c828b..532840e 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ Cargo.lock.orig .nexus/ bench/*.txt bench/tools/bin/ +bench/results/ +bench/__pycache__/ diff --git a/README.md b/README.md index 18750e9..1497840 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ # Noa Noa is a GPU-accelerated terminal emulator for macOS, implemented independently -in Rust with `winit` and `wgpu`. It aims for observable compatibility with -[Ghostty](https://ghostty.org) while keeping the core terminal model reusable -and testable. +in Rust with `winit` and `wgpu`. It is built to be **the fastest terminal at +swallowing output floods** — `cat huge.log`, build logs, CI streams — and aims +for observable compatibility with [Ghostty](https://ghostty.org) while keeping +the core terminal model reusable and testable. Noa is under active development and is currently built from source. The macOS app bundle targets macOS 13 or later. Fixture-based regression tests cover @@ -78,6 +79,19 @@ git push origin :refs/tags/v0.1.0 git tag -d v0.1.0 ``` +## Performance + +Noa's defining trait is sustained output throughput: render throughput of +220–233 MB/s across plain, ANSI, and CJK workloads, with CJK as the fastest +variant — wide-character handling carries no cost. Command overhead (1336 µs, +zsh), idle input latency (1.2–1.3 ms), and settled idle memory (52 MB) round +out the profile. + +Every number is reproducible with the bundled benchmark harness +(`bench/run_all.sh`), and performance changes are gated on it. See +[Positioning](docs/positioning.md) for the full claim and +[Methodology](bench/METHODOLOGY.md) for how the numbers are measured. + ## Key features - GPU-rendered terminal grid with Kitty graphics support diff --git a/bench/METHODOLOGY.md b/bench/METHODOLOGY.md index 1027f67..c9bd894 100644 --- a/bench/METHODOLOGY.md +++ b/bench/METHODOLOGY.md @@ -1,6 +1,6 @@ # Benchmark Methodology -A reproducible 6-axis comparison of terminal emulators on macOS/Apple Silicon. +A reproducible 7-axis comparison of terminal emulators on macOS/Apple Silicon. Everything is driven by `bench/run_all.sh`; a third party can re-run the whole suite with one command and get a machine-readable `results.json` plus a human-readable `table.md` under `bench/results//`. @@ -13,6 +13,11 @@ human-readable `table.md` under `bench/results//`. | Ghostty | `ghostty --config-default-files=false --window-save-state=never -e ` | native `-e` | | Termy | `XDG_CONFIG_HOME= termy` with `$SHELL` = wrapper | `$SHELL` (no `-e` flag exists) | | kitty | `kitty --config NONE -o confirm_os_window_close=0 ` | trailing command arg | +| Alacritty | `XDG_CONFIG_HOME= alacritty -e ` | native `-e` (entry activates when installed) | +| iTerm2 | own instance launched directly, then AppleScript `create window with default profile command ` | AppleScript (no `-e`; `$SHELL` verified ignored) — see caveats below | +| Warp | `SHELL= Warp` | `$SHELL` — **UNVERIFIED**; **opt-in only** (excluded from the default set; include explicitly via `--only ...,warp`). UNMEASURED axes mean Warp ignored `$SHELL` | +| Terminal.app | own second instance launched directly with `` as document argument | document-open (verified: real tty, env inherited via `login -p`) | +| Rio | `RIO_CONFIG_HOME= rio -e ` | native `-e` (Rio ignores `XDG_CONFIG_HOME`; `RIO_CONFIG_HOME` verified via `--write-config`; iso config pins `confirm-before-quit=false`, kitty-precedent kill-safety) | The `` dir / `--config`-suppression flags are the **config isolation** mechanism (fresh-install defaults for every terminal) and @@ -21,18 +26,57 @@ added 2026-07-16 and documented in their own sections below. The noa binary defaults to `/target/release/noa` and can be overridden with the `NOA_BIN` env var (for measuring a candidate build). `--axes -throughput,scroll,latency,startup,memory,load` (any subset; this is also the -default set) restricts a run to those axes. +throughput,scroll,latency,startup,memory,load,fire` (any subset; this is also +the default set) restricts a run to those axes. Versions and machine details are captured automatically into `results.json` (`terminal_versions`, `machine`) at run time — read them there for the exact build numbers of the run you are looking at. At authoring time: Ghostty 1.3.1, Termy 0.2.21, kitty 0.47.4, noa release build of the current branch. -**Alacritty** was requested as an optional reference point but is **not -installed** on this machine (`/Applications/Alacritty.app` absent), so it is -omitted rather than faked. Install it (`brew install --cask alacritty`) and it -would need a launch entry in `run_all.sh` (`alacritty -e `). +### The 2026-07-17 additions (Alacritty / iTerm2 / Warp / Terminal.app) + +Four more terminals have launch entries; each activates only when installed +(`skip (not installed)` otherwise — never faked). What was **verified on this +machine** vs assumed: + +- **Alacritty** (not installed at authoring): standard `-e` launch; isolation + via `XDG_CONFIG_HOME` → iso dir containing an **empty `alacritty.toml`**, + so the first config candidate wins and a user's `~/.alacritty.toml` + fallback is never consulted. +- **iTerm2** (verified 2026-07-17): `$SHELL` is **ignored** (the default + profile execs the passwd login shell), so the pty child is created via + AppleScript `create window with default profile command` pointing at a + generated **env-wrapper** (AppleScript-launched commands get launchd's + env, not ours — verified; the wrapper re-exports `NOA_*` and execs + `wrapper.sh`; command runs on a real tty — verified). Three caveats, all + disclosed per run: (1) the AppleScript targets the app **by name**, so the + harness **skips iTerm2 entirely if a user instance is already running** + (collateral-safety invariant); (2) prefs are macOS defaults + (com.googlecode.iterm2) and are **not isolable** without mutating user + defaults — iTerm2 runs with the user's prefs, recorded in + `results.json.isolation`; (3) our instance may open restored/startup + windows running login shells alongside the command window (verified: 3 + windows on this machine) — a Ghostty-phantom-class contamination of the + memory/load axes that is visible in `multitab_procs`/`wincount` rather + than silently included; read iTerm2's memory/load numbers with that in + mind. Its startup number reflects the AppleScript window path, not a + plain process launch. +- **Warp**: launch entry uses `SHELL=`, **unverified** — if Warp + resolves the shell from user records instead, every axis reports + UNMEASURED (honest failure). Prefs and account state are not isolable; + disclosed. **Opt-in only**: excluded from the default terminal set — + include it explicitly with `--only ...,warp`. +- **Terminal.app** (verified 2026-07-17): launched as **our own second + instance** by exec'ing the app binary directly with the wrapper path as + document argument — verified to coexist with the user's running Terminal + (distinct pid, untouched), give the wrapper a real tty, and pass our + environment through its `login -p` wrapper. Never launched via + `open`/AppleScript (those route to the user's instance). Prefs + (com.apple.Terminal) are not isolable; disclosed. + +For iTerm2/Warp/Terminal.app the equalized re-run records +`NATIVE-DEFAULT(prefs-based)` — they expose no CLI size/font control. ## The uniform-workload design @@ -184,6 +228,68 @@ nor records that state; after the first post-fix run, even a flag-less 2026-07-16) because the stale state had been cleared. The flag stays in the launch line permanently so no future environment state can re-introduce it. +## Fullscreen measurement (2026-07-17) + +The render-path axes — **throughput, scroll, fire, latency, and +load-active** — measure every terminal in **native macOS fullscreen** by +default. Rationale: it gives every terminal the exact same physical geometry +without per-terminal grid flags, which also neutralizes the previously +disclosed equalization gaps (Termy/iTerm2/Warp/Terminal.app expose no size +control; noa was pinned to 120×40 while others ran native defaults). + +Mechanism, per measured launch: + +1. Launch as usual; poll pid-scoped `wincount` until a window exists. +2. **Native-flag terminals** (kitty `--start-as=fullscreen`, Alacritty + `-o window.startup_mode="Fullscreen"`) are launched fullscreen directly + — reliable, needs no Accessibility; the harness only waits out the Space + animation. (An AX verify on them can false-negative: kitty was observed + at a clearly-fullscreen 244×85 region while the AX read reported + failure.) **Ghostty is deliberately NOT native-flagged**: with + `--fullscreen`, ghostty 1.3.1 never executes its `-e` command (verified + 2026-07-17 — the pty child never starts), so it takes the AX path; its + `--window-position-x/y` flags are unaffected and still pin the display. + **The rest** (noa, ghostty, Termy, iTerm2, Warp, Terminal.app) get + `AXFullScreen` set on window 1 of OUR tracked process via System Events + `unix id` (never by app name), polled back until `true`, then settled + 0.8 s. +3. **Display pinning:** the target display — the one under the mouse cursor + at run start (`bench/tools/dispinfo`, recorded as `target_display`) — is + where every measured window is placed before fullscreening, since macOS + otherwise places a window (and thus its fullscreen Space) on whatever + display that app last used. Ghostty gets `--window-position-x/y` flags, + Alacritty `window.position` via `-o`; the AX-path terminals get an + AXPosition set before AXFullScreen. kitty exposes no position control — + it fullscreens on the display where its window opens (normally the + active one). An AX read that never confirms is no longer treated as + failure (the set can succeed while the read is denied — observed); the + per-rep fire `region` remains the ground truth. +4. Only then create the **gate file** (`NOA_GO`): the wrapper holds the + workload until the gate appears, so no bytes are consumed at pre- + fullscreen geometry. `load-active` additionally takes its "before" CPU + snapshot after the fullscreen settle and releases the gate afterwards, + keeping the transition's own CPU out of the "same work, less CPU" delta. + +Not fullscreen: **startup** (measures the plain launch itself; its sentinel +predates any window manipulation), **memory** and **load-idle** (multitab's +`wincount` fairness check counts on-screen layer-0 windows, which fullscreen +Spaces would hide; keeping the whole memory axis windowed keeps its scenarios +mutually consistent), and **--equalize** runs (pinned-grid geometry is the +point there). + +Setting `AXFullScreen` requires Accessibility permission for `osascript`. +The harness probes this **once, up front, all-or-nothing**: if denied, it +prints a warning and measures EVERY terminal windowed — a run never mixes +fullscreen and windowed terminals — and records the outcome in +`results.json.contention`-adjacent meta (`fullscreen` key). Per-window +failures (an app without the attribute) fall back to windowed for that rep +and are emitted as `fullscreen_window FAILED` meta rows rather than silently +passing. + +Numbers from fullscreen runs are not comparable to pre-2026-07-17 windowed +runs (visible grid area differs); as always, compare within one results +directory only. + ## The four axes ### 1. Throughput @@ -513,6 +619,56 @@ would rank driver-reclaim timing luck, not the terminals. parse/render CPU is being measured. Reported in ms total and ms-per-MiB processed (normalized). +### 7. Fire — DOOM-fire IO stress (fps) + +`bench/tools/fire.c` runs as the pty child (`NOA_MODE=fire`): it renders the +classic DOOM fire effect (Fabien Sanglard's algorithm, the workload +popularized as a terminal benchmark by +[DOOM-fire-zig](https://github.com/const-void/DOOM-fire-zig)) as truecolor +half-blocks — every frame repaints the whole region with per-cell +`SGR 38;2/48;2` RGB + `U+2584` and absolute cursor positioning. This is the +"animated TUI at max rate" shape none of the other axes exercise: +frame-structured, truecolor-dense, cursor-repositioning flood. After 60 +discarded warmup frames (glyph-atlas population, alt-screen entry) it renders +flat-out for 10 s (3 s in `--quick`) and reports fps; the harness runs 3 reps +(1 in `--quick`) and reports the median. + +**Render region follows the run's geometry mode** (recorded per run as +`fire_condition`, per rep as `region`): + +- **Fullscreen runs (default): full-window** — the upstream DOOM-fire-zig + official condition. The fullscreen gate guarantees the winsize read + happens at final geometry, and every terminal fills the same physical + screen; the resulting **cell count follows each terminal's own font + defaults** (exactly like upstream comparisons), so the per-terminal region + is printed next to every fps number rather than hidden. +- **Windowed fallback runs: fixed 80×24 region** — fps is inversely + proportional to cell count, so full-window fps on unequal window geometry + would measure the geometry lottery, not the terminal. The fixed region + fits every default grid (nothing clips) and gives **every terminal a + byte-identical stream** (fixed PRNG seed → deterministic frame sequence) + with constant frame size, so fps maps linearly to drain MiB/s. + +The two conditions are not comparable to each other; `table.md` states which +one produced its numbers. + +**What fps means here:** producer-side frames/second under pty flow control — +the same "consume the pipe" proxy as axis 1 (the pty's small kernel buffer +blocks the producer's `write` until the terminal drains). It is drain rate, +not photon rate: a display-paced consumer shows up as ~refresh-rate fps (the +signature this axis exists to detect), while an event-driven consumer reports +hundreds or more. The window is focused during the run (same PID-scoped +activation as the latency axis, applied uniformly) so no terminal is measured +under macOS occluded/unfocused throttling. + +**Anchor caveat:** published DOOM-fire-zig figures come from other machines +and full-window regions and are **not comparable** to this axis's numbers — +they motivated the axis, nothing more. This implementation reproduces the workload *shape*, +not upstream's exact bytes. As a CPU-bound axis it is covered by the +builder-quiescence gate; it is skipped under `--equalize` (the fixed region +makes it grid/font-independent by construction). Full design rationale: +`docs/specs/bench-doom-fire.md`. + ### Ghostty load-active timeout (baseline 2026-07-16) — root-caused & fixed The 20260716-084038 baseline reported Ghostty's two load-active rows as diff --git a/bench/aggregate.py b/bench/aggregate.py index 31d1785..25e2850 100644 --- a/bench/aggregate.py +++ b/bench/aggregate.py @@ -18,7 +18,11 @@ def sh(*args): def plist(app, key): p = f"/Applications/{app}/Contents/Info.plist" - return sh("/usr/libexec/PlistBuddy", "-c", f"Print :{key}", p) + return plist_at(p, key) + + +def plist_at(path, key): + return sh("/usr/libexec/PlistBuddy", "-c", f"Print :{key}", path) def terminal_versions(repo): @@ -30,6 +34,13 @@ def terminal_versions(repo): v["ghostty"] = plist("Ghostty.app", "CFBundleShortVersionString") v["termy"] = plist("Termy.app", "CFBundleShortVersionString") v["kitty"] = plist("kitty.app", "CFBundleShortVersionString") + v["alacritty"] = plist("Alacritty.app", "CFBundleShortVersionString") + v["iterm2"] = plist("iTerm.app", "CFBundleShortVersionString") + v["warp"] = plist("Warp.app", "CFBundleShortVersionString") + v["terminal"] = plist_at( + "/System/Applications/Utilities/Terminal.app/Contents/Info.plist", + "CFBundleShortVersionString") + v["rio"] = plist("Rio.app", "CFBundleShortVersionString") return {k: val for k, val in v.items() if val} @@ -156,6 +167,33 @@ def get(term, axis, variant, metric): lat[term] = {"status": "UNMEASURED"} results["axes"]["latency"] = lat +# fire (DOOM-fire IO stress — fixed 80x24 truecolor full-region repaint fps +# under pty flow control; see docs/specs/bench-doom-fire.md). Gated on +# axes_with_data: the axis only exists in raw files from harness >= 2026-07-17. +fire = {} +if "fire" in axes_with_data: + for term in terminals: + cell = agg.get((term, "fire", "-"), {}) + if cell.get("status") == "UNMEASURED": + fire[term] = {"status": "UNMEASURED"} + elif "fps" in cell: + reps = [float(r["value"]) for r in rows + if r["terminal"] == term and r["axis"] == "fire" + and r["metric"] == "fps" and r["rep"] not in ("median", "-")] + winsz = next((r["value"] for r in rows if r["terminal"] == term + and r["axis"] == "fire" and r["metric"] == "winsize"), None) + # region rows exist from harness >= 2026-07-17 (full-window fire); + # older raw files were always the fixed 80x24 region + region = next((r["value"] for r in rows if r["terminal"] == term + and r["axis"] == "fire" and r["metric"] == "region"), + "80x24") + fire[term] = {"fps_median": float(cell["fps"]), "fps_reps": reps, + "region": region, + **({"winsize": winsz} if winsz else {})} + else: + fire[term] = {"status": "UNMEASURED"} + results["axes"]["fire"] = fire + # startup st = {} for term in terminals: @@ -316,14 +354,30 @@ def fmt_st(c): hdr = "| Terminal | " + " | ".join(terminals) + " |" sep = "|---|" + "|".join(["---"] * len(terminals)) + "|" -lines.append("\n## Throughput — ASCII (MiB/s, higher better)") -lines.append(hdr.replace("Terminal", "Metric")); lines.append(sep) -lines.append("| ascii MiB/s | " + " | ".join(fmt_tp(tp[t]["ascii"]) for t in terminals) + " |") -lines.append("| unicode MiB/s | " + " | ".join(fmt_tp(tp[t]["unicode"]) for t in terminals) + " |") +if "throughput" in axes_with_data: + lines.append("\n## Throughput — ASCII (MiB/s, higher better)") + lines.append(hdr.replace("Terminal", "Metric")); lines.append(sep) + lines.append("| ascii MiB/s | " + " | ".join(fmt_tp(tp[t]["ascii"]) for t in terminals) + " |") + lines.append("| unicode MiB/s | " + " | ".join(fmt_tp(tp[t]["unicode"]) for t in terminals) + " |") -lines.append("\n## Frame / Scroll (time ms / MiB·s⁻¹, lower ms better)") -lines.append(hdr.replace("Terminal", "Metric")); lines.append(sep) -lines.append("| scroll_stress | " + " | ".join(fmt_sc(sc[t]) for t in terminals) + " |") +if "scroll" in axes_with_data: + lines.append("\n## Frame / Scroll (time ms / MiB·s⁻¹, lower ms better)") + lines.append(hdr.replace("Terminal", "Metric")); lines.append(sep) + lines.append("| scroll_stress | " + " | ".join(fmt_sc(sc[t]) for t in terminals) + " |") + +if "fire" in axes_with_data: + def fmt_fire(c): + if c.get("status") == "UNMEASURED": + return "UNMEASURED" + return f"{c['fps_median']:.1f} ({c.get('region', '?')})" + fire_cond = contention.get("fire_condition", "fixed 80x24 region (pre-2026-07-17 harness)") + lines.append("\n## Fire — DOOM-fire IO stress (fps, higher better)") + lines.append(f"Condition: {fire_cond}. Producer-side fps under pty flow control " + "(frames written ≈ frames consumed); per-terminal render region in " + "parentheses. Not comparable to published DOOM-fire-zig figures " + "(other machines/displays).") + lines.append(hdr.replace("Terminal", "Metric")); lines.append(sep) + lines.append("| fire fps (region) | " + " | ".join(fmt_fire(fire[t]) for t in terminals) + " |") if "latency" in axes_with_data: lines.append("\n## Input Latency — DSR round-trip proxy (median / p95 / p99 / max µs, lower better)") @@ -434,8 +488,8 @@ def fmt_load_active(c): lines.append("| active: throughput workload | " + " | ".join(fmt_load_active(load[t]["throughput"]) for t in terminals) + " |") lines.append("| active: scroll workload | " + " | ".join(fmt_load_active(load[t]["scroll"]) for t in terminals) + " |") -# noa rank per axis -lines.append("\n## noa rank per axis") +# noa rank per axis — only when noa was actually measured in this run +# (an all-n/a section on --only runs without noa reads like a failure) def rank(better_high, getval): # standard competition ranking: 1 + number of STRICTLY better entries, # so ties share a rank instead of being ordered arbitrarily @@ -487,11 +541,18 @@ def load_idle_val(t): def load_active_val(t, scenario): c = load[t][scenario]; return None if c.get("status") == "UNMEASURED" else c["cpu_ms"] -rank_items = [ - ("throughput ascii", rank(True, lambda t: tp_val(t, "ascii"))), - ("throughput unicode", rank(True, lambda t: tp_val(t, "unicode"))), - ("scroll (MiB/s)", rank(True, sc_val)), -] +def fire_val(t): + c = fire.get(t, {}) + return c.get("fps_median") + +rank_items = [] +if "throughput" in axes_with_data: + rank_items.append(("throughput ascii", rank(True, lambda t: tp_val(t, "ascii")))) + rank_items.append(("throughput unicode", rank(True, lambda t: tp_val(t, "unicode")))) +if "scroll" in axes_with_data: + rank_items.append(("scroll (MiB/s)", rank(True, sc_val))) +if "fire" in axes_with_data: + rank_items.append(("fire (fps)", rank(True, fire_val))) if "latency" in axes_with_data: rank_items.append(("latency (median)", rank(False, lat_val))) if "startup" in axes_with_data: @@ -507,9 +568,12 @@ def load_active_val(t, scenario): rank_items.append(("load idle (mean CPU%)", rank(False, load_idle_val))) rank_items.append(("load active (throughput, CPU-ms)", rank(False, lambda t: load_active_val(t, "throughput")))) rank_items.append(("load active (scroll, CPU-ms)", rank(False, lambda t: load_active_val(t, "scroll")))) -for label, r in rank_items: - pos, n = r - lines.append(f"- {label}: {'#'+str(pos)+' of '+str(n) if pos else 'n/a'}") +if "noa" in terminals: + ranked = [(label, r) for label, r in rank_items if r[0]] + if ranked: + lines.append("\n## noa rank per axis") + for label, (pos, n) in ranked: + lines.append(f"- {label}: #{pos} of {n}") with open(os.path.join(OUT_DIR, "table.md"), "w") as f: f.write("\n".join(lines) + "\n") diff --git a/bench/run_all.sh b/bench/run_all.sh index 177076d..6041bf9 100755 --- a/bench/run_all.sh +++ b/bench/run_all.sh @@ -5,7 +5,9 @@ # frame/scroll (SGR + scroll-region stress consume), warm startup, # memory (idle/scrollback/multitab/longevity footprint — every # scenario dual-reported as active @15s + settled @90s, ranked on -# settled), load (idle CPU% + active CPU-time-per-workload). +# settled), load (idle CPU% + active CPU-time-per-workload), +# fire (DOOM-fire IO stress: fixed 80x24 truecolor full-region +# repaint fps under pty flow control). # Terminals: noa (target/release), Ghostty, Termy, kitty — whichever exist. # # One command runs the whole suite and writes a machine-readable results.json @@ -32,13 +34,15 @@ TOOLS="$BENCH_DIR/tools/bin" WRAPPER="$BENCH_DIR/wrapper.sh" NOWNS="$TOOLS/nowns" PROBE="$TOOLS/dsr_probe" +FIRE="$TOOLS/fire" # ── options ──────────────────────────────────────────────────────── QUICK=0 ONLY="" -AXES="throughput,scroll,latency,startup,memory,load" +AXES="throughput,scroll,latency,startup,memory,load,fire" EQUALIZE=0 FORCE=0 +FULLSCREEN=1 # measure the render-path axes fullscreen (see below); --no-fullscreen opts out # equalized-condition targets (used only with --equalize) EQ_COLS=120; EQ_ROWS=40; EQ_FONT="Menlo"; EQ_FSIZE=14 while [ $# -gt 0 ]; do @@ -48,10 +52,14 @@ while [ $# -gt 0 ]; do --axes) AXES="$2"; shift ;; --equalize) EQUALIZE=1 ;; --force) FORCE=1 ;; + --no-fullscreen) FULLSCREEN=0 ;; *) echo "unknown arg: $1" >&2; exit 2 ;; esac shift done +# Fullscreen conflicts with --equalize's pinned grid — geometry equalization +# is the whole point there, so fullscreen is disabled under --equalize. +[ "$EQUALIZE" = 1 ] && FULLSCREEN=0 axis_selected() { case ",$AXES," in *",$1,"*) return 0 ;; *) return 1 ;; esac; } @@ -98,6 +106,42 @@ fi emit harness meta - - loadavg_start "$(sysctl -n vm.loadavg 2>/dev/null)" note emit harness meta - - uptime_start "$(uptime)" note +# ── fullscreen capability probe (all-or-nothing) ─────────────────── +# The render-path axes (throughput/scroll/fire/latency + load-active) are +# measured with every terminal's window in native macOS fullscreen: identical +# physical geometry for all terminals, which also neutralizes the +# per-terminal grid-pinning gaps (Termy/iTerm2/Warp/Terminal expose no size +# control). Setting AXFullScreen needs Accessibility permission for +# osascript; probe it ONCE up front and fall back to windowed FOR EVERY +# terminal if denied — a run never mixes fullscreen and windowed geometry. +if [ "$FULLSCREEN" = 1 ]; then + # Probe with a REAL AX attribute read. Two decoy probes are known-broken + # (both verified 2026-07-17 on macOS 26): process-name listing succeeds + # WITHOUT the permission, and `UI elements enabled` can return true while + # actual attribute access still fails -1719 (it reflects a global flag, + # not this process's TCC grant). Reading AXRole of the frontmost process + # exercises the same access class AXFullScreen needs. + if osascript -e 'tell application "System Events" to get value of attribute "AXRole" of (first process whose frontmost is true)' >/dev/null 2>&1; then + emit harness meta - - fullscreen "enabled (AXFullScreen via System Events, gated workload start)" note + else + FULLSCREEN=0 + emit harness meta - - fullscreen "DISABLED: osascript lacks Accessibility permission (System Events denied) — all terminals measured windowed" note + # Best-effort: a real AX access attempt makes macOS surface the + # "wants to control this computer using accessibility features" prompt + # for the host app — the passive `UI elements enabled` read never does. + osascript -e 'tell application "System Events" to get value of attribute "AXRole" of (first process whose frontmost is true)' >/dev/null 2>&1 || true + echo "WARNING: fullscreen requested, but the terminal you ran this from has no" + echo " macOS Accessibility permission, so AXFullScreen cannot be set." + echo " Measuring ALL terminals windowed (recorded in results)." + echo " To enable fullscreen runs:" + echo " System Settings > Privacy & Security > Accessibility >" + echo " add/enable the terminal app running this script, then rerun." + echo " (If a permission prompt just appeared, approving it is enough.)" + fi +else + emit harness meta - - fullscreen "disabled (--no-fullscreen or --equalize)" note +fi + # reps / timeouts (seconds) if [ "$QUICK" = 1 ]; then TP_REPS=1; SCROLL_REPS=1; LAT_RUNS=2; START_REPS=2 @@ -107,6 +151,7 @@ if [ "$QUICK" = 1 ]; then MEM_ACTIVE_AT_S=3; MEM_SAMPLE_EVERY_S=5; MEM_SETTLED_UNTIL_S=30 MEM_MULTITAB_N=3; MEM_LONGEVITY_CYCLES=2; MEM_LONGEVITY_IDLE_S=1 LOAD_IDLE_SETTLE_S=5; LOAD_IDLE_S=10 + FIRE_REPS=1; FIRE_SECS=3 else TP_REPS=3; SCROLL_REPS=3; LAT_RUNS=10; START_REPS=5 # latency (full): 10 process launches x 1000 kept iterations each (100 @@ -127,8 +172,12 @@ else # AppKit/first-frame transient decays through `ps pcpu`'s decaying average # for >10s), then sample the settled window for LOAD_IDLE_S. LOAD_IDLE_SETTLE_S=15; LOAD_IDLE_S=60 + # fire: median of FIRE_REPS runs, each FIRE_SECS of flat-out rendering + # after 60 discarded warmup frames (see docs/specs/bench-doom-fire.md). + FIRE_REPS=3; FIRE_SECS=10 fi TP_TIMEOUT=180; SCROLL_TIMEOUT=120; LAT_TIMEOUT=60; START_TIMEOUT=30 +FIRE_TIMEOUT=60 MEM_HOLD_TIMEOUT=20 # ── data files (only for the axes that consume them) ─────────────── @@ -147,7 +196,7 @@ fi # prebuilt dsr_probe would silently emit the old 4-field result format). tools_fresh() { local t - for t in nowns dsr_probe winwait wincount; do + for t in nowns dsr_probe winwait wincount fire dispinfo; do [ -x "$TOOLS/$t" ] || return 1 [ "$BENCH_DIR/tools/$t.c" -nt "$TOOLS/$t" ] && return 1 done @@ -156,10 +205,25 @@ tools_fresh() { tools_fresh || \ (cd "$BENCH_DIR/tools" && mkdir -p bin && \ cc -O2 -o bin/nowns nowns.c && cc -O2 -o bin/dsr_probe dsr_probe.c && \ + cc -O2 -o bin/fire fire.c && \ cc -O2 -framework ApplicationServices -o bin/winwait winwait.c && \ - cc -O2 -framework ApplicationServices -o bin/wincount wincount.c) + cc -O2 -framework ApplicationServices -o bin/wincount wincount.c && \ + cc -O2 -framework ApplicationServices -o bin/dispinfo dispinfo.c) chmod +x "$WRAPPER" +# ── target display (fullscreen runs) ─────────────────────────────── +# Pin every measured window to the display the harness was launched from +# (the one under the mouse cursor at run start) — without this, macOS may +# place a terminal's window (and thus its fullscreen Space) on whatever +# display that app last used. +DISP_X=""; DISP_Y="" +if [ "$FULLSCREEN" = 1 ]; then + read -r DISP_X DISP_Y DISP_W DISP_H < <("$TOOLS/dispinfo" 2>/dev/null || echo "") + if [ -n "$DISP_X" ]; then + emit harness meta - - target_display "${DISP_X},${DISP_Y} ${DISP_W}x${DISP_H} (display under the cursor at run start)" note + fi +fi + # ── config isolation: every terminal runs FRESH-INSTALL DEFAULTS ─── # The scored comparison is fresh defaults for everyone (METHODOLOGY.md # "Config isolation"). The user's real config files are NEVER read, written, @@ -178,8 +242,18 @@ chmod +x "$WRAPPER" # config.txt inside the iso dir on first launch) # kitty --config NONE (pure built-in defaults) ISO_XDG="$RUNTMP/xdg-config" -mkdir -p "$ISO_XDG/noa" "$ISO_XDG/termy" +mkdir -p "$ISO_XDG/noa" "$ISO_XDG/termy" "$ISO_XDG/alacritty" printf 'window-save-state = never\n' > "$ISO_XDG/noa/config" +# Alacritty consults XDG paths FIRST but falls back to ~/.alacritty.toml when +# nothing is found — an empty iso config makes the first candidate win so a +# user's home-dir config is never consulted. +: > "$ISO_XDG/alacritty/alacritty.toml" +# Rio ignores XDG_CONFIG_HOME but honors RIO_CONFIG_HOME (verified +# 2026-07-17 via --write-config). confirm-before-quit=false is the one +# harness-mechanics non-default (kitty precedent: kills must not hang on a +# confirm dialog); disclosed in the isolation note. +mkdir -p "$ISO_XDG/rio" +printf 'confirm-before-quit = false\n' > "$ISO_XDG/rio/config.toml" # kill_all_tracked is defined below (needs the PID registry); `|| true` keeps # an early exit (before definitions) from failing the trap. trap 'kill_all_tracked 2>/dev/null || true; rm -rf "$RUNTMP"' EXIT @@ -195,6 +269,8 @@ if [ "$EQUALIZE" = 1 ]; then # Termy has no CLI size/font control; font is settable via config only. # (Grid size has no config key -> stays at native default; documented.) printf 'font_family = %s\nfont_size = %s\n' "$EQ_FONT" "$EQ_FSIZE" > "$ISO_XDG/termy/config.txt" + # Rio: font settable via config; grid size only in pixels -> native default + printf 'confirm-before-quit = false\n[fonts]\nfamily = "%s"\nsize = %s\n' "$EQ_FONT" "$EQ_FSIZE" > "$ISO_XDG/rio/config.toml" fi # ── terminal registry ────────────────────────────────────────────── @@ -204,6 +280,11 @@ NOA_BIN="${NOA_BIN:-$REPO_DIR/target/release/noa}" GHOSTTY_BIN="/Applications/Ghostty.app/Contents/MacOS/ghostty" TERMY_BIN="/Applications/Termy.app/Contents/MacOS/termy" KITTY_BIN="/Applications/kitty.app/Contents/MacOS/kitty" +ALACRITTY_BIN="/Applications/Alacritty.app/Contents/MacOS/alacritty" +ITERM_BIN="/Applications/iTerm.app/Contents/MacOS/iTerm2" +WARP_BIN="/Applications/Warp.app/Contents/MacOS/stable" +TERMINAL_BIN="/System/Applications/Utilities/Terminal.app/Contents/MacOS/Terminal" +RIO_BIN="/Applications/Rio.app/Contents/MacOS/rio" term_present() { case "$1" in @@ -211,6 +292,11 @@ term_present() { ghostty) [ -x "$GHOSTTY_BIN" ] ;; termy) [ -x "$TERMY_BIN" ] ;; kitty) [ -x "$KITTY_BIN" ] ;; + alacritty) [ -x "$ALACRITTY_BIN" ] ;; + iterm2) [ -x "$ITERM_BIN" ] ;; + warp) [ -x "$WARP_BIN" ] ;; + terminal) [ -x "$TERMINAL_BIN" ] ;; + rio) [ -x "$RIO_BIN" ] ;; *) return 1 ;; esac } @@ -243,6 +329,11 @@ tracked_roots() { ghostty) want="$GHOSTTY_BIN" ;; termy) want="$TERMY_BIN" ;; kitty) want="$KITTY_BIN" ;; + alacritty) want="$ALACRITTY_BIN" ;; + iterm2) want="$ITERM_BIN" ;; + warp) want="$WARP_BIN" ;; + terminal) want="$TERMINAL_BIN" ;; + rio) want="$RIO_BIN" ;; *) want="" ;; esac while read -r p; do @@ -282,6 +373,45 @@ launch_noa() { # extra flags in "$@" fi } +# gen_env_wrapper — write a script that re-exports every NOA_* var of +# the CURRENT environment and execs $WRAPPER. Needed for terminals whose pty +# child is started through an out-of-process path that does not inherit our +# environment (iTerm2's AppleScript `create window ... command` — verified +# empirically 2026-07-17: the command runs on a real tty but with launchd's +# env, so the exported NOA_* contract never reaches wrapper.sh directly). +# %q-quotes values (NOA_BENCH_CMD contains quotes/spaces). +gen_env_wrapper() { + local f="$1" k v + { + echo '#!/bin/sh' + while IFS='=' read -r k v; do + printf 'export %s=%q\n' "$k" "$v" + done < <(env | grep '^NOA_') + printf 'exec %q\n' "$WRAPPER" + } > "$f" + chmod +x "$f" +} + +# iTerm2 has no `-e`/$SHELL path (verified 2026-07-17: SHELL= is +# ignored — the default profile execs the passwd login shell), so the pty +# child is created via AppleScript AFTER our own instance is up: +# 1. launch $ITERM_BIN directly (env-inherited child, pid registered); +# 2. poll `create window with default profile command ` until +# the app answers (fresh instance needs a few seconds to register). +# The AppleScript targets the app BY NAME, which is ambiguous if the user's +# own iTerm2 is already running — the selection gate below skips iterm2 +# entirely in that case, so this helper only ever talks to OUR instance. +iterm2_open_cmd_window() { + local eww="$1" i=0 + while [ $i -lt 30 ]; do + if osascript -e "tell application \"iTerm2\" to create window with default profile command \"$eww\"" >/dev/null 2>&1; then + return 0 + fi + sleep 0.5; i=$((i + 1)) + done + return 1 +} + # Launch a terminal (fresh process) running $WRAPPER as its pty child. # Env (NOA_MODE etc.) is already exported by the caller and inherited. # Ghostty always gets --window-save-state=never: without it, macOS state @@ -292,6 +422,64 @@ launch_noa() { # extra flags in "$@" # saved-state window"). kitty's confirm_os_window_close=0 is the one # non-default setting the harness needs (kills must not hang on a confirm # dialog); it is disclosed in METHODOLOGY.md. +# launch_term_default — the fresh-default launch of one terminal. +# Every branch must leave the app's pid in $! (iterm2 captures it before +# backgrounding its AppleScript helper, then restores it via `wait`-free +# subshell avoidance — see the branch). +launch_term_default() { + # NOA_GO non-empty == this launch is a fullscreen-gated measurement (set + # only by run_once/run_load_active under FULLSCREEN=1). Terminals with a + # NATIVE fullscreen flag get it at launch — reliable and needs no + # Accessibility; the rest go through fullscreen_term's AX path. + local fs="" + [ -n "${NOA_GO:-}" ] && fs=1 + case "$1" in + noa) launch_noa --cols 120 --rows 40 ;; + ghostty) + # NO --fullscreen here: ghostty 1.3.1 never executes its -e command + # when launched with --fullscreen (verified 2026-07-17: sentinel never + # appears with the flag, appears without it) — ghostty goes through + # the AX fullscreen path instead. The position flags are unaffected + # and keep the window on the target display. + "$GHOSTTY_BIN" --config-default-files=false --window-save-state=never \ + ${fs:+--window-position-x=$((DISP_X + 64))} ${fs:+--window-position-y=$((DISP_Y + 64))} \ + -e "$WRAPPER" >/dev/null 2>&1 & ;; + termy) XDG_CONFIG_HOME="$ISO_XDG" SHELL="$WRAPPER" "$TERMY_BIN" >/dev/null 2>&1 & ;; + kitty) "$KITTY_BIN" --config NONE -o confirm_os_window_close=0 \ + ${fs:+--start-as=fullscreen} "$WRAPPER" >/dev/null 2>&1 & ;; + alacritty) + if [ -n "$fs" ]; then + XDG_CONFIG_HOME="$ISO_XDG" "$ALACRITTY_BIN" -o 'window.startup_mode="Fullscreen"' \ + -o "window.position={x=$((DISP_X + 64)),y=$((DISP_Y + 64))}" -e "$WRAPPER" >/dev/null 2>&1 & + else + XDG_CONFIG_HOME="$ISO_XDG" "$ALACRITTY_BIN" -e "$WRAPPER" >/dev/null 2>&1 & + fi ;; + warp) SHELL="$WRAPPER" "$WARP_BIN" >/dev/null 2>&1 & ;; # UNVERIFIED (not installed here): if Warp ignores $SHELL the sentinel times out and axes report UNMEASURED honestly + rio) RIO_CONFIG_HOME="$ISO_XDG/rio" "$RIO_BIN" -e "$WRAPPER" >/dev/null 2>&1 & ;; + terminal) + # Direct binary launch = OUR OWN second instance (verified 2026-07-17: + # coexists with a user's running Terminal, env is inherited through + # its `login` wrapper, and the script argument becomes the pty child + # on a real tty). Never launched via `open`/AppleScript — those route + # through LaunchServices/name lookup to the USER'S instance. + "$TERMINAL_BIN" "$WRAPPER" >/dev/null 2>&1 & ;; + iterm2) + # Two-phase (see gen_env_wrapper/iterm2_open_cmd_window above): start + # our own instance, then AppleScript a command window running the + # env-wrapper. The app pid must be what $! carries out of this branch, + # so the helper's pid is captured and discarded in a subshell. + gen_env_wrapper "$RUNTMP/eww.iterm2.sh" + "$ITERM_BIN" >/dev/null 2>&1 & + local iterm_app_pid=$! + ( iterm2_open_cmd_window "$RUNTMP/eww.iterm2.sh" ) >/dev/null 2>&1 & + register_pid "$1" "$iterm_app_pid" + echo "$iterm_app_pid" + return 0 ;; + esac + register_pid "$1" "$!" + echo $! +} + launch_term() { if [ "$EQUALIZE" = 1 ]; then case "$1" in @@ -303,14 +491,20 @@ launch_term() { kitty) "$KITTY_BIN" --config NONE -o remember_window_size=no \ -o initial_window_width="${EQ_COLS}c" -o initial_window_height="${EQ_ROWS}c" \ -o font_family="$EQ_FONT" -o font_size="$EQ_FSIZE" -o confirm_os_window_close=0 "$WRAPPER" >/dev/null 2>&1 & ;; + alacritty) XDG_CONFIG_HOME="$ISO_XDG" "$ALACRITTY_BIN" \ + -o "window.dimensions.columns=$EQ_COLS" -o "window.dimensions.lines=$EQ_ROWS" \ + -o "font.normal.family=\"$EQ_FONT\"" -o "font.size=$EQ_FSIZE" \ + -e "$WRAPPER" >/dev/null 2>&1 & ;; + rio) RIO_CONFIG_HOME="$ISO_XDG/rio" "$RIO_BIN" -e "$WRAPPER" >/dev/null 2>&1 & ;; # font via iso config; grid px-only -> native default + iterm2|warp|terminal) + # no CLI size/font control (prefs-based) — equalize unsupported; + # native defaults, recorded via the equalized-notes emit below + launch_term_default "$1" + return 0 ;; esac else - case "$1" in - noa) launch_noa --cols 120 --rows 40 ;; - ghostty) "$GHOSTTY_BIN" --config-default-files=false --window-save-state=never -e "$WRAPPER" >/dev/null 2>&1 & ;; - termy) XDG_CONFIG_HOME="$ISO_XDG" SHELL="$WRAPPER" "$TERMY_BIN" >/dev/null 2>&1 & ;; - kitty) "$KITTY_BIN" --config NONE -o confirm_os_window_close=0 "$WRAPPER" >/dev/null 2>&1 & ;; - esac + launch_term_default "$1" + return 0 fi register_pid "$1" "$!" echo $! @@ -324,6 +518,57 @@ launch_term() { # by mistake. Falls back to `open -a ` (focus-only, never kills) only # if System Events automation is unavailable AND our instance is still alive — # `open -a` on an already-dead instance would launch a fresh one. +# fullscreen_term — put OUR tracked instance's window into native +# macOS fullscreen. PID-scoped: System Events is addressed via `unix id`, so +# a user's own instance of the same app is never touched. Sequence: wait for +# a window to exist (wincount over our pids), set AXFullScreen, poll the +# attribute back until true, then a short settle for the Space animation. +# Returns 0 on verified fullscreen, 1 otherwise (caller decides how to note). +fullscreen_term() { + local pid i st + pid="$(tracked_roots "$1")"; pid="$(echo $pid | awk '{print $1}')" + [ -z "$pid" ] && return 1 + # window materialization can lag the pid — poll pid-scoped wincount + for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16; do + # shellcheck disable=SC2046 + [ "$("$TOOLS/wincount" $(app_pids "$1") 2>/dev/null || echo 0)" -ge 1 ] && break + sleep 0.5 + done + # Native-flag terminals were launched fullscreen already (kitty + # --start-as=fullscreen, alacritty startup_mode; ghostty is NOT here — + # its --fullscreen suppresses the -e command, see launch_term_default) — + # just allow the Space animation to finish; no Accessibility needed. + # (Attempting the AX verify anyway would false-negative them as FAILED + # when the AX read is denied — observed with kitty 2026-07-17: region + # 244x85 = clearly fullscreen, yet flagged FAILED by the AX read.) + case "$1" in + kitty|alacritty) sleep 1.2; return 0 ;; + esac + # Pin the window to the target display BEFORE fullscreening — the + # fullscreen Space is created on the display containing the window. + # Tolerated to fail (some windows reject AXPosition); fullscreen then + # happens on whatever display the window opened on. + if [ -n "$DISP_X" ]; then + osascript -e "tell application \"System Events\" to tell (first process whose unix id is $pid) to set position of window 1 to {$((DISP_X + 64)), $((DISP_Y + 64))}" >/dev/null 2>&1 + sleep 0.3 + fi + if ! osascript -e "tell application \"System Events\" to tell (first process whose unix id is $pid) to set value of attribute \"AXFullScreen\" of window 1 to true" >/dev/null 2>&1; then + return 1 + fi + # Verify best-effort: the AX read can be denied while the set succeeded + # (observed 2026-07-17: kitty at a fullscreen-sized region while the read + # reported failure), so a set that raised no error counts as applied even + # if the read never confirms — the per-rep fire region stays the ground + # truth in raw.tsv. + for i in 1 2 3 4 5 6 7 8; do + st="$(osascript -e "tell application \"System Events\" to tell (first process whose unix id is $pid) to get value of attribute \"AXFullScreen\" of window 1" 2>/dev/null)" + [ "$st" = "true" ] && { sleep 0.8; return 0; } + sleep 0.5 + done + sleep 0.8 + return 0 +} + activate_term() { local pid; pid="$(tracked_roots "$1")"; pid="$(echo $pid | awk '{print $1}')" [ -z "$pid" ] && return 0 @@ -516,15 +761,31 @@ run_once() { kill_term "$term"; sleep 0.4 + # Fullscreen gate: workload modes hold at wait_go until the window reached + # its measurement geometry; startup is never gated (it measures the launch + # itself and its sentinel predates any window manipulation). + local go="" + if [ "$FULLSCREEN" = 1 ] && [ "$mode" != startup ]; then + go="$RUNTMP/${term}.${mode}.go.$RANDOM" + rm -f "$go" + fi export NOA_MODE="$mode" NOA_SENTINEL="$sentinel" NOA_NOWNS="$NOWNS" \ - NOA_PROBE="$PROBE" NOA_RESULT="$result" NOA_BENCH_CMD="$cmd" + NOA_PROBE="$PROBE" NOA_FIRE="$FIRE" NOA_RESULT="$result" \ + NOA_BENCH_CMD="$cmd" NOA_GO="$go" local t0 t1 t0="$("$NOWNS")" launch_term "$term" >/dev/null - if [ "$mode" = latency ]; then + if [ -n "$go" ]; then + fullscreen_term "$term" || emit "$term" meta - - fullscreen_window "FAILED (windowed for one $mode rep)" note + : > "$go" + fi + if [ "$mode" = latency ] || [ "$mode" = fire ]; then # Focus the fresh window so render-thread/focus-gated DSR responders - # (Termy) reply. Backgrounded with a delay so the launch timestamp path + # (Termy) reply — and, for the fire axis, so no terminal is measured + # while macOS throttles its unfocused/occluded window (display-paced + # consumers would be understated; focus is applied uniformly). + # Backgrounded with a delay so the launch timestamp path # is untouched; the probe's blocking read simply resumes once focus # lands. Repeated: app registration with the window server can lag the # first attempt on a cold start. The delays are RANDOMIZED (+-0.7s @@ -553,7 +814,7 @@ run_once() { startup) echo "$((t1 - t0))" ;; - latency) + latency|fire) cat "$result" ;; esac @@ -562,6 +823,10 @@ run_once() { median() { sort -n | awk '{a[NR]=$1} END{ if(NR==0){print 0} else if(NR%2){print a[(NR+1)/2]} else {print int((a[NR/2]+a[NR/2+1])/2)} }'; } +# median_f — float-preserving median (median() ints the even-count midpoint, +# which would truncate fps values like "812.4"). +median_f() { sort -n | awk '{a[NR]=$1} END{ if(NR==0){print 0} else if(NR%2){print a[(NR+1)/2]} else {printf "%.1f\n", (a[NR/2]+a[NR/2+1])/2} }'; } + # pooled_stats -> "median p95 p99 max count" over the # POOLED distribution (all kept iterations of all launches concatenated). # Nearest-rank percentiles matching dsr_probe's own convention @@ -621,7 +886,7 @@ run_mem_idle() { local term="$1" sentinel="$RUNTMP/${term}.memidle.sentinel" kill_term "$term"; sleep 0.4 rm -f "$sentinel" - export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" + export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" NOA_GO="" launch_term "$term" >/dev/null if ! wait_sentinel "$sentinel" "$MEM_HOLD_TIMEOUT"; then kill_term "$term"; sleep 0.3; echo ""; return; fi mem_dual_sample "$term" idle @@ -634,7 +899,7 @@ run_mem_scrollback() { local term="$1" sentinel="$RUNTMP/${term}.memscroll.sentinel" kill_term "$term"; sleep 0.4 rm -f "$sentinel" - export NOA_MODE=scroll NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="cat '$SCROLLF'" + export NOA_MODE=scroll NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="cat '$SCROLLF'" NOA_GO="" launch_term "$term" >/dev/null if ! wait_sentinel "$sentinel" "$SCROLL_TIMEOUT"; then kill_term "$term"; sleep 0.3; echo ""; return; fi mem_dual_sample "$term" scrollback @@ -659,7 +924,7 @@ run_mem_multitab() { for i in $(seq 1 "$n"); do local sentinel="$RUNTMP/${term}.multitab.$i.sentinel" rm -f "$sentinel" - export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" + export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" NOA_GO="" launch_term "$term" >/dev/null sleep 0.3 done @@ -699,7 +964,7 @@ run_mem_longevity() { # already sleeps $idle before every per-cycle sentinel, but nothing keeps # it alive after the final cycle's sentinel + the closing "$NOA_SENTINEL" # write, so without HOLD the process can exit before that last sample. - export NOA_MODE=longevity NOA_SENTINEL="$sentinel" NOA_CYCLES="$cycles" NOA_IDLE_S="$idle" \ + export NOA_MODE=longevity NOA_SENTINEL="$sentinel" NOA_CYCLES="$cycles" NOA_IDLE_S="$idle" NOA_GO="" \ NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="cat '$SCROLLF'" launch_term "$term" >/dev/null local samples="" done_cycles=0 i @@ -727,7 +992,7 @@ run_load_idle_sample() { local term="$1" dur="$LOAD_IDLE_S" sentinel="$RUNTMP/${term}.loadidle.sentinel" kill_term "$term"; sleep 0.4 rm -f "$sentinel" - export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" + export NOA_MODE=hold NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="" NOA_GO="" launch_term "$term" >/dev/null if ! wait_sentinel "$sentinel" "$MEM_HOLD_TIMEOUT"; then kill_term "$term"; sleep 0.3; echo ""; return; fi sleep "$LOAD_IDLE_SETTLE_S" # settle: launch transient fully discarded @@ -764,7 +1029,15 @@ run_load_active() { local sentinel="$RUNTMP/${term}.loadactive.${mode}.sentinel" kill_term "$term"; sleep 0.4 rm -f "$sentinel" - export NOA_MODE="$mode" NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" NOA_BENCH_CMD="$cmd" + # Same fullscreen geometry as the throughput/scroll axes (the "same work" + # comparison must run under the same conditions). The gate ordering below + # keeps the fullscreen transition's own CPU out of the measurement: the + # "before" snapshot is taken AFTER fullscreen settles, and only then is + # the workload released. + local go="" + if [ "$FULLSCREEN" = 1 ]; then go="$RUNTMP/${term}.loadactive.go.$RANDOM"; rm -f "$go"; fi + export NOA_MODE="$mode" NOA_SENTINEL="$sentinel" NOA_HOLD=1 NOA_NOWNS="$NOWNS" \ + NOA_BENCH_CMD="$cmd" NOA_GO="$go" launch_term "$term" >/dev/null local pids="" waited=0 while [ -z "$pids" ] && [ "$waited" -lt 50 ]; do @@ -772,7 +1045,9 @@ run_load_active() { [ -z "$pids" ] && sleep 0.1 waited=$((waited + 1)) done + [ -n "$go" ] && { fullscreen_term "$term" || true; } local before; before="$(cpu_time_cs "$pids")" + [ -n "$go" ] && : > "$go" if ! wait_sentinel "$sentinel" "$timeout"; then kill_term "$term"; sleep 0.3; echo ""; return; fi local after; after="$(cpu_time_cs "$(app_pids "$term")")" kill_term "$term"; sleep 0.4 @@ -782,13 +1057,32 @@ run_load_active() { } # ── select terminals ─────────────────────────────────────────────── -ALL="noa ghostty termy kitty" +# warp is opt-in only (`--only ...,warp`): its $SHELL launch is unverified +# and it carries account/AI-agent state that doesn't belong in a default +# sweep — excluded from the default set rather than skipped at runtime. +ALL="noa ghostty termy kitty alacritty iterm2 terminal rio" +case ",$ONLY," in *",warp,"*) ALL="$ALL warp" ;; esac SELECTED="" for t in $ALL; do if [ -n "$ONLY" ]; then case ",$ONLY," in *",$t,"*) ;; *) continue ;; esac; fi - if term_present "$t"; then SELECTED="$SELECTED $t"; else + if ! term_present "$t"; then echo "skip (not installed): $t"; emit "$t" meta - - present 0 bool + continue + fi + # iTerm2's pty child is created via AppleScript that targets the app BY + # NAME (no `-e`/$SHELL path exists — verified 2026-07-17). With a user's + # own iTerm2 running, that name lookup could create/act on windows in + # THEIR session, so the collateral-safety invariant forces a skip. + # Read-only scan; nothing is signaled. + if [ "$t" = iterm2 ]; then + live="$(ps -axo pid=,comm= 2>/dev/null | awk '$2 ~ /iTerm\.app\/Contents\/MacOS\/iTerm2$/ {printf "%s ", $1}')" + if [ -n "$live" ]; then + echo "skip (iterm2): user iTerm2 instance running ($live) — AppleScript window creation targets the app by name and could touch the user's session; quit iTerm2 and re-run to include it" + emit iterm2 meta - - skipped "user-instance-running:$live" note + continue + fi fi + SELECTED="$SELECTED $t" done echo "Terminals under test:$SELECTED" echo "Results -> $OUT_DIR" @@ -801,6 +1095,11 @@ for t in $SELECTED; do ghostty) emit ghostty meta - - isolation "--config-default-files=false (no config files read) + --window-save-state=never (phantom saved-state window suppressed)" note ;; termy) emit termy meta - - isolation "XDG_CONFIG_HOME=iso-dir (fresh defaults; verified: writes its default config.txt into the iso dir)" note ;; kitty) emit kitty meta - - isolation "--config NONE (built-in defaults) + -o confirm_os_window_close=0 (harness kill-safety, disclosed)" note ;; + alacritty) emit alacritty meta - - isolation "XDG_CONFIG_HOME=iso-dir with an empty alacritty.toml (first config candidate wins; user's ~/.alacritty.toml never consulted)" note ;; + iterm2) emit iterm2 meta - - isolation "NOT ISOLATED: user preferences (com.googlecode.iterm2 defaults plist) apply; prefs cannot be redirected without mutating user defaults. Restored/startup windows of our instance may add login-shell trees to memory/load — check multitab_procs/wincount" note ;; + warp) emit warp meta - - isolation "NOT ISOLATED: user config + account state apply; \$SHELL=wrapper launch is UNVERIFIED (Warp not installed at authoring) — UNMEASURED axes mean Warp ignored \$SHELL" note ;; + terminal) emit terminal meta - - isolation "NOT ISOLATED: user preferences (com.apple.Terminal defaults plist) apply; launched as our own second instance with the wrapper as document argument (verified: real tty, env inherited via login -p)" note ;; + rio) emit rio meta - - isolation "RIO_CONFIG_HOME=iso-dir (fresh defaults; XDG_CONFIG_HOME is ignored by Rio — verified via --write-config) + confirm-before-quit=false (harness kill-safety, disclosed)" note ;; esac done @@ -870,6 +1169,57 @@ for term in $SELECTED; do done fi # axis: scroll +# ── FIRE (DOOM-fire IO stress: truecolor full-region repaint fps) ── +# Fixed 80x24 region -> byte-identical stream for every terminal (fps scales +# with cell count and Termy's grid is not pinnable, so window-sized rendering +# would measure default geometry, not the terminal). Producer-side fps under +# pty flow control — same "consume the pipe" proxy as the throughput axis. +# Skipped under --equalize: the fixed region makes the workload grid/font- +# independent by construction. See docs/specs/bench-doom-fire.md. +if axis_selected fire && [ "$EQUALIZE" != 1 ]; then +export NOA_FIRE_SECS="$FIRE_SECS" +# Fire condition follows the run's geometry mode: on fullscreen runs the fire +# fills the live window (upstream DOOM-fire's official full-window condition +# — the fullscreen gate guarantees the winsize read happens at final +# geometry; cell count then follows each terminal's own font defaults, which +# is exactly the upstream comparison shape and is disclosed via the per-rep +# region). Windowed fallback runs keep the fixed 80x24 region — full-window +# fps on unequal window geometry would measure the geometry lottery. +if [ "$FULLSCREEN" = 1 ]; then + export NOA_FIRE_ARG=full + FIRE_COND="full-window @ fullscreen (upstream DOOM-fire condition; cell count follows each terminal's font defaults)" +else + export NOA_FIRE_ARG="" + FIRE_COND="fixed 80x24 region (windowed fallback; byte-identical stream)" +fi +emit harness meta - - fire_condition "$FIRE_COND" note +for term in $SELECTED; do + echo "[fire] $term ($FIRE_REPS reps x ${FIRE_SECS}s, $FIRE_COND, 60 warmup frames discarded)" + samples=""; got=0 + for r in $(seq 1 $FIRE_REPS); do + out="$(run_once "$term" fire "$FIRE_TIMEOUT")" || continue + set -- $out + frames="${1:-0}"; elapsed_ns="${2:-0}"; fps="${3:-0}"; winsz="${4:-unknown}"; region="${5:-unknown}" + case "$frames" in ''|*[!0-9]*) continue ;; 0) continue ;; esac + got=$((got + 1)) + emit "$term" fire - "$r" frames "$frames" count + emit "$term" fire - "$r" elapsed_ns "$elapsed_ns" ns + emit "$term" fire - "$r" fps "$fps" fps + emit "$term" fire - "$r" winsize "$winsz" cells + emit "$term" fire - "$r" region "$region" cells + samples="$samples$fps\n" + done + if [ "$got" -ge 1 ]; then + med="$(printf "$samples" | median_f)" + emit "$term" fire - median fps "$med" fps + echo " median ${med} fps" + else + emit "$term" fire - - status UNMEASURED timeout + echo " UNMEASURED (timeout)" + fi +done +fi # axis: fire + # Latency + startup are condition-independent; skip them under --equalize # (the equalized re-run targets the render-sensitive throughput+scroll axes). if [ "$EQUALIZE" != 1 ]; then @@ -1094,6 +1444,9 @@ if [ "$EQUALIZE" = 1 ]; then ghostty) emit "$term" meta - - equalized "grid=${EQ_COLS}x${EQ_ROWS};font=${EQ_FONT}@${EQ_FSIZE};clean-config" note ;; kitty) emit "$term" meta - - equalized "grid=${EQ_COLS}x${EQ_ROWS};font=${EQ_FONT}@${EQ_FSIZE};config=NONE" note ;; termy) emit "$term" meta - - equalized "grid=NATIVE-DEFAULT(no-size-key);font=${EQ_FONT}@${EQ_FSIZE}" note ;; + alacritty) emit "$term" meta - - equalized "grid=${EQ_COLS}x${EQ_ROWS};font=${EQ_FONT}@${EQ_FSIZE};via -o overrides" note ;; + iterm2|warp|terminal) emit "$term" meta - - equalized "NATIVE-DEFAULT(prefs-based; no CLI size/font control)" note ;; + rio) emit "$term" meta - - equalized "grid=NATIVE-DEFAULT(px-only window size);font=${EQ_FONT}@${EQ_FSIZE} via RIO_CONFIG_HOME config" note ;; esac done fi @@ -1114,6 +1467,7 @@ fi # ── aggregate → json + markdown ──────────────────────────────────── python3 "$BENCH_DIR/aggregate.py" "$RAW" "$OUT_DIR" "$TS" +python3 "$BENCH_DIR/visualize.py" "$OUT_DIR" >/dev/null 2>&1 || true cp "$BENCH_DIR/METHODOLOGY.md" "$OUT_DIR/METHODOLOGY.md" 2>/dev/null || true echo echo "===================================================================" @@ -1121,3 +1475,4 @@ cat "$OUT_DIR/table.md" echo "===================================================================" echo "JSON: $OUT_DIR/results.json" echo "Table: $OUT_DIR/table.md" +echo "HTML: $OUT_DIR/report.html" diff --git a/bench/run_benchmark.sh b/bench/run_benchmark.sh index 4d1f297..6f873fa 100755 --- a/bench/run_benchmark.sh +++ b/bench/run_benchmark.sh @@ -1,47 +1,158 @@ #!/bin/bash +# run_benchmark.sh — manual in-terminal benchmark. +# +# Runs any subset of the suite's workloads INSIDE the terminal you execute it +# from: run_all.sh measures terminals from the outside (fresh process, config +# isolation, PID-scoped lifecycle) and is the source of scored numbers; this +# script is the quick self-check you run by hand in whatever terminal you are +# sitting in. Same workload files and tools, no isolation, no ranking. +# +# Usage: +# bench/run_benchmark.sh # all tests, prompted one by one +# bench/run_benchmark.sh ascii fire # just these tests +# bench/run_benchmark.sh --yes scroll # no [Enter] prompts +# bench/run_benchmark.sh --list # list available tests +# +# Tests: +# ascii cat 150MB_ascii.txt (throughput, plain text) +# unicode cat 150MB_unicode.txt (throughput, CJK/emoji/CSI mix) +# scroll cat scroll_stress.txt (SGR churn + scroll regions, ~40MB) +# fire DOOM-fire IO stress (fixed 80x24 truecolor repaint, fps) +# latency DSR ESC[6n round-trip (parser-responsiveness proxy, µs) +# +# Env knobs: FIRE_SECS (default 10). The fire test renders to the LIVE +# window size by default (upstream DOOM-fire-zig's full-window +# condition, same as the harness's fullscreen runs) — fps scales +# ~1/cell-count, so compare numbers only at the same window +# geometry. FIRE_FIXED=1 switches to the fixed 80x24 region +# (byte-identical stream, geometry-independent). +# LAT_ITERS (default 1000), LAT_WARMUP (default 100). +set -u -# Ensure data files are generated -if [ ! -f "150MB_ascii.txt" ] || [ ! -f "150MB_unicode.txt" ]; then - echo "Benchmark files not found. Generating data files (150MB each)..." - python3 generate_data.py +BENCH_DIR="$(cd "$(dirname "$0")" && pwd)" +TOOLS="$BENCH_DIR/tools/bin" + +YES=0 +TESTS="" +for a in "$@"; do + case "$a" in + --yes|-y) YES=1 ;; + --list) echo "tests: ascii unicode scroll fire latency"; exit 0 ;; + ascii|unicode|scroll|fire|latency) TESTS="$TESTS $a" ;; + *) echo "unknown arg: $a (tests: ascii unicode scroll fire latency; flags: --yes, --list)" >&2; exit 2 ;; + esac +done +[ -z "$TESTS" ] && TESTS="ascii unicode scroll fire latency" + +if [ ! -t 1 ]; then + echo "WARNING: stdout is not a tty — without pty flow control these numbers" >&2 + echo "measure the pipe/file, not a terminal. Run this inside the terminal" >&2 + echo "you want to measure." >&2 fi -echo "=========================================" -echo "Terminal IO Throughput Benchmarks" -echo "=========================================" -echo "Note: To measure terminal rendering speed, the output must be displayed on screen." -echo "This will flood your terminal with text for a moment." -echo "" -echo "Ready to run: time cat 150MB_ascii.txt" -read -p "Press [Enter] to start ASCII benchmark..." - -echo "--- START: ASCII BENCHMARK ---" -# Render to screen (stdout), but capture time output (stderr) -{ time cat 150MB_ascii.txt; } 2> .ascii_time.txt -echo "--- END: ASCII BENCHMARK ---" -echo "" - -echo "Ready to run: time cat 150MB_unicode.txt" -read -p "Press [Enter] to start Unicode benchmark..." - -echo "--- START: UNICODE BENCHMARK ---" -# Render to screen (stdout), but capture time output (stderr) -{ time cat 150MB_unicode.txt; } 2> .unicode_time.txt -echo "--- END: UNICODE BENCHMARK ---" -echo "" - -# Parse elapsed times -ascii_real=$(grep real .ascii_time.txt | awk '{print $2}') -unicode_real=$(grep real .unicode_time.txt | awk '{print $2}') - -# Clean up temp files -rm -f .ascii_time.txt .unicode_time.txt +selected() { case " $TESTS " in *" $1 "*) return 0 ;; *) return 1 ;; esac; } + +confirm() { + [ "$YES" = 1 ] && return 0 + read -r -p "Press [Enter] to start the $1 test (this floods the terminal)..." +} + +# ── prerequisites (only for the selected tests) ───────────────────── +if { selected ascii || selected unicode; } && \ + { [ ! -f "$BENCH_DIR/150MB_ascii.txt" ] || [ ! -f "$BENCH_DIR/150MB_unicode.txt" ]; }; then + echo "Generating throughput data files (150MB each)..." + (cd "$BENCH_DIR" && python3 generate_data.py) +fi +if selected scroll && [ ! -f "$BENCH_DIR/scroll_stress.txt" ]; then + echo "Generating scroll stress file..." + (cd "$BENCH_DIR" && python3 gen_scroll.py 40 scroll_stress.txt) +fi +# nowns is needed by every test's timing; fire/latency need their own tool. +build_tool() { # name [extra cc flags...] + local n="$1"; shift + if [ ! -x "$TOOLS/$n" ] || [ "$BENCH_DIR/tools/$n.c" -nt "$TOOLS/$n" ]; then + mkdir -p "$TOOLS" + (cd "$BENCH_DIR/tools" && cc -O2 "$@" -o "bin/$n" "$n.c") || exit 1 + fi +} +build_tool nowns +selected fire && build_tool fire +selected latency && build_tool dsr_probe +NOWNS="$TOOLS/nowns" + +SUMMARY="" +note() { SUMMARY="$SUMMARY - $1\n"; } + +# ── cat-workload tests (ascii / unicode / scroll) ─────────────────── +run_cat_test() { # name file + local name="$1" file="$2" + local bytes; bytes=$(stat -f%z "$file") + confirm "$name" + echo "--- START: $name ---" + local t0 t1 + t0=$("$NOWNS") + cat "$file" + t1=$("$NOWNS") + # The workload files leave terminal state behind (scroll_stress sets a + # DECSTBM scroll region and homes the cursor every 500 lines; the unicode + # file embeds SGR test cases), so anything printed next would overlap the + # flood residue mid-screen. Reset attributes + scroll region and park the + # cursor on the last row so output continues scrolling normally. + printf '\033[0m\033[r\033[9999;1H\n' + echo "--- END: $name ---" + local ms mibs + ms=$(awk -v ns=$((t1 - t0)) 'BEGIN{printf "%.0f", ns/1e6}') + mibs=$(awk -v b="$bytes" -v ns=$((t1 - t0)) 'BEGIN{printf "%.1f", (b/1048576)/(ns/1e9)}') + echo "$name: ${ms} ms (${mibs} MiB/s)" + note "$name: ${ms} ms (${mibs} MiB/s)" +} + +selected ascii && run_cat_test ascii "$BENCH_DIR/150MB_ascii.txt" +selected unicode && run_cat_test unicode "$BENCH_DIR/150MB_unicode.txt" +selected scroll && run_cat_test scroll "$BENCH_DIR/scroll_stress.txt" + +# ── fire (DOOM-fire IO stress) ────────────────────────────────────── +if selected fire; then + FIRE_SECS="${FIRE_SECS:-10}" + fire_mode_arg="full" + fire_mode_desc="full window (upstream DOOM-fire condition; fps depends on window geometry)" + if [ "${FIRE_FIXED:-0}" = 1 ]; then + fire_mode_arg="" + fire_mode_desc="fixed 80x24 region (byte-identical stream, geometry-independent)" + fi + confirm "fire (${FIRE_SECS}s, ${fire_mode_desc})" + result=$(mktemp) || exit 1 + "$TOOLS/fire" "$FIRE_SECS" "$result" $fire_mode_arg + read -r frames elapsed_ns fps winsz region < "$result" + rm -f "$result" + echo "fire: ${fps} fps (${frames} frames / ${FIRE_SECS}s, region ${region}, winsize ${winsz})" + note "fire: ${fps} fps (${frames} frames, ${FIRE_SECS}s, region ${region})" +fi + +# ── latency (DSR round-trip proxy) ────────────────────────────────── +if selected latency; then + LAT_ITERS="${LAT_ITERS:-1000}" + LAT_WARMUP="${LAT_WARMUP:-100}" + confirm "latency (${LAT_ITERS} iterations)" + result=$(mktemp) || exit 1 + "$TOOLS/dsr_probe" "$LAT_ITERS" "$LAT_WARMUP" "$result" "" + read -r med p95 p99 max min count < "$result" + rm -f "$result" + if [ "${count:-0}" -gt 0 ] 2>/dev/null && [ "${med:-0}" -gt 0 ] 2>/dev/null; then + lat_line=$(awk -v m="$med" -v a="$p95" -v b="$p99" -v x="$max" -v c="$count" \ + 'BEGIN{printf "median %.1f / p95 %.1f / p99 %.1f / max %.1f µs (%d samples)", m/1000, a/1000, b/1000, x/1000, c}') + else + lat_line="UNMEASURED (no DSR reply — is this a real terminal?)" + fi + echo "latency: $lat_line" + note "latency: $lat_line" +fi +# ── summary ───────────────────────────────────────────────────────── +echo echo "=========================================" echo " BENCHMARK SUMMARY" echo "=========================================" -echo "Your Terminal Performance:" -echo " - ASCII (150MB_ascii.txt): $ascii_real" -echo " - Unicode (150MB_unicode.txt): $unicode_real" +printf "%b" "$SUMMARY" echo "=========================================" -echo "Benchmarks completed!" +echo "Scored cross-terminal comparison: bench/run_all.sh (see METHODOLOGY.md)." diff --git a/bench/tools/dispinfo.c b/bench/tools/dispinfo.c new file mode 100644 index 0000000..0382c66 --- /dev/null +++ b/bench/tools/dispinfo.c @@ -0,0 +1,34 @@ +// dispinfo — print "X Y W H" (global desktop coordinates) of the display +// containing the mouse cursor, i.e. the user's CURRENT display. +// +// Used by the fullscreen measurement mode to pin every terminal's window to +// the display the user launched the harness from — without pinning, macOS +// places windows on whatever display an app last used, so measured windows +// (and their fullscreen Spaces) could land on another monitor. +// +// Reading cursor location + display bounds needs no special permission. +// Falls back to the main display if the cursor is somehow on none. +#include +#include + +int main(void) { + CGEventRef e = CGEventCreate(NULL); + CGPoint p = CGEventGetLocation(e); + CFRelease(e); + + CGDirectDisplayID ids[16]; + uint32_t n = 0; + CGGetActiveDisplayList(16, ids, &n); + for (uint32_t i = 0; i < n; i++) { + CGRect b = CGDisplayBounds(ids[i]); + if (CGRectContainsPoint(b, p)) { + printf("%d %d %d %d\n", (int)b.origin.x, (int)b.origin.y, + (int)b.size.width, (int)b.size.height); + return 0; + } + } + CGRect b = CGDisplayBounds(CGMainDisplayID()); + printf("%d %d %d %d\n", (int)b.origin.x, (int)b.origin.y, + (int)b.size.width, (int)b.size.height); + return 0; +} diff --git a/bench/tools/fire.c b/bench/tools/fire.c new file mode 100644 index 0000000..051dd64 --- /dev/null +++ b/bench/tools/fire.c @@ -0,0 +1,210 @@ +// fire.c — DOOM-fire IO stress workload (bench axis "fire"). +// +// Renders the classic DOOM fire effect (Fabien Sanglard's algorithm, the +// workload popularized as a terminal benchmark by DOOM-fire-zig) into a +// FIXED 80x24 cell region as truecolor half-blocks: every frame repaints the +// whole region with per-cell SGR 38;2/48;2 RGB + U+2584. The region is fixed +// (not window-sized) so every terminal consumes a byte-identical stream — +// fps is inversely proportional to cell count and Termy's grid size is not +// pinnable (see docs/specs/bench-doom-fire.md, decision 2). +// +// Producer-side fps under pty flow control: the pty's small kernel buffer +// makes write() block until the terminal drains, so frames-written/second +// ~= frames-consumed/second — the same "consume the pipe" proxy as the +// throughput axis. Frame size is constant (no SGR run-length dedupe), so +// fps maps linearly to drain MiB/s. +// +// Usage: fire [full] +// 60 warmup frames (discarded: atlas population, alt-screen entry), then +// render flat-out for , then write +// " x x" +// to . +// +// Default: fixed 80x24 region (the SCORED harness condition — byte- +// identical stream for every terminal). `full` renders to the live window +// size instead, approximating upstream DOOM-fire-zig's full-window +// condition for manual anchor runs; fps scales ~1/cell-count, so full-mode +// numbers are window-geometry-dependent and never enter the scored axis. +// +// Deterministic: fixed xorshift32 seed -> identical frame sequence every +// run (verify: two runs redirected to files agree on their common prefix; +// total length differs only by how many frames fit in the wall-clock +// budget). + +#include +#include +#include +#include +#include +#include +#include +#include + +#define DEF_W 80 /* default region: cells wide */ +#define DEF_ROWS 24 /* default region: cell rows */ +#define MAX_W 1000 /* full-mode safety caps for buffer sizing */ +#define MAX_ROWS 500 +#define NPAL 37 +#define WARMUP_FRAMES 60 + +/* Region geometry — fixed 80x24 by default, live winsize in `full` mode. */ +static int W = DEF_W; /* cells wide = fire pixels wide */ +static int ROWS = DEF_ROWS; /* cell rows */ +static int PH = DEF_ROWS * 2; /* fire pixels tall (2 per cell row) */ + +/* Sanglard's 37-entry DOOM fire palette (black -> red -> yellow -> white). */ +static const uint8_t PAL[NPAL][3] = { + {0x07, 0x07, 0x07}, {0x1F, 0x07, 0x07}, {0x2F, 0x0F, 0x07}, + {0x47, 0x0F, 0x07}, {0x57, 0x17, 0x07}, {0x67, 0x1F, 0x07}, + {0x77, 0x1F, 0x07}, {0x8F, 0x27, 0x07}, {0x9F, 0x2F, 0x07}, + {0xAF, 0x3F, 0x07}, {0xBF, 0x47, 0x07}, {0xC7, 0x47, 0x07}, + {0xDF, 0x4F, 0x07}, {0xDF, 0x57, 0x07}, {0xDF, 0x57, 0x07}, + {0xD7, 0x5F, 0x07}, {0xD7, 0x5F, 0x07}, {0xD7, 0x67, 0x0F}, + {0xCF, 0x6F, 0x0F}, {0xCF, 0x77, 0x0F}, {0xCF, 0x7F, 0x0F}, + {0xCF, 0x87, 0x17}, {0xC7, 0x87, 0x17}, {0xC7, 0x8F, 0x17}, + {0xC7, 0x97, 0x1F}, {0xBF, 0x9F, 0x1F}, {0xBF, 0x9F, 0x1F}, + {0xBF, 0xA7, 0x27}, {0xBF, 0xA7, 0x27}, {0xBF, 0xAF, 0x2F}, + {0xB7, 0xAF, 0x2F}, {0xB7, 0xB7, 0x2F}, {0xB7, 0xB7, 0x37}, + {0xCF, 0xCF, 0x6F}, {0xDF, 0xDF, 0x9F}, {0xEF, 0xEF, 0xC7}, + {0xFF, 0xFF, 0xFF}, +}; + +/* Precomposed SGR strings per palette index (frame composition must stay far + * faster than the pty drain rate; per-cell sprintf would cap the producer). */ +static char sgr_bg[NPAL][24], sgr_fg[NPAL][24]; +static int sgr_bg_len[NPAL], sgr_fg_len[NPAL]; + +static uint8_t *fire; +static char *frame; + +static uint32_t rng = 0x9d2c5680u; /* fixed seed: deterministic stream */ +static uint32_t xorshift32(void) { + rng ^= rng << 13; + rng ^= rng >> 17; + rng ^= rng << 5; + return rng; +} + +/* One simulation step: propagate heat upward with horizontal drift+decay. */ +static void spread(void) { + for (int x = 0; x < W; x++) { + for (int y = 1; y < PH; y++) { + int src = y * W + x; + uint8_t p = fire[src]; + if (p == 0) { + fire[src - W] = 0; + } else { + int r = (int)(xorshift32() & 3); + int dst = src - r + 1; + if (dst < W) dst = W; /* keep the target row in-buffer */ + fire[dst - W] = (uint8_t)(p - (r & 1)); + } + } + } +} + +/* Compose one full-region repaint: per cell row an absolute CUP, then per + * cell bg=upper pixel, fg=lower pixel, U+2584 lower half block. */ +static size_t render(void) { + char *p = frame; + for (int row = 0; row < ROWS; row++) { + p += sprintf(p, "\x1b[%d;1H", row + 1); + const uint8_t *up = &fire[(row * 2) * W]; + const uint8_t *lo = &fire[(row * 2 + 1) * W]; + for (int x = 0; x < W; x++) { + memcpy(p, sgr_bg[up[x]], (size_t)sgr_bg_len[up[x]]); + p += sgr_bg_len[up[x]]; + memcpy(p, sgr_fg[lo[x]], (size_t)sgr_fg_len[lo[x]]); + p += sgr_fg_len[lo[x]]; + memcpy(p, "\xe2\x96\x84", 3); + p += 3; + } + } + memcpy(p, "\x1b[0m", 4); /* reset so attributes don't bleed past the region */ + p += 4; + return (size_t)(p - frame); +} + +static int write_all(const void *buf, size_t len) { + const char *p = buf; + while (len > 0) { + ssize_t w = write(STDOUT_FILENO, p, len); + if (w < 0) { + if (errno == EINTR) continue; + return -1; + } + p += w; + len -= (size_t)w; + } + return 0; +} + +static long long now_ns(void) { + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (long long)ts.tv_sec * 1000000000LL + ts.tv_nsec; +} + +int main(int argc, char **argv) { + int full = (argc == 4 && strcmp(argv[3], "full") == 0); + if (argc != 3 && !full) { + fprintf(stderr, "usage: %s [full]\n", argv[0]); + return 64; + } + double secs = atof(argv[1]); + if (secs <= 0) { + fprintf(stderr, "fire: bad duration '%s'\n", argv[1]); + return 64; + } + + struct winsize ws = {0}; + ioctl(STDOUT_FILENO, TIOCGWINSZ, &ws); /* audit; region source in full mode */ + if (full && ws.ws_col >= 2 && ws.ws_row >= 2) { + W = ws.ws_col; + ROWS = ws.ws_row; + if (W > MAX_W) W = MAX_W; + if (ROWS > MAX_ROWS) ROWS = MAX_ROWS; + PH = ROWS * 2; + } + fire = calloc((size_t)(PH * W), 1); + /* per cell worst case: two 19-byte SGRs + 3-byte glyph = 41; +16/row CUP */ + frame = malloc((size_t)ROWS * ((size_t)W * 41 + 16) + 64); + if (!fire || !frame) return 1; + + for (int i = 0; i < NPAL; i++) { + sgr_bg_len[i] = sprintf(sgr_bg[i], "\x1b[48;2;%d;%d;%dm", + PAL[i][0], PAL[i][1], PAL[i][2]); + sgr_fg_len[i] = sprintf(sgr_fg[i], "\x1b[38;2;%d;%d;%dm", + PAL[i][0], PAL[i][1], PAL[i][2]); + } + for (int x = 0; x < W; x++) fire[(PH - 1) * W + x] = NPAL - 1; + + /* alt screen + hidden cursor + clear; restored on exit */ + if (write_all("\x1b[?1049h\x1b[?25l\x1b[2J", 18) < 0) return 1; + + for (int i = 0; i < WARMUP_FRAMES; i++) { + spread(); + if (write_all(frame, render()) < 0) return 1; + } + + long frames = 0; + long long t0 = now_ns(), t1 = t0; + long long deadline = t0 + (long long)(secs * 1e9); + for (;;) { + spread(); + if (write_all(frame, render()) < 0) return 1; + frames++; + t1 = now_ns(); + if (t1 >= deadline) break; + } + + write_all("\x1b[0m\x1b[?25h\x1b[?1049l", 18); + + double fps = (double)frames / ((double)(t1 - t0) / 1e9); + FILE *f = fopen(argv[2], "w"); + if (!f) return 1; + fprintf(f, "%ld %lld %.1f %dx%d %dx%d\n", frames, t1 - t0, fps, + (int)ws.ws_col, (int)ws.ws_row, W, ROWS); + fclose(f); + return 0; +} diff --git a/bench/visualize.py b/bench/visualize.py new file mode 100644 index 0000000..a897d24 --- /dev/null +++ b/bench/visualize.py @@ -0,0 +1,440 @@ +#!/usr/bin/env python3 +"""Render a results directory (results.json) into a self-contained HTML report. + +stdlib only; inline CSS + inline SVG bar charts; zero external requests (works +offline from file://). One section per axis THAT HAS DATA — subset runs (a +single axis, a single terminal) are common, so absent/all-UNMEASURED axes are +skipped entirely rather than drawn as empty. noa's bar is accented; other +terminals are neutral gray. Renders only what is in the given results files — +no published/competitor figures are baked in. +""" +import json +import math +import os +import sys + +NOA_ACCENT = "#34d399" # noa's bar / highlight +BAR_NEUTRAL = "#9ca3af" # every other terminal +UNMEAS_FG = "#6b7280" # subtle "unmeasured" footnotes + +# fixed terminal display order (matches the harness roster); unknown names append +TERM_ORDER = ["noa", "ghostty", "termy", "kitty", "alacritty", + "iterm2", "warp", "terminal", "rio"] + + +def order_terms(terms): + known = [t for t in TERM_ORDER if t in terms] + extra = sorted(t for t in terms if t not in TERM_ORDER) + return known + extra + + +def esc(s): + return (str(s).replace("&", "&").replace("<", "<") + .replace(">", ">").replace('"', """)) + + +# ── SVG horizontal-bar chart ─────────────────────────────────────── +# rows: list of dicts {name, value (float>0), label (str), noa (bool)}. +# `unmeasured`: list of terminal names to list subtly beneath the chart. +def svg_hbars(rows, higher_better, log=False, unmeasured=None): + unmeasured = unmeasured or [] + if not rows: + return "" + label_w, bar_w, row_h, pad = 96, 300, 26, 6 + val_w = 150 + width = label_w + bar_w + val_w + height = len(rows) * row_h + 2 * pad + vals = [r["value"] for r in rows] + maxv = max(vals) if vals else 1.0 + + def frac(v): + if v <= 0: + return 0.0 + if log: + return math.log10(max(v, 1.0)) / math.log10(max(maxv, 10.0)) + return v / maxv if maxv > 0 else 0.0 + + parts = [f''] + for i, r in enumerate(rows): + y = pad + i * row_h + cy = y + row_h / 2 + w = max(2.0, frac(r["value"]) * bar_w) + color = NOA_ACCENT if r["noa"] else BAR_NEUTRAL + weight = "700" if r["noa"] else "400" + parts.append( + f'{esc(r["name"])}') + parts.append( + f'') + parts.append( + f'{esc(r["label"])}') + parts.append("") + note = "higher is better" if higher_better else "lower is better" + if log: + note += " · log-scaled bar length" + html = [f'
{"".join(parts)}
', + f'
{esc(note)}
'] + if unmeasured: + html.append('
unmeasured: ' + + ", ".join(esc(t) for t in unmeasured) + "
") + return "\n".join(html) + + +def cell_measured(c): + return isinstance(c, dict) and c.get("status") != "UNMEASURED" + + +# ── per-axis section builders. Each returns HTML or "" (skip if no data). ── +def sec_throughput(axis, terms): + blocks = [] + for var in ("ascii", "unicode"): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}).get(var, {}) + if cell_measured(c) and "mib_per_s" in c: + rows.append({"name": t, "value": c["mib_per_s"], + "label": f'{c["mib_per_s"]:.1f} MiB/s', "noa": t == "noa"}) + elif t in axis: + unmeas.append(t) + if rows: + blocks.append(f'

{var}

' + svg_hbars(rows, True, unmeasured=unmeas)) + if not blocks: + return "" + return section("Throughput", "PTY write → screen, grouped by charset.", "".join(blocks)) + + +def sec_scroll(axis, terms): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}) + if cell_measured(c) and "mib_per_s" in c: + ms = c.get("median_ms") + lbl = f'{c["mib_per_s"]:.1f} MiB/s' + (f' · {ms} ms' if ms is not None else "") + rows.append({"name": t, "value": c["mib_per_s"], "label": lbl, "noa": t == "noa"}) + elif t in axis: + unmeas.append(t) + if not rows: + return "" + return section("Scroll", "Scrollback flood throughput (frame time as label).", + svg_hbars(rows, True, unmeasured=unmeas)) + + +def sec_fire(axis, terms, contention): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}) + if cell_measured(c) and "fps_median" in c: + region = c.get("region", "?") + rows.append({"name": t, "value": c["fps_median"], + "label": f'{c["fps_median"]:.1f} fps ({region})', "noa": t == "noa"}) + elif t in axis: + unmeas.append(t) + if not rows: + return "" + cond = contention.get("fire_condition") + cap = f"DOOM-fire IO stress. Condition: {cond}." if cond else "DOOM-fire IO stress." + return section("Fire", cap, svg_hbars(rows, True, unmeasured=unmeas)) + + +def sec_latency(axis, terms): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}) + if cell_measured(c) and "median_us" in c: + extra = [] + if "p95_us" in c: + extra.append(f'p95 {c["p95_us"]}') + if "p99_us" in c: + extra.append(f'p99 {c["p99_us"]}') + tail = (" · " + " / ".join(extra)) if extra else "" + rows.append({"name": t, "value": c["median_us"], + "label": f'{c["median_us"]} µs{tail}', "noa": t == "noa"}) + elif t in axis: + unmeas.append(t) + if not rows: + return "" + return section("Input Latency", "DSR round-trip proxy, median µs (p95/p99 in labels).", + svg_hbars(rows, False, log=True, unmeasured=unmeas)) + + +def sec_startup(axis, terms): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}) + if cell_measured(c) and "median_ms" in c: + rows.append({"name": t, "value": c["median_ms"], + "label": f'{c["median_ms"]} ms', "noa": t == "noa"}) + elif t in axis: + unmeas.append(t) + if not rows: + return "" + return section("Warm Startup", "spawn → pty-ready.", svg_hbars(rows, False, unmeasured=unmeas)) + + +def sec_memory(axis, terms): + # settled MiB per scenario (longevity uses final-settled), lower better + scenarios = [("idle", "idle"), ("scrollback", "scrollback (post-flood)"), + ("multitab", "multitab"), ("longevity", "longevity (final-settled)")] + blocks = [] + for key, label in scenarios: + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}).get(key, {}) + if not cell_measured(c): + if key in axis.get(t, {}): + unmeas.append(t) + continue + if key == "longevity": + v = c.get("final_settled_mib", c.get("final_active_mib")) + else: + v = c.get("settled_mib", c.get("active_mib")) + if v is None: + unmeas.append(t) + continue + rows.append({"name": t, "value": v, "label": f"{v} MiB", "noa": t == "noa"}) + if rows: + blocks.append(f'

{esc(label)}

' + svg_hbars(rows, False, unmeasured=unmeas)) + if not blocks: + return "" + return section("Memory", "Settled physical footprint per scenario.", "".join(blocks)) + + +def sec_load(axis, terms): + blocks = [] + # idle mean CPU% + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}).get("idle", {}) + if cell_measured(c) and "cpu_pct_mean" in c: + mx = c.get("cpu_pct_max") + lbl = f'{c["cpu_pct_mean"]:.2f}%' + (f' · max {mx:.2f}%' if mx is not None else "") + rows.append({"name": t, "value": c["cpu_pct_mean"], "label": lbl, "noa": t == "noa"}) + elif "idle" in axis.get(t, {}): + unmeas.append(t) + if rows: + blocks.append('

idle mean CPU%

' + svg_hbars(rows, False, unmeasured=unmeas)) + # active cpu_ms per workload + for wk in ("throughput", "scroll"): + rows, unmeas = [], [] + for t in terms: + c = axis.get(t, {}).get(wk, {}) + if cell_measured(c) and "cpu_ms" in c: + per = c.get("cpu_ms_per_mib") + lbl = f'{c["cpu_ms"]} ms' + (f' · {per:.2f} ms/MiB' if per else "") + rows.append({"name": t, "value": c["cpu_ms"], "label": lbl, "noa": t == "noa"}) + elif wk in axis.get(t, {}): + unmeas.append(t) + if rows: + blocks.append(f'

active: {wk} workload (CPU-ms)

' + + svg_hbars(rows, False, unmeasured=unmeas)) + if not blocks: + return "" + return section("Load", "Idle CPU% and active CPU-time per workload.", "".join(blocks)) + + +def section(title, caption, body): + return (f'

{esc(title)}

' + f'

{esc(caption)}

{body}
') + + +# ── noa trend across multiple runs (simple bars, timestamps on x) ────── +def noa_trend(runs): + # runs: list of (timestamp, results). Extract one comparable noa scalar per + # axis and draw a simple bar per run. + def getter(res): + ax = res.get("axes", {}) + out = {} + tp = ax.get("throughput", {}).get("noa", {}) + for var in ("ascii", "unicode"): + c = tp.get(var, {}) + if cell_measured(c) and "mib_per_s" in c: + out[f"throughput {var} (MiB/s)"] = (c["mib_per_s"], True) + sc = ax.get("scroll", {}).get("noa", {}) + if cell_measured(sc) and "mib_per_s" in sc: + out["scroll (MiB/s)"] = (sc["mib_per_s"], True) + fi = ax.get("fire", {}).get("noa", {}) + if cell_measured(fi) and "fps_median" in fi: + out["fire (fps)"] = (fi["fps_median"], True) + la = ax.get("latency", {}).get("noa", {}) + if cell_measured(la) and "median_us" in la: + out["latency median (µs)"] = (la["median_us"], False) + st = ax.get("startup", {}).get("noa", {}) + if cell_measured(st) and "median_ms" in st: + out["startup (ms)"] = (st["median_ms"], False) + mi = ax.get("memory", {}).get("noa", {}).get("idle", {}) + if cell_measured(mi) and (mi.get("settled_mib") or mi.get("active_mib")): + out["memory idle (MiB)"] = (mi.get("settled_mib", mi.get("active_mib")), False) + li = ax.get("load", {}).get("noa", {}).get("idle", {}) + if cell_measured(li) and "cpu_pct_mean" in li: + out["load idle (CPU%)"] = (li["cpu_pct_mean"], False) + return out + + per_run = [(ts, getter(res)) for ts, res in runs] + metrics = [] + for _, d in per_run: + for k in d: + if k not in metrics: + metrics.append(k) + blocks = [] + for m in metrics: + rows = [] + higher = True + for ts, d in per_run: + if m in d: + v, higher = d[m] + rows.append({"name": ts, "value": v, "label": f"{v}", "noa": True}) + if len(rows) >= 2: + blocks.append(f'

{esc(m)}

' + svg_hbars(rows, higher)) + if not blocks: + return "" + return section("noa trend", "noa's per-axis values across the given runs.", "".join(blocks)) + + +# ── header ───────────────────────────────────────────────────────── +def header(res): + ts = res.get("timestamp", "?") + m = res.get("machine", {}) + mline = " · ".join(x for x in [ + m.get("chip") or m.get("model"), + f'{m.get("cores")} cores' if m.get("cores") else "", + f'macOS {m.get("os")}' if m.get("os") else "", + f'{m.get("build")}' if m.get("build") else "", + m.get("arch"), + ] if x) + vers = res.get("terminal_versions", {}) + vline = ", ".join(f"{k} {v}" for k, v in vers.items()) + cont = res.get("contention", {}) + meta = [] + for key in ("quiescence_check", "fullscreen", "fire_condition", "target_display"): + if cont.get(key): + meta.append(f'
{esc(key)} {esc(cont[key])}
') + h = [f'

Benchmark Report

', + f'
{esc(ts)}
'] + if mline: + h.append(f'
{esc(mline)}
') + if vline: + h.append(f'
{esc(vline)}
') + if meta: + h.append('
' + "".join(meta) + "
") + h.append("
") + return "\n".join(h) + + +CSS = """ +:root { color-scheme: dark; } +* { box-sizing: border-box; } +body { margin: 0; padding: 24px 28px 60px; background: #0b0f14; color: #e5e7eb; + font: 14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; } +h1 { font-size: 22px; margin: 0 0 2px; } +h2 { font-size: 17px; margin: 0 0 4px; color: #f9fafb; } +h3 { font-size: 13px; margin: 14px 0 2px; color: #9ca3af; font-weight: 600; } +header { border-bottom: 1px solid #1f2937; padding-bottom: 16px; margin-bottom: 8px; } +.ts { color: #34d399; font-weight: 600; font-variant-numeric: tabular-nums; } +.machine, .vers { color: #9ca3af; font-size: 12.5px; margin-top: 4px; } +.meta { margin-top: 10px; display: flex; flex-direction: column; gap: 3px; } +.metarow { font-size: 12px; color: #cbd5e1; } +.metarow span { color: #6b7280; display: inline-block; min-width: 130px; } +section { padding: 18px 0; border-bottom: 1px solid #131a22; } +.cap { color: #9ca3af; font-size: 12.5px; margin: 0 0 6px; } +.chart { overflow-x: auto; margin: 4px 0; } +.chart svg { max-width: 100%; } +.dir { font-size: 11px; color: #6b7280; margin: 2px 0 4px; } +.unmeas { font-size: 11px; color: #6b7280; font-style: italic; margin-bottom: 6px; } +footer { color: #4b5563; font-size: 11px; margin-top: 24px; } +""" + + +def present_axes(res): + return res.get("axes", {}) + + +def build_report(runs, primary): + """runs: list of (ts, results); primary: the results whose axes drive sections.""" + res = primary + axes = present_axes(res) + terms = order_terms([t for ax in axes.values() for t in ax]) + body = [f'', header(res)] + + builders = { + "throughput": lambda a: sec_throughput(a, terms), + "scroll": lambda a: sec_scroll(a, terms), + "fire": lambda a: sec_fire(a, terms, res.get("contention", {})), + "latency": lambda a: sec_latency(a, terms), + "startup": lambda a: sec_startup(a, terms), + "memory": lambda a: sec_memory(a, terms), + "load": lambda a: sec_load(a, terms), + } + any_section = False + for name, fn in builders.items(): + if name in axes: + html = fn(axes[name]) + if html: + body.append(html) + any_section = True + if not any_section: + body.append('

No measured axes in this run.

') + + if len(runs) > 1: + tr = noa_trend(runs) + if tr: + body.append(tr) + + body.append('
Generated by bench/visualize.py — renders only the ' + 'values present in the given results files.
') + return "\n" + "\n".join(body) + + +def newest_results_dir(bench_dir): + root = os.path.join(bench_dir, "results") + dirs = [os.path.join(root, d) for d in os.listdir(root) + if os.path.isdir(os.path.join(root, d)) + and os.path.exists(os.path.join(root, d, "results.json"))] + if not dirs: + return None + return max(dirs, key=lambda d: os.path.basename(d)) + + +def load(d): + with open(os.path.join(d, "results.json")) as f: + return json.load(f) + + +def main(argv): + out = None + dirs = [] + i = 0 + while i < len(argv): + a = argv[i] + if a == "--out": + out = argv[i + 1] + i += 2 + continue + dirs.append(a) + i += 1 + + bench_dir = os.path.dirname(os.path.abspath(__file__)) + if not dirs: + d = newest_results_dir(bench_dir) + if not d: + print("no results dirs found", file=sys.stderr) + return 1 + dirs = [d] + + runs = [(os.path.basename(os.path.normpath(d)), load(d)) for d in dirs] + # primary = last dir (its report.html, its axes drive the per-axis sections) + primary = runs[-1][1] + html = build_report(runs, primary) + + out = out or os.path.join(dirs[-1], "report.html") + with open(out, "w") as f: + f.write(html) + print(f"wrote {out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bench/wrapper.sh b/bench/wrapper.sh index ee386a2..61cd160 100755 --- a/bench/wrapper.sh +++ b/bench/wrapper.sh @@ -11,7 +11,7 @@ # ignored. Behavior is selected entirely by NOA_MODE + friends in the env. # # Env contract: -# NOA_MODE throughput | scroll | latency | startup | hold | longevity +# NOA_MODE throughput | scroll | latency | startup | hold | longevity | fire # NOA_SENTINEL file to create when the workload is done (launcher watches it) # NOA_NOWNS path to the `nowns` monotonic-ns helper # NOA_BENCH_CMD (throughput/scroll/longevity) shell command to run, e.g. `cat file` @@ -21,6 +21,16 @@ # NOA_SAMPLES (latency, optional) file dsr_probe writes every kept raw # sample into (one ns per line) so the harness can pool # samples across launches +# NOA_FIRE (fire) path to the `fire` DOOM-fire IO-stress tool +# NOA_FIRE_SECS (fire) measured duration in seconds (after 60 warmup frames) +# NOA_FIRE_ARG (fire, optional) "full" = render to the live window size +# (upstream DOOM-fire condition; the harness sets it on +# fullscreen runs). Empty = fixed 80x24 region. +# NOA_GO (workload modes, optional) gate file: when set, the +# workload starts only after the file appears — the harness +# creates it once the window reached its measurement +# geometry (fullscreen). 10s fallback so a lost gate can +# never hang the child forever. # NOA_HOLD (memory/load axes) if "1", after the mode's own work the pty # child sleeps instead of exiting, so the terminal window (and # process tree) stays alive for the harness to sample RSS/CPU @@ -29,8 +39,21 @@ # NOA_CYCLES (longevity) number of flood+idle cycles (default 5) # NOA_IDLE_S (longevity) idle seconds between cycles (default 3) +# wait_go — block until the harness signals final window geometry (NOA_GO +# file appears). No-op when NOA_GO is unset (memory/load scenarios reuse the +# workload modes without gating). Bounded so a lost gate cannot hang the run. +wait_go() { + [ -n "${NOA_GO:-}" ] || return 0 + i=0 + while [ ! -f "$NOA_GO" ] && [ "$i" -lt 200 ]; do + sleep 0.05 + i=$((i + 1)) + done +} + case "$NOA_MODE" in throughput|scroll) + wait_go start=$("$NOA_NOWNS") eval "$NOA_BENCH_CMD" end=$("$NOA_NOWNS") @@ -38,9 +61,21 @@ case "$NOA_MODE" in mv "$NOA_SENTINEL.part" "$NOA_SENTINEL" ;; latency) + wait_go "$NOA_PROBE" "${NOA_PROBE_ITERS:-200}" "${NOA_PROBE_WARMUP:-20}" "$NOA_RESULT" "${NOA_SAMPLES:-}" : > "$NOA_SENTINEL" ;; + fire) + wait_go + # DOOM-fire IO stress (docs/specs/bench-doom-fire.md): renders truecolor + # half-block fire flat-out for NOA_FIRE_SECS under pty flow control — + # full-window when NOA_FIRE_ARG=full (fullscreen runs; wait_go above + # guarantees the winsize read happens at final geometry), fixed 80x24 + # otherwise. Writes " " + # into NOA_RESULT. + "$NOA_FIRE" "${NOA_FIRE_SECS:-10}" "$NOA_RESULT" ${NOA_FIRE_ARG:-} + : > "$NOA_SENTINEL" + ;; startup) # Child reached exec == pty ready + window materialized. The launcher's # t0..sentinel delta is the warm-start proxy. diff --git a/docs/positioning.md b/docs/positioning.md new file mode 100644 index 0000000..b6b2c7c --- /dev/null +++ b/docs/positioning.md @@ -0,0 +1,60 @@ +# Noa Positioning: The Fastest Terminal + +> Status: draft (ratified upon merge). Source of truth for how "fastest" is +> claimed, defended, and measured. Benchmarks live in `bench/` +> (see `bench/METHODOLOGY.md`). + +## One-line definition + +**Noa is the terminal that swallows output floods fastest** — `cat huge.log`, +build logs, CI streams — with class-leading render throughput and no +wide-character (CJK) penalty. + +## The claim, precisely + +"Fastest" means **fastest at sustained output throughput and bulk scroll**. +Every number below is reproducible via `bench/run_all.sh`. + +| Axis | Measurement (2026-07-17, 5-run median) | Role | +|---|---|---| +| render.throughput (plain/ansi/cjk) | 220–233 MB/s, stable across all runs | **Headline** | +| scroll proxy | 54 ms | **Headline** | +| cmd.overhead (zsh) | 1336 µs | Supporting | +| idle RSS | 52 MB settled | Supporting | +| input.latency (idle) | 1.2–1.3 ms | Supporting | + +CJK being the fastest throughput variant (233 MB/s) is a distinctive +strength: wide-character handling carries no cost. + +## Design character + +Noa buys speed with parallelism — parse and render scale across cores under +load. This is a deliberate desktop-first choice: when output floods in, Noa +spends hardware to stay ahead of it. + +## Guardrails (what keeps "fastest" honest) + +A headline metric may never be bought by regressing elsewhere. Tracked on +every perf PR via `bench/run_all.sh`: + +- **input.latency under load** — keep tightening toward the idle-level + figure. PR #21 (isolate keyboard input path from pty output processing) + is the first step; re-measure post-#21 before citing numbers. +- **CPU under load** — stay within the current envelope. +- **idle RSS** — hold the 52 MB settled figure. + +A regression in any guardrail metric blocks the merge regardless of +headline gains. + +## Elevator pitch + +> The terminal that swallows output floods fastest — `cat huge.log`, build +> logs, CI streams — and proves it with a benchmark harness on every change. + +## What this positioning demands next + +1. Re-run `bench/` on post-#21 main; record the loaded input-latency figure. +2. Add an efficiency variant to `bench/` (throughput per watt) so battery + behavior is measured, not assumed. +3. Surface the headline numbers in `README.md` with a link here and to + `bench/METHODOLOGY.md` — claims without reproducible numbers are marketing. diff --git a/docs/specs/bench-doom-fire.md b/docs/specs/bench-doom-fire.md new file mode 100644 index 0000000..1c60230 --- /dev/null +++ b/docs/specs/bench-doom-fire.md @@ -0,0 +1,157 @@ +# Bench axis: `fire` — DOOM-Fire IO stress (design) + +Status: IMPLEMENTED (2026-07-17). Adds a 7th axis to `bench/run_all.sh` +reproducing the workload class of +[DOOM-fire-zig](https://github.com/const-void/DOOM-fire-zig) — the de-facto +community IO benchmark for terminals (Ghostty's release notes cite it). + +## Motivation + +DOOM-fire-zig fps figures are widely published for many terminals (various +machines/window sizes, **not** comparable to ours — see Fairness). + +What it exercises that the existing axes do not: a **frame-structured, +truecolor-dense, cursor-repositioning flood** — every frame repaints the +whole region with per-cell `SGR 38;2/48;2` RGB and half-block glyphs, under +pty flow control. `throughput` streams flat text; `scroll` exercises +scroll-regions; `fire` is the "animated TUI at max rate" shape (fps = drain +rate of full-screen repaints). It cleanly detects display-paced consumption: +such terminals plateau at ~display-refresh fps regardless of hardware. + +## Decision 1 — reimplement in C, don't vendor the Zig binary + +`bench/tools/fire.c`, built by the existing `cc -O2` tools block. Rationale: + +- The harness's tools are all single-file C built on the fly; requiring a + Zig toolchain (or vendoring an opaque prebuilt binary) breaks the + "third party re-runs everything with one command" property. +- Upstream DOOM-fire-zig has no fixed-duration mode and no machine-readable + output (fps is drawn on screen, runs until keypress) — automation would + need a fork anyway. +- The algorithm is tiny and public (Fabien Sanglard's DOOM fire): a 1-byte + heat buffer, per-frame decay/spread, 37-entry palette mapped to RGB, + rendered as `▄` half-blocks (fg = lower pixel, bg = upper pixel, 2 fire + rows per cell row). + +The port must match the upstream *workload shape* (truecolor half-block +full-region repaint per frame), not its exact bytes. Published fps numbers +are anchors for manual runs of the real DOOM-fire-zig, never pasted into our +results. + +## Decision 2 — render region follows the geometry mode (revised 2026-07-17) + +fps is inversely proportional to cell count, so the region choice is a +fairness decision: + +- **Fullscreen runs (the harness default since fullscreen measurement + landed): full-window** — matches upstream DOOM-fire-zig's official + condition. Every terminal fills the same physical screen; the cell count + then follows each terminal's font defaults (as in upstream comparisons) + and is disclosed per rep (`region` in raw.tsv, parenthesized in table.md). + The fullscreen gate guarantees the `TIOCGWINSZ` read happens at final + geometry. +- **Windowed fallback runs: fixed 80×24 cell region** (fire buffer 80×48 + pixels, top-left anchored via absolute `CUP`) — on unequal window + geometry, full-window fps would measure the geometry lottery, not the + terminal. The fixed region fits every default grid (no clipping) and + gives **every terminal a byte-identical stream**. + +Fixed PRNG seed → the frame sequence is deterministic across terminals and +runs in both modes. The two conditions are mutually incomparable; every +results dir records which one ran (`fire_condition`). + +## Decision 3 — producer-side fps under flow control (same proxy as axis 1) + +The pty has a small kernel buffer with flow control: the producer's `write` +blocks until the terminal drains. Frames written per second therefore ≈ +frames consumed per second — the same "consume the pipe" proxy the +throughput axis already documents. No screen-capture or vsync instrumentation +is attempted. + +`fire.c `: + +1. raw mode, enter alt screen (`CSI ?1049h`), hide cursor. +2. **Warmup: 60 frames, discarded** (glyph-atlas population, palette ramp, + alt-screen entry transients). +3. Render frames flat-out for `` (`CLOCK_MONOTONIC` bracket), counting + completed frames. Each frame is composed into one buffer and written with + a single retry-on-partial `write` loop. +4. Leave alt screen, restore tty. +5. Write `" x"` to the result file. + +## Integration + +### `bench/wrapper.sh` — new mode + +```sh +fire) + "$NOA_FIRE" "${NOA_FIRE_SECS:-10}" "$NOA_RESULT" + : > "$NOA_SENTINEL" + ;; +``` + +New env keys: `NOA_FIRE` (tool path), `NOA_FIRE_SECS`. Documented in the +header env contract. `NOA_HOLD` composes as usual (unused by this axis). + +### `bench/run_all.sh` + +- `AXES` default set gains `fire` (opt-out via `--axes` as with every axis). +- Params: full `FIRE_REPS=3`, `FIRE_SECS=10`; quick `FIRE_REPS=1`, + `FIRE_SECS=3`. Timeout `FIRE_TIMEOUT=60`. +- Tools block: add `fire` to `tools_fresh` + the `cc -O2` build line. +- Axis loop mirrors latency's result-file pattern: + `run_once "$term" fire "$FIRE_TIMEOUT"` → read result file → emit per-rep + `fps`, `frames`, `elapsed_ns`, plus `median fps`. Median of reps is the + headline (fps is higher-is-better, like `mib_per_s`). +- **Focus each terminal during the run** (reuse the latency axis's + `activate_term` schedule, PID-scoped): display-paced terminals throttle + unfocused/occluded windows, which would understate them; focused is the + representative condition and is applied uniformly. +- run_once needs a one-line extension: trigger the activation schedule for + `mode = latency or fire`. +- Contention: CPU-bound axis — already covered by the builder-quiescence + gate and loadavg bookends. +- `--equalize` mode: skip (the fixed 80×24 region makes the workload + grid/font-independent by construction; font rendering differences remain + part of each terminal's own cost, as with scroll). + +### `bench/aggregate.py` / `table.md` + +- New axis row: `fire — DOOM-fire proxy (fps, median of N×10 s)`, ranked + higher-is-better on median fps. +- `results.json`: per-terminal `{fps_median, fps_reps[], frames, region, + winsize}`. + +### `bench/METHODOLOGY.md` + +New "Axis 7: fire" section covering: lineage (DOOM-fire-zig / Sanglard +algorithm), the fixed-region fairness decision, the flow-control fps proxy +and its caveat (drain rate, not photon rate), the focus policy, and an +explicit note that published DOOM-fire-zig figures are other machines + +full-window regions and must not be compared to this axis's numbers. + +## Acceptance criteria + +1. `bench/run_all.sh --axes fire` produces per-terminal median fps for all + installed terminals with zero UNMEASURED rows on a quiet machine. +2. `raw.tsv` contains per-rep `fps/frames/elapsed_ns` and the winsize note. +3. Byte-stream identity: running `fire.c` twice redirected to a file yields + identical output (deterministic seed) — verifiable with `cmp`. +4. A visual spot check in noa shows the fire animating in the top-left + 80×24 region (alt screen, restored cleanly on exit). +5. Rankings appear in `table.md` with fps ranked higher-is-better. +6. The EXIT-trap / kill_term lifecycle leaves no processes behind (existing + invariant, unchanged). + +## Non-goals + +- Matching upstream DOOM-fire-zig's exact escape output or its published fps. +- Photon-to-glass frame-rate measurement (out of scope for the whole + harness; see METHODOLOGY axis 2 caveats). +- Window-size-scaled numbers in the scored axis. A manual full-window mode + exists for upstream-style anchor runs (`fire full`, or + `FIRE_FULL=1 bench/run_benchmark.sh fire`) — fps scales ~1/cell-count + (e.g. a 128×37 window has ~2.5× the cells of the fixed 80×24 region, so + fixed-region fps reads ~2.5× higher than full-window fps on the same + terminal), which is exactly why the scored axis pins the region: full-mode + numbers depend on window geometry and never enter ranking.