Skip to content

fix(bench): measure ferrflow_parallel only from the binary method#154

Merged
BryanFRD merged 1 commit into
mainfrom
fix/bench-parallel-binary-only
Jul 2, 2026
Merged

fix(bench): measure ferrflow_parallel only from the binary method#154
BryanFRD merged 1 commit into
mainfrom
fix/bench-parallel-binary-only

Conversation

@BryanFRD

@BryanFRD BryanFRD commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

The parallel result file (parallel/${fixture}-${cmd}.json) isn't scoped by install method, and ferrflow runs binary → npm → docker. Each method re-ran the parallel hyperfine into the same file, so docker (last) overwrote it — ferrflow_parallel was really measuring docker run container startup (~constant 165–280 ms), which made ferrflow's default mode look far slower than --jobs 1.

Verified on ubuntu-latest with --timing: ferrflow check is 4–6 ms in both --jobs 1 and default — no parallel-path overhead. The stat was a harness artifact.

Fix: set par_cmd (the all-cores variant) only for the binary method, so the parallel stat is binary all-cores vs binary --jobs 1. The --jobs 1 pinning still applies to every ferrflow method for the single stat.

Closes #153

Copilot AI review requested due to automatic review settings July 2, 2026 21:10
@BryanFRD BryanFRD enabled auto-merge (squash) July 2, 2026 21:10
@BryanFRD BryanFRD merged commit 90ce643 into main Jul 2, 2026
14 checks passed
@BryanFRD BryanFRD deleted the fix/bench-parallel-binary-only branch July 2, 2026 21:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a benchmarking harness artifact where ferrflow_parallel was unintentionally measuring npx/Docker startup overhead due to the parallel results file not being scoped by install method. It ensures the “parallel” stat reflects ferrflow’s binary all-cores execution (vs the binary --jobs 1 run), while still pinning --jobs 1 for all ferrflow install methods in the primary comparison.

Changes:

  • Only sets par_cmd for ferrflow when method == "binary" to prevent npm/docker runs from overwriting results/raw/parallel/${fixture}-${cmd_name}.json.
  • Keeps ferrflow pinned to --jobs 1 for the main (single-thread comparable) measurement across all install methods.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: ferrflow_parallel measures docker/npx startup, not all-cores latency

2 participants