Skip to content

fix: emit per-client metrics in benchmark summary#27

Merged
cbermudez97 merged 3 commits into
mainfrom
fix/multi-client-benchs
Jul 22, 2026
Merged

fix: emit per-client metrics in benchmark summary#27
cbermudez97 merged 3 commits into
mainfrom
fix/multi-client-benchs

Conversation

@cbermudez97

Copy link
Copy Markdown
Contributor

Problem

When a benchmark runs multiple clients, the end-of-run per-client metrics were not truly per-client on the default (non-Prometheus) path — the per-client/per-method entries came out empty or reflected data that wasn't actually separated by client.

Each client maps to a k6 scenario (scenario name == client name). The default metrics collector reads k6's summary.json (--summary-export) and expects per-client-per-method submetrics named like http_req_duration{scenario:C,req_name:M}. But k6 only writes a tag-filtered submetric into summary.json when a threshold is defined on it, and the generator only registered thresholds keyed by req_name alone (and only when a call defined thresholds). So no per-scenario (client) submetric was ever emitted, and the parser had nothing per-client to read. The base http_req_duration metric in the summary is aggregated across all clients.

The Prometheus remote-write path was unaffected — every exported series keeps its scenario label.

Fix

Register always-true per-client × per-method submetric thresholds in the generated k6 config so k6 emits the per-scenario breakdown into summary.json:

  • http_req_duration{scenario:C,req_name:M}max>=0
  • http_reqs{scenario:C,req_name:M}count>=0
  • http_req_failed{scenario:C,req_name:M}rate>=0

The conditions can never fail, so the k6 exit code is unaffected. Keys are unquoted and use {scenario:C,req_name:M} ordering to match the name lookup in summary_fallback.go — k6 stores submetric names verbatim (it does not strip quotes or re-sort tags). Existing user-configured thresholds are left untouched (still global across clients). No parser or downstream changes were needed.

Verification

Ran a real k6 benchmark against two mock RPC servers with 20 ms / 60 ms injected latency:

  • go build ./... passes.
  • summary.json now contains all per-client submetric keys in the exact format the parser matches.
  • P99 coverage went from 0% → 100% (4/4 methods).
  • Final exports are genuinely distinct per client: geth ~21 ms avg, nethermind ~61 ms avg — matching the injected delays — with per-method counts, latencies, and P99 all populated.
  • k6 exit code stayed 0 (no false threshold failures); Prometheus path unaffected.

The default (non-Prometheus) metrics path reads k6's summary.json, which
only contains a tag-filtered submetric when a threshold is defined on it.
The generator only registered thresholds keyed by req_name, so no
per-scenario (client) breakdown was ever written and per-client metrics
came out empty or aggregated across all clients.

Register always-true per-client x per-method submetric thresholds so k6
emits the per-scenario breakdown into summary.json for the parser to
read. The conditions can never fail, so the k6 exit code is unaffected,
and the keys are unquoted in {scenario:C,req_name:M} order to match the
name lookup in summary_fallback.go (k6 stores submetric names verbatim).
The Docker image scan fails on GHSA-hrxh-6v49-42gf (HIGH, gRPC-Go xDS
RBAC and HTTP/2 vulnerabilities), which lives in the prebuilt k6 binary
from the grafana/k6 base image. It bundles google.golang.org/grpc <
1.82.1 and no k6 release ships a patched grpc yet (latest v2.1.0 still
uses 1.81.1); we do not build k6 from source. The affected gRPC xDS/RBAC
paths are not exercised — this suite benchmarks JSON-RPC over HTTP and
never uses k6's gRPC module. Add a documented .trivyignore entry to be
removed once k6 bumps grpc, matching the existing convention in that
file.
Bump the runtime base from grafana/k6:1.8.0 to grafana/k6:2.1.0, which
ships a newer Alpine (3.24) and embedded Go toolchain. All k6 features
this tool relies on are unchanged in v2 (the experimental fs/csv script
modules, --summary-export, --summary-mode full, and the
experimental-prometheus-rw output), and the benchmark flow was verified
end-to-end on k6 v2.1.0.

The GHSA-hrxh-6v49-42gf grpc advisory is not resolved by the upgrade —
no k6 release (2.1.0 or master) yet bundles grpc >= 1.82.1 — so the
Trivy .trivyignore entry is retained, with its note updated to reflect
the pinned version.
@cbermudez97
cbermudez97 merged commit 41b1767 into main Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant