fix: emit per-client metrics in benchmark summary#27
Merged
Conversation
The default (non-Prometheus) metrics path reads k6's summary.json, which
only contains a tag-filtered submetric when a threshold is defined on it.
The generator only registered thresholds keyed by req_name, so no
per-scenario (client) breakdown was ever written and per-client metrics
came out empty or aggregated across all clients.
Register always-true per-client x per-method submetric thresholds so k6
emits the per-scenario breakdown into summary.json for the parser to
read. The conditions can never fail, so the k6 exit code is unaffected,
and the keys are unquoted in {scenario:C,req_name:M} order to match the
name lookup in summary_fallback.go (k6 stores submetric names verbatim).
The Docker image scan fails on GHSA-hrxh-6v49-42gf (HIGH, gRPC-Go xDS RBAC and HTTP/2 vulnerabilities), which lives in the prebuilt k6 binary from the grafana/k6 base image. It bundles google.golang.org/grpc < 1.82.1 and no k6 release ships a patched grpc yet (latest v2.1.0 still uses 1.81.1); we do not build k6 from source. The affected gRPC xDS/RBAC paths are not exercised — this suite benchmarks JSON-RPC over HTTP and never uses k6's gRPC module. Add a documented .trivyignore entry to be removed once k6 bumps grpc, matching the existing convention in that file.
Bump the runtime base from grafana/k6:1.8.0 to grafana/k6:2.1.0, which ships a newer Alpine (3.24) and embedded Go toolchain. All k6 features this tool relies on are unchanged in v2 (the experimental fs/csv script modules, --summary-export, --summary-mode full, and the experimental-prometheus-rw output), and the benchmark flow was verified end-to-end on k6 v2.1.0. The GHSA-hrxh-6v49-42gf grpc advisory is not resolved by the upgrade — no k6 release (2.1.0 or master) yet bundles grpc >= 1.82.1 — so the Trivy .trivyignore entry is retained, with its note updated to reflect the pinned version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a benchmark runs multiple clients, the end-of-run per-client metrics were not truly per-client on the default (non-Prometheus) path — the per-client/per-method entries came out empty or reflected data that wasn't actually separated by client.
Each client maps to a k6 scenario (scenario name == client name). The default metrics collector reads k6's
summary.json(--summary-export) and expects per-client-per-method submetrics named likehttp_req_duration{scenario:C,req_name:M}. But k6 only writes a tag-filtered submetric intosummary.jsonwhen a threshold is defined on it, and the generator only registered thresholds keyed byreq_namealone (and only when a call definedthresholds). So no per-scenario (client) submetric was ever emitted, and the parser had nothing per-client to read. The basehttp_req_durationmetric in the summary is aggregated across all clients.The Prometheus remote-write path was unaffected — every exported series keeps its
scenariolabel.Fix
Register always-true per-client × per-method submetric thresholds in the generated k6 config so k6 emits the per-scenario breakdown into
summary.json:http_req_duration{scenario:C,req_name:M}→max>=0http_reqs{scenario:C,req_name:M}→count>=0http_req_failed{scenario:C,req_name:M}→rate>=0The conditions can never fail, so the k6 exit code is unaffected. Keys are unquoted and use
{scenario:C,req_name:M}ordering to match the name lookup insummary_fallback.go— k6 stores submetric names verbatim (it does not strip quotes or re-sort tags). Existing user-configured thresholds are left untouched (still global across clients). No parser or downstream changes were needed.Verification
Ran a real k6 benchmark against two mock RPC servers with 20 ms / 60 ms injected latency:
go build ./...passes.summary.jsonnow contains all per-client submetric keys in the exact format the parser matches.