roachtest: admission-control/disk-bandwidth-limiter failed

[cockroach-teamcity]

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-26.1 @ 403b316710e30e0e37d57e17304a1e225e4c6b17:

Failed with:

(admission_control_disk_bandwidth_overload.go:220).3: too many errors retrieving metrics
(cluster.go:2670).Run: context canceled
(cluster.go:2670).Run: context canceled
(monitor.go:267).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Cluster Node to Ip Mapping:

Node	Public IP	Private IP
teamcity-21356638-1779430068-14-n2cpu8-0001	35.196.65.68	10.142.1.76
teamcity-21356638-1779430068-14-n2cpu8-0002	35.196.32.203	10.142.3.78

Parameters:

• MetamorphicFilesystem=ext4
• MetamorphicVolumeType=local-ssd
• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=8
• diskCount=0
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicLeases=leader
• runtimeAssertionsBuild=false

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

• roachtest: admission-control/disk-bandwidth-limiter failed #170790 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot T-storage branch-release-25.2 release-blocker]

/cc @cockroachdb/admission-control

This test on roachdash | Improve this report!

Jira issue: CRDB-64177

[pav-kv]

/investigate

[github-actions]

Investigation: admission-control/disk-bandwidth-limiter

Investigated failure: #170793
Failure SHA: 403b316710e30e0e37d57e17304a1e225e4c6b17
Confidence: high

What This Test Does

This roachtest verifies CockroachDB's admission control disk bandwidth limiter. It runs two KV workloads (a foreground "regular" priority workload and a background write-heavy workload with 1024 concurrency), limits disk bandwidth to 128 MiB/s via cgroups, then sets the AC provisioned bandwidth to 75 MiB/s. After a 65-minute warmup, a 20-minute monitoring loop queries Prometheus every 10 seconds for disk write and read bandwidth, asserting the write bandwidth stays within bounds.

Where the Failure Occurs

The test fails at admission_control_disk_bandwidth_overload.go:220 with:

too many errors retrieving metrics

This is the guard at the end of the monitoring loop: when numErrors > numSuccesses, the test fails. In this run, all 120 Prometheus queries failed (120 errors, 0 successes). Every single query returned the same warning:

PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" (1:6)

The warning is returned by Prometheus for the rate(sys_host_disk_write_bytes[1m]) query. The CollectPoint function in collector.go:98-100 treats any Prometheus warning as a hard error:

if len(warnings) > 0 {
    return nil, errors.Newf("found warnings querying prometheus: %s", warnings)
}

Analysis

Hypothesis: Prometheus 2.53.5 upgrade introduced informational warnings that break metric collection (likely cause, high confidence)

PR #170676 bumped the local Prometheus version from 2.27.1 to 2.53.5. This was merged to master on 2026-05-21 and cherry-picked to all release branches:

Branch	Cherry-pick SHA	In failure SHA?
release-26.1	c2cd9148d90	Yes
release-25.2	9ac5c3e9478	Yes
release-26.2	36940e54bc8	Yes
release-25.4	26b1c1270cb	Yes

Prometheus 2.53 introduced informational PromQL warnings when rate() is applied to metrics whose names don't follow counter naming conventions (i.e., don't end in _total, _sum, _count, or _bucket). The CockroachDB metric sys_host_disk_write_bytes does not follow this convention, so rate(sys_host_disk_write_bytes[1m]) now produces a warning.

The CollectPoint function in the clusterstats package treats any Prometheus warning as a fatal error, returning an error instead of the (valid) query result. This causes every single metric collection attempt to fail, resulting in 120/120 errors and the "too many errors retrieving metrics" failure.

Evidence:

1. The Prometheus bump (c2cd9148d90) is an ancestor of the failure SHA — timing matches exactly.
2. The same failure occurred simultaneously on release-25.2 (#170790), which also received the Prometheus bump.
3. Prior failure release-26.2: roachtest: admission-control/disk-bandwidth-limiter failed #169721 (release-26.2, 2026-05-05) did NOT have the Prometheus bump and failed with a completely different error mode ("mean total bandwidth below threshold"), confirming these are distinct failure modes.
4. The PR was validated only with admission-control/intent-resolution, which likely does not use rate() on non-counter-named metrics.

Fix options (in order of preference):

1. Downgrade warnings to non-fatal in CollectPoint: Change the warning handling in collector.go to log warnings instead of returning errors. The query results are still valid — Prometheus just emits an informational note. This would fix this test and all other affected tests.

2. Rename CockroachDB metrics to follow Prometheus conventions: Rename sys_host_disk_write_bytes to sys_host_disk_write_bytes_total (and similar). This is a larger change with compatibility implications.

3. Use increase() instead of rate(): While increase() also triggers the same warning, this isn't a solution by itself.

Other tests likely affected by the same issue (they use rate() on non-_total metrics via CollectPoint):

• admission-control/index-backfill — uses rate(sys_host_disk_write_bytes[1m]) and rate(sys_host_disk_read_bytes[1m])
• admission-control/single-node-index-backfill — same queries
• admission-control/disk-iops-overload — uses rate(sys_host_disk_write_count[1m]) and rate(sys_host_disk_read_count[1m])

Related Issues and PRs

• #170790 — Same failure, same root cause, on release-25.2. Same error pattern ("too many errors retrieving metrics").
• #170676 — The PR that bumped Prometheus from 2.27.1 to 2.53.5 (the root cause).
• #169721 — Different failure mode ("bandwidth below threshold") on release-26.2 pre-Prometheus-bump. Not the same root cause.
• #168953 — Same as release-26.2: roachtest: admission-control/disk-bandwidth-limiter failed #169721, different failure mode.

Timeline

Date	Event
2026-05-20	Prometheus 2.53.5 bump cherry-picked to release-26.1 (c2cd9148d90)
2026-05-21	PR #170676 merged to master
2026-05-22 06:11	Nightly test run starts on release-26.1
2026-05-22 07:17	Monitoring loop begins; every query fails with PromQL warning
2026-05-22 07:38	Test fails: 120 errors, 0 successes

Detailed evidence and investigation notes

Log Excerpts

From test.log, the monitoring phase (07:17–07:38) shows 120 consecutive identical errors:

2026/05/22 07:18:08 admission_control_disk_bandwidth_overload.go:165: could not query prom found warnings querying prometheus: [PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "sys_host_disk_write_bytes" (1:6)]
...
2026/05/22 07:38:00 admission_control_disk_bandwidth_overload.go:218: test status: done monitoring, errors: 120 successes: 0
2026/05/22 07:38:00 test_impl.go:486: test failure #1: full stack retained in failure_1.log: (admission_control_disk_bandwidth_overload.go:220).3: too many errors retrieving metrics

Note that the write bandwidth query fails first on every iteration, so the read bandwidth query is never attempted. Both queries would produce the same warning.

Code References

The warning-as-error behavior in CollectPoint:

// collector.go:91-122
func (cs *clusterStatCollector) CollectPoint(
    ctx context.Context, l *logger.Logger, at time.Time, q string,
) (map[string]map[string]StatPoint, error) {
    fromVal, warnings, err := cs.promClient.Query(ctx, q, at)
    if err != nil {
        return nil, err
    }
    if len(warnings) > 0 {
        return nil, errors.Newf("found warnings querying prometheus: %s", warnings)
    }
    // ... processes valid results
}

The PromQL queries constructed by the test:

// admission_control_disk_bandwidth_overload.go:160-161
writeBWMetric := divQuery("rate(sys_host_disk_write_bytes[1m])", 1<<20)
readBWMetric := divQuery("rate(sys_host_disk_read_bytes[1m])", 1<<20)

Things Ruled Out

• CockroachDB node crash or OOM: The node was healthy throughout. dmesg shows no OOM kills. The workloads ran fine.
• Prometheus server crash: Prometheus was responding to queries — it returned valid warnings alongside results. The issue is that the client-side code rejects responses with warnings.
• Metric not being exported: The metric exists and is being scraped. The query returns valid data, but the Go client sees the warning and discards the result.
• Prior failure mode (bandwidth below threshold): This is a completely different failure. Prior issues release-26.2: roachtest: admission-control/disk-bandwidth-limiter failed #169721 and roachtest: admission-control/disk-bandwidth-limiter failed #168953 were caused by marginal bandwidth readings, not metric collection failures.

Other Failure Occurrences

Issue	Branch	Error Mode	Prometheus Version	Same Root Cause?
#170793 (this)	release-26.1	"too many errors retrieving metrics"	2.53.5	Yes
#170790	release-25.2	"too many errors retrieving metrics"	2.53.5	Yes
#169721	release-26.2	"bandwidth below threshold"	2.27.1	No
#168953	master	"bandwidth below threshold"	2.27.1	No

Tooling Feedback

No issues with available tooling. All necessary artifacts and data were accessible.

---

Workflow run

Was this investigation helpful? Leave a 👍 or 👎 on this comment.