Mathematically bind the cardinality over thr 12 h window.

[aditigaur4]

18 month retention cannot be bounded. This commit is made under the assumption that this entire thing can be removed ASAP.

This commit is binding the failed jobs dashboard to only show certain failed states to avoid explosion in case all jobs fail and/or are cancelled.
This commit also adds an upper bound of 20 to the failed jobs that can be exported every 5 minutes. Details are in the code comment.

Pre-existing code already had a bug that it was exporting Query window's starttime and endtime as opposed to job's starttime/endtime which probably was the original intent. But keeping this bug actually helps us here in not exploding the backend. This bug compromises the failed jobs dashboard more than the cardinality measures-- but whatever.

18 month retention window is still a long term serious cost of this design. This entire section needs to be removed by next release.

[Copilot]

Pull request overview

This PR introduces a new per-job failed-jobs gauge (sacct_failed_jobs) in the AzSlurm exporter, but constrains its emitted series to bound metrics-backend cardinality by (1) restricting to specific failure states and (2) exporting only the top-N failures per interval (ranked by allocated node count).

Changes:

• Restrict per-job failed-job exporting to genuine failure states and cap exported jobs per interval (cardinality_bound, default 20).
• Rank failed jobs by node count and export only the top-N to prevent cardinality explosion.
• Add unit tests validating state filtering, capping behavior, ranking, and timestamp label behavior; document the new metric in the README.

Criterion	Max Points	Points Awarded	Notes
PR Description Accuracy	20	20	Description matches the intent and behavior changes.
PR Atomicity	20	20	Focused on bounding failed-jobs metric cardinality.
Logical Implementation	10	10	Logic is straightforward and covered by tests.
Regression Risk	10	0	Current implementation sorts the full failed set; could impact exporter performance under mass-failure scenarios.
Exception Handling	10	10	No new exception-handling regressions introduced in the diff.
Code Comments	10	0	New commentary includes unprofessional/alarmist wording that should be cleaned up before shipping.
Repetitive Code	10	10	No notable duplication introduced.
Spelling	5	5	No clear spelling errors flagged in stored comments.
Logging Quality	5	5	New debug logging is clear and appropriately leveled.

FINAL SCORE: 80/100

RECOMMENDATION: MERGE WITH FOLLOW-UPS
RATIONALE: The core behavior change is reasonable and has targeted tests, but a couple of documentation/performance items should be addressed to reduce operational risk and avoid misleading docs.
BLOCKERS:

• Replace full-list sorting with a top-N selection approach to reduce worst-case CPU/memory overhead when many jobs are failed.
• Align README wording with the actual configuration surface (currently appears hard-coded rather than user-configurable).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
azure-slurm-exporter/exporter/sacct.py	Adds bounded failed-jobs gauge export logic and selection by node count.
azure-slurm-exporter/test/test_sacct.py	Adds tests for state filtering, cardinality cap, ranking, and timestamp label behavior.
README.md	Documents the new sacct_failed_jobs metric and its intended bounding behavior.

[aditigaur4]

no longer needed