Skip to content

Remove job related metrics/dashboards #534

Merged
aditigaur4 merged 4 commits into
masterfrom
azreenzaman/exporter-sacct-fix
Jun 26, 2026
Merged

Remove job related metrics/dashboards #534
aditigaur4 merged 4 commits into
masterfrom
azreenzaman/exporter-sacct-fix

Conversation

@azreenz

@azreenz azreenz commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Remove metrics with high cardinality labels to reduce prometheus backend explosion

Copilot AI review requested due to automatic review settings June 26, 2026 01:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR removes per-job (high-cardinality) Prometheus metrics and the Grafana dashboards/panels that depended on them, to reduce Prometheus/Azure Monitor backend series growth and associated operational risk.

Changes:

  • Removed the per-job squeue_job_nodes_allocated metric from the squeue collector and updated unit tests accordingly.
  • Removed the per-job sacct_failed_jobs metric from the sacct collector and updated unit tests accordingly.
  • Deleted the “Failed Jobs” dashboard and removed job/failed-job panels from the “Extra Details” dashboard; updated dashboard navigation text and README exported-metrics documentation.
Criterion Max Points Points Awarded Notes
PR Description Accuracy 20 20 Matches changes (removal of high-cardinality metrics/dashboards).
PR Atomicity 20 20 Single cohesive change: remove job-level metrics and dependent dashboards/tests/docs.
Logical Implementation 10 10 Straightforward removals; remaining collectors align with updated behavior.
Regression Risk 10 10 Change is intentionally breaking for removed metrics/dashboards and is described.
Exception Handling 10 10 No degradation vs existing error handling patterns.
Code Comments 10 10 Docstrings updated to reflect the new single-metric outputs.
Repetitive Code 10 10 No new duplication introduced.
Spelling 5 5 No spelling issues found in modified text.
Logging Quality 5 5 No negative changes to logging clarity/levels.

FINAL SCORE: 100/100

RECOMMENDATION: MERGE
RATIONALE: The PR cleanly removes the targeted high-cardinality metrics and updates tests/docs/dashboards consistently.
BLOCKERS: None

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
README.md Removes documentation for the deleted job-level metrics.
azure-slurm-exporter/exporter/squeue.py Removes per-job node allocation metric; keeps partition/state job counts.
azure-slurm-exporter/exporter/sacct.py Removes per-job failed-jobs gauge; keeps terminal-jobs counter.
azure-slurm-exporter/test/test_squeue.py Updates expectations to the single remaining squeue metric.
azure-slurm-exporter/test/test_sacct.py Updates expectations to the single remaining sacct metric.
azure-slurm-exporter/dashboards/library-panels/azslurm-dashboard-buttons.json Updates button label text for the details dashboard.
azure-slurm-exporter/dashboards/failed-jobs.json Deletes the failed-jobs dashboard.
azure-slurm-exporter/dashboards/azslurm-details.json Removes job/failed-job panels and updates dashboard description/layout.

@azreenz

azreenz commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

new extra details dashbaord thast doesnt contain any job specific panels
Screenshot 2026-06-25 214050

bwatrous
bwatrous previously approved these changes Jun 26, 2026
@aditigaur4
aditigaur4 merged commit f77af43 into master Jun 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants