Add slurmrestd (REST API) collection mode to the Slurm integration

[zhuminyi]

What does this PR do?

Adds an optional REST-API collection mode to the Slurm integration: when slurm_rest_api_url is set, the check collects scheduler (sdiag), partition, and node (sinfo) metrics from slurmrestd over HTTP with a JWT token, instead of executing the Slurm CLI binaries. CLI mode remains the default and is completely unaffected when the new fields are absent.

• New module rest.py: a thin slurmrestd HTTP client (SlurmRestAPIClient) that never raises on transport/API errors — it logs and returns None so the caller can emit a slurm.rest.can_connect service check and skip metrics for that run, rather than crashing the check.
• New config: slurm_rest_api_url (its presence enables REST mode), slurm_rest_api_token/slurm_rest_api_token_file (the token file is re-read fresh every check run, so rotated tokens are picked up), slurm_openapi_version (default v0.0.42), and slurm_rest_api_user (optional — sent as X-SLURM-USER-NAME alongside the JWT, needed only when slurmrestd sits behind an authenticating proxy that reuses one privileged token for multiple users; a direct per-user JWT, e.g. via scontrol token, already identifies the user via its own claims and does not need this).
• Metric names and node/partition-per-tag shape mirror the existing CLI collectors (node.cpu_load correctly divided by 100 to match slurmrestd's scaling; one series per (node, partition) so slurm_partition_name stays single-valued; every numeric node/partition field routed through a shared unwrap_number helper, since slurmrestd's data_parser wraps some integer fields as {"set","infinite","number"} depending on OpenAPI/build version and an inconsistently-unwrapped field would otherwise be silently dropped rather than erroring).
• squeue/sacct/sshare/scontrol collection remains CLI-only; REST mode logs a warning (not a silent gap) if those are requested while REST mode is active.

Motivation

On Slurm-on-Kubernetes deployments (SUNK, Slinky), the existing CLI-based check requires the Datadog Agent to have direct access to the Slurm CLI binaries and munge/auth socket access — which in practice means either running the agent on a bare-metal/VM head node (the only currently-documented deployment) or injecting a sidecar with staged-in binaries into a login pod, with the accompanying helm-chart changes.

slurmrestd (Slurm's REST API daemon) is already commonly available on these clusters and only needs network access plus a JWT — no co-located binaries, no munge socket. This lets the check run as a normal Kubernetes cluster check (dispatched by the Cluster Agent against a slurmrestd Service), with no sidecar and no per-pod injection, and with no login node required at all if one isn't already present.

This was verified end-to-end on a real Slinky Slurm cluster: deployed as a cluster check against slurmrestd, confirmed metric correctness against the equivalent CLI-mode output, and confirmed it runs correctly alongside a separate node-agent feature (Slurm-job GPU tagging, DataDog/datadog-agent#53697) with no interference — the two changes are complementary pieces of removing the Slurm sidecar dependency entirely, each shipped independently.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1-2]

   

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

Validate repository | Run Validations / Validate     

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 94.13%
• Overall Coverage: 93.13% (+4.84%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a5ad900 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

Validation	Description	Status
config	Validate default configuration files against spec.yaml	❌
license-headers	Validate Python files have proper license headers	❌

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (19)

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run