ARO-26057: add ADX Grafana datasource provisioning

[swiencki]

ARO-26057

What

This PR adds the upstream ARO-HCP support required to provision and clean up ADX data sources in Grafana as part of the existing deployment flow.

Why

We want to display Kusto data sources in Grafana for SRE dashboarding.

Special notes for your reviewer

The paired rollout PR is https://dev.azure.com/msazure/AzureRedHatOpenShift/_git/sdp-pipelines/pullrequest/15393796, and it is the downstream sdp-pipelines SHA bump and rollout enablement for this change, so this PR should be reviewed and merged first.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: swiencki
Once this PR has been reviewed and has the lgtm label, please assign geoberle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• config/OWNERS
• dev-infrastructure/OWNERS
• docs/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds ARO-HCP support to provision and clean up Azure Data Explorer (Kusto) data sources in the shared Managed Grafana workspace, integrated into the existing geography rollout flow.

Changes:

• Extend Kusto IaC to optionally grant Grafana’s managed identity DB-level Viewer access and output the cluster URI.
• Add geography pipeline steps to fetch global Grafana outputs and provision/update ADX Grafana datasources via Azure CLI.
• Introduce configuration flags/schema + docs, and update the Kusto delete cleanup to remove the matching Grafana datasource.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/monitoring.md	Documents ADX/Kusto datasource provisioning gates, naming, validation, and teardown behavior.
dev-infrastructure/templates/kusto.bicep	Threads optional Grafana resource ID into the Kusto module and exposes kustoUri output.
dev-infrastructure/modules/logs/kusto/main.bicep	Grants Grafana MI Viewer on ServiceLogs when provided and surfaces cluster URI output.
dev-infrastructure/modules/logs/kusto/cluster.bicep	Exposes Kusto cluster uri as an output.
dev-infrastructure/geography-pipeline.yaml	Adds global output step and a shell step to create/update ADX Grafana datasources.
dev-infrastructure/configurations/kusto.tmpl.bicepparam	Adds grafanaResourceId parameter placeholder for pipeline substitution.
dev-infrastructure/cleanup/delete.kusto.instance.sh	Extends cleanup to also delete the corresponding Grafana datasource.
dev-infrastructure/cleanup/delete.kusto.instance.pipeline.yaml	Wires Grafana resource ID from global outputs into the cleanup script run.
config/rendered/dev/swft/uksouth.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/prow/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/pers/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/perf/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/dev/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/rendered/dev/cspr/westus3.yaml	Adds default monitoring flags for ADX datasource provisioning (disabled).
config/config.yaml	Introduces new monitoring defaults for ADX provisioning flags.
config/config.schema.json	Defines schema/validation for the new monitoring configuration knobs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[janboll]

/hold
please use https://github.com/Azure/ARO-Tools/blob/main/tools/grafanactl/cmd/modify/cmd.go#L32
for doing modifications to grafana. Ideally we should update grafana ONCE in the pipeline, cause running an update on the instance will block it for several minutes.

[swiencki]

Thanks for the feedback, I'll move this to grafanactl in Azure/ARO-Tools.

On the "update once" concern: the existing modify datasource reconcile runs globally because UpdateGrafanaIntegrations is an ARM-level mutation that blocks the Grafana instance for several minutes. ADX datasources go through the Grafana REST API directly (POST/PUT /api/datasources), which is a lightweight config write, no ARM-level instance lock. So per-geography invocation is safe here. Is this good with you?

[openshift-ci]

@swiencki: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/cspr	8d14303	link	true	/test cspr

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.