Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
apiVersion: runwhen.com/v1
kind: GenerationRules
spec:
platform: gcp
generationRules:
- resourceTypes:
- project
matchRules:
- type: pattern
pattern: ".+"
properties: [name]
mode: substring
slxs:
- baseName: gcp-artifact-registry-spend
qualifiers: ["project"]
baseTemplateName: gcp-artifact-registry-spend-analysis
levelOfDetail: basic
outputItems:
- type: slx
- type: sli
templateName: gcp-artifact-registry-spend-analysis-sli.yaml
- type: runbook
templateName: gcp-artifact-registry-spend-analysis-taskset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
apiVersion: runwhen.com/v1
kind: ServiceLevelIndicator
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
displayUnitsLong: Health Score
displayUnitsShort: score
locations:
- {{default_location}}
description: Measures Artifact Registry spend health for project {{ match_resource.project_name }} using anomaly, MoM growth, and concentration signals.
codeBundle:
{% if repo_url %}
repoUrl: {{repo_url}}
{% else %}
repoUrl: https://github.com/runwhen-contrib/rw-cli-codecollection.git
{% endif %}
{% if ref %}
ref: {{ref}}
{% else %}
ref: main
{% endif %}
pathToRobot: codebundles/gcp-artifact-registry-spend-analysis/sli.robot
intervalStrategy: intermezzo
intervalSeconds: 300
configProvided:
- name: GCP_PROJECT_IDS
value: "{{ match_resource.project_name }}"
- name: GCP_BILLING_EXPORT_TABLE
value: "{{ custom.gcp_billing_export_table | default('') }}"
- name: COST_ANALYSIS_LOOKBACK_DAYS
value: "{{ custom.cost_analysis_lookback_days | default('30') }}"
- name: ARTIFACT_COST_SPIKE_MULTIPLIER
value: "{{ custom.artifact_cost_spike_multiplier | default('2') }}"
- name: ARTIFACT_MOM_GROWTH_THRESHOLD_PERCENT
value: "{{ custom.artifact_mom_growth_threshold_percent | default('25') }}"
- name: ARTIFACT_PROJECT_COST_THRESHOLD_PERCENT
value: "{{ custom.artifact_project_cost_threshold_percent | default('20') }}"
- name: GCP_ORG_WIDE_REPORT
value: "{{ custom.gcp_org_wide_report | default('false') }}"
secretsProvided:
{% if wb_version %}
{% include "gcp-auth.yaml" ignore missing %}
{% else %}
- name: gcp_credentials
workspaceKey: AUTH DETAILS NOT FOUND
{% endif %}
alertConfig:
tasks:
persona: eager-edgar
sessionTTL: 10m
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
apiVersion: runwhen.com/v1
kind: ServiceLevelX
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
imageURL: https://storage.googleapis.com/runwhen-nonprod-shared-images/icons/gcp/billing/billing.svg
alias: {{ match_resource.project_name }} Artifact Registry Spend Analysis
asMeasuredBy: Artifact registry spend health based on cost anomalies, month-over-month growth, and project spend concentration.
configProvided:
- name: SLX_PLACEHOLDER
value: SLX_PLACEHOLDER
owners:
- {{workspace.owner_email}}
statement: GCP Artifact Registry and legacy Container Registry spend should remain predictable with no sustained anomalies or unchecked storage growth.
additionalContext:
{% include "gcp-hierarchy.yaml" ignore missing %}
qualified_name: "{{ match_resource.qualified_name }}"
tags:
{% include "gcp-tags.yaml" ignore missing %}
- name: cloud
value: gcp
- name: service
value: artifact-registry
- name: scope
value: project
- name: access
value: read-only
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
apiVersion: runwhen.com/v1
kind: Runbook
metadata:
name: {{slx_name}}
labels:
{% include "common-labels.yaml" %}
annotations:
{% include "common-annotations.yaml" %}
spec:
location: {{default_location}}
description: Analyze Artifact Registry and legacy GCR spend for project {{ match_resource.project_name }} using BigQuery billing export.
codeBundle:
{% if repo_url %}
repoUrl: {{repo_url}}
{% else %}
repoUrl: https://github.com/runwhen-contrib/rw-cli-codecollection.git
{% endif %}
{% if ref %}
ref: {{ref}}
{% else %}
ref: main
{% endif %}
pathToRobot: codebundles/gcp-artifact-registry-spend-analysis/runbook.robot
configProvided:
- name: GCP_PROJECT_IDS
value: "{{ match_resource.project_name }}"
- name: GCP_BILLING_EXPORT_TABLE
value: "{{ custom.gcp_billing_export_table | default('') }}"
- name: COST_ANALYSIS_LOOKBACK_DAYS
value: "{{ custom.cost_analysis_lookback_days | default('30') }}"
- name: ARTIFACT_COST_SPIKE_MULTIPLIER
value: "{{ custom.artifact_cost_spike_multiplier | default('2') }}"
- name: ARTIFACT_MOM_GROWTH_THRESHOLD_PERCENT
value: "{{ custom.artifact_mom_growth_threshold_percent | default('25') }}"
- name: ARTIFACT_PROJECT_COST_THRESHOLD_PERCENT
value: "{{ custom.artifact_project_cost_threshold_percent | default('20') }}"
- name: OUTPUT_FORMAT
value: "{{ custom.output_format | default('table') }}"
- name: GCP_ORG_WIDE_REPORT
value: "{{ custom.gcp_org_wide_report | default('false') }}"
secretsProvided:
{% if wb_version %}
{% include "gcp-auth.yaml" ignore missing %}
{% else %}
- name: gcp_credentials
workspaceKey: AUTH DETAILS NOT FOUND
{% endif %}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Test infrastructure

This CodeBundle analyzes BigQuery billing export data. Integration tests require an existing organization billing export table and GCP credentials with BigQuery read access.

Run static validation from this directory:

```bash
task
```

For live integration testing, configure `GCP_BILLING_EXPORT_TABLE` and `GOOGLE_APPLICATION_CREDENTIALS`, then execute individual task scripts from the bundle root.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: "3"

tasks:
default:
desc: "Run CodeBundle validation suite"
cmds:
- task: validate-structure

validate-structure:
desc: "Validate required files and basic script checks"
cmds:
- ./validate-all-tests.sh

clean:
desc: "Remove local test artifacts"
cmds:
- rm -f ../artifact_*.json ../artifact_*.txt

build-infra:
desc: "Placeholder for billing export test infra (requires org billing export)"
dir: terraform
cmds:
- echo "Billing export integration tests require an existing BigQuery billing export table."
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
terraform {
backend "local" {
path = "terraform.tfstate"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
terraform {
required_version = ">= 1.5.0"
}

# BigQuery billing export is organization-scoped and cannot be synthesized in CI.
# This placeholder keeps the standard .test/terraform layout for future fixture work.

output "note" {
value = "Use an existing GCP billing export table for integration testing."
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
output "billing_export_table" {
value = var.billing_export_table
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
terraform {
required_providers {}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
billing_export_table = ""
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
variable "billing_export_table" {
type = string
description = "Optional billing export table for manual integration tests"
default = ""
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"

need=(
runbook.robot
sli.robot
README.md
artifact-billing-common.sh
analyze-artifact-registry-spend.sh
report-top-artifact-cost-contributors.sh
compare-artifact-spend-mom.sh
detect-artifact-cost-anomalies.sh
generate-artifact-spend-recommendations.sh
artifact-spend-sli-check.sh
.runwhen/generation-rules/gcp-artifact-registry-spend-analysis.yaml
.runwhen/templates/gcp-artifact-registry-spend-analysis-slx.yaml
.runwhen/templates/gcp-artifact-registry-spend-analysis-taskset.yaml
.runwhen/templates/gcp-artifact-registry-spend-analysis-sli.yaml
)

for f in "${need[@]}"; do
if [[ ! -e "$f" ]]; then
echo "missing: $f" >&2
exit 1
fi
done

for script in *.sh; do
if [[ ! -x "$script" ]]; then
echo "not executable: $script" >&2
exit 1
fi
done

# Validate JSON issue structure helpers using mock billing rows (no live GCP required)
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT

cat > "$TMPDIR/mock_rows.json" <<'EOF'
[
{"project_id":"proj-a","project_name":"Project A","service_name":"Artifact Registry","sku_description":"Artifact Registry Storage","usage_date":"2026-05-01","total_cost":"100","usage_amount":"10","usage_unit":"gibibyte month"},
{"project_id":"proj-a","project_name":"Project A","service_name":"Container Registry","sku_description":"Container Registry Storage","usage_date":"2026-05-01","total_cost":"50","usage_amount":"5","usage_unit":"gibibyte month"},
{"project_id":"proj-b","project_name":"Project B","service_name":"Artifact Registry","sku_description":"Artifact Registry Storage","usage_date":"2026-05-01","total_cost":"10","usage_amount":"1","usage_unit":"gibibyte month"}
]
EOF

total=$(jq '[.[].total_cost | tonumber] | add' "$TMPDIR/mock_rows.json")
if [[ "$total" != "160" ]]; then
echo "mock aggregation failed: expected 160 got $total" >&2
exit 1
fi

echo "gcp-artifact-registry-spend-analysis structure OK"
70 changes: 70 additions & 0 deletions codebundles/gcp-artifact-registry-spend-analysis/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# GCP Artifact Registry Spend Analysis

Analyze Google Cloud Artifact Registry and legacy Container Registry (GCR) spend from BigQuery billing export. Surfaces storage and egress cost trends, top contributors, anomalies, and optimization recommendations so operators can align artifact spend with current or required artifacts rather than stale storage.

## Overview

- **Spend breakdown**: Per-project, per-SKU artifact spend with daily, weekly, and monthly rollups
- **Top contributors**: Rank projects and SKUs; flag projects exceeding spend share thresholds
- **Month-over-month trends**: Compare the last three complete calendar months and detect storage growth without pull activity
- **Anomaly detection**: Daily storage spikes (vs 7-day average) and sustained weekly deviations from 30-day trends
- **Optimization summary**: Actionable recommendations for cleanup policies, GCR migration, duplicate tags, and scanning scope
- **SLI health score**: 0–1 score from anomaly, MoM growth, and project concentration checks

## Configuration

### Required Variables

None. When `GCP_PROJECT_IDS` is blank, projects with artifact-related billing activity are auto-discovered from the export.

### Optional Variables

- `GCP_PROJECT_IDS`: Comma-separated GCP project IDs to analyze; blank auto-discovers from billing export (default: `""`)
- `GCP_BILLING_EXPORT_TABLE`: BigQuery billing export table (`project.dataset.gcp_billing_export_v1_*`); auto-discovered if unset (default: `""`)
- `COST_ANALYSIS_LOOKBACK_DAYS`: Days of billing history to analyze (default: `30`)
- `ARTIFACT_COST_SPIKE_MULTIPLIER`: Daily cost spike threshold as multiple of 7-day average (default: `2`)
- `ARTIFACT_MOM_GROWTH_THRESHOLD_PERCENT`: Month-over-month growth percentage that triggers an issue (default: `25`)
- `ARTIFACT_PROJECT_COST_THRESHOLD_PERCENT`: Project share of total artifact spend that triggers an issue; `0` disables (default: `20`)
- `OUTPUT_FORMAT`: Report format: `table`, `csv`, `json`, or `all` (default: `table`)
- `GCP_ORG_WIDE_REPORT`: When `true`, analyze org-wide artifact spend instead of filtering to `GCP_PROJECT_IDS` (default: `false`)

### Secrets

- `gcp_credentials`: GCP service account JSON with BigQuery billing export read access (`roles/bigquery.dataViewer`, `roles/bigquery.jobUser`, `roles/bigquery.metadataViewer` on the billing export project)

## Prerequisites

1. Enable [BigQuery billing export](https://cloud.google.com/billing/docs/how-to/export-data-bigquery) for your organization.
2. Grant the service account read access to the billing export dataset.
3. Install `gcloud`, `bq`, and `jq` in the execution environment.

Artifact billing SKUs are matched using `service.description` and `sku.description` patterns for Artifact Registry, Container Registry, storage, egress, and vulnerability scanning.

**Note:** BigQuery billing export does not expose individual repository names. Correlate high-spend projects with inventory output from the `gcp-artifact-registry-governance` bundle.

## Tasks Overview

### Analyze Artifact Registry Spend by Project and SKU

Queries billing export for artifact-related SKUs and produces per-project, per-SKU totals with daily, weekly, and monthly rollups for the lookback window.

### Report Top Artifact Registry Cost Contributors

Ranks projects and SKUs by artifact storage and transfer spend. Raises issues when a project exceeds `ARTIFACT_PROJECT_COST_THRESHOLD_PERCENT` or legacy GCR SKUs dominate spend.

### Compare Artifact Registry Spend Month-over-Month

Compares artifact costs across the last three complete calendar months. Raises issues when MoM growth exceeds `ARTIFACT_MOM_GROWTH_THRESHOLD_PERCENT` or storage grows without corresponding pull/transfer activity.

### Detect Artifact Storage Cost Anomalies

Detects daily storage cost spikes at `ARTIFACT_COST_SPIKE_MULTIPLIER` times the 7-day average and sustained weekly deviations from the 30-day trend.

### Generate Artifact Registry Spend Optimization Summary

Consolidates findings into recommendations: enable cleanup policies, retire legacy GCR, reduce duplicate tags, right-size scanning, and follow up on high-cost projects with the governance bundle.

## Related Bundles

- **gcp-project-cost-health**: Organization-wide GCP cost reporting and generic optimization recommendations
- **gcp-artifact-registry-governance**: Operational inventory and cleanup-policy checks for stale artifacts
Loading
Loading