From 341b021e5a1c4a5eb0534c1d0873b8ddd7d049d9 Mon Sep 17 00:00:00 2001 From: Sudhanshu Sali Date: Mon, 20 Jul 2026 20:47:58 -0400 Subject: [PATCH] feat: behavioral-anomaly CloudWatch alarms; document dep-pinning stance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Q10 (AppSec review): add optional fleet-wide anomaly alarms, provisioned by the CFN Launch Stack when BudgetEmailAddress is set (off otherwise). Two alarms — escalation spike (default 25/hr) and invocation spike (default 100/hr), tunable via new parameters — plus an SNS topic. Each alarm aggregates its metric across all dimension schemas with a Metrics Insights query (SELECT SUM(...) FROM "Shadow"): a fixed-dimension alarm would watch an empty stream, and SEARCH is illegal in alarms. Alarms are regional and depend on Shadow's metric emission, both documented. Contract test asserts each alarm aggregates an emitted metric and wires its own threshold param. Q8 (AppSec review): keep deps version-pinned, not hash-pinned — SHA-pinned actions are the required control; pip hash-pinning was disproportionate for 3 deps (recipe noted for future). Docs: README cost-protection/monitoring updated for the alarms (region + metric dependencies, honest cost framing) and a corrected email-only budget row; cloudwatch.py docstring notes the Reason dimension. 542 tests pass. --- README.md | 12 ++- infrastructure/shadow-iam-stack.yaml | 123 ++++++++++++++++++++++++--- requirements.txt | 5 +- src/scripts/shadow/cloudwatch.py | 3 +- tests/contract/test_cfn_template.py | 47 +++++++++- 5 files changed, 169 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 35a97fd..a2e5bbe 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,7 @@ You pay the bill — Bedrock is invoked from your AWS account on your credential | Override `BEDROCK_MODEL_ID` to Haiku 4.5 across all stages | ~80% cost cut | Quality cost — bench before adopting | | `paths-ignore: ['**/*.md', 'docs/**', '.github/**']` in caller workflow | Zero cost on docs PRs | Docs PRs unreviewed | | Run only on labelled PRs: `if: contains(github.event.pull_request.labels.*.name, 'needs-shadow')` | Linear with label use | Maintainer must label | -| AWS Budgets alarm at $50/month with auto-action `SHADOW_DISABLED=true` | Hard cap | Bot stops mid-month if blown | +| AWS Budgets alarm at $50/month (email alert) | Spend visibility | Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade, not automatic | The CloudFormation Launch Stack provisions the Budget for you; see [Cost protection](#cost-protection). @@ -291,7 +291,8 @@ The button opens AWS Console with [`infrastructure/shadow-iam-stack.yaml`](infra | **ShadowWorkflowRef** | `*` for quick start, a `refs/tags/v1.x` release tag (e.g. `refs/tags/v1.8`), or a 40-char SHA to pin trust to one audited revision | | **BedrockRegion** | Where Bedrock will be invoked. `us-east-1` / `us-west-2` / `us-east-2` are the validated combinations; other regions work if both Opus 4.8 and Haiku 4.5 are available there ([model-region matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html)). The region you pick here must match where you enable model access in the next step. | | **ExistingOidcProviderArn** | Leave blank if your account has no GitHub OIDC provider yet. **If your account already uses GitHub Actions OIDC, paste the existing provider ARN** (`aws iam list-open-id-connect-providers`). Leaving blank when one exists fails with `EntityAlreadyExists`. | -| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. Set both to enable an AWS Budget that emails at 80% / 100% of the cap. `0` / blank skips the alarm. | +| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. `MonthlyBudgetLimit > 0` + an email enables an AWS Budget emailing at 80% / 100% of the cap (`0` skips the Budget). **Supplying `BudgetEmailAddress` also provisions two behavioral-anomaly alarms** (escalation/invocation spikes — see [Cost protection](#cost-protection)) regardless of the limit; blank skips both. Confirm the SNS subscription email or alerts won't arrive. | +| **EscalationSpikeThreshold** / **InvocationSpikeThreshold** | Fleet-wide per-hour alarm thresholds (defaults `25` / `100`). Only used when `BudgetEmailAddress` is set. Raise them above your normal hourly volume to avoid false pages. | | **ProvisionGuardrail** | Default `true`. Provisions a Bedrock Guardrail with prompt-attack defense + PII blocks (see [Security model](#security-model)). Set to `false` only if you maintain a custom guardrail and want to point Shadow at it via the `GUARDRAIL_ID`/`GUARDRAIL_VERSION` secrets. | The stack creates the OIDC provider (if needed), an IAM role with the canonical `job_workflow_ref`-pinned trust policy, a Bedrock-invoke permission scoped to Anthropic models only, AND (by default) a Bedrock Guardrail with prompt-attack + PII filters. After deploy, copy these outputs into repo secrets: @@ -310,6 +311,8 @@ After the stack is up: python -m shadow.doctor --role-arn $ARN --region $REGION ``` +> **Deploy in your Bedrock region.** The button opens CloudFormation in `us-east-1`; if your `BedrockRegion`/`aws_region` is elsewhere, switch the console region before creating the stack. The optional anomaly alarms are regional and only see metrics emitted in their own region, so a region mismatch leaves them permanently green. + > **Heads-up on the template URL:** the Launch Stack button resolves `…/sudsali/shadow/v1.8/infrastructure/shadow-iam-stack.yaml` at click time — pinned to the `v1.8` release tag, so it's stable as long as you use this button. If you bump to a newer Shadow release, grab that release's Launch Stack URL. For fully reproducible IAM provisioning, download the YAML at a specific SHA and upload it manually. ### Manual setup (alternative) @@ -410,7 +413,7 @@ Every analyze run writes a `shadow_result.json` artifact, retained 7 days by Git - **`security_events`** — per-PR histogram of sanitizer blocks during the analyze stage. Categories like `aws_access_key`, `github_pat`, `jwt`, `ignore previous instructions` — never the matched value itself. The `by_category` list shows which categories fired and how often. (Sanitizer blocks during the act stage land in workflow logs, not the artifact.) - **`_integrity`** — SHA-256 of the artifact body, bound to `(repo, run_id, pr_number)`. The `act` job verifies this before posting; a replayed or tampered artifact is rejected. `RUN_ATTEMPT` is intentionally excluded so `gh run rerun --failed-only` still verifies. Set `SHADOW_VERIFY_ARTIFACT=false` to opt out (combined-job flows only). - **Refutation Trail in posted comments** — UPHELD findings include a collapsed `
` block showing the Investigator's hypothesis and the Critic's disprove attempt when either is non-empty. The disprove pattern is *auditable* because adopters can read why each surviving finding survived. -- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/PostFailures` (GitHub post failed after retries) and `Shadow/SlackDeliveryFailures` (an escalation's optional Slack ping failed to deliver) from the act step. Dimensions: `Repository`, `Pipeline`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace. +- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/Throttled` on throttled runs, and `Shadow/PostFailures` / `Shadow/SlackDeliveryFailures` from the act step. Dimensions: `Repository`, `Pipeline`, and `Reason`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace. --- @@ -421,6 +424,7 @@ The per-PR levers (under [What it costs](#what-it-costs)) bound a single review. - **Per-(repo, item) hourly rate limit** (`BOT_MAX_RUNS_PER_HOUR`, default `20`). Caps how many times a single PR or issue can trigger Shadow per rolling hour. Beyond the limit, the bot ESCALATES with a `:rate-limited` label instead of running the agent pipeline. Defends against an adversary closing/reopening or editing a PR title in a loop. Set to `0` to disable. **Issue/issue_comment events** require `run-name: "Shadow #${{ github.event.issue.number || ... }}"` in your caller workflow so the rate-limit gate can match prior runs (see [`examples/caller-workflow.yml`](examples/caller-workflow.yml)). - **Pre-flight diff/file caps** (`BOT_MAX_DIFF_FOR_REVIEW_CHARS`, `BOT_MAX_FILES_FOR_REVIEW`, defaults `100000` / `50`). A 50-file PR makes the Investigator read 5+ files, the Critic re-reads, the Reporter formats — costs multiply. Diff or file count above the cap → ESCALATE before any Bedrock call. Pre-flight escalation is ~$0; a runaway pipeline on a giant PR is $5+. - **AWS Budgets opt-in via CFN** (`MonthlyBudgetLimit` parameter on `shadow-iam-stack.yaml`). Set a positive USD amount + a `BudgetEmailAddress` and the stack creates an `AWS::Budgets::Budget` filtered to Amazon Bedrock spend, with email alerts at 80% and 100%. `0` skips Budget creation (default — AWS Budgets bills $0.02/budget/day, so opt-in only). Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade. +- **Behavioral-anomaly alarms via CFN** (provisioned when `BudgetEmailAddress` is set). Two CloudWatch alarms aggregate the `Shadow` namespace fleet-wide via a Metrics Insights query (`SELECT SUM(...) FROM "Shadow"`): an **escalation spike** (`EscalationSpikeThreshold`, default `25`/hr — flags prompt-injection/abuse bursts or systemic failures) and an **invocation spike** (`InvocationSpikeThreshold`, default `100`/hr — flags public-trigger floods driving Bedrock spend, faster than the monthly Budget). Both notify the `AlarmTopicArn` SNS topic (`BudgetEmailAddress` auto-subscribed — confirm the email). Detection latency is up to 1 hour; tune thresholds to your fleet volume to avoid false pages. **Deploy the stack in the same region as your `aws_region`/`BedrockRegion`** — CloudWatch alarms are regional and see only metrics emitted in their own region. The alarms depend on Shadow's metric emission, so they are blind if `SHADOW_CLOUDWATCH_DISABLED=true` (they sit green, not red — absence of data isn't a breach). These are Metrics Insights query alarms, billed per alarm plus the metrics each query scans (which grows with your repo/reason cardinality) — small but not flat; see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). --- @@ -462,7 +466,7 @@ If you previously tried to set `SHADOW_DISABLED` as a Secret rather than a Varia **Implemented** (shipped, covered by tests + CI): - BYO-AWS reusable workflow with two-job security split (`analyze` / `act`) -- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip) +- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, optional behavioral-anomaly CloudWatch alarms (escalation/invocation spikes), **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip) - `shadow doctor` preflight CLI (verifies role, Bedrock access, prompts) - Audit trail in artifact: prompt-hash provenance, security-events histogram, SHA-256 integrity stamp bound to `(repo, run_id, pr_number)` - Refutation Trail rendered into posted comments (`
` block per finding) diff --git a/infrastructure/shadow-iam-stack.yaml b/infrastructure/shadow-iam-stack.yaml index 211429c..585602b 100644 --- a/infrastructure/shadow-iam-stack.yaml +++ b/infrastructure/shadow-iam-stack.yaml @@ -15,19 +15,21 @@ Metadata: Parameters: [ShadowSourceRepo, ShadowWorkflowRef] - Label: { default: "Bedrock region" } Parameters: [BedrockRegion] - - Label: { default: "Cost protection (optional)" } - Parameters: [MonthlyBudgetLimit, BudgetEmailAddress] + - Label: { default: "Cost protection & anomaly alarms (optional)" } + Parameters: [MonthlyBudgetLimit, BudgetEmailAddress, EscalationSpikeThreshold, InvocationSpikeThreshold] - Label: { default: "Prompt-injection guardrail" } Parameters: [ProvisionGuardrail] ParameterLabels: - GitHubOrg: { default: "Your GitHub org or user (e.g., awslabs, sudsali)" } - GitHubRepo: { default: "Your repo name (use * for org-wide; audit first)" } - ShadowSourceRepo: { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" } - ShadowWorkflowRef: { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" } - BedrockRegion: { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" } - MonthlyBudgetLimit: { default: "Monthly Bedrock spend limit in USD (0 = disabled)" } - BudgetEmailAddress: { default: "Email for budget threshold notifications" } - ProvisionGuardrail: { default: "Provision a Bedrock Guardrail (recommended; default ON)" } + GitHubOrg: { default: "Your GitHub org or user (e.g., awslabs, sudsali)" } + GitHubRepo: { default: "Your repo name (use * for org-wide; audit first)" } + ShadowSourceRepo: { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" } + ShadowWorkflowRef: { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" } + BedrockRegion: { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" } + MonthlyBudgetLimit: { default: "Monthly Bedrock spend limit in USD (0 = disabled)" } + BudgetEmailAddress: { default: "Email for budget + anomaly-alarm notifications (also enables the alarms)" } + EscalationSpikeThreshold: { default: "Escalations/hour that trips the anomaly alarm (default 25)" } + InvocationSpikeThreshold: { default: "Invocations/hour that trips the anomaly alarm (default 100)" } + ProvisionGuardrail: { default: "Provision a Bedrock Guardrail (recommended; default ON)" } Parameters: GitHubOrg: @@ -102,10 +104,33 @@ Parameters: BudgetEmailAddress: Type: String Description: >- - Email to notify when monthly budget threshold is hit. Required only - if MonthlyBudgetLimit > 0. + Email to notify for budget threshold AND behavioral-anomaly alarms. + Supplying it provisions the anomaly alarms (see EscalationSpikeThreshold / + InvocationSpikeThreshold) regardless of MonthlyBudgetLimit; you must click + the SNS confirmation email for alerts to arrive. Leave blank to skip both. Default: "" + EscalationSpikeThreshold: + Type: Number + Description: >- + Fleet-wide Escalations/hour above which the anomaly alarm fires. Tune to + your traffic: escalations cover injection/abuse AND benign causes + (model/agent-stage failures, rate-limited retries, and every PR + when BOT_AGENT_PIPELINE=0), so set above your normal hourly escalation + volume to avoid paging on routine config. + Default: 25 + MinValue: 1 + + InvocationSpikeThreshold: + Type: Number + Description: >- + Fleet-wide Invocations/hour above which the anomaly alarm fires. One + Invocation = one analyze run. BOT_MAX_RUNS_PER_HOUR caps per-(repo,item) + at 20, so a busy repo with several active PRs/issues can legitimately run + 100+/hr — set above your expected fleet peak. + Default: 100 + MinValue: 1 + ProvisionGuardrail: Type: String Description: >- @@ -127,6 +152,9 @@ Conditions: - !Not [!Equals [!Ref MonthlyBudgetLimit, 0]] - !Not [!Equals [!Ref BudgetEmailAddress, ""]] CreateGuardrail: !Equals [!Ref ProvisionGuardrail, "true"] + # Behavioral alarms reuse BudgetEmailAddress as the notification target, so + # they light up whenever an operator supplies an email (no extra knob). + CreateAlarms: !Not [!Equals [!Ref BudgetEmailAddress, ""]] Resources: GitHubOidcProvider: @@ -369,6 +397,69 @@ Resources: - SubscriptionType: EMAIL Address: !Ref BudgetEmailAddress + ShadowAlarmTopic: + # Fan-out point so operators can add more subscribers (PagerDuty, extra + # emails) beyond the auto-subscribed BudgetEmailAddress. + Type: AWS::SNS::Topic + Condition: CreateAlarms + Properties: + TopicName: !Sub "shadow-bot-alarms-${GitHubOrg}" + # No KmsMasterKeyId: CloudWatch can't publish to an aws/sns-encrypted + # topic (can't grant the service principal on the managed key), and the + # payload is non-sensitive (metric counts + org name). + Subscription: + - Protocol: email + Endpoint: !Ref BudgetEmailAddress + + ShadowEscalationSpikeAlarm: + # Metrics Insights SQL (not a fixed-dimension metric or SEARCH — both are + # dead/invalid here) so SUM spans every Repository/Pipeline/Reason schema. + Type: AWS::CloudWatch::Alarm + Condition: CreateAlarms + Properties: + AlarmName: !Sub "shadow-bot-escalation-spike-${GitHubOrg}" + AlarmDescription: >- + Fleet-wide Escalations exceeded the threshold in a 1-hour window — + possible prompt-injection/abuse burst, or a systemic failure (e.g. + model/agent-stage failures escalating many items). Check recent PR/issue + activity and the analyze-job logs before assuming an attack. + EvaluationPeriods: 1 + Threshold: !Ref EscalationSpikeThreshold + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + Metrics: + - Id: esc_total + Label: EscalationsPerHourFleetWide + Expression: 'SELECT SUM("Escalations") FROM "Shadow"' + Period: 3600 + ReturnData: true + AlarmActions: + - !Ref ShadowAlarmTopic + + ShadowInvocationSpikeAlarm: + # Hourly-burst cost signal (faster than the monthly Budget); same + # Metrics-Insights-SQL aggregation as ShadowEscalationSpikeAlarm. + Type: AWS::CloudWatch::Alarm + Condition: CreateAlarms + Properties: + AlarmName: !Sub "shadow-bot-invocation-spike-${GitHubOrg}" + AlarmDescription: >- + Fleet-wide Invocations exceeded the threshold in a 1-hour window — + possible flood of public trigger events (issues/PRs) driving unexpected + Bedrock spend. Cross-check with BOT_MAX_RUNS_PER_HOUR and the Budget. + EvaluationPeriods: 1 + Threshold: !Ref InvocationSpikeThreshold + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + Metrics: + - Id: inv_total + Label: InvocationsPerHourFleetWide + Expression: 'SELECT SUM("Invocations") FROM "Shadow"' + Period: 3600 + ReturnData: true + AlarmActions: + - !Ref ShadowAlarmTopic + Outputs: ShadowRoleArn: Description: >- @@ -406,6 +497,14 @@ Outputs: repository secret. Numbered version is published immediately. Value: !GetAtt ShadowGuardrailVersion.Version + AlarmTopicArn: + Condition: CreateAlarms + Description: >- + SNS topic ARN for the behavioral-anomaly alarms. BudgetEmailAddress is + auto-subscribed (confirm the emailed link, or alerts never arrive); + subscribe additional endpoints (PagerDuty, extra emails) to this ARN. + Value: !Ref ShadowAlarmTopic + NextSteps: Description: What to do after this stack succeeds. Value: !If diff --git a/requirements.txt b/requirements.txt index c35af55..b8f3315 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,5 @@ -# Version-pinned for reproducibility. Not hash-pinned yet: to defend against -# PyPI compromise, regenerate with `pip-compile --generate-hashes` and call -# `pip install --require-hashes` in the workflow. +# Version-pinned, not hash-pinned: judged disproportionate for 3 deps. To add +# hash-pinning against a PyPI compromise: pip-compile --generate-hashes. requests==2.33.1 boto3==1.42.94 PyYAML==6.0.2 diff --git a/src/scripts/shadow/cloudwatch.py b/src/scripts/shadow/cloudwatch.py index 0dfab7d..abf6557 100644 --- a/src/scripts/shadow/cloudwatch.py +++ b/src/scripts/shadow/cloudwatch.py @@ -4,7 +4,8 @@ account so they can graph Shadow as part of their existing observability. Best-effort: a CloudWatch failure NEVER blocks the bot from posting findings. -Dimensions: Repository (GITHUB_REPOSITORY), Pipeline (agentic). +Dimensions: Repository (GITHUB_REPOSITORY), Pipeline, and Reason (the +action/skip cause) on every datum where a reason is set. Namespace: Shadow (the CFN Launch Stack policy is scoped to this exact namespace via a `cloudwatch:namespace` condition; setting SHADOW_CLOUDWATCH_NAMESPACE to anything else requires updating the IAM diff --git a/tests/contract/test_cfn_template.py b/tests/contract/test_cfn_template.py index 7c10292..f58709d 100644 --- a/tests/contract/test_cfn_template.py +++ b/tests/contract/test_cfn_template.py @@ -2,6 +2,7 @@ names, resource names, and output names are a public contract — adopter docs reference them, and a downstream `Launch Stack` URL would break if they change.""" +import re from pathlib import Path import pytest @@ -42,6 +43,8 @@ def test_template_top_level_shape(doc): assert set(doc["Resources"].keys()) == { "GitHubOidcProvider", "ShadowBotRole", "ShadowMonthlyBudget", "ShadowGuardrail", "ShadowGuardrailVersion", + "ShadowAlarmTopic", "ShadowEscalationSpikeAlarm", + "ShadowInvocationSpikeAlarm", } @@ -52,6 +55,7 @@ def test_required_parameters_present(doc): "ShadowWorkflowRef", "BedrockRegion", "ExistingOidcProviderArn", "MonthlyBudgetLimit", "BudgetEmailAddress", + "EscalationSpikeThreshold", "InvocationSpikeThreshold", "ProvisionGuardrail"} assert set(doc["Parameters"].keys()) == expected @@ -60,7 +64,7 @@ def test_required_outputs_present(doc): # README points adopters at these output names. expected = {"ShadowRoleArn", "OidcProviderArn", "TrustPolicyScope", "BedrockRegion", "NextSteps", - "GuardrailId", "GuardrailVersion"} + "GuardrailId", "GuardrailVersion", "AlarmTopicArn"} assert set(doc["Outputs"].keys()) == expected @@ -157,6 +161,47 @@ def test_monthly_budget_resource_present_when_set(doc): assert sub["SubscriptionType"] == "EMAIL" +def test_behavioral_alarms_aggregate_the_emitted_metrics(doc): + """Each alarm must aggregate its metric across all dimension schemas via a + Metrics Insights SQL query. A bare MetricName (no Dimensions) or a SEARCH + expression is a dead/invalid alarm — the metric is emitted only WITH + dimensions, and CloudWatch rejects SEARCH in an alarm. Metric names are + cross-checked against cloudwatch.py so an emitter rename can't leave the + alarm targeting a metric that's never published.""" + emitter_src = ( + Path(__file__).resolve().parents[2] + / "src/scripts/shadow/cloudwatch.py" + ).read_text() + alarms = { + "ShadowEscalationSpikeAlarm": ("Escalations", "EscalationSpikeThreshold"), + "ShadowInvocationSpikeAlarm": ("Invocations", "InvocationSpikeThreshold"), + } + for name, (metric, threshold_param) in alarms.items(): + # The metric the alarm targets must actually be emitted by the engine. + # Match _datum() tolerating whitespace/quote style + # so a benign reformat of cloudwatch.py doesn't false-fail this. + assert re.search(rf'_datum\(\s*[\'"]{metric}[\'"]', emitter_src), \ + f"{name} targets {metric}, which cloudwatch.py does not emit" + alarm = doc["Resources"][name] + assert alarm["Type"] == "AWS::CloudWatch::Alarm" + assert alarm["Condition"] == "CreateAlarms" + props = alarm["Properties"] + # A top-level MetricName with no Dimensions is the dead-alarm footgun. + assert "MetricName" not in props, f"{name} uses bare MetricName (dead-alarm risk)" + exprs = [m.get("Expression", "") for m in props.get("Metrics", [])] + joined = " ".join(exprs) + assert "SEARCH(" not in joined, f"{name} uses SEARCH (illegal in a CloudWatch alarm)" + assert "SELECT" in joined and "SUM(" in joined.upper(), \ + f"{name} is not a SELECT SUM(...) Metrics Insights query" + assert f'"{metric}"' in joined, f"{name} does not aggregate {metric}" + # Threshold must be wired to this alarm's own parameter (not swapped). + assert props["Threshold"] == threshold_param, \ + f"{name} Threshold is not !Ref {threshold_param}" + assert '"Shadow"' in joined, f"{name} not scoped to the Shadow namespace" + # Wired to the SNS topic. (_CfnLoader renders !Ref as the bare scalar.) + assert alarm["Properties"]["AlarmActions"] == ["ShadowAlarmTopic"] + + def test_shadow_source_repo_default_and_threading(doc): """ShadowSourceRepo defaults to upstream sudsali/shadow but threads through job_workflow_ref so a fork can stand up its own role without editing the