diff --git a/README.md b/README.md index 35a97fd..a2e5bbe 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,7 @@ You pay the bill — Bedrock is invoked from your AWS account on your credential | Override `BEDROCK_MODEL_ID` to Haiku 4.5 across all stages | ~80% cost cut | Quality cost — bench before adopting | | `paths-ignore: ['**/*.md', 'docs/**', '.github/**']` in caller workflow | Zero cost on docs PRs | Docs PRs unreviewed | | Run only on labelled PRs: `if: contains(github.event.pull_request.labels.*.name, 'needs-shadow')` | Linear with label use | Maintainer must label | -| AWS Budgets alarm at $50/month with auto-action `SHADOW_DISABLED=true` | Hard cap | Bot stops mid-month if blown | +| AWS Budgets alarm at $50/month (email alert) | Spend visibility | Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade, not automatic | The CloudFormation Launch Stack provisions the Budget for you; see [Cost protection](#cost-protection). @@ -291,7 +291,8 @@ The button opens AWS Console with [`infrastructure/shadow-iam-stack.yaml`](infra | **ShadowWorkflowRef** | `*` for quick start, a `refs/tags/v1.x` release tag (e.g. `refs/tags/v1.8`), or a 40-char SHA to pin trust to one audited revision | | **BedrockRegion** | Where Bedrock will be invoked. `us-east-1` / `us-west-2` / `us-east-2` are the validated combinations; other regions work if both Opus 4.8 and Haiku 4.5 are available there ([model-region matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html)). The region you pick here must match where you enable model access in the next step. | | **ExistingOidcProviderArn** | Leave blank if your account has no GitHub OIDC provider yet. **If your account already uses GitHub Actions OIDC, paste the existing provider ARN** (`aws iam list-open-id-connect-providers`). Leaving blank when one exists fails with `EntityAlreadyExists`. | -| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. Set both to enable an AWS Budget that emails at 80% / 100% of the cap. `0` / blank skips the alarm. | +| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. `MonthlyBudgetLimit > 0` + an email enables an AWS Budget emailing at 80% / 100% of the cap (`0` skips the Budget). **Supplying `BudgetEmailAddress` also provisions two behavioral-anomaly alarms** (escalation/invocation spikes — see [Cost protection](#cost-protection)) regardless of the limit; blank skips both. Confirm the SNS subscription email or alerts won't arrive. | +| **EscalationSpikeThreshold** / **InvocationSpikeThreshold** | Fleet-wide per-hour alarm thresholds (defaults `25` / `100`). Only used when `BudgetEmailAddress` is set. Raise them above your normal hourly volume to avoid false pages. | | **ProvisionGuardrail** | Default `true`. Provisions a Bedrock Guardrail with prompt-attack defense + PII blocks (see [Security model](#security-model)). Set to `false` only if you maintain a custom guardrail and want to point Shadow at it via the `GUARDRAIL_ID`/`GUARDRAIL_VERSION` secrets. | The stack creates the OIDC provider (if needed), an IAM role with the canonical `job_workflow_ref`-pinned trust policy, a Bedrock-invoke permission scoped to Anthropic models only, AND (by default) a Bedrock Guardrail with prompt-attack + PII filters. After deploy, copy these outputs into repo secrets: @@ -310,6 +311,8 @@ After the stack is up: python -m shadow.doctor --role-arn $ARN --region $REGION ``` +> **Deploy in your Bedrock region.** The button opens CloudFormation in `us-east-1`; if your `BedrockRegion`/`aws_region` is elsewhere, switch the console region before creating the stack. The optional anomaly alarms are regional and only see metrics emitted in their own region, so a region mismatch leaves them permanently green. + > **Heads-up on the template URL:** the Launch Stack button resolves `…/sudsali/shadow/v1.8/infrastructure/shadow-iam-stack.yaml` at click time — pinned to the `v1.8` release tag, so it's stable as long as you use this button. If you bump to a newer Shadow release, grab that release's Launch Stack URL. For fully reproducible IAM provisioning, download the YAML at a specific SHA and upload it manually. ### Manual setup (alternative) @@ -410,7 +413,7 @@ Every analyze run writes a `shadow_result.json` artifact, retained 7 days by Git - **`security_events`** — per-PR histogram of sanitizer blocks during the analyze stage. Categories like `aws_access_key`, `github_pat`, `jwt`, `ignore previous instructions` — never the matched value itself. The `by_category` list shows which categories fired and how often. (Sanitizer blocks during the act stage land in workflow logs, not the artifact.) - **`_integrity`** — SHA-256 of the artifact body, bound to `(repo, run_id, pr_number)`. The `act` job verifies this before posting; a replayed or tampered artifact is rejected. `RUN_ATTEMPT` is intentionally excluded so `gh run rerun --failed-only` still verifies. Set `SHADOW_VERIFY_ARTIFACT=false` to opt out (combined-job flows only). - **Refutation Trail in posted comments** — UPHELD findings include a collapsed `
` block showing the Investigator's hypothesis and the Critic's disprove attempt when either is non-empty. The disprove pattern is *auditable* because adopters can read why each surviving finding survived. -- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/PostFailures` (GitHub post failed after retries) and `Shadow/SlackDeliveryFailures` (an escalation's optional Slack ping failed to deliver) from the act step. Dimensions: `Repository`, `Pipeline`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace. +- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/Throttled` on throttled runs, and `Shadow/PostFailures` / `Shadow/SlackDeliveryFailures` from the act step. Dimensions: `Repository`, `Pipeline`, and `Reason`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace. --- @@ -421,6 +424,7 @@ The per-PR levers (under [What it costs](#what-it-costs)) bound a single review. - **Per-(repo, item) hourly rate limit** (`BOT_MAX_RUNS_PER_HOUR`, default `20`). Caps how many times a single PR or issue can trigger Shadow per rolling hour. Beyond the limit, the bot ESCALATES with a `:rate-limited` label instead of running the agent pipeline. Defends against an adversary closing/reopening or editing a PR title in a loop. Set to `0` to disable. **Issue/issue_comment events** require `run-name: "Shadow #${{ github.event.issue.number || ... }}"` in your caller workflow so the rate-limit gate can match prior runs (see [`examples/caller-workflow.yml`](examples/caller-workflow.yml)). - **Pre-flight diff/file caps** (`BOT_MAX_DIFF_FOR_REVIEW_CHARS`, `BOT_MAX_FILES_FOR_REVIEW`, defaults `100000` / `50`). A 50-file PR makes the Investigator read 5+ files, the Critic re-reads, the Reporter formats — costs multiply. Diff or file count above the cap → ESCALATE before any Bedrock call. Pre-flight escalation is ~$0; a runaway pipeline on a giant PR is $5+. - **AWS Budgets opt-in via CFN** (`MonthlyBudgetLimit` parameter on `shadow-iam-stack.yaml`). Set a positive USD amount + a `BudgetEmailAddress` and the stack creates an `AWS::Budgets::Budget` filtered to Amazon Bedrock spend, with email alerts at 80% and 100%. `0` skips Budget creation (default — AWS Budgets bills $0.02/budget/day, so opt-in only). Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade. +- **Behavioral-anomaly alarms via CFN** (provisioned when `BudgetEmailAddress` is set). Two CloudWatch alarms aggregate the `Shadow` namespace fleet-wide via a Metrics Insights query (`SELECT SUM(...) FROM "Shadow"`): an **escalation spike** (`EscalationSpikeThreshold`, default `25`/hr — flags prompt-injection/abuse bursts or systemic failures) and an **invocation spike** (`InvocationSpikeThreshold`, default `100`/hr — flags public-trigger floods driving Bedrock spend, faster than the monthly Budget). Both notify the `AlarmTopicArn` SNS topic (`BudgetEmailAddress` auto-subscribed — confirm the email). Detection latency is up to 1 hour; tune thresholds to your fleet volume to avoid false pages. **Deploy the stack in the same region as your `aws_region`/`BedrockRegion`** — CloudWatch alarms are regional and see only metrics emitted in their own region. The alarms depend on Shadow's metric emission, so they are blind if `SHADOW_CLOUDWATCH_DISABLED=true` (they sit green, not red — absence of data isn't a breach). These are Metrics Insights query alarms, billed per alarm plus the metrics each query scans (which grows with your repo/reason cardinality) — small but not flat; see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/). --- @@ -462,7 +466,7 @@ If you previously tried to set `SHADOW_DISABLED` as a Secret rather than a Varia **Implemented** (shipped, covered by tests + CI): - BYO-AWS reusable workflow with two-job security split (`analyze` / `act`) -- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip) +- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, optional behavioral-anomaly CloudWatch alarms (escalation/invocation spikes), **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip) - `shadow doctor` preflight CLI (verifies role, Bedrock access, prompts) - Audit trail in artifact: prompt-hash provenance, security-events histogram, SHA-256 integrity stamp bound to `(repo, run_id, pr_number)` - Refutation Trail rendered into posted comments (`
` block per finding) diff --git a/infrastructure/shadow-iam-stack.yaml b/infrastructure/shadow-iam-stack.yaml index 211429c..585602b 100644 --- a/infrastructure/shadow-iam-stack.yaml +++ b/infrastructure/shadow-iam-stack.yaml @@ -15,19 +15,21 @@ Metadata: Parameters: [ShadowSourceRepo, ShadowWorkflowRef] - Label: { default: "Bedrock region" } Parameters: [BedrockRegion] - - Label: { default: "Cost protection (optional)" } - Parameters: [MonthlyBudgetLimit, BudgetEmailAddress] + - Label: { default: "Cost protection & anomaly alarms (optional)" } + Parameters: [MonthlyBudgetLimit, BudgetEmailAddress, EscalationSpikeThreshold, InvocationSpikeThreshold] - Label: { default: "Prompt-injection guardrail" } Parameters: [ProvisionGuardrail] ParameterLabels: - GitHubOrg: { default: "Your GitHub org or user (e.g., awslabs, sudsali)" } - GitHubRepo: { default: "Your repo name (use * for org-wide; audit first)" } - ShadowSourceRepo: { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" } - ShadowWorkflowRef: { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" } - BedrockRegion: { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" } - MonthlyBudgetLimit: { default: "Monthly Bedrock spend limit in USD (0 = disabled)" } - BudgetEmailAddress: { default: "Email for budget threshold notifications" } - ProvisionGuardrail: { default: "Provision a Bedrock Guardrail (recommended; default ON)" } + GitHubOrg: { default: "Your GitHub org or user (e.g., awslabs, sudsali)" } + GitHubRepo: { default: "Your repo name (use * for org-wide; audit first)" } + ShadowSourceRepo: { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" } + ShadowWorkflowRef: { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" } + BedrockRegion: { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" } + MonthlyBudgetLimit: { default: "Monthly Bedrock spend limit in USD (0 = disabled)" } + BudgetEmailAddress: { default: "Email for budget + anomaly-alarm notifications (also enables the alarms)" } + EscalationSpikeThreshold: { default: "Escalations/hour that trips the anomaly alarm (default 25)" } + InvocationSpikeThreshold: { default: "Invocations/hour that trips the anomaly alarm (default 100)" } + ProvisionGuardrail: { default: "Provision a Bedrock Guardrail (recommended; default ON)" } Parameters: GitHubOrg: @@ -102,10 +104,33 @@ Parameters: BudgetEmailAddress: Type: String Description: >- - Email to notify when monthly budget threshold is hit. Required only - if MonthlyBudgetLimit > 0. + Email to notify for budget threshold AND behavioral-anomaly alarms. + Supplying it provisions the anomaly alarms (see EscalationSpikeThreshold / + InvocationSpikeThreshold) regardless of MonthlyBudgetLimit; you must click + the SNS confirmation email for alerts to arrive. Leave blank to skip both. Default: "" + EscalationSpikeThreshold: + Type: Number + Description: >- + Fleet-wide Escalations/hour above which the anomaly alarm fires. Tune to + your traffic: escalations cover injection/abuse AND benign causes + (model/agent-stage failures, rate-limited retries, and every PR + when BOT_AGENT_PIPELINE=0), so set above your normal hourly escalation + volume to avoid paging on routine config. + Default: 25 + MinValue: 1 + + InvocationSpikeThreshold: + Type: Number + Description: >- + Fleet-wide Invocations/hour above which the anomaly alarm fires. One + Invocation = one analyze run. BOT_MAX_RUNS_PER_HOUR caps per-(repo,item) + at 20, so a busy repo with several active PRs/issues can legitimately run + 100+/hr — set above your expected fleet peak. + Default: 100 + MinValue: 1 + ProvisionGuardrail: Type: String Description: >- @@ -127,6 +152,9 @@ Conditions: - !Not [!Equals [!Ref MonthlyBudgetLimit, 0]] - !Not [!Equals [!Ref BudgetEmailAddress, ""]] CreateGuardrail: !Equals [!Ref ProvisionGuardrail, "true"] + # Behavioral alarms reuse BudgetEmailAddress as the notification target, so + # they light up whenever an operator supplies an email (no extra knob). + CreateAlarms: !Not [!Equals [!Ref BudgetEmailAddress, ""]] Resources: GitHubOidcProvider: @@ -369,6 +397,69 @@ Resources: - SubscriptionType: EMAIL Address: !Ref BudgetEmailAddress + ShadowAlarmTopic: + # Fan-out point so operators can add more subscribers (PagerDuty, extra + # emails) beyond the auto-subscribed BudgetEmailAddress. + Type: AWS::SNS::Topic + Condition: CreateAlarms + Properties: + TopicName: !Sub "shadow-bot-alarms-${GitHubOrg}" + # No KmsMasterKeyId: CloudWatch can't publish to an aws/sns-encrypted + # topic (can't grant the service principal on the managed key), and the + # payload is non-sensitive (metric counts + org name). + Subscription: + - Protocol: email + Endpoint: !Ref BudgetEmailAddress + + ShadowEscalationSpikeAlarm: + # Metrics Insights SQL (not a fixed-dimension metric or SEARCH — both are + # dead/invalid here) so SUM spans every Repository/Pipeline/Reason schema. + Type: AWS::CloudWatch::Alarm + Condition: CreateAlarms + Properties: + AlarmName: !Sub "shadow-bot-escalation-spike-${GitHubOrg}" + AlarmDescription: >- + Fleet-wide Escalations exceeded the threshold in a 1-hour window — + possible prompt-injection/abuse burst, or a systemic failure (e.g. + model/agent-stage failures escalating many items). Check recent PR/issue + activity and the analyze-job logs before assuming an attack. + EvaluationPeriods: 1 + Threshold: !Ref EscalationSpikeThreshold + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + Metrics: + - Id: esc_total + Label: EscalationsPerHourFleetWide + Expression: 'SELECT SUM("Escalations") FROM "Shadow"' + Period: 3600 + ReturnData: true + AlarmActions: + - !Ref ShadowAlarmTopic + + ShadowInvocationSpikeAlarm: + # Hourly-burst cost signal (faster than the monthly Budget); same + # Metrics-Insights-SQL aggregation as ShadowEscalationSpikeAlarm. + Type: AWS::CloudWatch::Alarm + Condition: CreateAlarms + Properties: + AlarmName: !Sub "shadow-bot-invocation-spike-${GitHubOrg}" + AlarmDescription: >- + Fleet-wide Invocations exceeded the threshold in a 1-hour window — + possible flood of public trigger events (issues/PRs) driving unexpected + Bedrock spend. Cross-check with BOT_MAX_RUNS_PER_HOUR and the Budget. + EvaluationPeriods: 1 + Threshold: !Ref InvocationSpikeThreshold + ComparisonOperator: GreaterThanThreshold + TreatMissingData: notBreaching + Metrics: + - Id: inv_total + Label: InvocationsPerHourFleetWide + Expression: 'SELECT SUM("Invocations") FROM "Shadow"' + Period: 3600 + ReturnData: true + AlarmActions: + - !Ref ShadowAlarmTopic + Outputs: ShadowRoleArn: Description: >- @@ -406,6 +497,14 @@ Outputs: repository secret. Numbered version is published immediately. Value: !GetAtt ShadowGuardrailVersion.Version + AlarmTopicArn: + Condition: CreateAlarms + Description: >- + SNS topic ARN for the behavioral-anomaly alarms. BudgetEmailAddress is + auto-subscribed (confirm the emailed link, or alerts never arrive); + subscribe additional endpoints (PagerDuty, extra emails) to this ARN. + Value: !Ref ShadowAlarmTopic + NextSteps: Description: What to do after this stack succeeds. Value: !If diff --git a/requirements.txt b/requirements.txt index c35af55..b8f3315 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,6 +1,5 @@ -# Version-pinned for reproducibility. Not hash-pinned yet: to defend against -# PyPI compromise, regenerate with `pip-compile --generate-hashes` and call -# `pip install --require-hashes` in the workflow. +# Version-pinned, not hash-pinned: judged disproportionate for 3 deps. To add +# hash-pinning against a PyPI compromise: pip-compile --generate-hashes. requests==2.33.1 boto3==1.42.94 PyYAML==6.0.2 diff --git a/src/scripts/shadow/cloudwatch.py b/src/scripts/shadow/cloudwatch.py index 0dfab7d..abf6557 100644 --- a/src/scripts/shadow/cloudwatch.py +++ b/src/scripts/shadow/cloudwatch.py @@ -4,7 +4,8 @@ account so they can graph Shadow as part of their existing observability. Best-effort: a CloudWatch failure NEVER blocks the bot from posting findings. -Dimensions: Repository (GITHUB_REPOSITORY), Pipeline (agentic). +Dimensions: Repository (GITHUB_REPOSITORY), Pipeline, and Reason (the +action/skip cause) on every datum where a reason is set. Namespace: Shadow (the CFN Launch Stack policy is scoped to this exact namespace via a `cloudwatch:namespace` condition; setting SHADOW_CLOUDWATCH_NAMESPACE to anything else requires updating the IAM diff --git a/tests/contract/test_cfn_template.py b/tests/contract/test_cfn_template.py index 7c10292..f58709d 100644 --- a/tests/contract/test_cfn_template.py +++ b/tests/contract/test_cfn_template.py @@ -2,6 +2,7 @@ names, resource names, and output names are a public contract — adopter docs reference them, and a downstream `Launch Stack` URL would break if they change.""" +import re from pathlib import Path import pytest @@ -42,6 +43,8 @@ def test_template_top_level_shape(doc): assert set(doc["Resources"].keys()) == { "GitHubOidcProvider", "ShadowBotRole", "ShadowMonthlyBudget", "ShadowGuardrail", "ShadowGuardrailVersion", + "ShadowAlarmTopic", "ShadowEscalationSpikeAlarm", + "ShadowInvocationSpikeAlarm", } @@ -52,6 +55,7 @@ def test_required_parameters_present(doc): "ShadowWorkflowRef", "BedrockRegion", "ExistingOidcProviderArn", "MonthlyBudgetLimit", "BudgetEmailAddress", + "EscalationSpikeThreshold", "InvocationSpikeThreshold", "ProvisionGuardrail"} assert set(doc["Parameters"].keys()) == expected @@ -60,7 +64,7 @@ def test_required_outputs_present(doc): # README points adopters at these output names. expected = {"ShadowRoleArn", "OidcProviderArn", "TrustPolicyScope", "BedrockRegion", "NextSteps", - "GuardrailId", "GuardrailVersion"} + "GuardrailId", "GuardrailVersion", "AlarmTopicArn"} assert set(doc["Outputs"].keys()) == expected @@ -157,6 +161,47 @@ def test_monthly_budget_resource_present_when_set(doc): assert sub["SubscriptionType"] == "EMAIL" +def test_behavioral_alarms_aggregate_the_emitted_metrics(doc): + """Each alarm must aggregate its metric across all dimension schemas via a + Metrics Insights SQL query. A bare MetricName (no Dimensions) or a SEARCH + expression is a dead/invalid alarm — the metric is emitted only WITH + dimensions, and CloudWatch rejects SEARCH in an alarm. Metric names are + cross-checked against cloudwatch.py so an emitter rename can't leave the + alarm targeting a metric that's never published.""" + emitter_src = ( + Path(__file__).resolve().parents[2] + / "src/scripts/shadow/cloudwatch.py" + ).read_text() + alarms = { + "ShadowEscalationSpikeAlarm": ("Escalations", "EscalationSpikeThreshold"), + "ShadowInvocationSpikeAlarm": ("Invocations", "InvocationSpikeThreshold"), + } + for name, (metric, threshold_param) in alarms.items(): + # The metric the alarm targets must actually be emitted by the engine. + # Match _datum() tolerating whitespace/quote style + # so a benign reformat of cloudwatch.py doesn't false-fail this. + assert re.search(rf'_datum\(\s*[\'"]{metric}[\'"]', emitter_src), \ + f"{name} targets {metric}, which cloudwatch.py does not emit" + alarm = doc["Resources"][name] + assert alarm["Type"] == "AWS::CloudWatch::Alarm" + assert alarm["Condition"] == "CreateAlarms" + props = alarm["Properties"] + # A top-level MetricName with no Dimensions is the dead-alarm footgun. + assert "MetricName" not in props, f"{name} uses bare MetricName (dead-alarm risk)" + exprs = [m.get("Expression", "") for m in props.get("Metrics", [])] + joined = " ".join(exprs) + assert "SEARCH(" not in joined, f"{name} uses SEARCH (illegal in a CloudWatch alarm)" + assert "SELECT" in joined and "SUM(" in joined.upper(), \ + f"{name} is not a SELECT SUM(...) Metrics Insights query" + assert f'"{metric}"' in joined, f"{name} does not aggregate {metric}" + # Threshold must be wired to this alarm's own parameter (not swapped). + assert props["Threshold"] == threshold_param, \ + f"{name} Threshold is not !Ref {threshold_param}" + assert '"Shadow"' in joined, f"{name} not scoped to the Shadow namespace" + # Wired to the SNS topic. (_CfnLoader renders !Ref as the bare scalar.) + assert alarm["Properties"]["AlarmActions"] == ["ShadowAlarmTopic"] + + def test_shadow_source_repo_default_and_threading(doc): """ShadowSourceRepo defaults to upstream sudsali/shadow but threads through job_workflow_ref so a fork can stand up its own role without editing the