security: hash-pin deps (Q8) + behavioral anomaly alarms (Q10)

README.md

@@ -164,7 +164,7 @@ You pay the bill — Bedrock is invoked from your AWS account on your credential
 | Override `BEDROCK_MODEL_ID` to Haiku 4.5 across all stages | ~80% cost cut | Quality cost — bench before adopting |
 | `paths-ignore: ['**/*.md', 'docs/**', '.github/**']` in caller workflow | Zero cost on docs PRs | Docs PRs unreviewed |
 | Run only on labelled PRs: `if: contains(github.event.pull_request.labels.*.name, 'needs-shadow')` | Linear with label use | Maintainer must label |
-| AWS Budgets alarm at $50/month with auto-action `SHADOW_DISABLED=true` | Hard cap | Bot stops mid-month if blown |
+| AWS Budgets alarm at $50/month (email alert) | Spend visibility | Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade, not automatic |
 
 The CloudFormation Launch Stack provisions the Budget for you; see [Cost protection](#cost-protection).
 
@@ -291,7 +291,8 @@ The button opens AWS Console with [`infrastructure/shadow-iam-stack.yaml`](infra
 | **ShadowWorkflowRef** | `*` for quick start, a `refs/tags/v1.x` release tag (e.g. `refs/tags/v1.8`), or a 40-char SHA to pin trust to one audited revision |
 | **BedrockRegion** | Where Bedrock will be invoked. `us-east-1` / `us-west-2` / `us-east-2` are the validated combinations; other regions work if both Opus 4.8 and Haiku 4.5 are available there ([model-region matrix](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html)). The region you pick here must match where you enable model access in the next step. |
 | **ExistingOidcProviderArn** | Leave blank if your account has no GitHub OIDC provider yet. **If your account already uses GitHub Actions OIDC, paste the existing provider ARN** (`aws iam list-open-id-connect-providers`). Leaving blank when one exists fails with `EntityAlreadyExists`. |
-| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. Set both to enable an AWS Budget that emails at 80% / 100% of the cap. `0` / blank skips the alarm. |
+| **MonthlyBudgetLimit** + **BudgetEmailAddress** | Optional. `MonthlyBudgetLimit > 0` + an email enables an AWS Budget emailing at 80% / 100% of the cap (`0` skips the Budget). **Supplying `BudgetEmailAddress` also provisions two behavioral-anomaly alarms** (escalation/invocation spikes — see [Cost protection](#cost-protection)) regardless of the limit; blank skips both. Confirm the SNS subscription email or alerts won't arrive. |
+| **EscalationSpikeThreshold** / **InvocationSpikeThreshold** | Fleet-wide per-hour alarm thresholds (defaults `25` / `100`). Only used when `BudgetEmailAddress` is set. Raise them above your normal hourly volume to avoid false pages. |
 | **ProvisionGuardrail** | Default `true`. Provisions a Bedrock Guardrail with prompt-attack defense + PII blocks (see [Security model](#security-model)). Set to `false` only if you maintain a custom guardrail and want to point Shadow at it via the `GUARDRAIL_ID`/`GUARDRAIL_VERSION` secrets. |
 
 The stack creates the OIDC provider (if needed), an IAM role with the canonical `job_workflow_ref`-pinned trust policy, a Bedrock-invoke permission scoped to Anthropic models only, AND (by default) a Bedrock Guardrail with prompt-attack + PII filters. After deploy, copy these outputs into repo secrets:
@@ -310,6 +311,8 @@ After the stack is up:
 python -m shadow.doctor --role-arn $ARN --region $REGION
 ```
 
+> **Deploy in your Bedrock region.** The button opens CloudFormation in `us-east-1`; if your `BedrockRegion`/`aws_region` is elsewhere, switch the console region before creating the stack. The optional anomaly alarms are regional and only see metrics emitted in their own region, so a region mismatch leaves them permanently green.
+
 > **Heads-up on the template URL:** the Launch Stack button resolves `…/sudsali/shadow/v1.8/infrastructure/shadow-iam-stack.yaml` at click time — pinned to the `v1.8` release tag, so it's stable as long as you use this button. If you bump to a newer Shadow release, grab that release's Launch Stack URL. For fully reproducible IAM provisioning, download the YAML at a specific SHA and upload it manually.
 
 ### Manual setup (alternative)
@@ -410,7 +413,7 @@ Every analyze run writes a `shadow_result.json` artifact, retained 7 days by Git
 - **`security_events`** — per-PR histogram of sanitizer blocks during the analyze stage. Categories like `aws_access_key`, `github_pat`, `jwt`, `ignore previous instructions` — never the matched value itself. The `by_category` list shows which categories fired and how often. (Sanitizer blocks during the act stage land in workflow logs, not the artifact.)
 - **`_integrity`** — SHA-256 of the artifact body, bound to `(repo, run_id, pr_number)`. The `act` job verifies this before posting; a replayed or tampered artifact is rejected. `RUN_ATTEMPT` is intentionally excluded so `gh run rerun --failed-only` still verifies. Set `SHADOW_VERIFY_ARTIFACT=false` to opt out (combined-job flows only).
 - **Refutation Trail in posted comments** — UPHELD findings include a collapsed `<details>` block showing the Investigator's hypothesis and the Critic's disprove attempt when either is non-empty. The disprove pattern is *auditable* because adopters can read why each surviving finding survived.
-- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/PostFailures` (GitHub post failed after retries) and `Shadow/SlackDeliveryFailures` (an escalation's optional Slack ping failed to deliver) from the act step. Dimensions: `Repository`, `Pipeline`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace.
+- **CloudWatch custom metrics** — `Shadow/CostPerPR`, `Shadow/CriticOverturnRate`, `Shadow/InputTokens`, `Shadow/OutputTokens`, `Shadow/Invocations`, `Shadow/Escalations` per analyze run, plus `Shadow/Throttled` on throttled runs, and `Shadow/PostFailures` / `Shadow/SlackDeliveryFailures` from the act step. Dimensions: `Repository`, `Pipeline`, and `Reason`. Set `SHADOW_CLOUDWATCH_DISABLED=true` to opt out. CloudFormation grants `cloudwatch:PutMetricData` scoped to the `Shadow` namespace.
 
 ---
 
@@ -421,6 +424,7 @@ The per-PR levers (under [What it costs](#what-it-costs)) bound a single review.
 - **Per-(repo, item) hourly rate limit** (`BOT_MAX_RUNS_PER_HOUR`, default `20`). Caps how many times a single PR or issue can trigger Shadow per rolling hour. Beyond the limit, the bot ESCALATES with a `<bot.name>:rate-limited` label instead of running the agent pipeline. Defends against an adversary closing/reopening or editing a PR title in a loop. Set to `0` to disable. **Issue/issue_comment events** require `run-name: "Shadow #${{ github.event.issue.number || ... }}"` in your caller workflow so the rate-limit gate can match prior runs (see [`examples/caller-workflow.yml`](examples/caller-workflow.yml)).
 - **Pre-flight diff/file caps** (`BOT_MAX_DIFF_FOR_REVIEW_CHARS`, `BOT_MAX_FILES_FOR_REVIEW`, defaults `100000` / `50`). A 50-file PR makes the Investigator read 5+ files, the Critic re-reads, the Reporter formats — costs multiply. Diff or file count above the cap → ESCALATE before any Bedrock call. Pre-flight escalation is ~$0; a runaway pipeline on a giant PR is $5+.
 - **AWS Budgets opt-in via CFN** (`MonthlyBudgetLimit` parameter on `shadow-iam-stack.yaml`). Set a positive USD amount + a `BudgetEmailAddress` and the stack creates an `AWS::Budgets::Budget` filtered to Amazon Bedrock spend, with email alerts at 80% and 100%. `0` skips Budget creation (default — AWS Budgets bills $0.02/budget/day, so opt-in only). Email-only today; auto-shutdown via `SHADOW_DISABLED` is a planned upgrade.
+- **Behavioral-anomaly alarms via CFN** (provisioned when `BudgetEmailAddress` is set). Two CloudWatch alarms aggregate the `Shadow` namespace fleet-wide via a Metrics Insights query (`SELECT SUM(...) FROM "Shadow"`): an **escalation spike** (`EscalationSpikeThreshold`, default `25`/hr — flags prompt-injection/abuse bursts or systemic failures) and an **invocation spike** (`InvocationSpikeThreshold`, default `100`/hr — flags public-trigger floods driving Bedrock spend, faster than the monthly Budget). Both notify the `AlarmTopicArn` SNS topic (`BudgetEmailAddress` auto-subscribed — confirm the email). Detection latency is up to 1 hour; tune thresholds to your fleet volume to avoid false pages. **Deploy the stack in the same region as your `aws_region`/`BedrockRegion`** — CloudWatch alarms are regional and see only metrics emitted in their own region. The alarms depend on Shadow's metric emission, so they are blind if `SHADOW_CLOUDWATCH_DISABLED=true` (they sit green, not red — absence of data isn't a breach). These are Metrics Insights query alarms, billed per alarm plus the metrics each query scans (which grows with your repo/reason cardinality) — small but not flat; see [CloudWatch pricing](https://aws.amazon.com/cloudwatch/pricing/).
 
 ---
 
@@ -462,7 +466,7 @@ If you previously tried to set `SHADOW_DISABLED` as a Secret rather than a Varia
 **Implemented** (shipped, covered by tests + CI):
 
 - BYO-AWS reusable workflow with two-job security split (`analyze` / `act`)
-- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip)
+- One-click CloudFormation Launch Stack for IAM, OIDC trust, AWS Budget, optional behavioral-anomaly CloudWatch alarms (escalation/invocation spikes), **and a default Bedrock Guardrail** with prompt-attack + PII filters (set `ProvisionGuardrail=false` to skip)
 - `shadow doctor` preflight CLI (verifies role, Bedrock access, prompts)
 - Audit trail in artifact: prompt-hash provenance, security-events histogram, SHA-256 integrity stamp bound to `(repo, run_id, pr_number)`
 - Refutation Trail rendered into posted comments (`<details>` block per finding)

infrastructure/shadow-iam-stack.yaml

@@ -15,19 +15,21 @@ Metadata:
         Parameters: [ShadowSourceRepo, ShadowWorkflowRef]
       - Label: { default: "Bedrock region" }
         Parameters: [BedrockRegion]
-      - Label: { default: "Cost protection (optional)" }
-        Parameters: [MonthlyBudgetLimit, BudgetEmailAddress]
+      - Label: { default: "Cost protection & anomaly alarms (optional)" }
+        Parameters: [MonthlyBudgetLimit, BudgetEmailAddress, EscalationSpikeThreshold, InvocationSpikeThreshold]
       - Label: { default: "Prompt-injection guardrail" }
         Parameters: [ProvisionGuardrail]
     ParameterLabels:
-      GitHubOrg:           { default: "Your GitHub org or user (e.g., awslabs, sudsali)" }
-      GitHubRepo:          { default: "Your repo name (use * for org-wide; audit first)" }
-      ShadowSourceRepo:    { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" }
-      ShadowWorkflowRef:   { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" }
-      BedrockRegion:       { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" }
-      MonthlyBudgetLimit:  { default: "Monthly Bedrock spend limit in USD (0 = disabled)" }
-      BudgetEmailAddress:  { default: "Email for budget threshold notifications" }
-      ProvisionGuardrail:  { default: "Provision a Bedrock Guardrail (recommended; default ON)" }
+      GitHubOrg:                { default: "Your GitHub org or user (e.g., awslabs, sudsali)" }
+      GitHubRepo:               { default: "Your repo name (use * for org-wide; audit first)" }
+      ShadowSourceRepo:         { default: "Shadow source repo (sudsali/shadow upstream, or your fork)" }
+      ShadowWorkflowRef:        { default: "Shadow workflow ref pin (use * for quick start; tag or SHA for prod)" }
+      BedrockRegion:            { default: "Bedrock region (us-east-1, us-west-2, us-east-2)" }
+      MonthlyBudgetLimit:       { default: "Monthly Bedrock spend limit in USD (0 = disabled)" }
+      BudgetEmailAddress:       { default: "Email for budget + anomaly-alarm notifications (also enables the alarms)" }
+      EscalationSpikeThreshold: { default: "Escalations/hour that trips the anomaly alarm (default 25)" }
+      InvocationSpikeThreshold: { default: "Invocations/hour that trips the anomaly alarm (default 100)" }
+      ProvisionGuardrail:       { default: "Provision a Bedrock Guardrail (recommended; default ON)" }
 
 Parameters:
   GitHubOrg:
@@ -102,10 +104,33 @@ Parameters:
   BudgetEmailAddress:
     Type: String
     Description: >-
-      Email to notify when monthly budget threshold is hit. Required only
-      if MonthlyBudgetLimit > 0.
+      Email to notify for budget threshold AND behavioral-anomaly alarms.
+      Supplying it provisions the anomaly alarms (see EscalationSpikeThreshold /
+      InvocationSpikeThreshold) regardless of MonthlyBudgetLimit; you must click
+      the SNS confirmation email for alerts to arrive. Leave blank to skip both.
     Default: ""
 
+  EscalationSpikeThreshold:
+    Type: Number
+    Description: >-
+      Fleet-wide Escalations/hour above which the anomaly alarm fires. Tune to
+      your traffic: escalations cover injection/abuse AND benign causes
+      (model/agent-stage failures, rate-limited retries, and every PR
+      when BOT_AGENT_PIPELINE=0), so set above your normal hourly escalation
+      volume to avoid paging on routine config.
+    Default: 25
+    MinValue: 1
+
+  InvocationSpikeThreshold:
+    Type: Number
+    Description: >-
+      Fleet-wide Invocations/hour above which the anomaly alarm fires. One
+      Invocation = one analyze run. BOT_MAX_RUNS_PER_HOUR caps per-(repo,item)
+      at 20, so a busy repo with several active PRs/issues can legitimately run
+      100+/hr — set above your expected fleet peak.
+    Default: 100
+    MinValue: 1
+
   ProvisionGuardrail:
     Type: String
     Description: >-
@@ -127,6 +152,9 @@ Conditions:
     - !Not [!Equals [!Ref MonthlyBudgetLimit, 0]]
     - !Not [!Equals [!Ref BudgetEmailAddress, ""]]
   CreateGuardrail: !Equals [!Ref ProvisionGuardrail, "true"]
+  # Behavioral alarms reuse BudgetEmailAddress as the notification target, so
+  # they light up whenever an operator supplies an email (no extra knob).
+  CreateAlarms: !Not [!Equals [!Ref BudgetEmailAddress, ""]]
 
 Resources:
   GitHubOidcProvider:
@@ -369,6 +397,69 @@ Resources:
             - SubscriptionType: EMAIL
               Address: !Ref BudgetEmailAddress
 
+  ShadowAlarmTopic:
+    # Fan-out point so operators can add more subscribers (PagerDuty, extra
+    # emails) beyond the auto-subscribed BudgetEmailAddress.
+    Type: AWS::SNS::Topic
+    Condition: CreateAlarms
+    Properties:
+      TopicName: !Sub "shadow-bot-alarms-${GitHubOrg}"
+      # No KmsMasterKeyId: CloudWatch can't publish to an aws/sns-encrypted
+      # topic (can't grant the service principal on the managed key), and the
+      # payload is non-sensitive (metric counts + org name).
+      Subscription:
+        - Protocol: email
+          Endpoint: !Ref BudgetEmailAddress
+
+  ShadowEscalationSpikeAlarm:
+    # Metrics Insights SQL (not a fixed-dimension metric or SEARCH — both are
+    # dead/invalid here) so SUM spans every Repository/Pipeline/Reason schema.
+    Type: AWS::CloudWatch::Alarm
+    Condition: CreateAlarms
+    Properties:
+      AlarmName: !Sub "shadow-bot-escalation-spike-${GitHubOrg}"
+      AlarmDescription: >-
+        Fleet-wide Escalations exceeded the threshold in a 1-hour window —
+        possible prompt-injection/abuse burst, or a systemic failure (e.g.
+        model/agent-stage failures escalating many items). Check recent PR/issue
+        activity and the analyze-job logs before assuming an attack.
+      EvaluationPeriods: 1
+      Threshold: !Ref EscalationSpikeThreshold
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      Metrics:
+        - Id: esc_total
+          Label: EscalationsPerHourFleetWide
+          Expression: 'SELECT SUM("Escalations") FROM "Shadow"'
+          Period: 3600
+          ReturnData: true
+      AlarmActions:
+        - !Ref ShadowAlarmTopic
+
+  ShadowInvocationSpikeAlarm:
+    # Hourly-burst cost signal (faster than the monthly Budget); same
+    # Metrics-Insights-SQL aggregation as ShadowEscalationSpikeAlarm.
+    Type: AWS::CloudWatch::Alarm
+    Condition: CreateAlarms
+    Properties:
+      AlarmName: !Sub "shadow-bot-invocation-spike-${GitHubOrg}"
+      AlarmDescription: >-
+        Fleet-wide Invocations exceeded the threshold in a 1-hour window —
+        possible flood of public trigger events (issues/PRs) driving unexpected
+        Bedrock spend. Cross-check with BOT_MAX_RUNS_PER_HOUR and the Budget.
+      EvaluationPeriods: 1
+      Threshold: !Ref InvocationSpikeThreshold
+      ComparisonOperator: GreaterThanThreshold
+      TreatMissingData: notBreaching
+      Metrics:
+        - Id: inv_total
+          Label: InvocationsPerHourFleetWide
+          Expression: 'SELECT SUM("Invocations") FROM "Shadow"'
+          Period: 3600
+          ReturnData: true
+      AlarmActions:
+        - !Ref ShadowAlarmTopic
+
 Outputs:
   ShadowRoleArn:
     Description: >-
@@ -406,6 +497,14 @@ Outputs:
       repository secret. Numbered version is published immediately.
     Value: !GetAtt ShadowGuardrailVersion.Version
 
+  AlarmTopicArn:
+    Condition: CreateAlarms
+    Description: >-
+      SNS topic ARN for the behavioral-anomaly alarms. BudgetEmailAddress is
+      auto-subscribed (confirm the emailed link, or alerts never arrive);
+      subscribe additional endpoints (PagerDuty, extra emails) to this ARN.
+    Value: !Ref ShadowAlarmTopic
+
   NextSteps:
     Description: What to do after this stack succeeds.
     Value: !If

requirements.txt

@@ -1,6 +1,5 @@
-# Version-pinned for reproducibility. Not hash-pinned yet: to defend against
-# PyPI compromise, regenerate with `pip-compile --generate-hashes` and call
-# `pip install --require-hashes` in the workflow.
+# Version-pinned, not hash-pinned: judged disproportionate for 3 deps. To add
+# hash-pinning against a PyPI compromise: pip-compile --generate-hashes.
 requests==2.33.1
 boto3==1.42.94
 PyYAML==6.0.2

src/scripts/shadow/cloudwatch.py

@@ -4,7 +4,8 @@
 account so they can graph Shadow as part of their existing observability.
 Best-effort: a CloudWatch failure NEVER blocks the bot from posting findings.
 
-Dimensions: Repository (GITHUB_REPOSITORY), Pipeline (agentic).
+Dimensions: Repository (GITHUB_REPOSITORY), Pipeline, and Reason (the
+action/skip cause) on every datum where a reason is set.
 Namespace: Shadow (the CFN Launch Stack policy is scoped to this exact
 namespace via a `cloudwatch:namespace` condition; setting
 SHADOW_CLOUDWATCH_NAMESPACE to anything else requires updating the IAM