[AWS] Story 5: Observability (CloudWatch)

[mfittko]

Problem Statement

Production deployment needs usable logs, metrics, dashboards, and alarms so the
service can be operated and failures can be detected quickly.

Scope

In scope:

• CloudWatch log groups
• Dashboards for key metrics
• Alarms and SNS notifications
• Structured log expectations

Out of scope:

• Replacing application logging format wholesale
• External observability platforms
• Non-AWS incident tooling

Technical Approach

• Create service log groups and retention policy.
• Build a dashboard from ALB, ECS, Aurora, and Redis metrics.
• Define alarms for error rate, latency, host health, and CPU.
• Validate one trigger and recovery path per alarm class.

Dependencies

Hard dependencies:

• CDK foundation and config contract
• Compute layer
• Networking layer
  Blocks:
• Production readiness gate

Acceptance Criteria

• Logs flow to CloudWatch with structured JSON
• Dashboard shows all key metrics
• Alarms trigger on error conditions
• SNS notifications work (if configured)
• Log retention policy applied

Proposed Definition Of Done

• Dashboard screenshots or exports are attached.
• Alarm trigger and recovery evidence is attached.
• Sample structured log validation is attached.
• Notification path evidence is attached when SNS is configured.

Validation Plan

1. Generate controlled traffic and confirm logs reach CloudWatch.
2. Confirm dashboard widgets show non-empty data.
3. Trigger each alarm class in a test-safe way.
4. Confirm SNS notifications are delivered when configured.
5. Confirm alarms recover to OK after remediation.

Risks And Mitigations

• Risk: Dashboards exist but remain empty or stale.
  • Mitigation: Require controlled-traffic evidence.
• Risk: Alarm definitions exist but are never tested.
  • Mitigation: Require one trigger-and-recover cycle.

Handoff Notes

The readiness gate should reuse the dashboards and alarms defined here.

AC/DoD Coverage Matrix

Item	Type (AC/DoD/Non-goal)	Status (Met/Partial/Unmet/Unverified)	Evidence (spec/tests/behavior)	Notes
Logs flow to CloudWatch with structured JSON	AC	Unverified	Log samples	Source AC
Dashboard shows all key metrics	AC	Unverified	Dashboard export	Source AC
Alarms trigger on error conditions	AC	Unverified	Alarm test evidence	Source AC
SNS notifications work (if configured)	AC	Unverified	Notification evidence	Source AC
Log retention policy applied	AC	Unverified	Log group config	Source AC
Dashboard screenshots or exports are attached.	DoD	Unverified	Issue evidence	Proposed DoD
Alarm trigger and recovery evidence is attached.	DoD	Unverified	Issue evidence	Proposed DoD
Sample structured log validation is attached.	DoD	Unverified	Issue evidence	Proposed DoD
Notification path evidence is attached when SNS is configured.	DoD	Unverified	Issue evidence	Proposed DoD

[mfittko]

Refinement update applied. This issue now has explicit scope, hard dependencies, measurable acceptance criteria, a proposed Definition of Done, a validation plan, and an AC/DoD coverage matrix so it is ready for assignment.