Skip to content

cleanup script: scope cleanup blocks to FORMAE_TEST_RUN_ID instead of just prefix-matching #96

Description

@JeroenSoeters

Background

The scripts/ci/clean-environment.sh cleanup script identifies test-owned resources by positive prefix-match on formae-plugin-sdk-test-* (and a couple of other prefixes for legacy fixtures). Every cleanup block uses this pattern. PR #95 added seven new blocks (CloudFront Functions, KVS, three policy types, OAC, ACM Certificate) following the same convention.

Problem

This works today because:

  • The aws-conformance-tests concurrency group in .github/workflows/ci.yml and .github/workflows/nightly.yml ensures only one CI run is in-flight at a time
  • Developers are unlikely to manually create resources with that prefix outside conformance

But the convention isn't load-bearing — a parallel CI job in another repo, a developer experimenting locally with a matching prefix, or a future migration to non-serialized CI would all let the cleanup script delete resources it doesn't own. Codex flagged this in the 2026-06-03 adversarial review of the CloudFront/ACM design spec.

Proposed fix

Every cleanup block should require BOTH the test prefix AND a match against the active FORMAE_TEST_RUN_ID (the env var the conformance harness injects into every fixture).

Approach options:

  1. Tag everything with FormaeTestRunID = $RUN_ID at create time, and filter on that tag in cleanup. Most robust but requires tagging support on every resource type (e.g., KVS doesn't carry tags via formae today).
  2. Encode the run-id into the resource name suffix (already true for most fixtures — e.g., formae-plugin-sdk-test-secret-abc123 where abc123 is the run-id). Cleanup grep can then check endswith against the run-id from CI.
  3. Hybrid: tag where possible, name-suffix elsewhere.

Probably option 2 is the cheapest: extend every cleanup block's JMESPath filter to also require the run-id substring. CI exposes FORMAE_TEST_RUN_ID as an env var; the cleanup script can either consume it directly or accept it as an argument.

Scope

This is a script-wide refactor — touches every cleanup block, not just the new ones from PR #95. Should be a single coherent PR rather than incremental per-block changes. Consider it a follow-up to PR #95 rather than blocking that merge.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions