Skip to content

Serialize EKS lifecycle state across CLI processes #6204

Description

@devantler

🤖 Generated by the Daily AI Engineer

Evidence

The capacity snapshot added by #6201 preserves EKS nodegroup sizes for a normal stop-to-start round trip, and the local API now rejects overlapping operations within one Service instance. Separate CLI processes can still interleave the same cluster lifecycle:

  1. Stop reads a positive nodegroup and saves its desired, minimum, and maximum capacity.
  2. A concurrent Start reads the still-positive nodegroup, treats it as already restored, verifies it, and removes the snapshot.
  3. Stop then scales the nodegroup to zero.

The cluster is left stopped with no restore snapshot. The same class of race can occur between lifecycle and deletion from different processes because the state files are atomic but the full read, persist, cloud mutation, verify, and cleanup transition is not locked.

No existing open issue matched a search for EKS lifecycle concurrency or snapshot locking. Related to #6087, #6202, and #6203.

Affected audience and impact

Users or automation invoking EKS start, stop, or delete concurrently from separate terminals or processes can lose the exact restore capacity or race state cleanup. A later start then fails closed, leaving the cluster unavailable until capacity is repaired manually.

Expected behavior

Only one destructive or capacity-changing EKS lifecycle transition may own a canonical cluster target at a time, including across KSail processes. A competing operation must wait with context cancellation or fail explicitly before reading or mutating snapshot or cloud state.

Acceptance criteria

  • Add an OS-backed cross-process lock keyed by canonical EKS target identity, coordinated with the immutable identity work in Bind EKS lifecycle ownership to immutable AWS identity #6202.
  • Hold the lock across region and ownership resolution, nodegroup read, snapshot load/save, scaling, verification, snapshot cleanup, and cluster-state deletion where applicable.
  • Store lock metadata outside the per-cluster state directory so successful deletion cannot remove a lock that is still held.
  • Context cancellation or timeout exits without cloud or state mutation; process exit releases the lock without stale-file recovery hazards.
  • Tests deterministically reproduce concurrent stop/start and stop/delete interleavings and prove capacity state cannot be deleted or resurrected out of order.
  • Preserve the local API immediate-conflict behavior added in fix(eks): route standalone lifecycle commands #6201.

Rough size

Medium: shared lifecycle locking plus deterministic multi-process or lock-seam tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Fields

    No fields configured for Bug.

    Projects

    Status
    📥 Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions