Skip to content

Isolate net-new repo bootstrap CRs so a pending manual bootstrap can't freeze all app delivery #2626

Description

@devantler

🤖 Generated by the Daily AI Engineer
Problem (observed live 2026-07-13). A single net-new repository that cannot self-create froze all app delivery. The github-config Kustomization (wait: true) manages the Crossplane Repository/TeamRepository/DefaultBranch CRs for devantler-tech/kyverno-policies (declared in .github#94). That repo can't be created declaratively — the provider's create POST /orgs/.../repos omits the org's 5 required custom properties (no CRD field), returning 422, so the CR sits InProgress awaiting a one-time org-admin bootstrap. Because github-config uses wait: true, it never reaches Ready; and because the parent apps Kustomization also uses wait: true on its children, apps never reaches Ready either → every deploy-carrying merge_group run fails its apps health check, and renovate/own PRs get evicted from the merge queue. This stayed hidden while the cluster was frozen on an unrelated pull-secret issue (host-audit #8c); the moment that cleared, this became the sole delivery-freeze cause.

Why it matters. Org-config housekeeping (a not-yet-bootstrapped shared repo) should never be able to freeze application delivery. The two concerns are coupled far more tightly than they need to be, turning a benign "pending manual bootstrap" into a portfolio-wide delivery outage.

Proposed direction. Decouple bootstrap-pending, manually-gated repo CRs from the readiness that gates app delivery. Options to evaluate:

  • Place net-new repo CRs whose creation requires a manual org-admin bootstrap in a separate Kustomization with wait: false (or exclude them from health checks) until they are adopted, so a pending bootstrap no longer blocks github-config/apps readiness.
  • Alternatively, relax the appsgithub-config readiness coupling so org-config health does not gate app delivery.
    Whichever path, add a guard/test so a future un-bootstrappable repo CR degrades gracefully (surfaced, not delivery-freezing).

Size. M. Part of the resilient-GitOps theme (#2043); relates to the repo-creation bootstrap pattern (#2325) and .github#94.

Note. The immediate freeze self-heals once an org admin bootstraps kyverno-policies (one-click already on .github#94); this issue is the durable fix so the class can't recur.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions