Skip to content

feat(eks-ci): add the GitHub-OIDC CI role for the ksail EKS smoke test#4

Closed
devantler wants to merge 4 commits into
mainfrom
claude/eks-ci-role
Closed

feat(eks-ci): add the GitHub-OIDC CI role for the ksail EKS smoke test#4
devantler wants to merge 4 commits into
mainfrom
claude/eks-ci-role

Conversation

@devantler

@devantler devantler commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

🤖 Generated by the Daily AI Engineer

Why

The ksail EKS smoke test still cannot run: the AWS-side identity it must assume does not exist yet. Step 1 (the GitHub OIDC provider) now reconciles on prod after the platform egress fix, which surfaced the exact provider ARN this role's trust policy requires.

What

Adds the eks-ci IAM role: trust locked to the ksail repo's ci environment via GitHub OIDC (no long-lived keys), a 2-hour session cap, and an inline least-privilege policy scoped to the smoke test's st-eks-*/eksctl-st-eks-* resource naming wherever IAM supports it. Once it reconciles, its ARN goes to the ksail ci environment as AWS_OIDC_ROLE_ARN (the workflow's activation gate — merging this creates the role but activates nothing in CI).

Part of devantler-tech/platform#2326 (child 2, step 2 of 2).

Merge order: devantler-tech/platform#2612 (activates the Policy managed-resource kind) must merge first, or the new permissions-boundary manifest sits pending; ksail#6082 then wires the boundary into the generated eks.yaml before the workflow's first dispatch.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce9566858c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deploy/eks-ci/eks-ci-role.yaml Outdated
The EKS API's documented accessEntryArn uses the accessEntry/ segment while
the service authorization convention documents access-entry/; a non-matching
pattern grants nothing, so cover both to keep access-entry management working
for API-mode clusters.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7417e4810d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread deploy/eks-ci/eks-ci-role.yaml
Comment thread deploy/eks-ci/eks-ci-role.yaml
…rovider lifecycle perms

Codex P1s on #4: (1) eksctl with iam.withOIDC needs Create/Delete/Tag
OpenIDConnectProvider — granted scoped to oidc-provider/oidc.eks.* so the
GitHub Actions provider stays untouchable; (2) iam:CreateRole is now
conditioned on the eks-ci-smoke-boundary permissions boundary (new Policy MR)
so a workflow assuming eks-ci cannot mint an unbounded role and pass it to
EC2 to escalate beyond the smoke-test scope.
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Both P1s fixed at 8d57d5c (threads resolved with details); merge-order gates added to the body.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 8d57d5c5a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@devantler
devantler marked this pull request as ready for review July 13, 2026 06:40
@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

⛔ HOLD — do not merge this PR yet (merge-order gate)

This PR is CLEAN and promoted, so the merge policy would otherwise pick it up. It must not merge in its current shape.

Why. deploy/eks-ci/eks-ci-smoke-boundary.yaml defines the permissions boundary that caps every IAM role the eks-ci principal is allowed to create. But this repo is public, and the platform applies it via Flux as the namespace-scoped aws ServiceAccount — which platform#2612 granted create/update/delete on IAM Policy objects. So the boundary would live inside, and be mutable from, the very artifact it is supposed to constrain: anything that can change this repo could rewrite the boundary to allow iam:*/sts:AssumeRole*, and the cap would be worthless. The active Role kind can also attach managed policies via managedPolicyArns, so not activating RolePolicyAttachment does not contain it.

Raised by the codex review on platform#2612 — which auto-merged before it could be addressed.

Sequence to unblock:

  1. platform#2631 (draft) lands: the boundary Policy becomes platform-owned, and the tenant ServiceAccount loses IAM-policy write access entirely.
  2. This PR drops deploy/eks-ci/eks-ci-smoke-boundary.yaml and keeps eks-ci-role.yaml unchanged — the role already references the boundary by ARN (arn:aws:iam::939001610192:policy/eks-ci-smoke-boundary), so no capability is lost.
  3. Then this merges, and the rollout continues to the ARN read → ksail ci env wiring → first System-Test-EKS dispatch.

I will make edit (2) once #2631 merges. Flagging here so no run merges this in the meantime.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

⛔ HOLD still stands — and the scope grew

Update to my earlier hold: the review on platform#2631 showed that moving only the boundary policy to the platform fixed nothing.

The cap on what eks-ci may create is an iam:PermissionsBoundary condition inside eks-ci-role.yaml's own inlinePolicy — not a field on the Policy object. So whoever can edit the role can delete the condition and mint unbounded roles without ever touching the policy. Since this repo is public and Flux applies it as the aws ServiceAccount, that meant the artifact could still remove its own cap.

So both files move to the platform, not just one. platform#2631 now owns:

  • policy-eks-ci-smoke-boundary.yamlk8s/providers/hetzner/apps/aws/policy-eks-ci-smoke-boundary.yaml
  • eks-ci-role.yamlk8s/providers/hetzner/apps/aws/role-eks-ci.yaml

and the tenant ServiceAccount now grants only openidconnectproviders — no write on iam roles or policies at all.

What this PR becomes: deploy/eks-ci/github-oidc-provider.yaml (already on main) stays here — the OIDC provider is what this artifact legitimately owns. This PR should drop both eks-ci-role.yaml and eks-ci-smoke-boundary.yaml, which leaves it with no remaining content, so it will most likely be closed rather than edited.

I will do that once #2631 merges. Until then: do not merge this.

@devantler

Copy link
Copy Markdown
Contributor Author

🤖 Generated by the Daily AI Engineer

Closing — superseded by devantler-tech/platform#2631 (merged)

This PR added the eks-ci IAM Role and its eks-ci-smoke-boundary permissions-boundary Policy to the public devantler-tech/aws artifact, owned by the aws tenant ServiceAccount. That ownership turned out to be the problem, not the plan.

The boundary is what caps the roles eks-ci can mint — but the cap is an iam:PermissionsBoundary condition inside the eks-ci Role's own inline policy, so whoever can edit the Role can delete the condition. Since this artifact is applied by the aws tenant ServiceAccount, keeping the Role here meant the bounded party could edit its own boundary — a boundary the bounded party can edit caps nothing.

platform#2631 (merged 15:06Z) resolves that by making both objects platform-owned:

  • k8s/providers/hetzner/apps/aws/role-eks-ci.yaml
  • k8s/providers/hetzner/apps/aws/policy-eks-ci-smoke-boundary.yaml

and narrowing the tenant ServiceAccount's RBAC to openidconnectproviders only.

So this PR is not merely redundant — merging it now would break the tenant reconcile, because the aws ServiceAccount no longer holds create/update/patch/delete on roles or policies and would be denied applying these two objects.

Net effect

devantler-tech/aws keeps exactly what it should: deploy/eks-ci/github-oidc-provider.yaml (already on main, reconciled and Ready=True). The privileged IAM objects live on the platform side, where the tenant cannot rewrite them.

Next in the chain (platform#2326 child 2)

The Role ARN is now produced by the platform, so the remaining steps are unchanged: read the Role ARN → set AWS_OIDC_ROLE_ARN in ksail's ci environment → dispatch System-Test-EKS for the first end-to-end run. ksail#6082 (wiring the boundary fields into the generated eks.yaml) still lands before that first dispatch, since iam:CreateRole is conditioned on the boundary.

Closing as superseded. No content is lost — both files are on main in the platform repo.

@devantler devantler closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant