feat: PeEKS BackupPolicy RGD + peeks-gp3 StorageClass (rebased on PR #642)#644
Conversation
When identityCenter.adminGroupId AND identityCenter.adminUserId are both pre-populated in config, skip the entire group/user/membership creation logic in idc:setup. This enables bootstrap from a role that lacks identitystore:Create* permissions on the target IdC — typical setup when AWS IAM Identity Center lives in the AWS Organization parent account and the workload account assumes a least-privilege role. Existing behavior is preserved when both fields are empty: the task still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user user, and wires the membership. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…cluster readiness The previous pipeline ran hub:ingress immediately after hub:claim, but hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that resolve at task entry. Since ACK provisions VPC + EKS asynchronously (15-25 min), hub:ingress consistently failed with ResourceNotFoundException. This change inserts a hub:wait-for-eks step that polls: 1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min) 2. AWS EKS cluster.status until ACTIVE (5 min cap) Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC (default 30). Idempotent — re-runs poll the in-flight RGD. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…om EC2 When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio runner, dev VM), the previous credentials:refresh logic queried IMDS first and silently used the host's instance role — overriding any aws.profile explicitly set in config.local.yaml. ACK controllers then provisioned the hub VPC/EKS in the EC2 host's account instead of the user's intended target account, failing with UnauthorizedOperation on ec2:CreateVpc. New precedence (highest to lowest): 1. aws.profile from config (if set and not 'default') 2. EC2 IMDS instance role 3. AWS default profile The wrong-account failure mode is silent (controllers just keep retrying on the wrong account), so make the chosen credential source observable by logging which path was taken. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…stanceSynced) The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced condition that KRO 0.9.2 never emits on instance objects. KRO publishes InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the canonical 'all good' signal. This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch) Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1) as a regular addon in gitops/addons/registry/platform.yaml. Why an addon and not the EKS ACK capability? The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT yet bundled in the managed 'ACK' capability. Until AWS ships it as part of the capability, it is deployed via Helm. How does it get its IAM? Through the new enableACKKRO=true path on appset-chart (previous commit). That path renders gitops/addons/charts/kro/instances/pod-identity, which expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy + eks.PodIdentityAssociation for SA ack-backup-controller in ack-system. Policy follows least-privilege: - backup:* + backup-storage:* on all resources - iam:PassRole scoped to role/* (controller passes backup service roles) - kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults Enabled only on the hub cluster via 'enable_ack_backup: true' in hub-config.yaml; spokes keep it false. Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan, BackupSelection). Adapted from upstream PR aws-samples#631.
Port from PR aws-samples#633 (feat/peeks-backup-policy-rgd) to the new structure introduced by PR aws-samples#642 (feature/platform-cluster-kro-ack). The RGD provisions BackupVault + BackupPlan + BackupSelection per tier via ACK backup-controller. Tier semantics live on the BackupPolicy CR, not on the StorageClass — backup selection happens at cluster/namespace level via tag matching (peeks.io/backup-tier + peeks.io/spoke). - gold: RPO 1h, retention 30d, cold 90d, cross-region copy - silver: RPO 24h, retention 14d, cross-region copy - bronze: RPO 24h, retention 7d, same-region only
Port the storageclass-resources chart from PR aws-samples#633 into the new platform-manifests-bootstrap chart from PR aws-samples#642 — single bootstrap chart for early cluster resources (NodePools, ClusterSecretStore, StorageClasses). - New template peeks-storageclass.yaml renders ebs/efs SCs from values - New values.yaml with defaults (gpu/customNodepools/storageClasses) - peeks-gp3: single PeEKS default SC, tier semantics live on BackupPolicy - Volume tags propagated via tagSpecification_N parameters for AWS Backup selection (peeks.io/spoke, peeks.io/managed-by) - README-storageclasses.md preserves design notes from the standalone chart
The kind-kro-ack provider was using `{{.ROOT_DIR}}/../../gitops` to
locate the gitops/ tree. However, when this Taskfile is included from
the root Taskfile via `includes: kind-kro-ack: {dir: ./cluster-providers/kind-kro-ack}`,
`{{.ROOT_DIR}}` resolves to the root project directory (where the root
Taskfile lives), not to the included Taskfile's directory.
This caused `task validate` and `task install` to fail with:
Error: open /opt/.../<repo>/../../gitops/addons/registry/core.yaml: no such file
Aligned with the kind-crossplane provider which correctly uses
`{{.ROOT_DIR}}/gitops`.
Validated: `task validate CONFIG_FILE=$PWD/config.local.yaml` now
passes pre-flight checks.
When identityCenter.adminGroupId AND identityCenter.adminUserId are both pre-populated in config, skip the entire group/user/membership creation logic in idc:setup. This enables bootstrap from a role that lacks identitystore:Create* permissions on the target IdC — typical setup when AWS IAM Identity Center lives in the AWS Organization parent account and the workload account assumes a least-privilege role. Existing behavior is preserved when both fields are empty: the task still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user user, and wires the membership. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…cluster readiness The previous pipeline ran hub:ingress immediately after hub:claim, but hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that resolve at task entry. Since ACK provisions VPC + EKS asynchronously (15-25 min), hub:ingress consistently failed with ResourceNotFoundException. This change inserts a hub:wait-for-eks step that polls: 1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min) 2. AWS EKS cluster.status until ACTIVE (5 min cap) Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC (default 30). Idempotent — re-runs poll the in-flight RGD. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…om EC2 When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio runner, dev VM), the previous credentials:refresh logic queried IMDS first and silently used the host's instance role — overriding any aws.profile explicitly set in config.local.yaml. ACK controllers then provisioned the hub VPC/EKS in the EC2 host's account instead of the user's intended target account, failing with UnauthorizedOperation on ec2:CreateVpc. New precedence (highest to lowest): 1. aws.profile from config (if set and not 'default') 2. EC2 IMDS instance role 3. AWS default profile The wrong-account failure mode is silent (controllers just keep retrying on the wrong account), so make the chosen credential source observable by logging which path was taken. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…stanceSynced) The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced condition that KRO 0.9.2 never emits on instance objects. KRO publishes InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the canonical 'all good' signal. This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync. Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch) Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go
…K capability role Add inline policy 'ManagePodIdentityAssociations' to the ACK capability role so that ACK controllers managed by the EKS ACK capability can create EKS Pod Identity associations and pass scoped workload roles. Required for any new ACK controller (e.g. ack-backup) wired through kro pod-identity instances. - iam:PassRole scoped to role/<name>-* with iam:PassedToService = pods.eks.amazonaws.com - eks:*PodIdentityAssociation* + tagging actions scoped to cluster/<name> and podidentityassociation/<name>/* - accountId and name come from schema.spec for cross-account / multi-cluster reuse
Wire a new 'enableACKKRO' flag on per-addon chart configs in the appset-chart. When set, the addon ApplicationSet gets a second source pointing to gitops/addons/charts/kro/instances/pod-identity, which materializes ACK iam.Role + iam.Policy + eks.PodIdentityAssociation via the podidentity.kro.run RGD instead of the legacy Crossplane path. - 'enableACKKRO' and 'enableAckPodIdentity' are mutually exclusive; the template fails fast if both are set on the same addon. - All existing addons keep their behavior (no flag flip). - Required for ACK controllers not yet bundled in the EKS ACK capability (e.g. ack-backup, currently in preview at v0.1.1). Adapted from upstream PR aws-samples#631 to fit the appset-chart structure on the feature/platform-cluster-kro-ack base (PR aws-samples#642). Credit to the upstream author.
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1) as a regular addon in gitops/addons/registry/platform.yaml. Why an addon and not the EKS ACK capability? The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT yet bundled in the managed 'ACK' capability. Until AWS ships it as part of the capability, it is deployed via Helm. How does it get its IAM? Through the new enableACKKRO=true path on appset-chart (previous commit). That path renders gitops/addons/charts/kro/instances/pod-identity, which expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy + eks.PodIdentityAssociation for SA ack-backup-controller in ack-system. Policy follows least-privilege: - backup:* + backup-storage:* on all resources - iam:PassRole scoped to role/* (controller passes backup service roles) - kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults Enabled only on the hub cluster via 'enable_ack_backup: true' in hub-config.yaml; spokes keep it false. Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan, BackupSelection). Adapted from upstream PR aws-samples#631.
The 'kro-manifests-hub' ApplicationSet recursively applies every YAML under gitops/addons/charts/kro/resource-groups/manifests/. Three sample BackupPolicy instances under .../backup-policy/examples/ were therefore applied by Argo CD on the hub before the BackupPolicy CRD existed, which broke the whole sync run with: resource mapping not found for kind "BackupPolicy" in version "kro.run/v1alpha1" That fail-fast aborted the apply of the *RGDs themselves* (backup-policy, pod-identity, eks, etc.), so no RGD ever became Active on the hub — chicken-and-egg. Move the three sample instances to gitops/addons/charts/kro/resource-groups/examples/backup-policy/, which sits outside the apply path. The RGD ships alone in manifests/, like the other RGDs (eks, pod-identity, ray-service, huggingface). Add a short README in the new directory documenting the manual-apply workflow once the RGD is Active and ACK Backup CRDs exist.
The eks-capabilities-kro ClusterRoleBinding only bound the synthetic User "capabilities.eks.amazonaws.com" which EKS uses for some internal calls. When the KRO capability assumes its IAM role to act on K8s resources (e.g. while reconciling the podidentity RGD which declares iam.services.k8s.aws/Policy and Role children), the API server sees the STS username 'arn:aws:sts::<acct>:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO' instead. Without this second subject, KRO fails with: policies.iam.services.k8s.aws is forbidden: User "arn:aws:sts::...:assumed-role/peeks-hub-kro-capability-role/KRO" cannot get resource "policies" in API group "iam.services.k8s.aws" This was not exposed before because the podidentity RGD had never been instantiated (the kro+ACK pod-identity path enableACKKRO=true is new in this branch, used by ack-backup-controller). RBAC subjects do not support wildcards, so the templated literal ARN is rendered from chart values (accountId/resourcePrefix/clusterName), which are now plumbed via the multi-acct entry in addons/registry/platform.yaml.
The KRO ResourceGraphDefinition 'backuppolicy.kro.run' instantiates BackupVault / BackupPlan / BackupSelection (apiGroup backup.services.k8s.aws). Without this rule, KRO reconciles fail with: backupvaults.backup.services.k8s.aws is forbidden: User "...:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO" cannot get resource "backupvaults" Mirrors the existing rules for ec2/eks/iam/ecr ACK groups.
35f0a43 to
4d8b31e
Compare
Harmonise ALB/SG/CloudFront naming on RESOURCE_PREFIX so multi-env deployments don't collide. Touches kind-kro-ack only; kind-crossplane unchanged. Sets up dual ingress mode work (tls/cloudfront).
New config.ingress section drives hub ALB exposure: - mode=cloudfront (default): ALB internal + CloudFront VPC Origin - mode=tls: ALB internet-facing HTTPS:443, requires certificateArn Preflight task validates mode + cert + domain before hub:ingress runs. Default cloudfront preserves current behaviour; Task 2 will switch the ALB scheme to internal in cloudfront mode.
- mode=cloudfront: scheme=internal, *private* subnets (internal-elb tag), SG ingress 80 restricted to VPC CIDR - mode=tls: unchanged (scheme=internet-facing, *public* subnets, 80+443 open) - Subnet selection prefers role tags with name-based fallback Mode switch after creation requires manual teardown (AWS ALB scheme is immutable). Default cloudfront only reachable end-to-end after Task 3 (CloudFront VPC Origin).
- VPC Origin (post-2024 CF feature) reaches internal ALB without
public exposure, replacing CustomOriginConfig http-only
- list-vpc-origins lookup makes the task idempotent
- Polls VPC Origin Status until Deployed (up to 15 min) before
creating the distribution
- Early-exit when ingress.mode != cloudfront (status precondition
+ cmd guard) — pure no-op for tls mode
- Distribution config preserves CachePolicyId/OriginRequestPolicyId,
PriceClass_100, http2, redirect-to-https
- cloudfront.domain still written back to {{.CONFIG_FILE}}
hub:ingress was creating an ALB without any listener — VPC Origin and CloudFront would then have nothing to forward to (502). Add a default listener after ALB creation: - mode=cloudfront: HTTP:80 fixed-response 404 - mode=tls: HTTPS:443 fixed-response 404 + HTTP:80 → 443 redirect AWS LBC will replace these defaults with rules pointing at K8s target groups when Ingresses are reconciled.
CIDR-only ingress (VPC CIDR :80) is insufficient for CloudFront VPC Origin traffic — CF routes through a dedicated AWS-managed SG 'CloudFront-VPCOrigins-Service-SG' that gets created when the first VPC Origin lands in the VPC. Without an explicit SG-to-SG rule the ALB returns 504 to all CloudFront requests. Lookup the service SG by name and authorize it. Print a warning and proceed if not yet present (chicken-and-egg on first run — re-run hub:ingress after hub:cloudfront has created the VPC Origin).
Hub ingress dual-mode (cloudfront/tls) — implementation + E2E validationFollowing the plan in Commits added to this PR
E2E validation
|
hub:ingress runs before any VPC Origin exists, so the AWS-managed 'CloudFront-VPCOrigins-Service-SG' is not yet present. The previous fix in hub:ingress prints a warning and continues — but then the user must remember to re-run hub:ingress manually after hub:cloudfront. Move the SG-to-SG authorization into hub:cloudfront, immediately after the VPC Origin reaches Deployed (which is when AWS guarantees the service SG exists). One hub:ingress + hub:cloudfront pass is now sufficient end-to-end. The hub:ingress best-effort attempt is kept as belt-and-braces for re-runs.
cainjector OOMKilled (exit 137) on cluster bootstrap when caching all CABundle injectables across CRDs (cert-manager + crossplane + ack-* providers + kro). 128Mi is too tight for a hub cluster with many CRDs; 256Mi is the upstream default and resolves the loop.
The BackupPolicy RGD's BackupSelection matches on 'aws:ResourceTag/peeks.io/backup-tier' but no StorageClass currently emits this tag, so volumes never enter a BackupPlan. Add three tier-labeled StorageClasses — peeks-gold-gp3, peeks-silver-gp3, peeks-bronze-gp3 — with backup-tier tags propagated to EBS volumes via the existing 'tags:' helm value. Workloads opt into a tier by picking the matching SC in their PVC manifest, so backup tier is workload-driven (no central ConfigMap, no platform ticket). The legacy ebs-sc-gp3 and the default peeks-gp3 are preserved unchanged. Bronze keeps reclaimPolicy=Delete (cheap/ephemeral); gold and silver keep Retain. Tier policies (gold/silver/bronze RPO + retention + cross-region copy) are defined in the BackupPolicy RGD — this commit only wires the emitter side. Ported from feat/peeks-backup-policy-rgd commit 6862a00 onto the rebased platform-manifests-bootstrap chart layout.
…stanceSynced) (#662) The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced condition that KRO 0.9.2 never emits on instance objects. KRO publishes InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the canonical 'all good' signal. This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync. Refs: PR #642 (parent), PR #644 (this branch) Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go Co-authored-by: Sébastien Allamand <allamand@users.noreply.github.com>
…dC, EKS gate, profile-first creds) (#664) * fix(kind-kro-ack): GITOPS_ROOT path resolution The kind-kro-ack provider was using `{{.ROOT_DIR}}/../../gitops` to locate the gitops/ tree. However, when this Taskfile is included from the root Taskfile via `includes: kind-kro-ack: {dir: ./cluster-providers/kind-kro-ack}`, `{{.ROOT_DIR}}` resolves to the root project directory (where the root Taskfile lives), not to the included Taskfile's directory. This caused `task validate` and `task install` to fail with: Error: open /opt/.../<repo>/../../gitops/addons/registry/core.yaml: no such file Aligned with the kind-crossplane provider which correctly uses `{{.ROOT_DIR}}/gitops`. Validated: `task validate CONFIG_FILE=$PWD/config.local.yaml` now passes pre-flight checks. * feat(kind-kro-ack): support BYO IdC (cross-account adminGroup/adminUser) When identityCenter.adminGroupId AND identityCenter.adminUserId are both pre-populated in config, skip the entire group/user/membership creation logic in idc:setup. This enables bootstrap from a role that lacks identitystore:Create* permissions on the target IdC — typical setup when AWS IAM Identity Center lives in the AWS Organization parent account and the workload account assumes a least-privilege role. Existing behavior is preserved when both fields are empty: the task still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user user, and wires the membership. Refs: PR #642 (parent), PR #644 (this branch) * feat(kind-kro-ack): add hub:wait-for-eks step to gate hub:ingress on cluster readiness The previous pipeline ran hub:ingress immediately after hub:claim, but hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that resolve at task entry. Since ACK provisions VPC + EKS asynchronously (15-25 min), hub:ingress consistently failed with ResourceNotFoundException. This change inserts a hub:wait-for-eks step that polls: 1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min) 2. AWS EKS cluster.status until ACTIVE (5 min cap) Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC (default 30). Idempotent — re-runs poll the in-flight RGD. Refs: PR #642 (parent), PR #644 (this branch) * fix(kind-kro-ack): prefer aws.profile over IMDS when bootstrapping from EC2 When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio runner, dev VM), the previous credentials:refresh logic queried IMDS first and silently used the host's instance role — overriding any aws.profile explicitly set in config.local.yaml. ACK controllers then provisioned the hub VPC/EKS in the EC2 host's account instead of the user's intended target account, failing with UnauthorizedOperation on ec2:CreateVpc. New precedence (highest to lowest): 1. aws.profile from config (if set and not 'default') 2. EC2 IMDS instance role 3. AWS default profile The wrong-account failure mode is silent (controllers just keep retrying on the wrong account), so make the chosen credential source observable by logging which path was taken. Refs: PR #642 (parent), PR #644 (this branch) --------- Co-authored-by: Sébastien Allamand <allamand@users.noreply.github.com>
shapirov103
left a comment
There was a problem hiding this comment.
Merge conflicts.
I see the feature and I think it is valuable for the solution. Is this something you plan to highlight in the workshop content as well? What is the GTM for this feature?
Context
This PR ports the changes originally proposed in #633 onto the new structure introduced by #642 (
feature/platform-cluster-kro-ack).PR #633 was opened against
mainbefore #642 reorganized the platform charts. A mechanical rebase was not viable (thegitops/addons/charts/storageclass-resourcesandgitops/addons/charts/kro-rgds/manifests/backup-policy/paths from #633 either disappeared or moved). This PR is the manual port. #633 will be closed once this one is reviewed.Closes #643 (functional requirements for AWS Backup for EKS DR test).
Changes
1.
BackupPolicyRGD (kro)Path:
gitops/addons/charts/kro/resource-groups/manifests/backup-policy/A KRO
ResourceGraphDefinitionthat materializesBackupVault+BackupPlan+BackupSelection(all via ACKbackup.services.k8s.aws/v1alpha1) for three tiers:Tier semantics live on the
BackupPolicyCR, not on the StorageClass. Backup selection happens at cluster/namespace level via tag matching (peeks.io/backup-tier+peeks.io/spoke). Includes 3 example instances (examples/instance-{gold,silver,bronze}.yaml) and a README.2.
peeks-gp3StorageClass merged intoplatform-manifests-bootstrapPath:
gitops/addons/charts/platform-manifests-bootstrap/Rather than reintroducing the standalone
storageclass-resourceschart that #642 removed, the StorageClass template is merged intoplatform-manifests-bootstrap— aligned with #642's design of a single bootstrap chart for early cluster resources (NodePools, ClusterSecretStore, StorageClasses).peeks-storageclass.yamlrenders EBS/EFS SCs from valuesvalues.yamlwith defaults (gpu,customNodepools,storageClasses)peeks-gp3: single PeEKS default SC (no more gold/silver/bronze SC variants — collapsed to one)tagSpecification_Nparameters for AWS Backup selectionREADME-storageclasses.mdpreserves the design notes from the original standalone chartValidation
helm template platform-manifests-bootstrap --show-only templates/peeks-storageclass.yamlrenders bothebs-sc-gp3(legacy) andpeeks-gp3(PeEKS default) cleanly with tag propagationrg-s3-bucket.yamlneighbor)iamRoleARNas a parameterNext steps (out of scope for this PR)
BackupPolicyinstance on the hub, verify BackupVault/Plan/Selection are reconciled by ACKBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.