Skip to content

feat: PeEKS BackupPolicy RGD + peeks-gp3 StorageClass (rebased on PR #642)#644

Draft
allamand wants to merge 22 commits into
aws-samples:feature/platform-cluster-kro-ackfrom
allamand:feat/peeks-backup-policy-rgd-on-pr642
Draft

feat: PeEKS BackupPolicy RGD + peeks-gp3 StorageClass (rebased on PR #642)#644
allamand wants to merge 22 commits into
aws-samples:feature/platform-cluster-kro-ackfrom
allamand:feat/peeks-backup-policy-rgd-on-pr642

Conversation

@allamand

@allamand allamand commented May 14, 2026

Copy link
Copy Markdown
Contributor

Context

This PR ports the changes originally proposed in #633 onto the new structure introduced by #642 (feature/platform-cluster-kro-ack).

PR #633 was opened against main before #642 reorganized the platform charts. A mechanical rebase was not viable (the gitops/addons/charts/storageclass-resources and gitops/addons/charts/kro-rgds/manifests/backup-policy/ paths from #633 either disappeared or moved). This PR is the manual port. #633 will be closed once this one is reviewed.

Closes #643 (functional requirements for AWS Backup for EKS DR test).

Changes

1. BackupPolicy RGD (kro)

Path: gitops/addons/charts/kro/resource-groups/manifests/backup-policy/

A KRO ResourceGraphDefinition that materializes BackupVault + BackupPlan + BackupSelection (all via ACK backup.services.k8s.aws/v1alpha1) for three tiers:

Tier RPO Retention Cold storage Cross-region copy
gold 1h 30d 90d yes
silver 24h 14d yes
bronze 24h 7d no

Tier semantics live on the BackupPolicy CR, not on the StorageClass. Backup selection happens at cluster/namespace level via tag matching (peeks.io/backup-tier + peeks.io/spoke). Includes 3 example instances (examples/instance-{gold,silver,bronze}.yaml) and a README.

2. peeks-gp3 StorageClass merged into platform-manifests-bootstrap

Path: gitops/addons/charts/platform-manifests-bootstrap/

Rather than reintroducing the standalone storageclass-resources chart that #642 removed, the StorageClass template is merged into platform-manifests-bootstrap — aligned with #642's design of a single bootstrap chart for early cluster resources (NodePools, ClusterSecretStore, StorageClasses).

  • New template peeks-storageclass.yaml renders EBS/EFS SCs from values
  • New values.yaml with defaults (gpu, customNodepools, storageClasses)
  • peeks-gp3: single PeEKS default SC (no more gold/silver/bronze SC variants — collapsed to one)
  • Volume tags propagated via tagSpecification_N parameters for AWS Backup selection
  • README-storageclasses.md preserves the design notes from the original standalone chart

Validation

  • helm template platform-manifests-bootstrap --show-only templates/peeks-storageclass.yaml renders both ebs-sc-gp3 (legacy) and peeks-gp3 (PeEKS default) cleanly with tag propagation
  • RGD compatible with ACK backup-controller v1.3.4 (same API group as the existing rg-s3-bucket.yaml neighbor)
  • No coupling to Pod Identity — the RGD receives iamRoleARN as a parameter

Next steps (out of scope for this PR)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@allamand allamand changed the title Feat/peeks backup policy rgd on pr642 feat: PeEKS BackupPolicy RGD + peeks-gp3 StorageClass (rebased on PR #642) May 14, 2026
allamand added a commit to allamand/appmod-blueprints that referenced this pull request May 14, 2026
When identityCenter.adminGroupId AND identityCenter.adminUserId are
both pre-populated in config, skip the entire group/user/membership
creation logic in idc:setup.

This enables bootstrap from a role that lacks identitystore:Create*
permissions on the target IdC — typical setup when AWS IAM Identity
Center lives in the AWS Organization parent account and the workload
account assumes a least-privilege role.

Existing behavior is preserved when both fields are empty: the task
still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user
user, and wires the membership.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
allamand added a commit to allamand/appmod-blueprints that referenced this pull request May 14, 2026
…cluster readiness

The previous pipeline ran hub:ingress immediately after hub:claim, but
hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that
resolve at task entry. Since ACK provisions VPC + EKS asynchronously
(15-25 min), hub:ingress consistently failed with ResourceNotFoundException.

This change inserts a hub:wait-for-eks step that polls:
1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min)
2. AWS EKS cluster.status until ACTIVE (5 min cap)

Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC
(default 30). Idempotent — re-runs poll the in-flight RGD.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
allamand added a commit to allamand/appmod-blueprints that referenced this pull request May 14, 2026
…om EC2

When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio
runner, dev VM), the previous credentials:refresh logic queried IMDS first
and silently used the host's instance role — overriding any aws.profile
explicitly set in config.local.yaml. ACK controllers then provisioned the
hub VPC/EKS in the EC2 host's account instead of the user's intended
target account, failing with UnauthorizedOperation on ec2:CreateVpc.

New precedence (highest to lowest):
  1. aws.profile from config (if set and not 'default')
  2. EC2 IMDS instance role
  3. AWS default profile

The wrong-account failure mode is silent (controllers just keep retrying
on the wrong account), so make the chosen credential source observable by
logging which path was taken.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
allamand added a commit to allamand/appmod-blueprints that referenced this pull request May 14, 2026
…stanceSynced)

The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced
condition that KRO 0.9.2 never emits on instance objects. KRO publishes
InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the
canonical 'all good' signal.

This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when
the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking
hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go
allamand added a commit to allamand/appmod-blueprints that referenced this pull request May 15, 2026
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1)
as a regular addon in gitops/addons/registry/platform.yaml.

Why an addon and not the EKS ACK capability?
The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT
yet bundled in the managed 'ACK' capability. Until AWS ships it as part
of the capability, it is deployed via Helm.

How does it get its IAM?
Through the new enableACKKRO=true path on appset-chart (previous commit).
That path renders gitops/addons/charts/kro/instances/pod-identity, which
expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy +
eks.PodIdentityAssociation for SA ack-backup-controller in ack-system.

Policy follows least-privilege:
- backup:* + backup-storage:* on all resources
- iam:PassRole scoped to role/* (controller passes backup service roles)
- kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults

Enabled only on the hub cluster via 'enable_ack_backup: true' in
hub-config.yaml; spokes keep it false.

Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize
its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan,
BackupSelection).

Adapted from upstream PR aws-samples#631.
allamand added 13 commits May 15, 2026 20:32
Port from PR aws-samples#633 (feat/peeks-backup-policy-rgd) to the new structure
introduced by PR aws-samples#642 (feature/platform-cluster-kro-ack).

The RGD provisions BackupVault + BackupPlan + BackupSelection per tier
via ACK backup-controller. Tier semantics live on the BackupPolicy CR,
not on the StorageClass — backup selection happens at cluster/namespace
level via tag matching (peeks.io/backup-tier + peeks.io/spoke).

- gold: RPO 1h, retention 30d, cold 90d, cross-region copy
- silver: RPO 24h, retention 14d, cross-region copy
- bronze: RPO 24h, retention 7d, same-region only
Port the storageclass-resources chart from PR aws-samples#633 into the new
platform-manifests-bootstrap chart from PR aws-samples#642 — single bootstrap
chart for early cluster resources (NodePools, ClusterSecretStore,
StorageClasses).

- New template peeks-storageclass.yaml renders ebs/efs SCs from values
- New values.yaml with defaults (gpu/customNodepools/storageClasses)
- peeks-gp3: single PeEKS default SC, tier semantics live on BackupPolicy
- Volume tags propagated via tagSpecification_N parameters for AWS Backup
  selection (peeks.io/spoke, peeks.io/managed-by)
- README-storageclasses.md preserves design notes from the standalone chart
The kind-kro-ack provider was using `{{.ROOT_DIR}}/../../gitops` to
locate the gitops/ tree. However, when this Taskfile is included from
the root Taskfile via `includes: kind-kro-ack: {dir: ./cluster-providers/kind-kro-ack}`,
`{{.ROOT_DIR}}` resolves to the root project directory (where the root
Taskfile lives), not to the included Taskfile's directory.

This caused `task validate` and `task install` to fail with:
  Error: open /opt/.../<repo>/../../gitops/addons/registry/core.yaml: no such file

Aligned with the kind-crossplane provider which correctly uses
`{{.ROOT_DIR}}/gitops`.

Validated: `task validate CONFIG_FILE=$PWD/config.local.yaml` now
passes pre-flight checks.
When identityCenter.adminGroupId AND identityCenter.adminUserId are
both pre-populated in config, skip the entire group/user/membership
creation logic in idc:setup.

This enables bootstrap from a role that lacks identitystore:Create*
permissions on the target IdC — typical setup when AWS IAM Identity
Center lives in the AWS Organization parent account and the workload
account assumes a least-privilege role.

Existing behavior is preserved when both fields are empty: the task
still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user
user, and wires the membership.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…cluster readiness

The previous pipeline ran hub:ingress immediately after hub:claim, but
hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that
resolve at task entry. Since ACK provisions VPC + EKS asynchronously
(15-25 min), hub:ingress consistently failed with ResourceNotFoundException.

This change inserts a hub:wait-for-eks step that polls:
1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min)
2. AWS EKS cluster.status until ACTIVE (5 min cap)

Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC
(default 30). Idempotent — re-runs poll the in-flight RGD.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…om EC2

When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio
runner, dev VM), the previous credentials:refresh logic queried IMDS first
and silently used the host's instance role — overriding any aws.profile
explicitly set in config.local.yaml. ACK controllers then provisioned the
hub VPC/EKS in the EC2 host's account instead of the user's intended
target account, failing with UnauthorizedOperation on ec2:CreateVpc.

New precedence (highest to lowest):
  1. aws.profile from config (if set and not 'default')
  2. EC2 IMDS instance role
  3. AWS default profile

The wrong-account failure mode is silent (controllers just keep retrying
on the wrong account), so make the chosen credential source observable by
logging which path was taken.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
…stanceSynced)

The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced
condition that KRO 0.9.2 never emits on instance objects. KRO publishes
InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the
canonical 'all good' signal.

This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when
the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking
hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync.

Refs: PR aws-samples#642 (parent), PR aws-samples#644 (this branch)
Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go
…K capability role

Add inline policy 'ManagePodIdentityAssociations' to the ACK capability role
so that ACK controllers managed by the EKS ACK capability can create EKS Pod
Identity associations and pass scoped workload roles. Required for any new
ACK controller (e.g. ack-backup) wired through kro pod-identity instances.

- iam:PassRole scoped to role/<name>-* with iam:PassedToService = pods.eks.amazonaws.com
- eks:*PodIdentityAssociation* + tagging actions scoped to cluster/<name>
  and podidentityassociation/<name>/*
- accountId and name come from schema.spec for cross-account / multi-cluster reuse
Wire a new 'enableACKKRO' flag on per-addon chart configs in the
appset-chart. When set, the addon ApplicationSet gets a second source
pointing to gitops/addons/charts/kro/instances/pod-identity, which
materializes ACK iam.Role + iam.Policy + eks.PodIdentityAssociation via
the podidentity.kro.run RGD instead of the legacy Crossplane path.

- 'enableACKKRO' and 'enableAckPodIdentity' are mutually exclusive;
  the template fails fast if both are set on the same addon.
- All existing addons keep their behavior (no flag flip).
- Required for ACK controllers not yet bundled in the EKS ACK
  capability (e.g. ack-backup, currently in preview at v0.1.1).

Adapted from upstream PR aws-samples#631 to fit the
appset-chart structure on the feature/platform-cluster-kro-ack base
(PR aws-samples#642). Credit to the upstream author.
Register the ack-backup-controller (chart oci://public.ecr.aws/aws-controllers-k8s/backup-chart:0.1.1)
as a regular addon in gitops/addons/registry/platform.yaml.

Why an addon and not the EKS ACK capability?
The backup ACK controller is in PREVIEW (v0.1.1, May 2026) and is NOT
yet bundled in the managed 'ACK' capability. Until AWS ships it as part
of the capability, it is deployed via Helm.

How does it get its IAM?
Through the new enableACKKRO=true path on appset-chart (previous commit).
That path renders gitops/addons/charts/kro/instances/pod-identity, which
expands via the podidentity.kro.run RGD to ACK iam.Role + iam.Policy +
eks.PodIdentityAssociation for SA ack-backup-controller in ack-system.

Policy follows least-privilege:
- backup:* + backup-storage:* on all resources
- iam:PassRole scoped to role/* (controller passes backup service roles)
- kms:Decrypt/Encrypt/GenerateDataKey/CreateGrant for encrypted vaults

Enabled only on the hub cluster via 'enable_ack_backup: true' in
hub-config.yaml; spokes keep it false.

Required prerequisite for the BackupPolicy RGD (PR aws-samples#644) to materialize
its referenced backup.services.k8s.aws CRDs (BackupVault, BackupPlan,
BackupSelection).

Adapted from upstream PR aws-samples#631.
The 'kro-manifests-hub' ApplicationSet recursively applies every YAML
under gitops/addons/charts/kro/resource-groups/manifests/. Three sample
BackupPolicy instances under .../backup-policy/examples/ were therefore
applied by Argo CD on the hub before the BackupPolicy CRD existed,
which broke the whole sync run with:

  resource mapping not found for kind "BackupPolicy" in version "kro.run/v1alpha1"

That fail-fast aborted the apply of the *RGDs themselves*
(backup-policy, pod-identity, eks, etc.), so no RGD ever became Active
on the hub — chicken-and-egg.

Move the three sample instances to
gitops/addons/charts/kro/resource-groups/examples/backup-policy/, which
sits outside the apply path. The RGD ships alone in manifests/, like
the other RGDs (eks, pod-identity, ray-service, huggingface).

Add a short README in the new directory documenting the manual-apply
workflow once the RGD is Active and ACK Backup CRDs exist.
The eks-capabilities-kro ClusterRoleBinding only bound the synthetic
User "capabilities.eks.amazonaws.com" which EKS uses for some internal
calls. When the KRO capability assumes its IAM role to act on K8s
resources (e.g. while reconciling the podidentity RGD which declares
iam.services.k8s.aws/Policy and Role children), the API server sees the
STS username 'arn:aws:sts::<acct>:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO'
instead.

Without this second subject, KRO fails with:
  policies.iam.services.k8s.aws is forbidden: User
  "arn:aws:sts::...:assumed-role/peeks-hub-kro-capability-role/KRO"
  cannot get resource "policies" in API group "iam.services.k8s.aws"

This was not exposed before because the podidentity RGD had never been
instantiated (the kro+ACK pod-identity path enableACKKRO=true is new in
this branch, used by ack-backup-controller). RBAC subjects do not
support wildcards, so the templated literal ARN is rendered from
chart values (accountId/resourcePrefix/clusterName), which are now
plumbed via the multi-acct entry in addons/registry/platform.yaml.
The KRO ResourceGraphDefinition 'backuppolicy.kro.run' instantiates
BackupVault / BackupPlan / BackupSelection (apiGroup
backup.services.k8s.aws). Without this rule, KRO reconciles fail with:

  backupvaults.backup.services.k8s.aws is forbidden:
  User "...:assumed-role/<prefix>-<cluster>-kro-capability-role/KRO"
  cannot get resource "backupvaults"

Mirrors the existing rules for ec2/eks/iam/ecr ACK groups.
@allamand allamand force-pushed the feat/peeks-backup-policy-rgd-on-pr642 branch from 35f0a43 to 4d8b31e Compare May 15, 2026 20:32
allamand added 6 commits May 16, 2026 16:40
Harmonise ALB/SG/CloudFront naming on RESOURCE_PREFIX so multi-env
deployments don't collide. Touches kind-kro-ack only; kind-crossplane
unchanged. Sets up dual ingress mode work (tls/cloudfront).
New config.ingress section drives hub ALB exposure:
- mode=cloudfront (default): ALB internal + CloudFront VPC Origin
- mode=tls: ALB internet-facing HTTPS:443, requires certificateArn

Preflight task validates mode + cert + domain before hub:ingress runs.
Default cloudfront preserves current behaviour; Task 2 will switch
the ALB scheme to internal in cloudfront mode.
- mode=cloudfront: scheme=internal, *private* subnets (internal-elb tag),
  SG ingress 80 restricted to VPC CIDR
- mode=tls: unchanged (scheme=internet-facing, *public* subnets, 80+443 open)
- Subnet selection prefers role tags with name-based fallback

Mode switch after creation requires manual teardown (AWS ALB scheme is
immutable). Default cloudfront only reachable end-to-end after Task 3
(CloudFront VPC Origin).
- VPC Origin (post-2024 CF feature) reaches internal ALB without
  public exposure, replacing CustomOriginConfig http-only
- list-vpc-origins lookup makes the task idempotent
- Polls VPC Origin Status until Deployed (up to 15 min) before
  creating the distribution
- Early-exit when ingress.mode != cloudfront (status precondition
  + cmd guard) — pure no-op for tls mode
- Distribution config preserves CachePolicyId/OriginRequestPolicyId,
  PriceClass_100, http2, redirect-to-https
- cloudfront.domain still written back to {{.CONFIG_FILE}}
hub:ingress was creating an ALB without any listener — VPC Origin
and CloudFront would then have nothing to forward to (502).

Add a default listener after ALB creation:
- mode=cloudfront: HTTP:80 fixed-response 404
- mode=tls: HTTPS:443 fixed-response 404 + HTTP:80 → 443 redirect

AWS LBC will replace these defaults with rules pointing at K8s
target groups when Ingresses are reconciled.
CIDR-only ingress (VPC CIDR :80) is insufficient for CloudFront VPC
Origin traffic — CF routes through a dedicated AWS-managed SG
'CloudFront-VPCOrigins-Service-SG' that gets created when the first
VPC Origin lands in the VPC. Without an explicit SG-to-SG rule the
ALB returns 504 to all CloudFront requests.

Lookup the service SG by name and authorize it. Print a warning and
proceed if not yet present (chicken-and-egg on first run — re-run
hub:ingress after hub:cloudfront has created the VPC Origin).
@allamand

Copy link
Copy Markdown
Contributor Author

Hub ingress dual-mode (cloudfront/tls) — implementation + E2E validation

Following the plan in .hermes/plans/2026-05-16-hub-ingress-dual-mode.md, this branch now ships the dual-mode hub ingress (CloudFront VPC Origin or TLS internet-facing). Validated end-to-end against AWS account 586794472760 / eu-west-1.

Commits added to this PR

SHA Type Summary
8e80e17c refactor Rename hub ALB to {{.RESOURCE_PREFIX}}-hub-ingress
3bf31533 feat Add ingress.mode config (cloudfront/tls) with preflight validation
04831bcd feat hub:ingress mode-aware ALB scheme/subnets/SG
9a336f36 feat hub:cloudfront uses VPC Origin (post-2024 CF feature), skips in tls mode
2c250fc1 fix Create default 404 fixed-response listener on hub ALB (was missing → CF had nothing to forward to)
3c5f21f9 fix Authorize CloudFront-VPCOrigins-Service-SG on ALB SG (CIDR-only ingress was insufficient → 504 Gateway Timeout)

E2E validation

  • ALB peeks-hub-ingress (internal, private subnets, SG limited to VPC CIDR + CF service SG)
  • CloudFront VPC Origin Deployed (~10 min provisioning)
  • Distribution d3****.cloudfront.net returns HTTP 404 NotFound in ~57ms — full path client → CF edge → VPC Origin ENI → ALB → fixed-response 404 works as expected (404 is the placeholder until AWS LBC reconciles K8s Ingress rules)
  • task hub:seed ran cleanly; ArgoCD root appset applied; 34/53 hub apps Synced/Healthy

⚠️ Required IAM additions (not yet in repo policies)

The CloudFront VPC Origin feature (2024) is not covered by the AWS-managed CloudFrontFullAccess policy. The role bootstrapping the cluster needs an additional inline policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "cloudfront:CreateVpcOrigin",
      "cloudfront:UpdateVpcOrigin",
      "cloudfront:DeleteVpcOrigin",
      "cloudfront:GetVpcOrigin",
      "cloudfront:ListVpcOrigins",
      "ec2:DescribeNetworkInterfaces",
      "ec2:CreateNetworkInterface",
      "ec2:DeleteNetworkInterface",
      "ec2:CreateTags"
    ],
    "Resource": "*"
  }]
}

I'll fold this into a follow-up commit on the IAM policy templates (or happy to take a pointer if there's a preferred file under cluster-providers/).

Pitfall worth documenting

First-run chicken-and-egg: CloudFront-VPCOrigins-Service-SG is only created after the first VPC Origin lands in the VPC. Initial hub:ingress run prints a warning and continues; re-running hub:ingress after hub:cloudfront adds the SG-to-SG rule. Could be solved by making hub:cloudfront reconcile the ALB SG after VPC Origin creation — happy to refactor if reviewers prefer that ordering.

Out of scope (pre-existing on base branch)

A few hub apps are Degraded for reasons unrelated to ingress: gitlab (initial_root_password < 8 chars), kargo (kargo-admin-credentials secret missing), cert-manager-cainjector (CrashLoopBackOff), jupyterhub (config). These reproduce on feature/platform-cluster-kro-ack without my changes — flagging for tracker context, not blocking this PR.

allamand added 3 commits May 17, 2026 14:31
hub:ingress runs before any VPC Origin exists, so the AWS-managed
'CloudFront-VPCOrigins-Service-SG' is not yet present. The previous
fix in hub:ingress prints a warning and continues — but then the
user must remember to re-run hub:ingress manually after hub:cloudfront.

Move the SG-to-SG authorization into hub:cloudfront, immediately
after the VPC Origin reaches Deployed (which is when AWS guarantees
the service SG exists). One hub:ingress + hub:cloudfront pass is
now sufficient end-to-end. The hub:ingress best-effort attempt is
kept as belt-and-braces for re-runs.
cainjector OOMKilled (exit 137) on cluster bootstrap when caching
all CABundle injectables across CRDs (cert-manager + crossplane +
ack-* providers + kro). 128Mi is too tight for a hub cluster with
many CRDs; 256Mi is the upstream default and resolves the loop.
The BackupPolicy RGD's BackupSelection matches on
'aws:ResourceTag/peeks.io/backup-tier' but no StorageClass currently
emits this tag, so volumes never enter a BackupPlan.

Add three tier-labeled StorageClasses — peeks-gold-gp3, peeks-silver-gp3,
peeks-bronze-gp3 — with backup-tier tags propagated to EBS volumes via
the existing 'tags:' helm value. Workloads opt into a tier by picking
the matching SC in their PVC manifest, so backup tier is workload-driven
(no central ConfigMap, no platform ticket).

The legacy ebs-sc-gp3 and the default peeks-gp3 are preserved unchanged.
Bronze keeps reclaimPolicy=Delete (cheap/ephemeral); gold and silver
keep Retain.

Tier policies (gold/silver/bronze RPO + retention + cross-region copy)
are defined in the BackupPolicy RGD — this commit only wires the
emitter side. Ported from feat/peeks-backup-policy-rgd commit 6862a00
onto the rebased platform-manifests-bootstrap chart layout.
allamand added a commit that referenced this pull request May 18, 2026
…stanceSynced) (#662)

The eksclusterwithvpc RGD's eks node was waiting for an InstanceSynced
condition that KRO 0.9.2 never emits on instance objects. KRO publishes
InstanceManaged / GraphResolved / ResourcesReady / Ready — Ready is the
canonical 'all good' signal.

This caused the eksclusterwithvpc parent to remain IN_PROGRESS even when
the underlying EKS cluster was ACTIVE in AWS for 30+ minutes, blocking
hub:ingress, hub:cloudfront, hub:seed and hub:wait-for-sync.

Refs: PR #642 (parent), PR #644 (this branch)
Refs: KRO 0.9.2 condition types — see kro/api/v1alpha1/instance_status.go

Co-authored-by: Sébastien Allamand <allamand@users.noreply.github.com>
allamand added a commit that referenced this pull request May 18, 2026
…dC, EKS gate, profile-first creds) (#664)

* fix(kind-kro-ack): GITOPS_ROOT path resolution

The kind-kro-ack provider was using `{{.ROOT_DIR}}/../../gitops` to
locate the gitops/ tree. However, when this Taskfile is included from
the root Taskfile via `includes: kind-kro-ack: {dir: ./cluster-providers/kind-kro-ack}`,
`{{.ROOT_DIR}}` resolves to the root project directory (where the root
Taskfile lives), not to the included Taskfile's directory.

This caused `task validate` and `task install` to fail with:
  Error: open /opt/.../<repo>/../../gitops/addons/registry/core.yaml: no such file

Aligned with the kind-crossplane provider which correctly uses
`{{.ROOT_DIR}}/gitops`.

Validated: `task validate CONFIG_FILE=$PWD/config.local.yaml` now
passes pre-flight checks.

* feat(kind-kro-ack): support BYO IdC (cross-account adminGroup/adminUser)

When identityCenter.adminGroupId AND identityCenter.adminUserId are
both pre-populated in config, skip the entire group/user/membership
creation logic in idc:setup.

This enables bootstrap from a role that lacks identitystore:Create*
permissions on the target IdC — typical setup when AWS IAM Identity
Center lives in the AWS Organization parent account and the workload
account assumes a least-privilege role.

Existing behavior is preserved when both fields are empty: the task
still auto-creates admin/editor/viewer groups, the eks-argocd-admin-user
user, and wires the membership.

Refs: PR #642 (parent), PR #644 (this branch)

* feat(kind-kro-ack): add hub:wait-for-eks step to gate hub:ingress on cluster readiness

The previous pipeline ran hub:ingress immediately after hub:claim, but
hub:ingress declares vars with 'sh: aws eks describe-cluster ...' that
resolve at task entry. Since ACK provisions VPC + EKS asynchronously
(15-25 min), hub:ingress consistently failed with ResourceNotFoundException.

This change inserts a hub:wait-for-eks step that polls:
1. EksclusterWithVpc/<hub>.status.state until ACTIVE (default 30 min)
2. AWS EKS cluster.status until ACTIVE (5 min cap)

Tunables via vars: WAIT_TIMEOUT_SEC (default 1800), WAIT_INTERVAL_SEC
(default 30). Idempotent — re-runs poll the in-flight RGD.

Refs: PR #642 (parent), PR #644 (this branch)

* fix(kind-kro-ack): prefer aws.profile over IMDS when bootstrapping from EC2

When the bootstrap host is itself an EC2 instance (Cloud9, Workshop Studio
runner, dev VM), the previous credentials:refresh logic queried IMDS first
and silently used the host's instance role — overriding any aws.profile
explicitly set in config.local.yaml. ACK controllers then provisioned the
hub VPC/EKS in the EC2 host's account instead of the user's intended
target account, failing with UnauthorizedOperation on ec2:CreateVpc.

New precedence (highest to lowest):
  1. aws.profile from config (if set and not 'default')
  2. EC2 IMDS instance role
  3. AWS default profile

The wrong-account failure mode is silent (controllers just keep retrying
on the wrong account), so make the chosen credential source observable by
logging which path was taken.

Refs: PR #642 (parent), PR #644 (this branch)

---------

Co-authored-by: Sébastien Allamand <allamand@users.noreply.github.com>

@shapirov103 shapirov103 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge conflicts.

I see the feature and I think it is valuable for the solution. Is this something you plan to highlight in the workshop content as well? What is the GTM for this feature?

@allamand allamand marked this pull request as draft May 22, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants