feat(cloudfront): CloudFront exposure mode for domain-less deployments#709
Open
allamand wants to merge 119 commits into
Open
feat(cloudfront): CloudFront exposure mode for domain-less deployments#709allamand wants to merge 119 commits into
allamand wants to merge 119 commits into
Conversation
punkwalker
reviewed
May 27, 2026
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
allamand
pushed a commit
that referenced
this pull request
May 28, 2026
1. Remove exposure.mode config field — derive from domain (empty = cloudfront) 2. Group cloudfront tasks into hub:setup-cloudfront-exposure subtask 3. Fix CodeQL clear-text storage — use 0600 permissions on credentials file
Contributor
Author
|
I've made good progress on this: cluster installation and all addons deployment OK. Now needs to configure backstage to read from github, an from configmap for the dynamic configuration part that was initially done in gitlab custom |
Contributor
Author
|
Need to fix Gitlab SSO |
punkwalker
reviewed
Jun 3, 2026
Collaborator
There was a problem hiding this comment.
PR is Great! But I feel some of the workshop specific task steps and scripts are creeping into this. @allamand Could you please evaluate if the workshop specific configuration tasks/steps/scripts like Gitlab, ~/.bashrc.d/platform.sh creation etc, can be moved out of this PR?
This was referenced Jun 4, 2026
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
8ecd584 to
6e20787
Compare
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
8a19c73 to
0d79541
Compare
allamand
pushed a commit
that referenced
this pull request
Jun 6, 2026
…p hardcoded password and vestigial .exposure schema - gitlab.yaml/init-job.yaml: GitLab root password now sourced from git_token (= workshop USER_PASSWORD/IDE_PASSWORD); initial_root_password kept only as fallback - control-plane/gitlab/values.yaml: remove hardcoded initial_root_password (PR #709 review) - config.schema.json: remove vestigial .exposure block (EXPOSURE_MODE derives from .domain)
allamand
pushed a commit
that referenced
this pull request
Jun 6, 2026
…709 review) Base default is now empty; the value is injected per-deployment via the registry valuesObject from cluster metadata (gitlab_domain_name annotation).
allamand
pushed a commit
that referenced
this pull request
Jun 6, 2026
…loudfront.yaml (PR #709 review) Move the cloudfront-mode tasks (setup-exposure, create-alb, hub/gitlab distributions, NLB, sync-origins) into a dedicated included Taskfile under the 'cloudfront:' namespace, passing required vars via includes. Per-task status: guards keep them cloudfront-mode-only. Validated with 'task --list-all'.
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
ec078d4 to
179a088
Compare
allamand
added a commit
that referenced
this pull request
Jun 12, 2026
1. Remove exposure.mode config field — derive from domain (empty = cloudfront) 2. Group cloudfront tasks into hub:setup-cloudfront-exposure subtask 3. Fix CodeQL clear-text storage — use 0600 permissions on credentials file
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
cae6520 to
efce048
Compare
allamand
added a commit
that referenced
this pull request
Jun 12, 2026
…p hardcoded password and vestigial .exposure schema - gitlab.yaml/init-job.yaml: GitLab root password now sourced from git_token (= workshop USER_PASSWORD/IDE_PASSWORD); initial_root_password kept only as fallback - control-plane/gitlab/values.yaml: remove hardcoded initial_root_password (PR #709 review) - config.schema.json: remove vestigial .exposure block (EXPOSURE_MODE derives from .domain)
allamand
added a commit
that referenced
this pull request
Jun 12, 2026
…709 review) Base default is now empty; the value is injected per-deployment via the registry valuesObject from cluster metadata (gitlab_domain_name annotation).
allamand
added a commit
that referenced
this pull request
Jun 12, 2026
…loudfront.yaml (PR #709 review) Move the cloudfront-mode tasks (setup-exposure, create-alb, hub/gitlab distributions, NLB, sync-origins) into a dedicated included Taskfile under the 'cloudfront:' namespace, passing required vars via includes. Per-task status: guards keep them cloudfront-mode-only. Validated with 'task --list-all'.
punkwalker
reviewed
Jun 23, 2026
| @@ -243,6 +364,8 @@ tasks: | |||
| | kubectl apply --server-side --force-conflicts -f - 2>&1 | grep -v "ProviderConfig" || true | |||
| - kubectl wait --for=condition=Healthy --timeout=300s providers.pkg.crossplane.io --all | |||
| - kubectl wait --for=condition=Healthy --timeout=300s functions.pkg.crossplane.io --all | |||
| # Grant cluster-admin to provider-kubernetes (needed to manage Jobs, Secrets, etc.) | |||
| - kubectl create clusterrolebinding provider-kubernetes-admin --clusterrole=cluster-admin --serviceaccount=crossplane-system:provider-kubernetes 2>/dev/null || true | |||
Collaborator
There was a problem hiding this comment.
Crossplane Manages k8s Jobs? Are we using kubernetes provider of crossplane?
…o ClusterRole kro needs permissions to create/manage RayService (ray.io) and SparkApplication (sparkoperator.k8s.io) resources for Module 40 (AI/ML Delivery). Without these, kro gets RBAC denied when the Backstage template creates a RayService or Spark job kro instance. Peeks test agent identified this as the root cause of Module 40 Phase 40.3 block.
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
d21fee4 to
61f331b
Compare
The Ray serve app expects requests at /generate but the ingress routes /ray-serve/<name>/generate without stripping the prefix. Add transforms annotations to rewrite /ray-serve/<name>/* to /* and /ray-dashboard/<name>/* to /* so both the serve endpoint and dashboard work through CloudFront.
allamand
force-pushed
the
feature/cloudfront-on-agent-platform
branch
from
61f331b to
4def3df
Compare
Contributor
Author
|
@elamaran11 @punkwalker @shapirov103 p’ease look at this new version. All is almost working, we have a dedicated workshop path, it uses different git overlay so we can just use GitHub as mainstream and other git to customize. Happy to chat about it |
… Playwright The master realm returns HTTP 200 quickly, but the platform realm SAML descriptor takes much longer to initialize after Keycloak boot. The script was launching Playwright immediately after master realm 200, then spending 30min waiting for the SAML descriptor to appear — ultimately timing out. Fix: poll the SAML descriptor endpoint (max 5min / 20x15s) before launching Playwright. If it's ready, proceed normally. This avoids the 30-minute configure_identity_center.py internal timeout.
The global DOMAIN Task var is evaluated at Taskfile load time — before private/cloudfront-domain is written by the background create-config.sh job. This caused ingress_domain_name='empty' in peeks-hub/config Secrets Manager, which then made Keycloak start with hostname=https:// (empty) and crash-loop. Fix: read domain from private/cloudfront-domain at shell runtime inside secrets-manager:seed, with fallbacks to config.local.yaml and AWS API. This ensures Keycloak always gets a valid hostname even on the first deploy.
KC26 HostnameV2 uses hostname AS-IS for all generated URLs (issuer, SAML metadata entityID/Location). Without /keycloak in hostname, the issuer and SAML URLs become https://domain/realms/platform (missing context path), causing ArgoCD SSO and IDC SAML federation to redirect to wrong URLs. The ALB ingress uses pathType:Prefix on /keycloak (no prefix stripping), so Keycloak receives /keycloak/* paths. Both hostname and http-relative-path must specify /keycloak — this is correct for KC26 when the ALB does not strip the context path prefix before forwarding to Keycloak.
The check for spoke ArgoCD cluster secrets was timing out after 40x15s=10min. In practice the secret was created ~5min after the warning, so 15min budget (60 retries) avoids the false-positive non-fatal warning.
- GRAFANA_URL: resolve AMG workspace endpoint instead of /grafana path (was serving Kargo SPA — blank page for all Grafana steps in workshop) - AWS_REGION + AWS_ACCOUNT_ID: export to platform.sh so content commands work - generated/genaitemplates/: pre-create directory for Kiro AI generation exercise (replaces stale amazonq/ path that no longer exists)
Two issues causing 'No token found' on the Keycloak path: 1. argocd_token_automation.py was missing from scripts/ (only existed in platform/infra/terraform/scripts/) so the shell script couldn't find it. Copied to scripts/ alongside argocd-refresh-token.sh. 2. Post-Keycloak wait was too short (5s + 15 poll iterations = 20s max) for the full SAML round-trip: Keycloak → AWS IDC ACS → ArgoCD callback. Increased to 8s initial wait + 30 poll iterations (30s max).
…ycloak auth IDE_PASSWORD is the code-server/IDE password (different source). USER1_PASSWORD is the Keycloak user1 password set during task install. They can differ — using IDE_PASSWORD caused Keycloak login to fail silently, leaving argocd-refresh-token unable to obtain a token.
…lete terraform/ Active scripts moved to scripts/: - configure_identity_center.py (IDC config — called by workshop + crossplane Taskfiles) - colors.sh (terminal colors — used by argocd-sync, check-*, recover-argocd-apps) - recover-argocd-apps.sh (ArgoCD recovery — called by argocd-sync in ssm-setup-ide-logs) - check-ray-build.sh (Ray build checker — called by check-workshop-setup + ssm-setup) - check-workshop-setup.sh (workshop health check — called by ssm-setup-ide-logs) recover-argocd-apps.sh: removed unused 'source argocd-utils.sh' (functions used are print_info/warning/success which come from colors.sh, not argocd-utils.sh) References updated: - workshop/Taskfile.yaml: configure_identity_center.py path - cluster-providers/kind-crossplane/Taskfile.yaml: configure_identity_center.py path - hack/.bashrc.d/ssm-setup-ide-logs.sh: all 4 refs (argocd-sync, check-ray-build, check-workshop-setup, argocd-refresh-token) Deleted platform/infra/terraform/ entirely: - scripts/: stale scripts (0-init, 1-tools-urls, 2-gitlab-init, utils, argocd-utils, backstage-utils, build_backstage, force-unlock-helper, monitor-consolidation) - cluster/, common/, database/, identity-center/: Terraform modules for EKS+VPC+RDS that were superseded by the kind-kro-ack cluster provider (GitOps-native) - As noted in UPGRADE-FROM-MAIN-2026-05.md: 'Phase 4 Terraform ports went beyond scope; primary deploy path is task install via kind-crossplane, not terraform' Docs updated: README.md, backstage/docs/build-and-deployment-process.md, .kiro/steering/project.md, hack/.kiro/steering/project.md
argocd login with --auth-token still prompts for Username: on some versions. Redirecting stdin from /dev/null suppresses the prompt.
…y-repo TLS handshake timeout was observed on fresh accounts when set-overlay-repo ran immediately after the EKS cluster reached ACTIVE state. Root cause: EKS ACTIVE != API server reachable — SG rule propagation + internal EKS control plane startup adds ~30-60s after ACTIVE before TLS connections are accepted from the IDE VPC. Added a poll loop (30 × 5s = 150s max) that verifies the API server actually accepts kubectl connections before attempting the kubectl apply of the fleet-config-repo and gitlab-user-repo-creds secrets.
…in to suppress Username: prompt
…URL logic
- Add PREFIX="{{.RESOURCE_PREFIX}}" to inject Taskfile var into shell
- Initialize AMG_ENDPOINT="" before conditional to prevent unbound var error
- Fix missing newline between set_var AWS_ACCOUNT_ID and PREFIX assignment
The final kubectl context restore in task install was swallowing errors with 2>/dev/null || true on both the update-kubeconfig and use-context calls. If update-kubeconfig created the hub context under a different name than expected, or the call failed for any other reason, use-context failed silently and the active context was left on whatever spoke cluster was touched last by an earlier install step (wait-for-spokes, idc:configure, spoke enablement) — observed on a live event where the default kubeconfig ended up on peeks-spoke-prod instead of peeks-hub, with no error printed anywhere. Make the restore fail loudly: check update-kubeconfig's exit status explicitly, and only run use-context (unguarded) if it succeeded so a real failure surfaces instead of being masked.
Restore platform-manifests-bootstrap addon (sync-wave -1) that creates the custom system-peeks/general-purpose-peeks/gpu/trainium NodePools BEFORE any addon that targets them (previously the nodepools lived in platform-manifests at wave 6, too late). enable_platform_manifests_bootstrap is already wired in rg-eks-vpc.yaml (default true). Flip in-cluster addon nodeSelectors from the built-in 'system' nodepool to the custom 'system-peeks' so we control disruption/consolidation budgets: kubevela, keycloak, cert-manager, crossplane, external-secrets, argo-events/rollouts/ workflows, aws-efs-csi-driver, efs-chart. Left on built-in 'system' (managed capabilities / not in-cluster pods): argocd, kro, ACK *-chart controllers, gitlab (runs on EC2). Built-in nodepools remain enabled as a bootstrap fallback. Requires a fresh-install validation (no Pending pods; -peeks nodepools created at wave -1 before dependent addons).
Vendor 6 APEX EKS skills from aws-samples/sample-apex-skills @ v1.2.0 into hack/.kiro/skills/ so they ship by default in every participant's Kiro 'peeks' agent (no manual install, no boot-time network dependency). - scripts/sync-apex-skills.sh: idempotent maintainer script, pinned tag+SHA, collision guard for PEEKS skills, clean failure if tag/network unavailable. - hack/.kiro/skills/APEX_VERSION: records tag v1.2.0 + commit SHA. - peeks.json unchanged: its file://.kiro/skills (**/SKILL.md) resource indexes them automatically. - README: APEX Skills section (deterministic vendoring, refresh procedure, documented clone-at-startup alternative left disabled).
aws eks update-kubeconfig switches current-context on every call and the cluster order from list-clusters is not guaranteed, so setup could leave the active context on a spoke. Explicitly select the hub context at the end.
commit 8937815 deleted platform/infra/terraform/ (Terraform -> CDK/kro migration) but did not relocate Dockerfile.ray-vllm, which Taskfile.ray.yaml's ray:setup task still referenced by its old path. This made the background vLLM image build fail on every task install with 'unable to prepare context: path .../terraform/common not found', retrying every ~20s for the full 30min wait-image timeout — non-fatal (falls back to the pre-staged TinyLlama model) but always failing and wasting the full 30 minutes. Restore the Dockerfile next to the Taskfile that uses it (cluster-providers/common/) and update DOCKERFILE_PATH accordingly.
…ckstage template Same terraform/ cleanup (8937815) that deleted Dockerfile.ray-vllm also deleted Dockerfile.ray-neuron, used by the ray-serve-trainium Backstage template (platform/backstage/templates/ray-serve-trainium/, registered in catalog-info.yaml). Restore it next to Dockerfile.ray-vllm. NOT YET WIRED: Taskfile.ray.yaml only builds/pushes one image ({resourcePrefix}-ray-vllm-custom:latest) used by all Ray Serve templates. To actually run ray-serve-trainium on Neuron hardware, a follow-up is needed to add a separate build+push task for this Dockerfile (e.g. {resourcePrefix}-ray-vllm-neuron:latest) and have the kro RayService RGD select the image based on acceleratorType.
The ray-service.yaml RGD already had a full rayserviceNeuron block
(nodeSelector: karpenter.sh/nodepool: trainium, tolerations, Neuron
env vars, aws.amazon.com/neuron resources) referencing an image named
${resourcePrefix}-ray-neuron-custom:latest — but nothing ever built or
pushed that image. The trainium NodePool (platform-manifests-bootstrap/
templates/trainium-nodepool.yaml) is also already deployed unconditionally
alongside the GPU NodePool.
Add 'ray:setup-neuron' (ECR repo + background build/push from the
restored Dockerfile.ray-neuron) and 'ray:wait-neuron-image', mirroring
the existing vLLM/GPU pattern. Kept opt-in and separate from ray:setup
— Trainium quota/hardware is not available in every account, so this
should be run explicitly before deploying the ray-serve-trainium
Backstage template, not on every task install.
NOT tested against real Trainium hardware (trn1 instances) — the
Dockerfile and Taskfile wiring are mechanically correct but unverified
end-to-end.
prod/enabled-addons.yaml was missing kro_manifests: true (present in dev/enabled-addons.yaml). Without it, the peeks-spoke-prod cluster secret never gets the enable_kro_manifests label, so the kro-manifests-<cluster> ApplicationSet never generates an app for prod — the AppmodService/S3Bucket/PodIdentity/Eks*/Vpc kro CRDs are installed on spoke-dev but never on spoke-prod. This breaks every prod flow that deploys an AppmodService (e.g. Kargo dev->prod promotion errors on 'no matches for kind AppmodService'; the first java-prod-cd ArgoCD sync fails validation because the CRD itself doesn't exist, not just the namespace — CreateNamespace=true was already correctly set on argo-app-prod.yaml, this was never a namespace ordering problem).
…platform Resolved 3 real conflicts (verified branch-by-branch with explicit diff labels to avoid ambiguity): - gitops/addons/registry/observability.yaml: auto-merged cleanly. Kept exposure_mode (source) + upstream's newer Helm chart version bumps (kube-state-metrics 7.3.0, prometheus-node-exporter 4.55.0, opentelemetry-operator 0.112.1, cni-metrics-helper 1.21.1). Kept aws-for-fluentbit tag 3.4.6 (source's more recent pin) over upstream's 3.4.5. - gitops/overlays/environments/control-plane/enabled-addons.yaml: kept source (feature/cloudfront-on-agent-platform) entirely. Upstream (feature/agent-platform) had this file far behind — most addons still false (cert_manager, argo_events, argo_rollouts, kargo, kro_manifests_hub, multi_acct, platform_manifests*, kubevela, devlake, jupyterhub, ray_operator, spark_operator) — these are all required by this workshop and were already enabled on source. - gitops/addons/charts/backstage/templates/install.yaml: auto-merged cleanly except the final Ingress block. Kept source's dual-mode (insecure/ALB vs nginx/domain) ingress, matching the same established pattern already used in grafana and argo-workflows ingress templates (host: omitted in ALB/insecure mode since CloudFront routes by path on a single domain, not by host).
Contributor
Author
|
Updated all the recommendations, and tested it for workshop in workshop studio. all is working. Workshop is a separate entry point. Gitlab is managed apart. All good from my side |
shapirov103
reviewed
Jul 13, 2026
| {{- /* insecure: HTTP-only ingress, no cert (consumer terminates TLS upstream). | ||
| Explicit .Values.global.insecure wins; falls back to the legacy | ||
| exposure_mode == "cloudfront" toggle during the transition. */ -}} | ||
| {{- $insecure := or .Values.global.insecure (eq (default "domain" .Values.global.exposure_mode) "cloudfront") -}} |
Contributor
There was a problem hiding this comment.
This still talks about cloudfront - should be removed. Probably a sweep over the PR to make sure references to cloudfront at localized to the workshop folder
shapirov103
requested changes
Jul 13, 2026
Contributor
There was a problem hiding this comment.
Very cool, getting there!
I stared with comment: https://github.com/aws-samples/appmod-blueprints/pull/709/changes#r3571491629
Here is review based on arch constraints that we agreed upon. Please check the section Leaks to remove for constraint #1
Architecture review: CloudFront exposure must not leak into the solution
Reviewed against three constraints: (1) no CloudFront-specific logic in the solution — all workshop customization localized to workshop/; (2) workshop provisioning is a private case of solution instantiation; (3) the solution supports two modes — an externally-provisioned domain and an insecure mode (for services like Keycloak).
| Constraint | Status |
|---|---|
| #3 — two modes (external domain + insecure) | ✅ Satisfied |
| #2 — workshop = private case of solution instantiation | ✅ Satisfied in structure |
#1 — no CloudFront leak; workshop specifics localized to workshop/ |
✅ What's right
- The core abstraction is generic and correct:
config.schema.jsondefinesdomain(consumer-provisioned — "the platform never creates it"),insecure, andcertificateArn. Ingress templates switch on.Values.global.insecure(keycloak/templates/ingress.yaml:4,backstage/templates/install.yaml:790,argo-workflows/templates/ingress.yaml:3). A consumer settinginsecure: truegets HTTP mode without ever naming CloudFront. Constraint #3 met. workshop/is the CloudFront consumer:workshop/Taskfile.yamlprovisions the distribution and writesprivate/cloudfront-domain/private/gitlab-cloudfront-domain. The generic provider documents spoke/overlay/teardown as consumer concerns. Constraint #2 met structurally.
⚠️ Leaks to remove for constraint #1
-
gitops/addons/registry/core.yaml:211-216—ingress-class-albscheme: internalandloadBalancerNameare gated onexposure_mode == "cloudfront". CloudFront-topology logic (VPC origin ⇒ internal ALB) keyed to the consumer's edge name in the core registry. Make it a generic knob (drive frominsecure, or an explicitalbScheme). -
Legacy
exposure_mode == "cloudfront"shim in ingress templates (keycloak, backstage, argo-workflows — comments say "legacy … during transition") OR'd withinsecure. Collapse toinsecureonly so the string"cloudfront"disappears from solution charts. -
gitops/overlays/environments/control-plane/kargo/values.yaml:1— self-described "cloudfront variant (HTTP:80 behind the CloudFront origin)" living in the base solution overlays. Belongs in the workshopfleet-configoverlay. -
cluster-providers/kind-crossplane/Taskfile.yamlembeds workshop/IDE specifics that already exist inworkshop/Taskfile.yaml:EXPOSURE_MODEderives the literal"cloudfront"frominsecure.- reads
private/cloudfront-domain/private/gitlab-cloudfront-domain(lines 794, 951, 1102, 1425). peeks-ideCFN stack +SharedRole+ Playwright insideidc:configure(1434-1461);IDE_PASSWORD(1009); "git_token seeded by CDK" (1086).- generic
install:callsinstall:phase1-playwright-prefetch(193-204) — Playwright is only needed for the workshop's IDC SAML automation.
idc:configureand the cloudfront-domain reads are duplicated inworkshop/Taskfile.yaml(their correct home), so the copies here are the leak to delete.
Doc drift (not a code leak)
.kiro/steering/troubleshooting.md references cluster-providers/common/Taskfile.cloudfront.yaml, install:phase1-cloudfront, and create-alb — none exist on the branch. The generic install does not invoke CloudFront provisioning; please update or remove those sections.
Bottom line
The two-mode design is sound and domain-mode installs work (the CloudFront references are gated fallbacks). To fully satisfy "no CloudFront logic in the solution," please (a) collapse exposure_mode=="cloudfront" to insecure, (b) make ALB scheme generic in core.yaml, (c) move the Kargo cloudfront overlay + the cloudfront-domain/peeks-ide/IDE_PASSWORD/Playwright logic into workshop/, and (d) fix the stale steering references.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CloudFront exposure mode for domain-less deployments
Enables the platform to run without a custom domain by using CloudFront distributions for HTTPS ingress.
Key Changes
Infrastructure & Bootstrap
create-mgmt-roles, fix elapsed time display, addspokes:enable-crossplane/spokes:enable-krotaskssecrets-manager:seed-secretstask for platform user credentialssts:AssumeRolepolicy to ACK capability role for cluster-mgmt roleseks:*permissions tocluster-mgmt-eksroleArgoCD 3.x (EKS Capability) Compatibility
digwithindex/orin ApplicationSet templates (ArgoCD 3.x strict typing)project: defaultto cluster secrets (required by ArgoCD 3.x)indexfor optional annotations withmissingkey=errorredirectUrito use GitLab CloudFront domainFleet Management & Spoke Clusters
kro-values/→spoke-values/, separatecrossplane-clusters/andkro-clusters/clusters-kroApplicationSet: one app per cluster (aligned with main),CreateNamespace=truesecretsmanagerIAMRoleSelector for ACK spoke provisioningresourcePrefixvsclusterNamemismatch in IAMRoleSelectorsrg-eks.yamlandrg-eks-vpc.yamlwith main (removeenable_ack_*, single NAT Gateway, camelCase annotations)GitLab
ROOT_GITLAB_TOKENusage in init job (was using raw password as API token)aws-load-balancer-nameannotation for NLB adoptiongitlab_domain_namein Keycloak client redirect URIObservability
ttlStrategyto devlake workflow for pod cleanupreadyWhen:eks.status.clusterStateinstead ofeks.status.stateDocumentation
Additional fixes (follow-up commits)
Bootstrap robustness
crossplane-pod-identity,enableCertManagerfor the LBC webhook,pod_identitiesmoved aftercrossplane-basein sync-wave, andwait-for-full-syncnow surfaces per-app errors and fails fast on CRD-ordering failures.EksclusterWithVpcRGD defaultaddonsRepoBasePath(gitops/addons/→gitops/); the doubled value produced non-existent overlay paths (gitops/addons/addons/...) that left all of a spoke's appsUnknown.Spoke provisioning (kro / Crossplane)
addonsRepoRevision/fleetRepoRevision) viaclusters-kroglobal values instead of defaulting tomain(charts that exist only on the deploy branch were unresolvable on spokes).clusters-krogit generator reads per-cluster trigger JSONs from the overlay (fleet-config) with a base fallback, sotask spokes:enable-kroactually drives generation; removed the redundant base-repokro-clusters/clusters/peeks-spoke-prod.json.fleet_member=spoke, matching kro-provisioned spokes.Developer experience / docs
spokes:enable-crossplane/spokes:enable-krorefresh the local~/environment/fleet-configcheckout after pushing (best-effort, non-fatal).platform/fleet-config/README.md) by the GitLab init-job (single source of truth), replacing the inline placeholder.Spoke access entries & Crossplane provider bootstrap
hub_cluster_nameon the KROEksClusterargocdSecretso the appset-chartdestinationHubresolution targets the hub. Thepod-identitiesbootstrap app now deploys on the hub (whose Crossplane providers already have credentials) and creates the spoke'sprovider-aws-iam/provider-aws-eksPod Identity associations, instead of falling back to the credential-less spoke. This chicken-and-egg previously leftcrossplane-base/pod-identitiesDegradedon KRO-provisioned spokes (no EC2 IMDS role found). Dev worked only because its Crossplanefleet-secretalready carried the annotation.grant-ide-accesstask (and extendkubeconfig) to grant the current caller — the IDE EC2 instance role (*SharedRole*) — cluster-admin access entries on the hub and every<prefix>-*cluster.create-config.shdeliberately excludes the instance role fromadminRoleName, so it had no access entry on KRO-provisioned spokes (kubectl401). Idempotent and additive.adminRoleName(string) →adminRoles([]{label, roleName}) on theEksCluster/EksclusterWithVpcRGDs, turning theaccessEntryAdminresource into a KRO v0.9forEachthat creates one cluster-adminAccessEntryper role — so a cluster can grant the console/participant role and the IDE/instance role at the same time.labelis a DNS-1123-safe discriminator (IAM role names may contain uppercase/illegal chars). Backward-compatible:clusters.yamlwraps a legacy singleadminRoleNameinto a one-element list.