Skip to content

Jojung/promote test envs#143

Draft
jojung1 wants to merge 5 commits into
mainfrom
jojung/promote-test-envs
Draft

Jojung/promote test envs#143
jojung1 wants to merge 5 commits into
mainfrom
jojung/promote-test-envs

Conversation

@jojung1

@jojung1 jojung1 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Draft — stacked on #134. This is the protected promote half of the test/test01 GitOps deploy flow; #134 (the untrusted build stage) must merge first. Kept as a draft until then. The kiwi-argocd ApplicationSet/values changes are a separate MR.

Adds the protected promote pipeline that deploys a PR to the shared test / test01 environments by consuming the immutable artifacts built in #134 and committing a machine-written deploy-state.yaml to the env branch in the ArgoCD values repo. It targets only test/test01 (every rule requires NVCM_PROMOTE_ENV ∈ {test, test01}); production stays on the tag-driven release flow.

Run on the mirror's default branch with NVCM_PROMOTE_PR + NVCM_PROMOTE_ENV. Jobs (.gitlab/ci/promote-test-envs.yml):

  • test-promote-build — resolves the vetted pull-request/<n> snapshot, reuses or triggers the Add secret-free build stage for untrusted PRs #134 build, and gates provenance on GitLab pipeline metadata (sha/ref/status/source) — never on anything the untrusted build wrote.
  • test-promote-push-images — downloads the image tarballs by job id, docker load, retag to 0.0.0-pr<n>.<sha>, push, capture registry digests.
  • test-promote-chart — downloads the chart .tgz packaged in Add secret-free build stage for untrusted PRs #134's secret-free stage, validates the render against the env's baseline+overrides+digests, and publishes it (no PR checkout, no repackaging — helm template only, which is sandboxed).
  • test-promote-deploy-state — commits deploy-state.yaml to the env branch (NV_CONFIG_MANAGER_VALUES_PUSH_TOKEN), owning the GitLab Environment.
  • test-rollback-env / test-release-env — commit a prior deploy-state / reset to canonicals.

Security model: the promote stage operates only on already-built immutable artifacts, so no untrusted PR code executes with secrets. Provenance is established from GitLab metadata; the build stage (#134) holds no secrets.

Default-branch job guards: a web-source pipeline on the default branch with NVCM_PROMOTE_ENV set would otherwise also match docker-build-main-*, test-main-*, sonar, and pulse (their rules:changes evaluate true on non-push pipelines). Each gets a leading - if: $NVCM_PROMOTE_ENV / when: never so a promote run doesn't rebuild/retest main as a side effect. Normal push/MR/tag pipelines are unaffected.

Chart source: the promote publishes the dev chart to the existing NGC Helm registry (https://helm.ngc.nvidia.com/nvidian/cfa) via the ngc target; ArgoCD pulls it by exact version. (Confirmed exact-prerelease pins resolve there.)

Required protected CI/CD variables: NVCM_MIRROR_API_TOKEN, NVCM_TEST_ENV_TARGETS, NVCM_CHART_REPO, NV_CONFIG_MANAGER_VALUES_PUSH_TOKEN, NVCM_VALUES_REPO_PATH, plus the existing image/chart target vars. Documented in .gitlab/ci/README.md.

Validation

  • yq parse of .gitlab-ci.yml + promote-test-envs.yml + the four guarded CI files; bash -n on all four promote scripts.

  • Confirmed the promote source allowlist and default-branch guards behave as intended.

  • Exercised the build→promote chart handoff locally: packaged the chart as Add secret-free build stage for untrusted PRs #134's pr-build-chart does, then ran the promote render gate against the real test baseline+overrides+digests → renders nvcr.io/nvidian/cfa/...@sha256:....

  • End-to-end pipeline validation is gated on Add secret-free build stage for untrusted PRs #134 merging (nothing here runs until the build stage exists) and the protected variables being provisioned.

  • Standard CI passes.

  • Kind integration passes, or this PR explains why it was not run.

Kind integration not run — not applicable. This PR only adds GitLab CI job definitions and shell scripts for the deploy pipeline; it changes no application code or Helm chart runtime output, and nothing the Kind suite exercises.

Checklist

  • I am familiar with the contributing guidelines in CONTRIBUTING.md.
  • Commits are signed off for DCO compliance.
  • New or existing tests cover these changes, or the PR explains why tests are not needed.
  • Documentation is updated for user-facing behavior changes.
  • Generated artifacts are updated when applicable, such as OpenAPI specs, docs screenshots, or Helm/rendered outputs.

Summary by CodeRabbit

  • New Features

    • Added a protected workflow for promoting verified pull-request builds to the test and test01 environments.
    • Added support for promoting immutable images, charts, and deployment state.
    • Added manual rollback and environment release operations.
    • Added configuration validation and safeguards for stale builds, held environments, and invalid deployment data.
  • Documentation

    • Documented promotion, rollback, free-slot procedures, required settings, and operational variables.
  • Bug Fixes

    • Prevented promotion pipelines from unnecessarily running build, test, security scan, and publishing jobs.

@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 82c57055-cb6f-4d91-b5ac-b691a4e98cee

📥 Commits

Reviewing files that changed from the base of the PR and between c4b2daf and dc51bc2.

📒 Files selected for processing (11)
  • .gitlab-ci.yml
  • .gitlab/ci/README.md
  • .gitlab/ci/build.yml
  • .gitlab/ci/promote-test-envs.yml
  • .gitlab/ci/pulse.yml
  • .gitlab/ci/scripts/promote_build_pipeline.sh
  • .gitlab/ci/scripts/promote_push_images.sh
  • .gitlab/ci/scripts/test_env_config.sh
  • .gitlab/ci/scripts/write_deploy_state.sh
  • .gitlab/ci/sonar.yml
  • .gitlab/ci/test.yml
🚧 Files skipped from review as they are similar to previous changes (10)
  • .gitlab-ci.yml
  • .gitlab/ci/sonar.yml
  • .gitlab/ci/test.yml
  • .gitlab/ci/scripts/promote_push_images.sh
  • .gitlab/ci/scripts/test_env_config.sh
  • .gitlab/ci/scripts/promote_build_pipeline.sh
  • .gitlab/ci/README.md
  • .gitlab/ci/build.yml
  • .gitlab/ci/scripts/write_deploy_state.sh
  • .gitlab/ci/promote-test-envs.yml

📝 Walkthrough

Walkthrough

Adds a protected GitLab promotion workflow for test and test01, including PR provenance validation, artifact promotion, chart rendering, deployment-state commits, rollback, and environment release operations. Promotion pipelines skip unrelated build, test, scan, and Sonar jobs.

Changes

Test-environment promotion flow

Layer / File(s) Summary
Promotion pipeline wiring and isolation
.gitlab-ci.yml, .gitlab/ci/promote-test-envs.yml, .gitlab/ci/README.md, .gitlab/ci/{build,pulse,sonar,test}.yml
Adds the protected promotion pipeline, documents required settings and procedures, and skips unrelated jobs for test and test01 promotion pipelines.
Promotion provenance resolution
.gitlab/ci/scripts/promote_build_pipeline.sh
Resolves mirror-vetted PR snapshots, verifies PR and pipeline provenance, reuses or triggers builds, and exports artifact job identifiers.
Image and chart promotion
.gitlab/ci/promote-test-envs.yml, .gitlab/ci/scripts/promote_push_images.sh
Downloads build artifacts, pushes digest-pinned images, validates Helm rendering, and publishes chart targets.
Environment state update
.gitlab/ci/promote-test-envs.yml, .gitlab/ci/scripts/test_env_config.sh, .gitlab/ci/scripts/write_deploy_state.sh
Resolves environment configuration, enforces holds, renders deployment state, and commits it to the downstream environment branch.
Rollback and environment release
.gitlab/ci/promote-test-envs.yml, .gitlab/ci/README.md
Adds manual rollback to historical deployment state and release operations that restore canonical values-repository state.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested reviewers: ryanheffernan, dmathur0

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title is too terse and branch-like, so it doesn't clearly describe the promotion pipeline changes. Rename it to a concise summary such as "Add protected test environment promotion pipeline".
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jojung/promote-test-envs

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.gitlab/ci/build.yml:
- Around line 215-218: Replace each broad NVCM_PROMOTE_ENV-only exclusion with
the full promote predicate used in .gitlab/ci/promote-test-envs.yml: web/api
pipeline source, default branch, and NVCM_PROMOTE_ENV matching test or test01.
Apply this consistently at .gitlab/ci/build.yml lines 215-218, 243-245, and
278-280; .gitlab/ci/pulse.yml lines 37-40; .gitlab/ci/sonar.yml lines 81-84 and
119-121; and .gitlab/ci/test.yml lines 416-419, 466-469, 502-505, and 538-541.

In @.gitlab/ci/promote-test-envs.yml:
- Around line 183-193: Pin the baseline revision across validation and
deployment: in .gitlab/ci/promote-test-envs.yml at lines 183-193, resolve the
commit used by the Helm render and export its SHA as a dotenv value; in
.gitlab/ci/scripts/write_deploy_state.sh at lines 73-76, consume that exported
SHA instead of fetching and recomputing origin/main.
- Around line 206-220: Add the same per-environment resource_group configuration
to test-promote-deploy-state, test-rollback-env, and test-release-env in
.gitlab/ci/promote-test-envs.yml, using $NVCM_PROMOTE_ENV as the shared resource
key so all environment-branch mutations are serialized.
- Around line 135-136: Update the yq download steps in the promote-test-envs job
to use a fixed release URL instead of the floating latest endpoint, then verify
the downloaded binary’s SHA-256 against the expected checksum before chmod or
any execution. Keep chmod and subsequent use only after checksum validation
succeeds.

In @.gitlab/ci/scripts/write_deploy_state.sh:
- Around line 36-47: Normalize values-repository URLs consistently across all
clone operations: in .gitlab/ci/scripts/write_deploy_state.sh lines 36-47, clone
NV_CONFIG_MANAGER_VALUES_REPO_URL directly, while constructing the GitLab URL
only when using NVCM_VALUES_REPO_PATH; apply the same distinction in
.gitlab/ci/promote-test-envs.yml lines 172-177, 254-259, and 314-319 for chart
validation, rollback, and release respectively.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e9a786c0-da4a-4bb5-a804-c989fa62084a

📥 Commits

Reviewing files that changed from the base of the PR and between 73aba7f and d959484.

📒 Files selected for processing (15)
  • .gitlab-ci.yml
  • .gitlab/ci/README.md
  • .gitlab/ci/build.yml
  • .gitlab/ci/common.yml
  • .gitlab/ci/pr-build.yml
  • .gitlab/ci/promote-test-envs.yml
  • .gitlab/ci/pulse.yml
  • .gitlab/ci/scripts/pr_ref_guard.sh
  • .gitlab/ci/scripts/promote_build_pipeline.sh
  • .gitlab/ci/scripts/promote_push_images.sh
  • .gitlab/ci/scripts/test_env_config.sh
  • .gitlab/ci/scripts/write_deploy_state.sh
  • .gitlab/ci/sonar.yml
  • .gitlab/ci/test.yml
  • Makefile

Comment thread .gitlab/ci/build.yml
Comment thread .gitlab/ci/promote-test-envs.yml Outdated
Comment thread .gitlab/ci/promote-test-envs.yml
Comment thread .gitlab/ci/promote-test-envs.yml
Comment thread .gitlab/ci/scripts/write_deploy_state.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.gitlab/ci/scripts/write_deploy_state.sh (1)

49-59: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Do not log the raw values-repository URL.

NV_CONFIG_MANAGER_VALUES_REPO_URL may contain authentication credentials, but values_repo is printed in the commit/error messages and embedded in the final view URL. This can expose tokens in CI logs and produces a malformed view URL for full-URL overrides. Use a sanitized display label, and construct the view URL only from the repository path when available.

Also applies to: 139-139

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.gitlab/ci/scripts/write_deploy_state.sh around lines 49 - 59, Sanitize the
repository identifier used by the deploy-state logging and final view URL
construction. Update the relevant commit/error messages to use a credential-free
display label instead of raw values_repo or NV_CONFIG_MANAGER_VALUES_REPO_URL,
and derive the view URL from the repository path only when available so full-URL
overrides cannot embed credentials or produce malformed URLs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.gitlab/ci/scripts/write_deploy_state.sh:
- Around line 49-59: Sanitize the repository identifier used by the deploy-state
logging and final view URL construction. Update the relevant commit/error
messages to use a credential-free display label instead of raw values_repo or
NV_CONFIG_MANAGER_VALUES_REPO_URL, and derive the view URL from the repository
path only when available so full-URL overrides cannot embed credentials or
produce malformed URLs.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: eff0741c-d9e4-4719-90c0-d1d3c0cb04f0

📥 Commits

Reviewing files that changed from the base of the PR and between d959484 and 89d04aa.

📒 Files selected for processing (6)
  • .gitlab/ci/build.yml
  • .gitlab/ci/promote-test-envs.yml
  • .gitlab/ci/pulse.yml
  • .gitlab/ci/scripts/write_deploy_state.sh
  • .gitlab/ci/sonar.yml
  • .gitlab/ci/test.yml
🚧 Files skipped from review as they are similar to previous changes (5)
  • .gitlab/ci/sonar.yml
  • .gitlab/ci/pulse.yml
  • .gitlab/ci/build.yml
  • .gitlab/ci/test.yml
  • .gitlab/ci/promote-test-envs.yml

@jojung1
jojung1 force-pushed the jojung/promote-test-envs branch 2 times, most recently from 354a47e to c4b2daf Compare July 21, 2026 14:22
jojung1 added 5 commits July 24, 2026 17:25
Signed-off-by: Joseph Jung <jojung@nvidia.com>
Signed-off-by: Joseph Jung <jojung@nvidia.com>
Signed-off-by: Joseph Jung <jojung@nvidia.com>
Signed-off-by: Joseph Jung <jojung@nvidia.com>
…mages

Signed-off-by: Joseph Jung <jojung@nvidia.com>
@jojung1
jojung1 force-pushed the jojung/promote-test-envs branch from c4b2daf to dc51bc2 Compare July 24, 2026 21:28
@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant