Skip to content

Describe steps for safely releasing provision script changes #2384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
djeebus wants to merge 2 commits into
base: main
Choose a base branch
from
+18 −0
Open

Describe steps for safely releasing provision script changes #2384

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
677bcff
describe steps for releasing provision script changes
djeebus Apr 13, 2026
1da7437
Update documentation to be more accurate, less judgy
djeebus Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions packages/orchestrator/pkg/template/build/phases/base/provision.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
The provison.sh script file sets up any base dependencies required for running envd and sdk commands.

Check warning on line 1 in packages/orchestrator/pkg/template/build/phases/base/provision.md

View check run for this annotation

Claude / Claude Code Review

Typo in opening sentence: 'provison.sh' should be 'provision.sh' 

Line 1 of provision.md contains a typo: 'provison.sh' is missing the second 'i' and should be 'provision.sh'. This is a minor documentation inconsistency since the rest of the file (lines 5, 15) spells it correctly.
Copy link
Copy Markdown

@claude claude bot Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Line 1 of provision.md contains a typo: 'provison.sh' is missing the second 'i' and should be 'provision.sh'. This is a minor documentation inconsistency since the rest of the file (lines 5, 15) spells it correctly.

Extended reasoning...

What the bug is: The very first sentence of the newly added provision.md file contains a misspelling: 'The provison.sh script file...' — the word 'provison' is missing the second 'i', making it different from the actual filename 'provision.sh'.

The specific code path: Line 1 of the diff reads: 'The provison.sh script file sets up any base dependencies required for running envd and sdk commands.' The misspelling exists only in this opening sentence.

Why existing code doesn't prevent it: This is a documentation file with no automated spell-checking or linting in place to catch typos. The rest of the document consistently uses the correct spelling 'provision.sh' (lines 5, 7, 8, 15), which confirms the opening sentence is a one-off typo rather than a systemic naming issue.

Impact: The impact is minimal — this is documentation-only and has no runtime effect. However, a misspelling in the very first sentence can cause confusion for readers trying to locate or reference the actual script file named 'provision.sh'.

How to fix it: Change 'provison.sh' to 'provision.sh' on line 1.

Step-by-step proof:

  1. Open the new file at packages/orchestrator/pkg/template/build/phases/base/provision.md
  2. Read line 1: 'The provison.sh script file sets up any base dependencies...'
  3. Count the letters: p-r-o-v-i-s-o-n → 'provison' has no second 'i', but the correct name is p-r-o-v-i-s-i-o-n → 'provision'
  4. Compare with line 5: 'The provision.sh script is executed immediately...' — correctly spelled
  5. Compare with line 15: 'Deploy template managers with the new version of provision.sh.' — correctly spelled
  6. Conclusion: line 1 is a typo that contradicts the rest of the document and the actual filename.


# change rollout process

The provision.sh script is executed immediately after the docker image is pulled from the registry, and before running
any other build commands, and the result is cached. As such, it is most likely cached for any future builds. The caching
means that the provision.sh script will most likely not be executed if the template is rebuilt.

There are two considerations to be made when rolling out new versions of the template:
- We don't want to overwhelm the template managers by invalidating all cache at once.
- We want to communicate to the end users that, in order to take advantage of new features, customers need to rebuild their templates.

Current process:

1. Deploy template managers with the new version of `provision.sh`.
2. Increment the `build-provision-version` environment variable in the template manager deployment.

Check failure on line 16 in packages/orchestrator/pkg/template/build/phases/base/provision.md

View check run for this annotation

Claude / Claude Code Review

build-provision-version is a LaunchDarkly feature flag, not an environment variable 

Step 2 of the rollout guide incorrectly describes `build-provision-version` as an environment variable in the template manager deployment, but it is actually a LaunchDarkly feature flag — updating it requires the LaunchDarkly dashboard, not a deployment manifest change. This distinction matters operationally: env var changes require pod restarts, while feature flag changes take effect immediately and support per-template/per-team targeting for gradual rollout.
Copy link
Copy Markdown

@claude claude bot Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Step 2 of the rollout guide incorrectly describes build-provision-version as an environment variable in the template manager deployment, but it is actually a LaunchDarkly feature flag — updating it requires the LaunchDarkly dashboard, not a deployment manifest change. This distinction matters operationally: env var changes require pod restarts, while feature flag changes take effect immediately and support per-template/per-team targeting for gradual rollout.

Extended reasoning...

What the bug is

Line 16 of provision.md instructs: "Increment the build-provision-version environment variable in the template manager deployment." This is factually incorrect. build-provision-version is a LaunchDarkly integer feature flag, not a deployment environment variable.

The code path

In packages/shared/pkg/featureflags/flags.go:166, the flag is declared as:

BuildProvisionVersion = newIntFlag("build-provision-version", 0)

In packages/orchestrator/pkg/template/build/phases/base/hash.go:35-43, it is read via the LaunchDarkly client:

bb.featureFlags.IntFlag(ctx, featureflags.BuildProvisionVersion, ...)

with per-template and per-team targeting contexts passed in. There is no environment variable named build-provision-version anywhere in the codebase.

Why existing docs don't prevent this

The document is newly introduced in this PR, so there is no prior documentation to catch the error. The flag name closely resembles what an env var might be called, which makes the mistake easy to introduce.

Impact

An operator following this runbook during a live rollout would search the deployment YAML or Helm charts for an env var that does not exist. They would be unable to execute step 2, blocking the rollout. Even if they eventually discovered the correct mechanism, the delay could mean either missed coordinated rollout timing or incorrect configuration methods being applied (e.g., accidentally adding a spurious env var to the deployment).

Additionally, the document misses an important capability: because this is a LaunchDarkly feature flag with per-template and per-team targeting, operators can perform a gradual rollout to a subset of templates or teams — which directly addresses the cache stampede concern raised by the document itself. By calling it an env var, this capability is obscured from the reader.

How to fix

Replace line 16 with accurate guidance, e.g.: "Increment the build-provision-version LaunchDarkly feature flag via the LaunchDarkly dashboard. Because this flag supports per-template and per-team targeting, you can roll out the cache invalidation gradually to avoid overwhelming template managers."

Step-by-step proof

  1. An operator reads step 2: "Increment the build-provision-version environment variable in the template manager deployment."
  2. They open the template manager Kubernetes deployment manifest or Helm values file and search for build-provision-version.
  3. They find no such env var — because it does not exist as an env var anywhere in the deployment configuration.
  4. They are stuck and must escalate or investigate the codebase independently.
  5. Eventually they find packages/shared/pkg/featureflags/flags.go:166 and hash.go:35-43, realizing the value must be updated in LaunchDarkly — a completely separate system requiring different tooling and access.

Note on duplicate reports

Multiple verifiers flagged this as a duplicate of other bugs (bug_003/bug_006). The synthesis agent has canonicalized these into merged_bug_001. The underlying issue is confirmed by all verifiers: the documentation is factually wrong about the mechanism for this flag.

3. Bump the envd version number and release it. Gate the release of new envd/sdk features with the new envd version.
4. (optional) Update the dashboard to alert users to rebuild their templates to take advantage of the new features.
Loading