[BUG] `BUILDKITE_COMMIT` remains "HEAD" in build steps despite `resolve-commit-after-checkout` experiment

[aas47]

Describe the bug

When using the resolve-commit-after-checkout experimental feature, the Buildkite agent logs indicate that BUILDKITE_COMMIT is being updated from "HEAD" to the resolved commit hash, but the environment variable is not actually available to build steps, which continue to see "HEAD".

Steps to Reproduce

1. Deploy Buildkite Kubernetes stack version 0.35.0 with the following configuration:

config:
  agent-config:
    experiment:
      - "resolve-commit-after-checkout"

2. Run a build with the following pipeline steps:

steps:
  - label: testing
    command:
      - echo ${BUILDKITE_COMMIT}
      - echo $${BUILDKITE_COMMIT}

Expected behavior

The build steps should output the resolved commit hash instead of "HEAD".

Environment

• agent-stack-k8s version: 0.35.0
• Kubernetes version: 1.34.1
• Deployment method: Helm chart

Logs

# Using resolve-commit-after-checkout experiment 🧪
# Updating BUILDKITE_COMMIT from "HEAD" to "050ada53c89d97ff1b6fd77a754155f5115d2a51"
# Checking to see if git commit information needs to be sent to Buildkite...
$ /usr/local/bin/buildkite-agent meta-data exists buildkite:git:commit
# Sending Git commit information back to Buildkite
$ /usr/local/bin/buildkite-agent meta-data set buildkite:git:commit < /dev/stdin

Additional context

The agent appears to be resolving the commit hash correctly (as evidenced by the log output), but the environment variable update is not being propagated to the build step execution environment. This suggests the variable is being set in a context that is not accessible to the build steps.

Screenshots

resolve-commit-after-checkout working

command output

[zhming0]

Hi @aas47, thank you very much for raising this issue to us. As you are aware after chatting with my colleagues, this is a limitation caused by the fact that checkout phase and command phase lives in different containers within the same pod.

(Context: the reason why they are separated is so that people can use arbitrary container image in command step, e.g. vanilla golang:alpine)

This category issues are known to us, we are still in planning mode weighing various solution's pros and cons, we welcome your suggestion too!

In the meantime, is running export BUILDKITE_COMMIT=$(git rev-parse $BUILDKITE_COMMIT) in command, or pre-command hook a viable workaround for your use case? We understand the implication that it means git needs to present in the command image.

[aas47]

Thanks for the response, @zhming0.
The workaround is fine for now. The main downside is having to install git in the command image, but I can work around that for the time being.

[aas47]

@zhming0 any update on this issue?