Skip to content

test(integration): de-flake otel pre-job config test #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cbartz wants to merge 2 commits into
base: main
Choose a base branch
from
+26 −11
Open

test(integration): de-flake otel pre-job config test #803

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ffc30a5
test(integration): de-flake otel pre-job config test
cbartz Jun 29, 2026
75567c1
Merge branch 'main' into fix/deflake-otel-pre-job-test
cbartz Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 26 additions & 11 deletions tests/integration/test_charm_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,21 +216,36 @@ def test_otel_collector_endpoint_pre_job_installs_config(
)
wait_for_runner_ready(juju, app)

dispatch_workflow(
# Dispatch the long-running wait workflow (instead of a quick one) so the runner stays
# busy while we inspect it. An ephemeral runner's VM is torn down on job completion, which
# races the inspection below; keeping the job in progress guarantees the VM is still alive.
workflow = dispatch_workflow(
app_name=app,
branch=test_github_branch,
github_repository=github_repository,
conclusion="success",
workflow_id_or_name=DISPATCH_TEST_WORKFLOW_FILENAME,
dispatch_input={"runner": app},
workflow_id_or_name=DISPATCH_WAIT_TEST_WORKFLOW_FILENAME,
dispatch_input={"runner": app, "minutes": "5"},
wait=False,
)
wait_for(lambda: workflow.update() or workflow.status == "in_progress")

@yhaliaw yhaliaw Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An AI has flagged the follwoing, which seems reasonable:

The lambda: workflow.update() or workflow.status == "in_progress" expression does not reliably wait for the run to reach in_progress. PyGithub's CompletableGithubObject.update() returns True whenever it performs a successful HTTP GET (200 OK) and False only on a 304 Not Modified. Because update() is the left operand of or, any 200 response short-circuits the expression before status is ever consulted. On the first loop iteration the run object was returned from a list call and has no stored etag, so the first GET is unconditional — it returns True immediately, regardless of what status actually is.

The guard therefore degrades to "refresh once and continue" rather than "wait until the job is running." This undercuts the invariant that lines 219–221 rely on — that the VM is guaranteed to be alive when the inspection begins. The downstream _read_otel_config poll (120 s) masks this in typical conditions, but the structural guarantee the PR intends is lost and the effective startup budget is reduced to a single wait.

The fix is to call update() for its side effect and evaluate status in a separate step:

def _in_progress() -> bool:
    workflow.update()
    return workflow.status == "in_progress"

 wait_for(_in_progress, timeout=300, check_interval=10)


exit_code, stdout, stderr = instance_helper.run_in_instance(
unit_name=f"{app}/0",
command="sudo cat /etc/otelcol/config.d/github.yaml",
)
def _read_otel_config() -> str:
"""Read the otel config written by the pre-job script, retrying until it exists.

The pre-job hook may not have finished writing the file the moment the job is reported
in progress, so callers poll on a truthy (non-empty) return value.

Returns:
The otel config file content, or an empty string if not yet written.
"""
exit_code, stdout, _ = instance_helper.run_in_instance(
unit_name=f"{app}/0",
command="sudo cat /etc/otelcol/config.d/github.yaml",
)
return stdout if exit_code == 0 and stdout else ""

config = wait_for(_read_otel_config, timeout=120, check_interval=10)

assert exit_code == 0, stderr
assert stdout is not None
assert "exporters:" in stdout
assert f"endpoint: {endpoint}" in stdout
assert "exporters:" in config
assert f"endpoint: {endpoint}" in config
Loading