test: fix flaky CI failures for the arc-geneva-poc branch#1271
Open
ryanzhang-oss wants to merge 1 commit intoAzure:sehobbs/fleet-arc-geneva-pocfrom
Open
test: fix flaky CI failures for the arc-geneva-poc branch#1271ryanzhang-oss wants to merge 1 commit intoAzure:sehobbs/fleet-arc-geneva-pocfrom
ryanzhang-oss wants to merge 1 commit intoAzure:sehobbs/fleet-arc-geneva-pocfrom
Conversation
Two separate CI flakiness fixes: 1. pkg/controllers/workapplier/suite_test.go: Increase GracefulShutdownTimeout from the default 30s to 2 minutes for all four controller managers in the integration test suite. With four managers running concurrently (each with multiple controllers), the default 30s grace period is insufficient to drain all runnables on a loaded CI runner, causing AfterSuite teardown to fail with 'context deadline exceeded' even though all 290 specs pass. 2. test/e2e/utils_test.go: Widen the per-CPU-core and per-GB-memory cost property tolerance from 0.002 to 0.005. The Azure Retail Prices API can return values that differ from the locally-computed expected value by exactly 0.002 (e.g. got=0.141, want=0.143), which hits the strict boundary of the original threshold and causes BeforeSuite to fail, aborting the entire custom e2e suite. A margin of 0.005 provides sufficient headroom while still catching genuine property provider bugs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Applies the two CI flakiness fixes from #1270 onto the
sehobbs/fleet-arc-geneva-pocbranch so that PR #1269 CI passes.Fixes included
1. e2e cost property tolerance (
test/e2e/utils_test.go)The
e2e-tests (custom)BeforeSuite was failing with:The diff of exactly
0.002hit the strict> 0.002boundary. Widened tolerance to0.005to absorb Azure Retail Prices API fluctuations.2. workapplier AfterSuite teardown timeout (
pkg/controllers/workapplier/suite_test.go)All 290 specs pass but AfterSuite fails with:
Four concurrent managers can't drain within 30s on a loaded CI runner. Set
GracefulShutdownTimeout: 2*time.Minuteon all 4 managers.cc @Ealianis — this should unblock the CI on #1269.