unflake a System Test: 'test_remote_sampling_rules_retention'

[vlad-scherbich]

https://datadoghq.atlassian.net/browse/PROF-13796

Motivation

test_remote_sampling_rules_retention fails intermittently in CI. Observed flaking in a dd-trace-py PR that only changes CI workflow config (zero relation to test logic). See failed job.

The root cause is a race condition in set_and_wait_rc: its wait signals (telemetry event + RC ACKNOWLEDGED state) are not config-version-specific, so stale events from a prior RC config can satisfy them before the new sampling rules are actually active in the library.

Changes

Adds a retry loop around the first trace assertion in test_remote_sampling_rules_retention to tolerate the brief propagation window between RC acknowledgment and actual rule application. This is consistent with how other tests in the same file handle similar timing sensitivity (e.g., get_sampled_trace).

Testing

$ ./run.sh PARAMETRIC -L python -k "retention"
...
============================================================================= test context ==============================================================================
Scenario: PARAMETRIC
Logs folder: ./logs_parametric
Library: python@4.4.0
========================================================================== test session starts ==========================================================================
gw0 [1] / gw1 [1] / gw2 [1] / gw3 [1] / gw4 [1] / gw5 [1] / gw6 [1] / gw7 [1] / gw8 [1] / gw9 [1] / gw10 [1] / gw11 [1] / gw12 [1] / gw13 [1] / gw14 [1] / gw15 [1]
.                                                                                                                                                                 [100%]
--------------------------- generated xml file: /Users/vlad.scherbich/go/src/github.com/DataDog/system-tests/logs_parametric/reportJunit.xml ----------------------------
========================================================================== 1 passed in 32.80s ===========================================================================
COMP-LR7JK0FKW1:system-tests vlad.scherbich$ 

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_dynamic_configuration.py                          @DataDog/system-tests-core @DataDog/apm-sdk-capabilities

[KowalskiThomas]

If I understand correctly, we try up to 30 times (waiting 0.1 seconds between each attempt) to see a span whose metadata shows that the new sampling rules were correctly received and applied?

[vlad-scherbich]

Exactly!

[cbeauchesne]

Would it be possible to improve the wait_rc condition to wait for the precise new config ?

[vlad-scherbich]

@cbeauchesne , I like this idea.
set_and_wait_rc is just a helper function used only in test_dynamic_configuration.py at 20 call sites.

I'll open a separate PR for the proposed fix, which would be to clear the session before setting the new RC config.

[vlad-scherbich]

@cbeauchesne , new PR implementing your suggestion: #6349

Lots of unrelated system test fails, do I just re-run them until they pass?

[vlad-scherbich]

@cbeauchesne I tried to make this work, but seems like #6349 has broken .Net and potentially PHP and Ruby parametric tests. It seems the general fix will be more involved than I thought.

Should we just merge this to fix flaky Python tests for now, or do you have some pointers on how to make the generalized fix work for all?

[datadog-datadog-prod-us1]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: be1f7fd | Docs | Datadog PR Page | Was this helpful? Give us feedback!}