Skip to content

Comments

[system_tests] Fix flaky test_asm_standalone.py tests#6377

Open
vlad-scherbich wants to merge 4 commits intomainfrom
vlad/unflake_test_asm_standalone
Open

[system_tests] Fix flaky test_asm_standalone.py tests#6377
vlad-scherbich wants to merge 4 commits intomainfrom
vlad/unflake_test_asm_standalone

Conversation

@vlad-scherbich
Copy link
Contributor

@vlad-scherbich vlad-scherbich commented Feb 24, 2026

https://datadoghq.atlassian.net/browse/PROF-13796

Motivation

Out of the latest 100 flaked system-test CI runs on dd-trace-py main, 75 are dynamic configuration tests (58% of total). See JIRA for details.

Similar motivation as #6371. Test suite in question: test_asm_standalone.py::Test_IastStandalone_UpstreamPropagation_V2::test_no_appsec_upstream__no_asm_event__is_kept_with_priority_1__from_0

These tests are flaky because the trace sometimes wasn't flushed to the agent before the scenario stopped. The tracer sends traces asynchronously; without a short delay, the trace could still be buffered when containers shut down.

Changes

  • Add a deterministic poll loop (same approach as the dynamic config fix): poll interfaces.library.get_spans(request) every 0.5s until the trace appears (5s timeout). Exits as soon as the trace is captured, or fails with a clear error on timeout.
  • Add _iast_standalone_wait_for_trace(request) helper and apply it in setup for all 9 flaky IAST tests in Test_IastStandalone_UpstreamPropagation_V2. Scoped to iast_standalone only so appsec_standalone (e.g. PHP) is unchanged.

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Contributor

CODEOWNERS have been resolved as:

tests/appsec/test_asm_standalone.py                                     @DataDog/asm-libraries @DataDog/system-tests-core

@datadog-datadog-prod-us1-2
Copy link

datadog-datadog-prod-us1-2 bot commented Feb 24, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8f85efa | Docs | Datadog PR Page | Was this helpful? Give us feedback!

@vlad-scherbich vlad-scherbich changed the title attempt to fix flaky test_asm_standalone.py tests [system_tests] Fix flaky test_asm_standalone.py tests Feb 24, 2026
@vlad-scherbich vlad-scherbich marked this pull request as ready for review February 25, 2026 01:20
@vlad-scherbich vlad-scherbich requested review from a team as code owners February 25, 2026 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant