[python] implement /flush endpoint in python weblogs

[smola]

Motivation

Faster run, better guarantees for telemetry arrival.

Changes

Add GET /flush to flask, fastapi, django, and tornado weblogs. The endpoint calls tracer.flush() and telemetry_writer.periodic(force_flush=True) to force all pending data to be sent to the agent.

Enable python in the flush() call in containers.py, and set library_interface_timeout to 0 for python (alongside nodejs and ruby) now that explicit flushing is supported.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/build/docker/python/django/app/urls.py                            @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/fastapi/main.py                               @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/flask/app.py                                  @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core
utils/build/docker/python/tornado/main.py                               @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core

[datadog-official]

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 132 Tests failed

tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedHeader_ExtendedLocation.test_extended_location_data[flask-poc] from system_tests_suite     (Fix with Cursor)

AssertionError: Expected a single vulnerability with the matching criteria
assert 2 == 1
 +  where 2 = len([{'evidence': {'valueParts': [{'source': 0, 'value': 'http://dummy.location.com'}]}, 'hash': 1185812129, 'location': {...ted_redirect_insecure_header', 'path': 'app.py', 'spanId': 11628164113005246748, ...}, 'type': 'UNVALIDATED_REDIRECT'}])

self = <tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedHeader_ExtendedLocation object at 0x7fb6483b85c0>

    def test_extended_location_data(self):
>       validate_extended_location_data(self.r, self.vulnerability_type)

tests/appsec/iast/sink/test_unvalidated_redirect.py:126: 
...

tests.appsec.rasp.test_api10.Test_API10_downstream_ssrf_telemetry.test_api10_req[django-poc] from system_tests_suite     (Fix with Cursor)

AssertionError: assert False
 +  where False = validate_metric_variant_v2_exists('rasp.rule.match', 'ssrf', 'response', [{'_computed_namespace': 'appsec', 'common': True, 'metric': 'rasp.rule.match', 'points': [[1776764919, 3.0]], ...}, {'_computed_namespace': 'appsec', 'common': True, 'metric': 'rasp.rule.match', 'points': [[1776764919, 2.0]], ...}])

self = <tests.appsec.rasp.test_api10.Test_API10_downstream_ssrf_telemetry object at 0x7f968523ce30>

    def test_api10_req(self):
        series_eval = find_series("appsec", "rasp.rule.eval", is_metrics=True)
        assert series_eval
        assert validate_metric_variant_v2_exists("rasp.rule.eval", "ssrf", "request", series_eval)
        assert validate_metric_variant_v2_exists("rasp.rule.eval", "ssrf", "response", series_eval)
...

tests.appsec.rasp.test_api10.Test_API10_downstream_ssrf_telemetry.test_api10_req[flask-poc] from system_tests_suite     (Fix with Cursor)

assert []

self = <tests.appsec.rasp.test_api10.Test_API10_downstream_ssrf_telemetry object at 0x7f3d07e10c50>

    def test_api10_req(self):
        series_eval = find_series("appsec", "rasp.rule.eval", is_metrics=True)
>       assert series_eval
E       assert []

tests/appsec/rasp/test_api10.py:270: AssertionError

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4bc8a12 | Docs | Datadog PR Page | Give us feedback!}