[Tracing] Implement initial OTLP traces weblog tests by zacharycmontoya · Pull Request #6363 · DataDog/system-tests

zacharycmontoya · 2026-02-20T22:46:45Z

Motivation

We are seeing increased demand for exporting traces as OTLP from our DD SDKs (rather than Datadog-proprietary MessagePack), so we are prototyping and establishing requirements for generating OTLP traces payloads. This is only the first of a series of PRs to establish clear expectations for what the generated OTLP traces and trace stats will look like.

Changes

Scenario: Adds a new APM_TRACING_OTLP scenario to test the weblog application with the configuration needed for the DD SDK to export traces using OTLP. This also adds a include_opentelemetry property to the EndToEndScenario to set up the OpenTelemetry interface.
Tests: Adds tests/otel/test_tracing_otlp.py::Test_Otel_Tracing_OTLP::test_tracing to send a request to the weblog app using weblog.get("/") and asserts properties of the OTLP trace payload
Interfaces: Updates the OpenTelemetry interface with methods get_otel_spans and get_trace_stats to retrieve the OTLP payloads for test assertions

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…ple. Notably, this creates a new scenario APM_TRACING_OTLP to enable the environment variables needed to configure the SDK to export traces as OTLP.

…is yet

…e only use the "/v1/metrics" subpath

github-actions · 2026-02-20T22:47:11Z

CODEOWNERS have been resolved as:

tests/otel/test_tracing_otlp.py                                         @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/interfaces/_open_telemetry.py                                     @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core

datadog-official · 2026-02-20T23:15:31Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 7 New flaky tests detected

tests.appsec.test_request_blocking.Test_AppSecRequestBlocking.test_request_blocking[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_request_blocking.Test_AppSecRequestBlocking object at 0x7f6608b4a570>

    def test_request_blocking(self):
        """Test requests are blocked by rules in blocking mode"""
    
        assert self.config_state.state == remote_config.ApplyState.ACKNOWLEDGED
    
        assert self.blocked_requests1.status_code == 403
...

tests.appsec.test_traces.Test_AppSecObfuscator.test_appsec_obfuscator_key[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_traces.Test_AppSecObfuscator object at 0x7f6608815250>

    @missing_feature(
        context.library < "nodejs@5.57.0" and context.weblog_variant == "fastify",
        reason="Query string not supported yet",
    )
    def test_appsec_obfuscator_key(self):
        """General obfuscation test of several attacks on several rule addresses."""
...

tests.appsec.test_traces.Test_AppSecObfuscator.test_appsec_obfuscator_value[haproxy-spoa] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: No appsec event validate this condition

self = <tests.appsec.test_traces.Test_AppSecObfuscator object at 0x7f6608814dd0>

    @missing_feature(
        context.library < "nodejs@5.57.0" and context.weblog_variant == "fastify",
        reason="Query string not supported yet",
    )
    def test_appsec_obfuscator_value(self):
        """Obfuscation test of a matching rule parameter value containing a sensitive keyword."""
...

View all

🧪 1 Test failed

tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP.test_tracing[nginx] from system_tests_suite (Datadog) (Fix with Cursor)

assert 0 == 1
 +  where 0 = len([])

self = <tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP object at 0x7f701fb7b860>

    def test_tracing(self):
        data = list(interfaces.open_telemetry.get_otel_spans(self.req))
>       assert len(data) == 1
E       assert 0 == 1
E        +  where 0 = len([])
...

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 86b284d | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

ida613 · 2026-02-24T20:36:51Z

tests/otel/test_tracing_otlp.py

+
+        # Assert spans
+        assert span.get("name") == "GET /"
+        assert span.get("kind") == "SPAN_KIND_SERVER"


should this one be 2?

Ah yes, for JSON this should be 2, but I don't know what this will end up being for http/protobuf. We can update it to 2 for now and then modify it when we get to exporting http/protobuf

ida613 · 2026-02-24T20:53:38Z

tests/otel/test_tracing_otlp.py

+        assert "telemetry.sdk.version" in attributes
+        assert "git.commit.sha" in attributes
+        assert "git.repository_url" in attributes
+        assert "runtime-id" in attributes


are these attributes from config.tags which are sent in the header?

These are an example of attributes that we set on all spans in .NET, so I think it makes sense for us to represent some of them as resource attributes rather than individual span tags

ida613 · 2026-02-24T21:04:27Z

tests/otel/test_tracing_otlp.py

+        assert "runtime-id" in attributes
+
+        # Assert spans
+        assert span.get("name") == "GET /"


where does this one come from? for name I used span.name and got 'express.request'

I hardcoded this specifically for the .NET test app, but perhaps we can make a dictionary of language->expected name

zacharycmontoya added 5 commits February 20, 2026 14:29

Implement initial OTLP traces + trace stats with the .NET weblog exam…

dd04bd9

…ple. Notably, this creates a new scenario APM_TRACING_OTLP to enable the environment variables needed to configure the SDK to export traces as OTLP.

Skip assertions for the Trace Metrics, as we don't have a spec for th…

ebbbb5a

…is yet

Run formatter

7fee5a2

Remove "/api/v0.2/stats" subpath for the OpenTelemetry interface so w…

5f8b2a8

…e only use the "/v1/metrics" subpath

Add manifest entries for languages

221b031

Merge branch 'main' into zach.montoya/weblog-traces-otlp

86b284d

ida613 reviewed Feb 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[Tracing] Implement initial OTLP traces weblog tests#6363

[Tracing] Implement initial OTLP traces weblog tests#6363
zacharycmontoya wants to merge 6 commits intomainfrom
zach.montoya/weblog-traces-otlp

zacharycmontoya commented Feb 20, 2026

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

datadog-official bot commented Feb 20, 2026 •

edited

Loading

Uh oh!

ida613 Feb 24, 2026

Uh oh!

zacharycmontoya Feb 25, 2026

Uh oh!

ida613 Feb 24, 2026

Uh oh!

zacharycmontoya Feb 25, 2026

Uh oh!

ida613 Feb 24, 2026

Uh oh!

zacharycmontoya Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

zacharycmontoya commented Feb 20, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

datadog-official bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

Uh oh!

ida613 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

ida613 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

ida613 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-official bot commented Feb 20, 2026 •

edited

Loading