fix(opentelemetry): prevent OTLP logs self-export loop#19266
Open
bm1549 wants to merge 2 commits into
Open
Conversation
…r's own logs When DD_LOGS_OTEL_ENABLED=true, the OpenTelemetry SDK attaches a LoggingHandler to the Python root logger that captures every propagated log record. Because ddtrace's internal loggers (ddtrace.*) and the OpenTelemetry exporter's loggers (opentelemetry.*) propagate to root, the exporter's own log records -- the per-batch export debug line and the exporter's export-failure warnings/errors -- were captured, exported, and captured again, producing a self-amplifying export loop (triggered by DD_TRACE_DEBUG or by export failures at the default log level). Attach a logging.Filter to the OTLP LoggingHandler that rejects records whose logger name is in the ddtrace or opentelemetry namespaces, so the tracer never feeds its own telemetry-pipeline logs back into its own log export. Application logs are still captured and exported. The filter is attached only to the handler ddtrace itself installs (identified by diffing the root logger's handlers), so application-configured handlers are left untouched. Because it binds to the handler instance, it also persists across logging.basicConfig reconfiguration, which the OpenTelemetry SDK patches to re-add that same handler instance (dictConfig/fileConfig are not patched by the SDK). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
added
the
AI Generated
This was referenced Jul 24, 2026
Contributor
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 1 job - 1 passed on retry 🔗 Commit SHA: b35341a | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-07-24 02:13:52 Comparing candidate commit 31b6da5 in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics. scenario:iastaspects-lstrip_aspect
scenario:iastaspects-translate_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:telemetryaddmetric-1-count-metric-1-times
|
…logs-self-telemetry-loop
Circular import analysis
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When
DD_LOGS_OTEL_ENABLED=true, the OTLP logs pipeline attaches an OpenTelemetryLoggingHandlerto the Python root logger. ddtrace's own loggers (ddtrace.*) and the OpenTelemetry exporter's loggers (opentelemetry.*) propagate to root, so the handler captured the tracer's own log records: the per-export debug line and the exporter's export-failure warnings/errors. Exporting those records emits more log records, which were captured and exported again. The result is a self-amplifying export loop. This happens whenever those records reach the effective log level: withDD_TRACE_DEBUG=true(the per-export debug line), or on export failures (warnings/errors at the default level).The fix attaches a self-telemetry exclusion filter to the OTLP
LoggingHandlerthat drops records whose logger name is in theddtraceoropentelemetrynamespaces. Matching is exact-or-dotted-prefix (ddtrace/ddtrace.*,opentelemetry/opentelemetry.*), so application loggers such asddtrace_myappare not affected. The filter is attached only to the handler ddtrace itself installs; application-configured handlers are not modified. Application logs are still captured and exported.This was discovered during cross-tracer startup-log work. It is an independent, pre-existing bug, so it ships on its own branch.
Testing
Adds
tests/opentelemetry/test_logs.py::test_otel_logs_exporter_excludes_self_telemetry. WithDD_LOGS_OTEL_ENABLED=true, it emits addtrace.*record, anopentelemetry.*record, and an application record, flushes the exporter to a mock gRPC log service, and asserts the application record is exported while the two self-telemetry records are not. The test fails without the fix and passes with it. It runs across the OpenTelemetry versions already covered by theopentelemetrytest venv.Risks
Low. The change only adds a logging filter to the handler ddtrace creates. Records from the
ddtraceandopentelemetrylogger namespaces are no longer exported through this pipeline; application logs are unchanged. The filter binds to the handler instance, so it persists acrosslogging.basicConfigreconfiguration, which is the reconfiguration API the OpenTelemetry SDK patches to re-add the same handler instance (dictConfig/fileConfigare not patched by the SDK).Additional Notes
Includes a reno release note. No configuration or public API changes.
Context
Independent, pre-existing bug surfaced while adding OTLP export-status fields to the tracer startup log across dd-trace-* (the Python system-test for the logs field hung on this loop). Related: