Skip to content

Fix concurrent span event recording in OTel shim#12053

Open
Meemaw wants to merge 1 commit into
DataDog:masterfrom
Meemaw:fix-otel-span-events-concurrency
Open

Fix concurrent span event recording in OTel shim#12053
Meemaw wants to merge 1 commit into
DataDog:masterfrom
Meemaw:fix-otel-span-events-concurrency

Conversation

@Meemaw

@Meemaw Meemaw commented Jul 23, 2026

Copy link
Copy Markdown

What Does This Do

Makes span-event handling in the OpenTelemetry shim (OtelSpan) thread-safe.

Span events were stored in a plain ArrayList that addEvent() / recordException() mutated without synchronization, and that onSpanFinished() iterated at span finish. When events are recorded from multiple threads (e.g. GraphQL DataLoaders running on virtual threads) while the span is being finished on another thread, the backing list gets corrupted, leaving a null hole that triggers a NullPointerException in OtelSpanEvent.toTag:

java.lang.NullPointerException: Cannot invoke
  "datadog.opentelemetry.shim.trace.OtelSpanEvent.toJson()" because "event" is null
    at datadog.opentelemetry.shim.trace.OtelSpanEvent.toTag(OtelSpanEvent.java:53)
    at datadog.opentelemetry.shim.trace.OtelConventions.setEventsAsTag(OtelConventions.java:143)
    at datadog.opentelemetry.shim.trace.OtelSpan.onSpanFinished(OtelSpan.java:182)
    at datadog.trace.agent.core.DDSpan.finishAndAddToTrace(DDSpan.java:162)

That NPE escapes span finish and can surface to the application as a request failure (observed in production as a jakarta.servlet.ServletException → HTTP 500 on a request whose spans were otherwise healthy).

Motivation

A tracer-internal data race should never turn into an application-visible error. This was hit in production on a Spring Boot + GraphQL service using virtual threads, where DataLoaders record span events concurrently on a span that is then finished from a completion callback.

Change

  • All event-list mutations go through a private synchronized void addEvent(OtelSpanEvent).
  • onSpanFinished() copies the list under the lock and serializes the private snapshot, so iteration can never observe a concurrent mutation.
  • events is volatile so onSpanFinished() keeps a lock-free fast path (a single volatile read) for the common case of a span with no events — no monitor acquire and no extra allocation on the hot finish path. The lock/copy cost is paid only by spans that actually record events. Synchronization is on this, matching the OTel SDK's own SdkSpan, so no per-span lock object is allocated.

Testing

Added OpenTelemetry14Test.testConcurrentAddEvents (8 threads × 2000 events, then finish). It fails with the exact NPE against the current code and passes with the fix. The existing OpenTelemetry14Test suite remains green.

OtelSpan stored span events in a plain ArrayList that was mutated by
addEvent()/recordException() without synchronization and iterated at
span finish. When events are recorded from multiple threads (e.g.
GraphQL DataLoaders running on virtual threads) while the span is being
finished, the backing list gets corrupted, leaving a null hole that
triggers a NullPointerException in OtelSpanEvent.toTag. That NPE escapes
span finish (DDSpan.finishAndAddToTrace) and can surface to the
application as a request failure.

Guard all event-list mutations by synchronizing on the span, and copy
the list under lock at finish so serialization iterates a private
snapshot. The events field is volatile so onSpanFinished keeps a
lock-free fast path (single volatile read) for the common case of a
span with no events, avoiding overhead on the hot finish path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Meemaw
Meemaw marked this pull request as ready for review July 23, 2026 11:41
@Meemaw
Meemaw requested review from a team as code owners July 23, 2026 11:41
@Meemaw
Meemaw requested review from mcculls and ygree and removed request for a team July 23, 2026 11:41
@mcculls mcculls added type: bug fix Bug fix tag: community Community contribution inst: opentelemetry OpenTelemetry instrumentation labels Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: opentelemetry OpenTelemetry instrumentation tag: community Community contribution type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants