Fix concurrent span event recording in OTel shim#12053
Open
Meemaw wants to merge 1 commit into
Open
Conversation
OtelSpan stored span events in a plain ArrayList that was mutated by addEvent()/recordException() without synchronization and iterated at span finish. When events are recorded from multiple threads (e.g. GraphQL DataLoaders running on virtual threads) while the span is being finished, the backing list gets corrupted, leaving a null hole that triggers a NullPointerException in OtelSpanEvent.toTag. That NPE escapes span finish (DDSpan.finishAndAddToTrace) and can surface to the application as a request failure. Guard all event-list mutations by synchronizing on the span, and copy the list under lock at finish so serialization iterates a private snapshot. The events field is volatile so onSpanFinished keeps a lock-free fast path (single volatile read) for the common case of a span with no events, avoiding overhead on the hot finish path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Meemaw
marked this pull request as ready for review
July 23, 2026 11:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Makes span-event handling in the OpenTelemetry shim (
OtelSpan) thread-safe.Span events were stored in a plain
ArrayListthataddEvent()/recordException()mutated without synchronization, and thatonSpanFinished()iterated at span finish. When events are recorded from multiple threads (e.g. GraphQL DataLoaders running on virtual threads) while the span is being finished on another thread, the backing list gets corrupted, leaving anullhole that triggers aNullPointerExceptioninOtelSpanEvent.toTag:That NPE escapes span finish and can surface to the application as a request failure (observed in production as a
jakarta.servlet.ServletException→ HTTP 500 on a request whose spans were otherwise healthy).Motivation
A tracer-internal data race should never turn into an application-visible error. This was hit in production on a Spring Boot + GraphQL service using virtual threads, where DataLoaders record span events concurrently on a span that is then finished from a completion callback.
Change
private synchronized void addEvent(OtelSpanEvent).onSpanFinished()copies the list under the lock and serializes the private snapshot, so iteration can never observe a concurrent mutation.eventsisvolatilesoonSpanFinished()keeps a lock-free fast path (a single volatile read) for the common case of a span with no events — no monitor acquire and no extra allocation on the hot finish path. The lock/copy cost is paid only by spans that actually record events. Synchronization is onthis, matching the OTel SDK's ownSdkSpan, so no per-span lock object is allocated.Testing
Added
OpenTelemetry14Test.testConcurrentAddEvents(8 threads × 2000 events, then finish). It fails with the exact NPE against the current code and passes with the fix. The existingOpenTelemetry14Testsuite remains green.