Fix DDTraceId/DD64bTraceId class-initialization deadlock#11509
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 9 commits intoJun 3, 2026
Merged
Conversation
bm1549
added
type: bug
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56ea720eb8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
dougqh
reviewed
May 29, 2026
Contributor
dougqh
left a comment
dougqh
left a comment
There was a problem hiding this comment.
Overall, it looks good to me.
But before merging, double check that the Codex comment isn't a problem
DD64bTraceId is a subclass of DDTraceId, so the JVM must initialize DDTraceId before DD64bTraceId. DDTraceId.<clinit> in turn initialized DD64bTraceId by building its ZERO/ONE constants via DD64bTraceId.from(), a circular initialization dependency. When the two classes were first touched concurrently from opposite ends -- the service-discovery task (muteTracing() -> blackholeSpan() -> DDTraceId.ZERO) racing the application's first span (IdGenerationStrategy.generateTraceId() -> DD64bTraceId.from()) -- each thread held one class-init lock and waited for the other, hanging trace creation. This surfaced as recurring 30s LogInjectionSmokeTest timeouts in CI (latent since #9705 added the scheduled muteTracing task). Break the cycle at its source while keeping DDTraceId.ZERO/ONE as public fields (preserving the API restored in #5021): ZERO/ONE are now instances of a private DDTraceId subtype (a sibling of DD64bTraceId), so DDTraceId.<clinit> no longer references the subclass. Replace the fragile "== DDTraceId.ZERO" identity checks with a value-based DDTraceId.isZero(). Those identity checks relied on every zero id being the single ZERO instance; isZero() recognizes a zero id of any concrete type, so the factories need not route 0 to the singleton and the propagation codecs no longer mishandle a zero parsed via the direct 64-bit factories. Add a forked regression test that initializes the two classes concurrently from opposite ends (deadlocks without the fix), plus isZero() coverage across the DDTraceId subtypes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
force-pushed
the
brian.marks/fix-ddtraceid-clinit-deadlock
branch
from
b04a0d1 to
0e15d6c
Compare
Debugger benchmarksParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics. See unchanged results
Request duration reports for reportsgantt
title reports - request duration [CI 0.99] : candidate=None, baseline=None
dateFormat X
axisFormat %s
section baseline
noprobe (340.653 µs) : 311, 370
. : milestone, 341,
basic (298.082 µs) : 291, 305
. : milestone, 298,
loop (8.981 ms) : 8976, 8987
. : milestone, 8981,
section candidate
noprobe (341.882 µs) : 302, 382
. : milestone, 342,
basic (294.321 µs) : 288, 300
. : milestone, 294,
loop (8.982 ms) : 8977, 8988
. : milestone, 8982,
|
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
mcculls
reviewed
Jun 1, 2026
mcculls
reviewed
Jun 1, 2026
mcculls
reviewed
Jun 1, 2026
mcculls
previously requested changes
Jun 1, 2026
Contributor
mcculls
left a comment
mcculls
left a comment
There was a problem hiding this comment.
I'd prefer to have a method isValid rather than isZero (where isValid() should return true when the id is not zero)
Also I'm worried that we could now create lots of DD64bTraceId(0) instances with the change to that class.
Finally some of the non-test comments are too verbose and should be simplified
…constant
- Rename DDTraceId.isZero() to value-based isValid() (OTel-aligned; mcculls);
flip the boolean at all 7 production call sites.
- Restore a cached zero singleton in DD64bTraceId so from(0)/fromHex("0") do
not allocate per call (mcculls).
- Extract the ZERO/ONE backing type to a dedicated DDTraceIdConstant class
(PerfectSlayer); update the JaCoCo coverage exclusion.
- Preserve equals/hashCode: ZERO/ONE stay value-equal (both directions) to the
equivalent DD64bTraceId, matching pre-PR behavior.
- Trim verbose non-test comments (mcculls, PerfectSlayer).
- Parameterize and rename TraceIdIsZeroTest -> TraceIdIsValidTest (PerfectSlayer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Trim the verbose DDTraceId/DD64bTraceId/DDTraceIdConstant/XRayHttpCodec comments to terse navigation aids (review feedback). - Suppress EQ_CHECK_FOR_OPERAND_NOT_COMPATIBLE_WITH_THIS on the two cross-type equals methods (intentional value-equality between ZERO/ONE and the equivalent DD64bTraceId); fixes the check_base SpotBugs failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mcculls
reviewed
Jun 2, 2026
| * <p>Only ever initialize this through {@link DDTraceId}'s constants. Initializing it independently | ||
| * (e.g. a static-member access) would bring back the class-initialization deadlock. | ||
| */ | ||
| final class DDTraceIdConstant extends DDTraceId { |
Contributor
There was a problem hiding this comment.
I do wonder if there's a simpler fix that doesn't require introducing another class, as this is making the equals matrix harder to maintain. I'll need to think this through, but eliminating its construction via the static method might be enough. We may need to allow 2 copies of ZERO - a public one in DDTraceId and a private one in DD64bTraceId - but assuming we use isValid() and avoid any identity check then that should break the cycle and still keep the changes minimal.
Contributor
There was a problem hiding this comment.
After verifying that the deadlock could potentially still occur even with the sibling class (if something happened to touch it at the wrong time) and the widening ripple of changes caused by that change I went with a more focused fix - effectively making sure that DDTraceId.ZERO is touched first in IdGenerationStrategy.
This fixes the observed issue without introducing any API compatibility issues.
Ideally we'd revisit the DDTraceId class to avoid it having constants that depend on a subclass - but since this is part of the public API we are limited in what we can do...
Contributor
Author
There was a problem hiding this comment.
That's an incredibly clever workaround - thanks @mcculls!
Contributor
PerfectSlayer
left a comment
PerfectSlayer
left a comment
There was a problem hiding this comment.
Keep adding more feedback to my review.
Important question: did you blindly replace all == ZERO checks by isValid() or did you make sure the TID are always non null first? I want to make sure the isValid() call won't raise NPE and all call sites have been verified.
…raceid-clinit-deadlock
… introducing a new sub-class
…ild time due to the added static initializer
mcculls
approved these changes
Jun 3, 2026
PerfectSlayer
approved these changes
Jun 3, 2026
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
5 tasks
bm1549
added a commit
that referenced
this pull request
Jun 3, 2026
…1509 The DDTraceId/DD64bTraceId class-initialization deadlock was the root cause of the traceCount=0 flakiness that prompted these diagnostics. With the deadlock fixed, the diagnostic instrumentation is no longer needed. Removes from #11075 and #11400: - waitForTraceCountAlive() and all helpers (dumpThreadStacks, filterThreadDump, isBoilerplateThread, isHighPriorityThread, getTestedProcessPid, runJstack, tailProcessLog, INLINE_DUMP_CAP) - The OperatingSystem import that was only used by dumpThreadStacks Keeps the actual fixes from #10999, #11375: - BaseApplication TIMEOUT_IN_NANOS 10s→30s (prevents app-side timeout racing the test-side timeout under CI load) - @flaky(isZulu8()) and JavaVirtualMachine.isZulu8() (covers a separate JUL race on Zulu 8 that is not fixed by #11509) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
gh-worker-dd-mergequeue-cf854d
Bot
merged commit 28344fa
into
master
758 of 763 checks passed
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
brian.marks/fix-ddtraceid-clinit-deadlock
branch
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Jun 3, 2026
…1509 (#11548) Remove log-injection diagnostic code now that deadlock is fixed in #11509 The DDTraceId/DD64bTraceId class-initialization deadlock was the root cause of the traceCount=0 flakiness that prompted these diagnostics. With the deadlock fixed, the diagnostic instrumentation is no longer needed. Removes from #11075 and #11400: - waitForTraceCountAlive() and all helpers (dumpThreadStacks, filterThreadDump, isBoilerplateThread, isHighPriorityThread, getTestedProcessPid, runJstack, tailProcessLog, INLINE_DUMP_CAP) - The OperatingSystem import that was only used by dumpThreadStacks Keeps the actual fixes from #10999, #11375: - BaseApplication TIMEOUT_IN_NANOS 10s→30s (prevents app-side timeout racing the test-side timeout under CI load) - @flaky(isZulu8()) and JavaVirtualMachine.isZulu8() (covers a separate JUL race on Zulu 8 that is not fixed by #11509) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Merge branch 'master' into brian.marks/remove-log-injection-diagnostics Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Fixes a class-initialization deadlock between
DDTraceIdandDD64bTraceIdthat can hang trace creation at startup.DDTraceId.ZEROis now eagerly initialized byIdGenerationStrategyto break the observed cycle without introducing any API compatibility issues.IdGenerationStrategyis guaranteed to be touched very early on when config is built.Motivation
DD64bTraceIdextendsDDTraceId, so the JVM initializesDDTraceIdfirst. ButDDTraceId.<clinit>built itsZERO/ONEconstants viaDD64bTraceId.from(...), which initializes the subclass while theDDTraceIdinit lock is held. When the two classes are first touched concurrently from opposite ends, each thread ends up holding one class-init lock and waiting for the other:dd-task-scheduler: the service-discovery task added in Add support for service discovery using JNA #9705 runsmuteTracing()->blackholeSpan()->DDTraceId.ZEROmain: the application's first span runsIdGenerationStrategy.generateTraceId()->DD64bTraceId.from()Trace creation then hangs. This surfaced as recurring ~30s
LogInjectionSmokeTesttimeouts on master (traceCount=0, process.alive=true, RC polls received: ~135). The forked-process thread dumps added in #11400 confirmed the cycle, and it reproduces deterministically.Additional Notes
DDTraceIdClinitDeadlockForkedTestruns in a fresh JVM and initializes the two classes concurrently from opposite ends; it deadlocks without the fix and passes with it.muteTracing()task; it began manifesting recently on CI as startup timing shifted.Contributor Checklist
type:andcomp:labels assignedJira ticket: N/A