Fix native OOM from unbounded CallTraceHashTable expansion by jbachorik · Pull Request #670 · DataDog/java-profiler

jbachorik · 2026-07-17T18:25:43Z

Problem

Under live-heap profiling (memory=...:l) with a long-running, unrotated recording, the profiler can exhaust native address space and crash with os::commit_memory ... failed; error='Not enough space' (errno=12) / pthread_create failed (EAGAIN). The failure is vm.max_map_count (VMA) exhaustion, not RAM: a captured memory map showed tens of thousands of 8 MB (and 8 MB-multiple) anonymous regions.

Root cause

CallTraceHashTable grows its backing LongHashTable by doubling capacity, allocating each generation from a LinearAllocator whose chunks are a fixed 8 MB (CALL_TRACE_CHUNK). Once a table's byte size exceeds one chunk (capacity ≥ ~512K entries), LinearAllocator::alloc() can never place it: the bump-allocator's inner loop only fits offs + size <= _chunk_size, so an oversized request falls through to getNextChunk(), which allocates a fresh 8 MB chunk that also can't fit it — looping and allocating chunks until mmap fails.

The active table only grows this large between processTraces() rotations (which reset it to the initial capacity). In a single start-to-end JFR recording no rotation happens, so under the high distinct-trace volume that live-heap sampling sustains, the table crosses the one-chunk threshold and the allocator runs away.

Fix

expandTableIfNeeded() now refuses to expand when the next generation's getSize() would exceed LinearAllocator::maxAllocatableSize() (one chunk minus its header). Instead of an unbounded allocation loop, put() keeps working on the current table at a higher load factor; the next rotation resets it. The bound is derived from the chunk size, so it stays correct if CALL_TRACE_CHUNK changes. A new calltrace_storage_expansion_skipped counter records when the cap engages.

This also includes a related per-thread native leak fix in LivenessTracker: its subsampling RNG ThreadLocals (gen/dis/skipped) were reclaimed only via pthread-key destructors, which never fire when a JNI-attached thread detaches on a reused OS thread. They are now released explicitly from Profiler::onThreadEnd(), matching how the CPU/wall engines and ProfiledThread are already torn down.

Validation

Reproducer (renaissance akka-uct -r 20 under memory=64:l) OOM'd at ~90 s / 13 iterations before the fix. With the fix it completes all 20 iterations with peak VMA count in the hundreds (was pinned at the 65530 limit). Compiles clean; verified the change is compatible with the existing expandTableIfNeeded/overflow-guard unit tests by inspection (the new check sits alongside the existing guards and only engages for tables that exceed one chunk).

🤖 Generated with Claude Code

dd-octo-sts · 2026-07-17T18:56:45Z

Benchmark Results (commit `75285e9`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125374352 Commit: 75285e92c193b1b4eedd58201cecc5e0f6694060

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10181 ms (21 iters)	✅ 10290 ms (21 iters)	≈ +1.1% (±10.8%)	— / —
finagle-chirper	21	✅ 5949 ms (33 iters)	✅ 5906 ms (33 iters)	≈ -0.7% (±25.3%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5510 ms (36 iters)	✅ 5495 ms (36 iters)	≈ -0.3% (±25.2%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2716 ms (69 iters)	✅ 2726 ms (68 iters)	≈ +0.4% (±2.7%)	— / —
fj-kmeans	25	✅ 2866 ms (66 iters)	✅ 2832 ms (66 iters)	≈ -1.2% (±2.5%)	— / —
future-genetic	21	✅ 2037 ms (90 iters)	✅ 2073 ms (90 iters)	≈ +1.8% (±2.6%)	— / —
future-genetic	25	✅ 2063 ms (90 iters)	✅ 2048 ms (90 iters)	≈ -0.7% (±2.9%)	— / —
naive-bayes	21	✅ 1299 ms (132 iters)	✅ 1277 ms (134 iters)	≈ -1.7% (±32.4%)	— / —
naive-bayes	25	✅ 1018 ms (168 iters)	✅ 1004 ms (170 iters)	≈ -1.4% (±31.4%)	— / —
reactors	21	✅ 16455 ms (15 iters)	✅ 16332 ms (15 iters)	≈ -0.7% (±8.1%)	— / —
reactors	25	✅ 18441 ms (15 iters)	✅ 18882 ms (15 iters)	≈ +2.4% (±4.1%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	1 / 2	1952 / 1942	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	1 / 3	2169 / 2356	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	4 / 2	8479 / 8210	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	✅ / ✅	8416 / 8155	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	1 / 5	1282 / 1277	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	3 / 3	1298 / 1255	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	2 / 2	2922 / 2931	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	2 / 1	2888 / 2859	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	4 / 4	3531 / 3536	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	3 / 4	3463 / 3494	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	✅ / ✅	1612 / 1596	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	✅ / 1	1952 / 1935	✅ / ✅	✅ / ✅

zhengyu123 · 2026-07-20T13:27:39Z

Could you rebase the PR? seems it has conflicts.

CallTraceHashTable grows its backing LongHashTable by doubling capacity from a LinearAllocator whose chunks are a fixed 8MB. Once a table exceeds one chunk (capacity >= 512K entries), LinearAllocator::alloc() can never place it: the bump-allocator loops allocating fresh chunks that still cannot fit the oversized request, until mmap fails - a native OOM / vm.max_map_count exhaustion, not a heap-space error. This is reachable under live-heap profiling (memory=...:l) with a long, unrotated recording, where the active table grows past that threshold between processTraces() rotations. Cap expansion at the largest table that fits in one allocator chunk (LinearAllocator::maxAllocatableSize()); further growth degrades put() to a higher load factor and recovers on the next rotation. Deriving the bound from the chunk size keeps it correct if the chunk size ever changes. Also fix a per-thread native leak in LivenessTracker's subsampling RNG state: the gen/dis/skipped ThreadLocals were reclaimed only via pthread-key destructors, which never fire when a JNI-attached thread detaches on a reused OS thread. Release them explicitly from Profiler::onThreadEnd(), matching how the CPU/wall engines and ProfiledThread are already torn down. Environment: Datadog workspace Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

dd-octo-sts · 2026-07-20T13:59:29Z

CI Test Results

Run: #29814088134 | Commit: a81a45d | Duration: 15m 58s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

Updated: 2026-07-21 09:05:25 UTC

…abot Dependabot bumped analyze to v4.37.1 without init, causing CodeQL's "Loaded a configuration file for version X, but running version Y" error.

dd-octo-sts · 2026-07-20T15:27:54Z

Benchmark Results (commit `d1d5076`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125698320 Commit: d1d5076612a3fda65dd1f7d127e02b995b55fc59

⚠️ Significant outliers

🔴 fj-kmeans (JDK 21): runtime +6.1% (2663→2825 ms)
🔴 future-genetic (JDK 25): runtime +5% (2022→2124 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10276 ms (21 iters)	✅ 10386 ms (21 iters)	≈ +1.1% (±10.9%)	— / —
akka-uct	25	✅ 8869 ms (24 iters)	✅ 8909 ms (24 iters)	≈ +0.5% (±10.4%)	— / —
finagle-chirper	21	✅ 5960 ms (33 iters)	✅ 5984 ms (33 iters)	≈ +0.4% (±25.1%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5462 ms (36 iters)	✅ 5542 ms (36 iters)	≈ +1.5% (±24.9%)	⚠️ W:3 / ⚠️ W:3
fj-kmeans	21	✅ 2663 ms (70 iters)	✅ 2825 ms (66 iters)	🔴 +6.1%	— / —
fj-kmeans	25	✅ 2834 ms (66 iters)	✅ 2818 ms (66 iters)	≈ -0.6% (±2.6%)	— / —
future-genetic	21	✅ 2058 ms (90 iters)	✅ 2044 ms (90 iters)	≈ -0.7% (±2.5%)	— / —
future-genetic	25	✅ 2022 ms (91 iters)	✅ 2124 ms (87 iters)	🔴 +5%	— / —
naive-bayes	21	✅ 1233 ms (138 iters)	✅ 1245 ms (137 iters)	≈ +1% (±33.1%)	— / —
naive-bayes	25	✅ 1009 ms (169 iters)	✅ 1010 ms (169 iters)	≈ +0.1% (±31.5%)	— / —
reactors	21	✅ 16270 ms (15 iters)	✅ 15718 ms (15 iters)	≈ -3.4% (±6%)	— / —
reactors	25	✅ 19058 ms (15 iters)	✅ 18294 ms (15 iters)	≈ -4% (±4.9%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	2 / 3	2042 / 1952	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	2 / 4	2251 / 2204	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	4 / 3	8718 / 8771	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	✅ / 2	8527 / 8342	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	1 / 4	1270 / 1292	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	2 / 1	3042 / 2917	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	✅ / ✅	2833 / 2919	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	7 / 7	3462 / 3491	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	4 / 3	3463 / 3490	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	1 / ✅	1696 / 1521	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	2 / ✅	1978 / 1919	✅ / ✅	✅ / ✅

dd-octo-sts · 2026-07-20T15:30:21Z

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/125698277

zhengyu123

LGTM

kaahos

apart from the missing copyrights, it looks good to me!

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

datadog-datadog-prod-us1 · 2026-07-21T08:51:18Z

Tests

🔄 Datadog auto-retried 2 jobs - 2 passed on retry

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b159f4e | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-21T08:53:59Z

Benchmark Results (commit `b159f4e`)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125887870 Commit: b159f4e0e4a132c2d43e443f179f44663931cb94

⚠️ Significant outliers

🟢 future-genetic (JDK 25): runtime -3.2% (2074→2008 ms)

Runtime details (per benchmark × JDK)

Benchmark	JDK	Latest	Dev	Δ (dev vs latest)	Issues L/D
akka-uct	21	✅ 10319 ms (21 iters)	✅ 10341 ms (21 iters)	≈ +0.2% (±11.1%)	— / —
akka-uct	25	✅ 8856 ms (24 iters)	✅ 8805 ms (24 iters)	≈ -0.6% (±10.3%)	— / —
finagle-chirper	21	✅ 5940 ms (33 iters)	✅ 5938 ms (33 iters)	≈ -0% (±25.7%)	⚠️ W:3 / ⚠️ W:3
finagle-chirper	25	✅ 5533 ms (35 iters)	✅ 5496 ms (36 iters)	≈ -0.7% (±24.5%)	⚠️ W:3 / ⚠️ W:4
fj-kmeans	21	✅ 2784 ms (67 iters)	✅ 2770 ms (67 iters)	≈ -0.5% (±2.7%)	— / —
fj-kmeans	25	✅ 2774 ms (67 iters)	✅ 2819 ms (66 iters)	≈ +1.6% (±2.7%)	— / —
future-genetic	21	✅ 2095 ms (88 iters)	✅ 2112 ms (87 iters)	≈ +0.8% (±2.9%)	— / —
future-genetic	25	✅ 2074 ms (90 iters)	✅ 2008 ms (93 iters)	🟢 -3.2%	— / —
naive-bayes	21	✅ 1267 ms (135 iters)	✅ 1231 ms (139 iters)	≈ -2.8% (±32.3%)	— / —
naive-bayes	25	✅ 1010 ms (169 iters)	✅ 1015 ms (168 iters)	≈ +0.5% (±31.7%)	— / —
reactors	21	✅ 15746 ms (15 iters)	✅ 16228 ms (15 iters)	≈ +3.1% (±6.1%)	— / —
reactors	25	✅ 17941 ms (15 iters)	✅ 18765 ms (15 iters)	≈ +4.6% (±5.6%)	— / —

Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark	JDK	Dropped rec	Dropped jvmti	Dropped trace	Skipped WC	AGCT fail	Unwind fail
akka-uct	21	✅ / ✅	✅ / ✅	1 / ✅	1925 / 2009	✅ / ✅	✅ / ✅
akka-uct	25	✅ / ✅	✅ / ✅	2 / 3	2253 / 2239	✅ / ✅	✅ / ✅
finagle-chirper	21	✅ / ✅	✅ / ✅	2 / 2	8371 / 8786	✅ / ✅	✅ / ✅
finagle-chirper	25	✅ / ✅	✅ / ✅	2 / 2	8047 / 8448	✅ / ✅	✅ / ✅
fj-kmeans	21	✅ / ✅	✅ / ✅	1 / 4	1252 / 1245	✅ / ✅	✅ / ✅
fj-kmeans	25	✅ / ✅	✅ / ✅	4 / ✅	1284 / 1261	✅ / ✅	✅ / ✅
future-genetic	21	✅ / ✅	✅ / ✅	1 / ✅	2935 / 2848	✅ / ✅	✅ / ✅
future-genetic	25	✅ / ✅	✅ / ✅	3 / 1	2984 / 2948	✅ / ✅	✅ / ✅
naive-bayes	21	✅ / ✅	✅ / ✅	3 / 6	3512 / 3484	✅ / ✅	✅ / ✅
naive-bayes	25	✅ / ✅	✅ / ✅	5 / 4	3469 / 3505	✅ / ✅	✅ / ✅
reactors	21	✅ / ✅	✅ / ✅	4 / 1	1562 / 1503	✅ / ✅	✅ / ✅
reactors	25	✅ / ✅	✅ / ✅	2 / ✅	1806 / 1940	✅ / ✅	✅ / ✅

jbachorik requested a review from a team as a code owner July 17, 2026 18:25

jbachorik force-pushed the jb/fix-liveheap-calltrace-oom branch from 75285e9 to c9d17a9 Compare July 20, 2026 13:38

Fix codeql-action init/analyze version mismatch, group them in depend…

d1d5076

…abot Dependabot bumped analyze to v4.37.1 without init, causing CodeQL's "Loaded a configuration file for version X, but running version Y" error.

zhengyu123 approved these changes Jul 20, 2026

View reviewed changes

kaahos approved these changes Jul 20, 2026

View reviewed changes

Comment thread ddprof-lib/src/main/cpp/livenessTracker.h Outdated

Comment thread ddprof-lib/src/main/cpp/callTraceHashTable.cpp Outdated

Update copyright

b159f4e

Copilot AI review requested due to automatic review settings July 21, 2026 08:23

Copilot AI reviewed Jul 21, 2026

jbachorik merged commit 77c2122 into main Jul 21, 2026
180 of 182 checks passed

jbachorik deleted the jb/fix-liveheap-calltrace-oom branch July 21, 2026 09:11

github-actions Bot added this to the 1.47.0 milestone Jul 21, 2026

rkennke mentioned this pull request Jul 23, 2026

Bump java-profiler (ddprof) to 1.47.0 DataDog/dd-trace-java#12055

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix native OOM from unbounded CallTraceHashTable expansion#670

Fix native OOM from unbounded CallTraceHashTable expansion#670
jbachorik merged 3 commits into
mainfrom
jb/fix-liveheap-calltrace-oom

jbachorik commented Jul 17, 2026

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Uh oh!

zhengyu123 commented Jul 20, 2026

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026

Uh oh!

zhengyu123 left a comment

Uh oh!

kaahos left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jbachorik commented Jul 17, 2026

Problem

Root cause

Fix

Validation

Uh oh!

dd-octo-sts Bot commented Jul 17, 2026

Benchmark Results (commit 75285e9)

✅ Within expected boundaries

Uh oh!

zhengyu123 commented Jul 20, 2026

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI Test Results

Status Overview

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026

Benchmark Results (commit d1d5076)

⚠️ Significant outliers

Uh oh!

dd-octo-sts Bot commented Jul 20, 2026

Reliability & Chaos Results

Uh oh!

zhengyu123 left a comment

Choose a reason for hiding this comment

Uh oh!

kaahos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026

Benchmark Results (commit b159f4e)

⚠️ Significant outliers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Benchmark Results (commit `75285e9`)

dd-octo-sts Bot commented Jul 20, 2026 •

edited

Loading

Benchmark Results (commit `d1d5076`)

datadog-datadog-prod-us1 Bot commented Jul 21, 2026 •

edited by datadog-official Bot

Loading

Benchmark Results (commit `b159f4e`)