Skip to content

feat: emit TaskBlock events for native socket I/O#666

Draft
kaahos wants to merge 19 commits into
mainfrom
paul.fournillon/taskblock-native-io
Draft

feat: emit TaskBlock events for native socket I/O#666
kaahos wants to merge 19 commits into
mainfrom
paul.fournillon/taskblock-native-io

Conversation

@kaahos

@kaahos kaahos commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?:

Adds native socket I/O TaskBlock instrumentation on Linux by interposing blocking socket APIs, classifying socket file descriptors, and recording synchronous TaskBlock events for owned native I/O waits.

Motivation:

Blocking native I/O is a major source of wall-clock time. Recording these intervals directly avoids intentionally signaling threads while they are blocked in recognized socket operations.

Additional Notes:

This PR includes Linux native interposition, socket/Fd classification, lifecycle handling for unloadable libraries, integration tests, and a native socket I/O benchmark.

How to test the change?:

Automated coverage includes native unit tests for interposition, socket sampling, native block handling, thread-filter suppression, and Java integration tests for native socket TaskBlock behavior.

  • ./.claude/commands/build-and-summarize testDebug

Native socket integration tests are Linux-specific and skip on macOS.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review (run the dd:platform-security-review skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: [JIRA-XXXX]

@datadog-datadog-prod-us1

This comment has been minimized.

Sign in to view
@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #29847483834 | Commit: b4cbf90 | Duration: 14m 58s (longest job)

❌ 1 of 32 test jobs failed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ❌ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Failed Tests

glibc-amd64/debug / 8-ibm

Job: View logs

No detailed failure information available. Check the job logs.

Summary: Total: 32 | Passed: 31 | Failed: 1

Updated: 2026-07-21 16:34:06 UTC

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-native-io branch 5 times, most recently from 5de84e2 to c4a2496 Compare July 17, 2026 09:44
@dd-octo-sts

dd-octo-sts Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 4077c10)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/125241502 Commit: 4077c10d812d39470e41a46ffb6dd02dfe10ae05

⚠️ Significant outliers

  • 🟢 future-genetic (JDK 25): runtime -3.2% (2072→2005 ms)
Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10318 ms (21 iters) ✅ 10297 ms (21 iters) ≈ -0.2% (±11.5%) — / —
akka-uct 25 ✅ 8877 ms (24 iters) ✅ 8872 ms (24 iters) ≈ -0.1% (±10%) — / —
finagle-chirper 21 ✅ 5924 ms (33 iters) ✅ 6023 ms (33 iters) ≈ +1.7% (±25.7%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5440 ms (36 iters) ✅ 5447 ms (36 iters) ≈ +0.1% (±24%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 25 ✅ 2819 ms (66 iters) ✅ 2768 ms (68 iters) ≈ -1.8% (±2.7%) — / —
future-genetic 21 ✅ 2100 ms (89 iters) ✅ 2118 ms (87 iters) ≈ +0.9% (±2.8%) — / —
future-genetic 25 ✅ 2072 ms (89 iters) ✅ 2005 ms (93 iters) 🟢 -3.2% — / —
naive-bayes 25 ✅ 990 ms (172 iters) ✅ 1017 ms (169 iters) ≈ +2.7% (±31.9%) — / —
reactors 21 ✅ 16610 ms (15 iters) ✅ 16761 ms (15 iters) ≈ +0.9% (±8%) — / —
reactors 25 ✅ 18536 ms (15 iters) ✅ 18473 ms (15 iters) ≈ -0.3% (±2.8%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ ✅ / 1 2000 / 2002 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 1 / 4 2240 / 2285 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 3 / 4 8783 / 8895 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 2 / 3 8381 / 7985 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 4 / ✅ 1321 / ✅ ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 2 / 3 1268 / 1276 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ ✅ / 2 2969 / 2966 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ 3 / 1 2828 / 2888 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 1 / 4 3471 / 3500 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1675 / 1607 ✅ / ✅ ✅ / ✅
reactors 25 ✅ / ✅ ✅ / ✅ 1 / ✅ 1935 / 1843 ✅ / ✅ ✅ / ✅

@kaahos
kaahos force-pushed the paul.fournillon/taskblock-native-io branch 3 times, most recently from 8ad829e to 7a6627e Compare July 20, 2026 14:35
@kaahos
kaahos force-pushed the paul.fournillon/taskblock-native-io branch from 7a6627e to 0aadd79 Compare July 21, 2026 07:39
Copilot AI review requested due to automatic review settings July 21, 2026 07:39
Copilot AI reviewed Jul 21, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds TaskBlock event production for native socket I/O waits (Linux) by introducing native interposition + fd classification, expanding TaskBlock recording infrastructure (native + Java API), and updating wall-clock precheck behavior/tests to reflect lifecycle-owned suppression semantics.

Changes:

  • Introduces datadog.TaskBlock JFR event end-to-end: metadata, recording, counters, and Java public paired API (beginTaskBlock / endTaskBlock).
  • Implements Linux native socket I/O interposition + fd classification and emits TaskBlock for owned native I/O waits (plus lifecycle-safe patch/unpatch handling).
  • Expands/updates Java + C++ test coverage and adds a wall-clock precheck overhead benchmark + native test fixture build support.

Reviewed changes

Copilot reviewed 91 out of 91 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/WallclockMitigationsCombinedTest.java Updates assertions to reflect context-scoped threads not being suppression candidates.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/VirtualNativeIoTaskBlockTest.java Adds (currently disabled) virtual-thread native I/O attribution coverage.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckTest.java Adds unfiltered (filter=) precheck behavior tests for owned-block suppression.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/UnfilteredWallPrecheckRestartTest.java Verifies owned-block registry activation doesn’t leak across restart sequences.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/TaskBlockAssertions.java Adds shared assertions for the datadog.TaskBlock event contract.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckTest.java Updates precheck tests/counters for lifecycle-owned suppression and unfiltered sampling.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/PrecheckEfficiencyTest.java Adjusts benchmark-like tests to run under explicit filter= semantics.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/ParkTaskBlockTest.java Adds TaskBlock coverage for Java-owned park hooks + suppression behavior.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/NativeSocketTaskBlockLifecycleTest.java Adds (disabled) lifecycle tests for native socket hooks across stop/restart/uninstall.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/NativeIoBlockHelper.java Adds JNI helper entrypoints for blocking Linux syscalls used in tests.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/MonitorTaskBlockTest.java Adds TaskBlock coverage for monitor wait/contended callbacks and restart recovery.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedUnfilteredWallPrecheckTest.java Runs unfiltered precheck coverage with delegated JVMTI stack collection.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedPrecheckTest.java Ensures delegated stacks path uses explicit unfiltered recording flags.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedParkTaskBlockTest.java Verifies TaskBlock production with delegated wall-clock stacks enabled.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedNativeSocketTaskBlockTest.java Adds (disabled) native socket TaskBlock coverage with delegated stacks.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JvmtiBasedMonitorTaskBlockTest.java Verifies monitor TaskBlock production with delegated stacks enabled.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockPreExistingThreadTest.java Ensures pre-existing threads can initialize TaskBlock TLS and record events.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockDisabledTest.java Confirms paired API remains inactive under filter=0 (no all-thread scope).
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/JavaProfilerTaskBlockApiTest.java End-to-end tests for the paired public TaskBlock API and eligibility rules.
ddprof-test/src/test/java/com/datadoghq/profiler/wallclock/J9WallClockPrecheckCapabilityTest.java Ensures unsupported wall engine does not activate unfiltered precheck tracking on J9.
ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerTest.java Adds agentpath-based startup tests and pre-existing monitor callback coverage.
ddprof-test/src/test/java/com/datadoghq/profiler/JavaProfilerApiSurfaceTest.java Locks public API surface for TaskBlock and monitor ownership/delegation.
ddprof-test/src/test/java/com/datadoghq/profiler/ExternalLauncher.java Adds child-process scenarios for agent initialization + pre-existing monitor callbacks.
ddprof-test/src/test/java/com/datadoghq/profiler/context/OtelContextStorageModeTest.java Ensures trace context is cleared during cleanup.
ddprof-test/src/test/java/com/datadoghq/profiler/context/AllNativeContextTest.java Ensures trace context is cleared during cleanup.
ddprof-test/src/test/java/com/datadoghq/profiler/AbstractProfilerTest.java Adds beforeProfilerStart() hook and clears trace context on stop.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/WallClockPrecheckBenchmarkHooks.java Exposes package-scoped owned-block hooks to JMH benchmarks.
ddprof-stresstest/src/jmh/java/com/datadoghq/profiler/stresstest/scenarios/throughput/WallClockPrecheckOverheadBenchmark.java Adds throughput benchmark for precheck overhead under increasing owned-block populations.
ddprof-lib/src/test/resources/native-libs/unloadable-io-lib/unloadable_io.c Adds native fixture library used by unload/unpatch safety tests.
ddprof-lib/src/test/resources/native-libs/unloadable-io-lib/Makefile Build recipe for unloadable native fixture library.
ddprof-lib/src/test/resources/native-libs/reladyn-lib/reladyn.c Expands relocation fixture to exercise multiple import slots for same symbol.
ddprof-lib/src/test/cpp/wallprecheck_args_ut.cpp Adds capability + filter parsing tests for unfiltered wall precheck.
ddprof-lib/src/test/cpp/wallClockCounters_ut.cpp Updates counter tests to new “suppressed owned block” naming.
ddprof-lib/src/test/cpp/wallClockCandidateSelector_ut.cpp Adds tests for candidate selection/backfill logic used in wall-clock sampling.
ddprof-lib/src/test/cpp/park_state_ut.cpp Extends tests for park + monitor block state snapshotting and slot transitions.
ddprof-lib/src/test/cpp/jvmSupport_ut.cpp Adds tests for JNI-based platform vs virtual thread classification.
ddprof-lib/src/test/cpp/frame_ut.cpp Adds test ensuring JVMTI→ASGCT frame copy preserves overlapping fields.
ddprof-lib/src/test/cpp/elfparser_ut.cpp Adds import-location coverage + skips a deliberate race test under TSan.
ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java Adds TaskBlock public API, monitor delegation flag, and thread-aware native hook signatures.
ddprof-lib/src/main/cpp/wallClockCounters.h Renames/repurposes suppression counter for lifecycle-owned blocked intervals.
ddprof-lib/src/main/cpp/wallClockCandidateSelector.h Adds bounded backfill selector for wall-clock sampling candidates.
ddprof-lib/src/main/cpp/wallClock.h Integrates bounded backfill selection + adds unfiltered-precheck capability on wall engines.
ddprof-lib/src/main/cpp/vmEntry.h Adds monitor ownership/delegation init-result plumbing to bridge initialization.
ddprof-lib/src/main/cpp/threadState.h Adds IO_WAIT OS thread state.
ddprof-lib/src/main/cpp/threadLocalData.h Adds per-thread TaskBlock, park, and monitor state tracking for event emission.
ddprof-lib/src/main/cpp/taskBlockRecorder.h Declares TaskBlock record/finish helpers and eligibility logic.
ddprof-lib/src/main/cpp/taskBlockRecorder.cpp Implements min-duration thresholding and “finish-at-exit” recording path.
ddprof-lib/src/main/cpp/symbols.h Makes UnloadProtection movable and adds release() for retained handles.
ddprof-lib/src/main/cpp/symbols_macos.cpp Implements move operations + release for UnloadProtection.
ddprof-lib/src/main/cpp/symbols_linux.cpp Hardens ELF address computations, adds relocation parsing fixes, and sanitizer-specific VDSO behavior.
ddprof-lib/src/main/cpp/profiler.h Adds TaskBlock recording entrypoints/rotation state and test hooks.
ddprof-lib/src/main/cpp/nativeSocketSampler.h Refactors sampling to use shared fd classifier and adds active/disable helpers.
ddprof-lib/src/main/cpp/nativeSocketSampler.cpp Integrates shared fd classifier, improves lifecycle handling around patch failures, and adds test observers.
ddprof-lib/src/main/cpp/nativeSocketInterposer.h Adds Linux native I/O interposer API and hook specs for blocking syscalls.
ddprof-lib/src/main/cpp/nativeFdClassifier.h Adds shared fd type classifier with epoch+incarnation tagging for reuse safety.
ddprof-lib/src/main/cpp/nativeBlock.h Adds native owned-block scope for TaskBlock emission around blocking syscalls.
ddprof-lib/src/main/cpp/nativeBlock.cpp Implements native blocking scope lifecycle + TaskBlock recording integration.
ddprof-lib/src/main/cpp/libraryPatcher.h Extends patch tracking for native I/O hook sets and safe restoration across unload.
ddprof-lib/src/main/cpp/jvmThread.h Adds capability query for resolving native thread IDs beyond the caller.
ddprof-lib/src/main/cpp/jvmThread.cpp Implements native thread ID lookup capability helper.
ddprof-lib/src/main/cpp/jvmSupport.h Adds platform-thread classification helper.
ddprof-lib/src/main/cpp/jvmSupport.cpp Implements JNI IsVirtualThread-based platform-thread classification.
ddprof-lib/src/main/cpp/jfrMetadata.h Adds JFR type ID for TaskBlock.
ddprof-lib/src/main/cpp/jfrMetadata.cpp Defines datadog.TaskBlock event schema and updates wall epoch suppression field name.
ddprof-lib/src/main/cpp/frames.h Adds safe JVMTI→ASGCT frame copy helper.
ddprof-lib/src/main/cpp/flightRecorder.h Adds TaskBlock recording entrypoints.
ddprof-lib/src/main/cpp/flightRecorder.cpp Emits TaskBlock events and adds IO_WAIT thread state to constant pool.
ddprof-lib/src/main/cpp/event.h Adds TaskBlock event struct and renames wall epoch suppression field.
ddprof-lib/src/main/cpp/engine.h Adds wall-engine capability hook for unfiltered precheck consumption.
ddprof-lib/src/main/cpp/counters.h Adds counters for TaskBlock and precheck backfill/lookup budgeting.
ddprof-lib/src/main/cpp/codeCache.h Redesigns import tracking to support multiple slots per import + patch preparation.
ddprof-lib/src/main/cpp/codeCache.cpp Implements multi-location import storage, patchability prep, and broader import discovery.
ddprof-lib/build.gradle.kts Wires native fixture build directories into gtest extension configuration.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/gtest/GtestPlugin.kt Adds native fixture build task and CI-friendly build task dependency behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread ddprof-lib/src/main/cpp/profiler.h
Comment on lines +478 to +480
int taskBlockInflightForTest() const {
return _task_block_inflight.load(std::memory_order_acquire);
}
Comment thread ...-test/src/test/java/com/datadoghq/profiler/wallclock/NativeSocketTaskBlockLifecycleTest.java
Comment on lines +180 to +182
if (!taskBlockEvents.hasItems()) {
assertTrue(false, missingTaskBlockDiagnostic());
}
@kaahos
kaahos force-pushed the paul.fournillon/taskblock-native-io branch from 0aadd79 to 18130c4 Compare July 21, 2026 14:33
@kaahos
kaahos force-pushed the paul.fournillon/taskblock-native-io branch from 18130c4 to 4698b06 Compare July 21, 2026 16:11
@dd-octo-sts

dd-octo-sts Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (commit 4698b06)

Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126018479 Commit: 4698b06106e60ae0c890bef42375f831660f5f82

✅ Within expected boundaries

No significant runtime deltas (all within run-to-run noise) and no internal-counter outliers.

Runtime details (per benchmark × JDK)
Benchmark JDK Latest Dev Δ (dev vs latest) Issues L/D
akka-uct 21 ✅ 10187 ms (21 iters) ✅ 10361 ms (21 iters) ≈ +1.7% (±11.1%) — / —
akka-uct 25 ✅ 8903 ms (24 iters) ✅ 8898 ms (24 iters) ≈ -0.1% (±9.9%) — / —
finagle-chirper 21 ✅ 5980 ms (33 iters) ✅ 5974 ms (33 iters) ≈ -0.1% (±25.1%) ⚠️ W:3 / ⚠️ W:3
finagle-chirper 25 ✅ 5485 ms (36 iters) ✅ 5457 ms (36 iters) ≈ -0.5% (±23.9%) ⚠️ W:3 / ⚠️ W:3
fj-kmeans 21 ✅ 2777 ms (67 iters) ✅ 2779 ms (68 iters) ≈ +0.1% (±2.7%) — / —
fj-kmeans 25 ✅ 2824 ms (66 iters) ✅ 2819 ms (66 iters) ≈ -0.2% (±2.6%) — / —
future-genetic 25 ✅ 2120 ms (88 iters) ✅ 2116 ms (88 iters) ≈ -0.2% (±2.4%) — / —
naive-bayes 21 ✅ 1252 ms (137 iters) ✅ 1254 ms (137 iters) ≈ +0.2% (±32.2%) — / —
naive-bayes 25 ✅ 1009 ms (169 iters) ✅ 987 ms (173 iters) ≈ -2.2% (±31.2%) — / —
reactors 21 ✅ 16229 ms (15 iters) ✅ 16443 ms (15 iters) ≈ +1.3% (±8.6%) — / —
Internal counter details (ddprof)

ddprof internal counters, latest / dev (✅ = 0, · = unavailable):

Benchmark JDK Dropped rec Dropped jvmti Dropped trace Skipped WC AGCT fail Unwind fail
akka-uct 21 ✅ / ✅ ✅ / ✅ 3 / 1 1903 / 1959 ✅ / ✅ ✅ / ✅
akka-uct 25 ✅ / ✅ ✅ / ✅ 2 / 1 2280 / 2206 ✅ / ✅ ✅ / ✅
finagle-chirper 21 ✅ / ✅ ✅ / ✅ 5 / 4 8883 / 8542 ✅ / ✅ ✅ / ✅
finagle-chirper 25 ✅ / ✅ ✅ / ✅ 1 / 1 8636 / 8339 ✅ / ✅ ✅ / ✅
fj-kmeans 21 ✅ / ✅ ✅ / ✅ 3 / 4 1236 / 1266 ✅ / ✅ ✅ / ✅
fj-kmeans 25 ✅ / ✅ ✅ / ✅ 3 / 3 1246 / 1275 ✅ / ✅ ✅ / ✅
future-genetic 21 ✅ / ✅ ✅ / ✅ 3 / 2 3093 / 2929 ✅ / ✅ ✅ / ✅
future-genetic 25 ✅ / ✅ ✅ / ✅ ✅ / 2 2950 / 2833 ✅ / ✅ ✅ / ✅
naive-bayes 21 ✅ / ✅ ✅ / ✅ 3 / 5 3498 / 3535 ✅ / ✅ ✅ / ✅
naive-bayes 25 ✅ / ✅ ✅ / ✅ 3 / 3 3491 / 3446 ✅ / ✅ ✅ / ✅
reactors 21 ✅ / ✅ ✅ / ✅ ✅ / 1 1697 / 1715 ✅ / ✅ ✅ / ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kaahos