Store known span tags densely in TagMap by tag-id (phase 2) by dougqh · Pull Request #11814 · DataDog/dd-trace-java

dougqh · 2026-06-30T20:12:02Z

Draft — reviewable; lands after the next release. Both the read-through and dense pieces stay behind experimental flags (off by default, flipped on internally to bake), so it's safe to hold across a release.

This branch is stacked on the level-split PRs — the read-through mechanism (#11789) and the mergedTracerTags consumer wiring (#11932) land first; this PR adds the dense known-tag store on top. (The read-through/consumer commits that used to ride on this branch have been split out into those PRs.)

Dense known-tag store (this PR's focus)

Known tags (those KnownTags.keyOf resolves to a stored id) store in dense parallel arrays (knownIds/knownValues, insertion-ordered, cap-8 ×2) with no per-tag Entry — the #1 tracer allocator. Coexists with the hash buckets (arbitrary tags); disjoint by global known-ness, so read-through shadow checks stay within-region.

keyOf substrate — KnownTags (id constants + registry) + KnownTagCodec (tagId encoding + resolver), with the open-addressed keyOf table via StringIndex
dense store woven into TagMap (set/get/remove/forEach/size/copy/putAll/clear/iterate)
lazy buckets — shared EMPTY_BUCKETS sentinel + copy-on-write; all-known maps allocate zero buckets
flyweight EntryReaderIterator — alloc-free dense reads on the serialize path; entrySet() keeps real retain-safe Entry
dormant behind dd.trace.dense.tags.enabled (experimental system property; → promote to a Config flag before ship)

Validation

Deterministic -prof gc (real resolver, realistic mixed span): per-build alloc −8% (7 tags) → −41% (12 tags); all-known / trace-tier shape −57% / −43%.
PetClinic macro (Xmx96m, counterbalanced A-B-B-A): GC pauses −2.8% (vs +6.6% before the flyweight fix), throughput break-even.
Coverage: TagMapDenseForkedTest (real registry + read-through union), TagMapDenseFuzzForkedTest (3 key regimes, HashMap oracle + checkIntegrity), DenseStoreAllocBenchmark.

The CPU upside (skip keyOf by setting tags via setTag(KnownTags.X_ID)) arrives via incremental instrumentation migration over time — no big-bang.

🤖 Generated with Claude Code

dd-octo-sts · 2026-06-30T20:32:34Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.75 s	14.49 s	[+1.2%; +2.5%] (significantly worse)
startup:insecure-bank:tracing:Agent	13.58 s	13.71 s	[-1.9%; +0.0%] (no difference)
startup:petclinic:appsec:Agent	16.25 s	16.79 s	[-7.3%; +1.0%] (no difference)
startup:petclinic:iast:Agent	16.78 s	16.43 s	[-2.4%; +6.6%] (no difference)
startup:petclinic:profiling:Agent	16.61 s	16.79 s	[-2.1%; -0.1%] (maybe better)
startup:petclinic:sca:Agent	16.80 s	16.67 s	[-0.2%; +1.7%] (no difference)
startup:petclinic:tracing:Agent	15.96 s	16.18 s	[-2.4%; -0.3%] (maybe better)

Commit: 0c0d740c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dougqh · 2026-07-15T20:32:53Z

Directional span-creation benchmark: dense off vs on (this branch)

Toggle A/B on this branch (-Ddd.trace.dense.tags.enabled), so it isolates the dense store from the rest of the stack — byte-identical code, just the flag. Grafted SpanCreationBenchmark (from #11915), not committed.

Config: -f 2 -wi 3 -i 3 -w 5 -r 5 -prof gc, @Threads(8), JDK 17. Realistic trace bundle via -jvmArgsAppend "-Ddd.env=bench -Ddd.version=1.0.0 -Ddd.tags=team:apm,tier:web,region:us1,shard:7,build:1234". Alloc = gc.alloc.rate.norm. Without bloom (#11900); alloc is bloom-independent, but throughput here reflects the slower linear-scan insertion — directional only.

method	off B/op	on B/op	Δ% (cap 12)	Δ% (cap 8)
bareStartSpan	52,429	52,779	+0.67%	+0.78%
bareBuildSpan	52,403	52,879	+0.91%	+0.65%
webServerSpan	62,800	63,156	+0.57%	+0.23%
webServerSpanViaBuilder	69,306	70,334	+1.48%	+1.25%
jdbcClientSpan	64,370	64,677	+0.48%	+0.75%

Honest read: dense in isolation is alloc-negative here (~+0.5–1.5%)

This is a local valley, not the destination — and the cause is instructive:

It's over-provisioning, not array growth. The regression is cap-insensitive — bumping KNOWN_INIT_CAP 8→12 shuffles within run noise and the worst case stayed worst. If growth were the driver, cap-12 (≥ every span's known count) would have clearly rescued the exit spans. It didn't. The cost is the fixed two-array allocation (long[]+Object[]) per span-with-known-tags; at real span tag counts that exceeds the Entry bytes eliminated, regardless of cap. So a stopgap size can't fix it.
The win is jointly gated on two upstack pieces:
1. Per-type exact sizing (tag registry) — a webServer span sized to its ~6 own known tags (long[6]+Object[6] ≈ 104 B) genuinely beats ~6 Entries (≈ 190 B). Generous fixed caps over-provision that away.
2. Level-split actually engaging — the trace bundle (env/version/…) is landing in every span's dense array (even bare spans regress), because an intercepted tag in mergedTracerTags gates read-through off and forces a copy. Stripping it out via read-through shrinks the per-span known count.

Reconciles with the macro TagMap$Entry→0 finding: dense removes the Entries but adds array bytes; the net only turns positive once the arrays are packed tight (registry) and the trace bundle is read through (level-split). Sizing is a generous fixed stopgap (KNOWN_INIT_CAP=12) until the registry lands.

From Claude: directional A/B run during the dense reconcile onto the level-split stack; posting the honest before/after and the coupled-gating that gets us out of the valley.

🤖 Generated with Claude Code

…scan) A per-map long bitmask (knownBloom) over the dense store: a set bit means a tagId MAY be present (scan to confirm), a clear bit means DEFINITELY absent — so the common per-build insert skips the linear knownIndexOf scan and appends in O(1). Crude position->bit map (fieldPos & 63); a collision-minimizing per-type coloring later only raises the hit rate — correctness never depends on it because the scan stays authoritative. Superset semantics: set on add, never cleared on remove (a stale bit costs a scan, never a wrong answer). Alloc-neutral (one long field, no extra allocation); the win is insertion CPU, moving the dense store toward HashMap insertion parity without the scan. Reconciled onto the folded-class dense store (#11814). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

datadog-datadog-prod-us1 · 2026-07-15T21:00:59Z

🎯 Code Coverage (details)
• Patch Coverage: 79.88%
• Overall Coverage: 57.32% (+0.09%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0c0d740 | Docs | Datadog PR Page | Give us feedback!}

…scan) A per-map long bitmask (knownBloom) over the dense store: a set bit means a tagId MAY be present (scan to confirm), a clear bit means DEFINITELY absent — so the common per-build insert skips the linear knownIndexOf scan and appends in O(1). Crude position->bit map (fieldPos & 63); a collision-minimizing per-type coloring later only raises the hit rate — correctness never depends on it because the scan stays authoritative. Superset semantics: set on add, never cleared on remove (a stale bit costs a scan, never a wrong answer). Alloc-neutral (one long field, no extra allocation); the win is insertion CPU, moving the dense store toward HashMap insertion parity without the scan. Reconciled onto the folded-class dense store (#11814). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

TagMap was an interface with a single implementation, OptimizedTagMap. The split was vestigial scaffolding from when a second (HashMap-backed) impl existed; with one impl it is false generalization. Collapse them into one `public final class TagMap`: - The interface's abstract method declarations are removed; OptimizedTagMap's bodies become TagMap's methods. - Nested types that were implicitly `public static` in the interface (EntryChange, EntryRemoval, EntryReader, Entry, Ledger) are now written out explicitly as `public static`. - Static factories (create/fromMap/ledger/...) and the EMPTY constant become explicit `public static` members; the EmptyHolder lazy-init note is updated now that there is no interface<->impl class-init cycle. - putAll(TagMap) loses its `instanceof` dispatch (always true once there is one class) and calls the fast path directly. No behavior change; motivation is code simplicity, not performance (a single final class is monomorphic by construction, but CHA already devirtualized the sole impl). Public API is preserved, so callers are unchanged; the 3 tests that referenced OptimizedTagMap now reference TagMap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Post-fold tidy, all TagMap-scoped: - Remove EmptyHolder: with one class there is no interface<->impl class-init cycle to break, and the private constructor reads no statics, so EMPTY is a direct `new TagMap(new Object[1], 0)` initializer. - Static factories (create/fromMap/ledger/...) are now `public static final` (not expressible on the old interface). - assertSize/assertNotEmpty/assertEmpty/checkIntegrity test helpers dropped their now-always-true `instanceof TagMap` guard + redundant cast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@nonnull

The interface -> final-class fold removed the TagMap interface decls, which carried the @Nullable/@nonnull param annotations from #11963. Re-home them onto the now-concrete methods: @nonnull tag keys and strict-setter values, @nullable on the set(EntryReader)/getAndSet(Entry) sinks (+ the getAndSet contract javadoc). The null-tolerance behavior was already preserved by the fold; this restores the self-describing contract on the write surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Per review: these factories NPE on a null map (map.size()/putAll), so the input is non-null by contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A fresh, mutable TagMap can read through to a frozen parent on local misses, so a span can layer its own tags over a shared, immutable set (e.g. merged tracer tags) without copying them. - createFromParent(parent): the only way to attach a parent; the parent must be frozen and is fixed at construction (no re-parenting), so read-through can treat it as stable. Single-parent by design in phase 1. - Reads resolve local-first, then the parent; a local entry shadows the parent's (local-wins). Removing a parent key locally records a lazy tombstone (removedFromParent) so it stops reading through; the tombstone set is null until first needed, keeping the hot paths untouched. - size()/isEmpty() are exact (Map contract) and resolve the parent; isDefinitelyEmpty()/estimateSize() are the cheap conservative variants for the hot path. copy() preserves the parent and tombstones; forEach walks local then parent. Built on the folded final-class TagMap (#11967); composes cleanly with the null-tolerant Entry pathway (#11963). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…plit phase 1) Attach the trace's merged tracer tags to each span's TagMap as a frozen read-through parent (via TagMap.createFromParent) at span construction, instead of copying them into every span. The span sees the shared tags on read and only stores its own local tags, so the common trace-level bundle is held once per trace rather than duplicated per span. - CoreTracer builds the frozen merged-tracer-tags parent once; config version is kept out of that bundle. - DDSpanContext attaches the parent at construction (fixed, no re-parenting). - Adds TagMapReadThroughBenchmark (copy-down vs read-through, -prof gc). Stacked on the read-through mechanism (#11789), which builds on the folded final-class TagMap (#11967). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

StringIndex is a compact open-addressed string→index structure (the keyOf substrate the dense tag store builds on): parallel hash/name arrays, linear probing, on par with HashSet on lookup at a smaller footprint. Includes unit tests, a footprint test (jol), and comparison benchmarks (vs HashSet/switch). No TagMap changes — standalone util. Rebased onto the level-split stack (consumer #11932) as the layer dense-store sits on. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… gate Re-applies the coverage fix dropped by the branch rebase/restack. jacocoTestCoverageVerification flags StringIndex at 0.7 instruction coverage (min 0.8): the instance long[] API (mapLongValues / lookup / lookupOrDefault) and the Support.numSlots(int[]) static were never exercised. Add two tests covering them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… capacityFor Two consistency changes in light of the FlatHashtable strategy family: - Rename Support -> EmbeddingSupport. StringIndex is the LightMap/object-side member (a real object with static factories), and its static-over-raw-arrays tier is exactly the "embed the backing arrays in your own fields" role that LightMap.EmbeddingSupport names. - Replace tableSizeFor with capacityFor(n[, loadFactor]) + DEFAULT_LOAD_FACTOR / LOW_LOAD_FACTOR, mirroring FlatHashtable's sizing (duplicated for now; the two branches are independent, to be unified when the family converges). This also tightens the sizing: the old `while (size <= n)` over-allocated 2x at power-of- two counts (capacityFor(16) is now 32, was 64) while still targeting load factor <= 0.5. capacityFor(0) stays valid (StringIndex allows the empty set). Updates StringIndexTest (sizing expectations + a rejects test) and the three benchmarks referencing the tier. Behavior unchanged except the tighter default table size. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Known tags (keyOf resolves to a stored id) are held in insertion-ordered parallel arrays (knownIds/knownValues) with NO per-tag Entry object — the allocation lever. Lazily allocated on the first known-tag write; custom tags stay in the hash buckets. Disjoint by construction (known-ness is global), so read-through shadow checks stay within-region and the bucket path is unchanged. - KnownTagCodec (id encoding + resolver) + hand-written KnownTags (keyOf substrate over StringIndex). Off-by-default: dormant until a resolver registers, so production is byte-identical. - CoreTracer flips it live behind `-Ddd.trace.dense.tags.enabled` for A/B. - Sizing is a generous fixed stopgap (KNOWN_INIT_CAP=12, the per-type max); exact per-type sizing comes with the tag registry. Reconciled onto the level-split stack (fold + read-through + StringIndex); built on the folded final-class TagMap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…scan) A per-map long bitmask (knownBloom) over the dense store: a set bit means a tagId MAY be present (scan to confirm), a clear bit means DEFINITELY absent — so the common per-build insert skips the linear knownIndexOf scan and appends in O(1). Crude position->bit map (fieldPos & 63); a collision-minimizing per-type coloring later only raises the hit rate — correctness never depends on it because the scan stays authoritative. Superset semantics: set on add, never cleared on remove (a stale bit costs a scan, never a wrong answer). Alloc-neutral (one long field, no extra allocation); the win is insertion CPU, moving the dense store toward HashMap insertion parity without the scan. Reconciled onto the folded-class dense store (#11814). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dougqh · 2026-07-23T01:10:17Z

Superseded by #12045 — the same dense known-tag store, rebuilt as a clean one-PR-per-layer stack on the #11660 base (tagset → dense #12045 → two-tier bloom #12046 → generator #12047). Closing in favor of #12045.

dougqh added type: feature Enhancements and improvements comp: core Tracer core tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM labels Jun 30, 2026

dougqh mentioned this pull request Jun 30, 2026

Store known span tags densely by tag-id in TagMap (concept draft) #11659

Closed

This was referenced Jul 9, 2026

Bloom-filter fast-path for the dense tag store (O(1) insertion) (phase 2) #11900

Closed

Id-keyed TagMap.set(long) + insertion-comparison benchmark (phase 2) #11901

Draft

dougqh force-pushed the dougqh/dense-store branch from 27d42d4 to cd8b73a Compare July 15, 2026 20:32

dougqh changed the base branch from master to dougqh/tagset July 15, 2026 20:32

dougqh force-pushed the dougqh/dense-store branch from cd8b73a to a5e12e4 Compare July 15, 2026 20:41

dougqh force-pushed the dougqh/tagset branch from a1a6332 to 36bb222 Compare July 15, 2026 21:22

dougqh force-pushed the dougqh/dense-store branch from a5e12e4 to 6c70848 Compare July 15, 2026 21:23

dougqh and others added 9 commits July 16, 2026 12:53

Mark fromMap/fromMapImmutable map param @nonnull

0737ca6

Per review: these factories NPE on a null map (map.size()/putAll), so the input is non-null by contract. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

dougqh force-pushed the dougqh/tagset branch from 8c68ec2 to f3c4c21 Compare July 20, 2026 15:50

dougqh force-pushed the dougqh/dense-store branch from 6c70848 to 0c0d740 Compare July 20, 2026 16:49

dougqh changed the title ~~Store known span tags densely in TagMap by tag-id~~ Store known span tags densely in TagMap by tag-id (phase 2) Jul 20, 2026

dougqh force-pushed the dougqh/tagset branch from f3c4c21 to fce3162 Compare July 23, 2026 00:05

dougqh mentioned this pull request Jul 23, 2026

Store known span tags densely in TagMap by tag-id (phase 2) #12045

Draft

dougqh closed this Jul 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Store known span tags densely in TagMap by tag-id (phase 2)#11814

Store known span tags densely in TagMap by tag-id (phase 2)#11814
dougqh wants to merge 10 commits into
dougqh/tagsetfrom
dougqh/dense-store

dougqh commented Jun 30, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

dougqh commented Jul 15, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

dougqh commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

dougqh commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dense known-tag store (this PR's focus)

Validation

Uh oh!

dd-octo-sts Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

dougqh commented Jul 15, 2026

Directional span-creation benchmark: dense off vs on (this branch)

Honest read: dense in isolation is alloc-negative here (~+0.5–1.5%)

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dougqh commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dougqh commented Jun 30, 2026 •

edited

Loading

dd-octo-sts Bot commented Jun 30, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 15, 2026 •

edited

Loading