Add streaming early-exit JsonPath extractor for simple linear paths

[xiangfu0]

Summary

Ingestion and query-time JsonPath extraction goes through Jayway
(parse(json).read(path)), which builds a full Jackson DOM of the document and only then walks to
the field — once per row per derived column, a major ingestion CPU cost. This adds a streaming
Jackson-based extractor for simple linear paths ($ followed only by .key, ['key'],
[int]), exposed as opt-in scalar functions so each field chooses independently, in two clearly
named families:

jsonPathStringFast(json, path, default)        jsonPathStringFirstMatch(json, path, default)
jsonPathLongFast  (json, path, default)        jsonPathLongFirstMatch  (json, path, default)
jsonPathDoubleFast(json, path, default)        jsonPathDoubleFirstMatch(json, path, default)

• *Fast — streaming full scan; identical result to the existing jsonPath* (Jayway), just
  faster. Use when the value matters and the result must match.
• *FirstMatch — streaming, stops at the addressed field; faster still when the field is early,
  at the cost of two documented behavior changes on undefined/corrupt input (duplicate keys resolve
  to the first occurrence; a document malformed strictly after the field is not rejected). Use
  only when the source JSON is known well-formed and duplicate-free.

Both resolve a simple linear path in a single streaming pass and fall back to Jayway for complex
paths (wildcards, deep scan .., filters, unions, slices, negative indices) and non-JSON input, and
also fall back to Jayway on any streaming exception (so an unforeseen streaming bug can only cost
a second parse, never a wrong result). The existing jsonPath* functions and the jsonExtractScalar
transform are unchanged — pure Jayway.

Like jsonPathString, these take an Object argument, so they are used through ingestion
transform configs (deriving columns), not as query-time scalars.

Design history. v1 gated the streaming path with a node-level config flag; per review
(@Jackie-Jiang) that became opt-in functions (a node flag can't
express a per-field early-exit choice). The result-affecting mode was then moved out of a positional
earlyExit boolean into the *Fast / *FirstMatch names, per the naming/config reviewers.
Server-local compute only — no wire/segment/config-schema change.

The single-pass multi-path extractor (the ~7-15x N-column win) and a streaming jsonExtractScalar
transform function remain follow-ups; their engine support (multi-path extract, byte[]
overloads) ships here, tested and benchmarked, but is not yet wired to a caller. Function names are
open for reviewer sign-off.

Semantics

The default (full-scan) path returns the same value as Jayway as Pinot configures it
(JacksonJsonProvider + JacksonMappingProvider + SUPPRESS_EXCEPTIONS) for every input, including
duplicate-key last-wins and raising InvalidJsonException on a document malformed anywhere inside its
root value. Content after the root value is ignored, exactly as ObjectMapper.readValue ignores it.

One deliberate difference. Jayway builds a DOM of the whole document, so it materializes and
validates values the path never addresses. This extractor skips those subtrees, so a value Jackson can
lex but not materialize makes Jayway reject the document while this returns the requested value.
Exactly two such values exist, and only when they sit outside the addressed path:

1. a string longer than StreamReadConstraints.maxStringLength (20,000,000 chars);
2. under useBigDecimal (which backs jsonExtractScalar(..., 'STRING') / 'BIG_DECIMAL'), a float
   literal whose exponent overflows an int, e.g. 1e999999999999.

When the offending value is the addressed one, both raise, so the results still agree. This is
inherent to the optimization — matching Jayway here would mean materializing every string in every
document, which is precisely the cost this change exists to avoid. The extractor never returns a
different value for the addressed path; it only declines to fail on values the caller did not ask
about, and it is the safer of the two (it never allocates the 20 MB string or the pathological
BigDecimal). Pinned by a test so it cannot widen unnoticed.

The *FirstMatch functions trade two behaviors for speed when the field appears early, since the
tail is never read: (1) duplicate keys resolve to the first occurrence; (2) a document malformed
strictly after the addressed field no longer raises. Both inputs are RFC-8259-undefined or corrupt,
and this is opt-in per expression (a distinct function, never the *Fast / Jayway path).

Correctness gate

StreamingJsonPathExtractorTest is a differential test with the production Jayway config as the
oracle: the *Fast functions and the raw extractor are compared against the existing Jayway
functions and PARSE_CONTEXT, over a hand-picked matrix (missing paths, invalid JSON,
null/empty/whitespace/BOM, type mismatches, nested
arrays, arrays of objects, deep nesting, unicode/surrogates, escaped/dotted/duplicate keys,
big/precise numbers) plus 60k randomized fuzz iterations, the byte[] overloads (incl. multibyte
and invalid-UTF-8), and the USE_BIG_DECIMAL_FOR_FLOATS context. Early-exit divergences are
asserted explicitly. JsonFunctionsFastPathTest and JsonExtractScalarTransformFunctionFastPathTest
re-run the existing corpora through the fast path.

Performance

JMH (BenchmarkJsonPathExtraction, Apple M, ~688-byte nested payload, 4×5s warmup / 6×5s measure);
the fastPathEnabled=false arm is the current Jayway baseline, measured in the same run:

jsonPathString, one column	ops/s	vs Jayway
Jayway (today)	458k	1.0x
Streaming, exact parity (full scan)	836k	1.8x
Streaming, early exit, field early	5.09M	11.1x
Streaming, early exit, field last	813k	1.8x

4 derived columns from one document	ops/s	vs Jayway
Jayway (4 parses)	106k	1.0x
Streaming, one pass, all 4 paths	763k	7.2x

1.8x is the Jackson lexing floor: exact parity requires reading the whole root value, so a bare
nextToken()-to-EOF loop (~907 ns) is already close to the full extract (~919 ns). The larger wins
come from early exit (undefined-input tradeoff) and from single-pass multi-path extraction, which
must walk the document anyway and so gets exact parity for free — the right target for
transformConfigs pulling N columns from one source.

On representative production logs

To validate on non-synthetic data, both the correctness and the throughput were re-measured on a
corpus of 50k Vector log lines (~1.4KB nested JSON each), in two variants: message as a
stringified-JSON blob, and message as a nested object (2% of object-variant rows carry a plain
non-JSON string message). message is the last top-level field, so $.message.* is a worst-case
late extraction. The transform extracts a realistic set of 12 columns (6 top-level dimensions plus
6 fields under message).

Correctness: 100k lines × 15 paths = 1.5M extractions compared against Jayway, 0 differences
(full-scan mode). Real production JSON — duplicate-free, deeply nested, mixed string/object
message, unicode — is a stronger differential corpus than the randomized fuzzer.

Throughput (JMH, ops/s; string / object variant):

Scenario	Jayway	Streaming	Speedup
1 column, late ($.message.status), full scan	357k / 396k	553k / 537k	1.55x / 1.36x
1 column, late, object, early exit	396k	725k	1.83x
jsonPathString end-to-end (fast path off → on)	356k / 372k	555k / 517k	1.56x / 1.39x
12 columns, one doc — single pass	31.5k / 32.9k	486k / 444k	15.5x / 13.5x

Single-column is a modest ~1.4–1.6x here (vs 1.8x on the smaller synthetic payload): these docs are
larger and the target fields sit at the end, so full-scan parity reads almost the whole document —
the lex floor. The multi-column single-pass number is 13–15x, larger than the synthetic 7.2x,
because Jayway re-parses the full ~1.4KB DOM once per column (jaywayMultiColumn ≈ single-field ÷
12) — exactly the per-row-per-column re-parse this change removes. This is the case for wiring the
single-pass extractor into jsonExtractScalar (follow-up).

(The production-log harness reads local NDJSON files and so is not committed; it is reproducible
against any NDJSON via -Dbench.dir. The committed BenchmarkJsonPathExtraction covers the
synthetic payload above.)

Testing

pinot-common json tests green (differential + fuzz + BigDecimal fuzz + the documented-divergence
pin), including a FunctionRegistry test that resolves and invokes the new *Fast / *FirstMatch functions through
the SQL path. jsonExtractScalar transform + query tests green (reverted to pure Jayway, unchanged).
spotless / checkstyle / license clean. The extractor was additionally validated against 100k real
production log lines (1.5M extractions) with zero differences from Jayway, as noted under
Performance.

Notes for reviewers

• Opt-in per expression, so there is no node config, no rolling-restart hazard, and no
  mixed-version concern — a server that doesn't recognize the overload isn't in the picture, and
  the existing functions are untouched.
• *FirstMatch is the only result-affecting choice, and it is explicit in the function name, chosen
  per expression by whoever writes it, on data they know. See Semantics for its two divergences.
• The streaming path in these functions falls back to Jayway on any exception, so an unforeseen
  streaming bug can only cost a second parse on a row, never change a result.
• Follow-ups (not in this PR): a streaming jsonExtractScalar transform function (the
  column-optimized ingestion path), and wiring the single-pass multi-path extract (the ~7-15x
  N-column win) into it.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.32932% with 44 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (master@00d1c88). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...ot/common/function/StreamingJsonPathExtractor.java	78.68%	16 Missing and 10 partials ⚠️
...g/apache/pinot/common/function/SimpleJsonPath.java	82.35%	11 Missing and 4 partials ⚠️
...he/pinot/common/function/scalar/JsonFunctions.java	92.85%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #18972   +/-   ##
=========================================
  Coverage          ?   65.26%           
  Complexity        ?     1405           
=========================================
  Files             ?     3422           
  Lines             ?   215271           
  Branches          ?    34082           
=========================================
  Hits              ?   140491           
  Misses            ?    63441           
  Partials          ?    11339           

Flag	Coverage Δ
custom-integration1	100.00% <ø> (?)
integration	100.00% <ø> (?)
integration1	100.00% <ø> (?)
integration2	0.00% <ø> (?)
java-21	65.26% <82.32%> (?)
temurin	65.26% <82.32%> (?)
unittests	65.25% <82.32%> (?)
unittests1	56.96% <82.32%> (?)
unittests2	37.63% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds a Jackson streaming-based JsonPath extractor optimized for simple linear paths and wires it into both query-time scalar functions and ingestion-time jsonExtractScalar, behind two default-OFF feature flags to preserve existing Jayway semantics.

Changes:

• Introduces SimpleJsonPath + StreamingJsonPathExtractor and a global mode switch (JsonPathFastPathMode) for fast-path enablement / early-exit behavior.
• Wires the fast path into JsonFunctions.jsonPath* and JsonExtractScalarTransformFunction, with startup-time config seeding via ServiceStartableUtils.
• Adds differential + fuzz testing for parity against the Jayway fallback, plus fast-path re-runs of existing JSON function / transform-function test suites, and a JMH benchmark.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java	Adds cluster config keys for fast-path enablement and early-exit.
pinot-common/src/main/java/org/apache/pinot/common/function/JsonPathFastPathMode.java	Adds global volatile switches (system-property seeded) for fast-path and early-exit.
pinot-common/src/main/java/org/apache/pinot/common/function/SimpleJsonPath.java	Compiles/caches the restricted “simple linear” JsonPath subset for streaming resolution.
pinot-common/src/main/java/org/apache/pinot/common/function/StreamingJsonPathExtractor.java	Implements single-/multi-path streaming extraction via Jackson JsonParser.
pinot-common/src/main/java/org/apache/pinot/common/function/scalar/JsonFunctions.java	Routes eligible jsonPath* calls through the streaming extractor when enabled.
pinot-common/src/main/java/org/apache/pinot/common/utils/ServiceStartableUtils.java	Seeds fast-path flags from cluster config during service startup.
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunction.java	Applies streaming fast path to ingestion jsonExtractScalar for STRING/BYTES inputs when eligible.
pinot-common/src/test/java/org/apache/pinot/common/function/StreamingJsonPathExtractorTest.java	Differential + fuzz parity tests (Jayway oracle), plus explicit early-exit divergence assertions.
pinot-common/src/test/java/org/apache/pinot/common/function/JsonFunctionsFastPathTest.java	Re-runs existing JsonFunctionsTest under fast-path enabled.
pinot-core/src/test/java/org/apache/pinot/core/operator/transform/function/JsonExtractScalarTransformFunctionFastPathTest.java	Re-runs existing JsonExtractScalarTransformFunctionTest under fast-path enabled.
pinot-common/src/test/java/org/apache/pinot/common/function/JsonPathFastPathModeTest.java	Tests startup wiring/config assignment for the two feature flags.
pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkJsonPathExtraction.java	Adds JMH benchmarks comparing Jayway vs streaming extraction (single + multi-path).

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

[yashmayya]

Reviewed for correctness and performance, focused on whether the streaming
extractor can ever diverge from Jayway on the addressed value, or regress the
hot path. No blockers — LGTM.

Correctness — the streaming path is value-equivalent to Jayway

• Number materialization is provably identical. Jayway's JacksonJsonProvider
  reads into Object.class via UntypedObjectDeserializer, which returns
  p.getNumberValue() for ints and useBigDecimal ? getDecimalValue() : getNumberValue()
  for floats — exactly what readValue() does. Containers go through the same
  unconfigured ObjectMapper, so nested contents (incl. duplicate-key last-wins) match.
• Tokenization is a safe subset. SimpleJsonPath's name-char set
  (letterOrDigit | _ | -) sits entirely inside Jayway's dot-notation property
  alphabet (Jayway reads any char until . / [ / ( / space), and the compiler
  bails to NOT_SIMPLE whenever a name run ends on anything but . / [ / EOS. So
  every path it accepts, Jayway segments identically — no "addresses a different field"
  risk.
• byte[] parity. Both extract(byte[]) and Jayway's parseUtf8 create a Jackson
  byte[] parser that auto-detects encoding, so they agree (incl. invalid UTF-8 → same
  exception type).
• The two documented divergences are real, safe-direction, and gated. Over-long
  string / bigDecimal-exponent-overflow only differ when the value sits outside the
  addressed path (skip-vs-materialize); the extractor declines to fail rather than
  returning a wrong value. I confirmed escape validation does happen during
  skipChildren (so a \uZZZZ in an unaddressed field still throws in both) — the
  "exactly two" claim is accurate.

I re-ran the differential-against-production-Jayway methodology with my own inputs
(not the committed corpus):

• 172 adversarial hand-picked cases — numeric/keyword/cyrillic dot keys ($.123,
  $.true, $.null, $.ключ), deep nested duplicates, leading-zero indices ([00]),
  number boundaries, trailing junk — diverged only on the 2 documented
  1e999999999999-outside-path cases under useBigDecimal.
• 80k fuzz iterations using dot-notation keys the committed fuzzer never emits (1,
  12, 1a, true, null, x-y, x_y) → 0 mismatches.

Production only uses the single-path extract; the multi-path walk is benchmark-only
for now, so the exhaustively-tested single-path is the entire prod surface.

Performance

No per-row cost added on the reject path (the transform returns the pure Jayway lambda
when the path isn't simple, and the kill switch is read per-block, not per-row); the
Guava→ConcurrentHashMap change removes the per-read segment-lock contention; and the
extractor skips the full-DOM allocation. Net positive with no regression when the fast
path declines.

Minor, non-blocking

• The single-path extract(String/byte[], …) overloads allocate a new Object[1] per
  row on the wired ingestion path. Negligible against the ~900 ns parse (and the JMH
  numbers already include it), but trivially avoidable — the single-pass multi-path you
  flagged as the follow-up is the bigger win anyway.
• The bytes×bigDecimal combination and the BYTES-column path through
  getResultExtractor aren't directly differential-tested. The parser-source and
  number-materialization dimensions are orthogonal so the risk is low, but worth a case
  or two if you want to close the matrix.

[Jackie-Jiang]

This should not be modeled as a node level config. It doesn't work if some field can leverage early-exist but some not.
We can add new scalar function for this optimized parsing

[xiangfu0]

Thanks @Jackie-Jiang — agreed on both points, and reworked accordingly (pushed).

No more node-level config. JsonPathFastPathMode, the two CommonConstants keys, and the ServiceStartableUtils wiring are all removed. jsonExtractScalar and the existing jsonPath* functions revert to pure Jayway — existing queries are unchanged.

Opt-in per expression instead, so early exit is a per-field choice as you noted (some columns can use it, others can't). New streaming overloads:

jsonPathString(json, path, defaultValue, earlyExit)
jsonPathLong  (json, path, defaultValue, earlyExit)
jsonPathDouble(json, path, defaultValue, earlyExit)

Each resolves a simple linear path in a single streaming pass and falls back to Jayway for complex paths / non-JSON, so the result matches the existing overload; earlyExit is the per-field knob for the two documented divergences. A FunctionRegistry test confirms the new 4-arg signatures resolve and the boolean literal converts through the SQL path.

Two open items for your call:

1. Naming — I used 4-arg overloads of the existing names (arity is already how jsonPathString overloads today) so there's nothing new to learn, but happy to switch to distinct names (e.g. a jsonExtract* family) if you'd prefer the engine choice be explicit rather than implied by the extra arg.
2. I kept this to the scalar functions per your suggestion. The typed jsonExtractScalar transform function (the column-optimized ingestion path) and the single-pass multi-path extractor are left as follow-ups — let me know if you'd rather fold a streaming transform variant into this PR.

[xiangfu0]

Follow-up on the earlyExit naming: implemented distinct names instead of the positional boolean, per the naming/config-backcompat reviewers.

• jsonPathStringFast / jsonPathLongFast / jsonPathDoubleFast — streaming, full scan, same result as the existing jsonPath* (Jayway).
• jsonPathStringFirstMatch / … — streaming, stops at the first match; the two divergences (duplicate-key first-wins, malformed-tail tolerated) are now signalled by the name rather than an unlabeled true.

Also fixed the integration test: these take an Object arg, so (like jsonPathString) they run through ingestion transform configs, not query-time SELECT — a query-time call hits Unsupported parameter type: OBJECT in ScalarTransformFunctionWrapper. JsonPathTest now derives columns with the new functions at ingestion and checks them against the Jayway-derived column on both engines.

Names are still your call — easy to adjust.