[Do not merge] URN Data Type

[dinoocch]

Adds a first-class URN logical data type for LinkedIn-style URN columns (urn:<namespace>:<type>:<key>), with compact per-entity-typed storage, polymorphic multi-entity dictionaries, and predicate / transform support tuned for the URN representation.

End-to-end JMH benchmarks show 6.9× to 11× speedups vs the equivalent STRING baseline on the queries that matter, and 32–42% smaller on-disk storage.

---

What this gives you

• A new logical DataType.URN you can declare on dimension columns.
• Optional per-column urnTypes schema annotation that declares which entity-type prefixes are valid (e.g. urn:li:memberId) and the physical key type (LONG / INT / STRING). Unspecified valueType defaults to STRING.
• One unified storage layout for every URN column, the polymorphic dictionary (see below). Single-entity columns are just polymorphic columns with one entry.
• Untyped URN columns (URN with no urnTypes) auto-detect entity types and per-entity value types at segment build — LONG when every key for an entity parses as a long, otherwise STRING.
• Two new transform functions: urnEntity(col) and urnKeyLong(col) (plus underscore aliases).
• Two new ingestion transformers: UrnValidationTransformer, UrnKeyEncodingTransformer.
• Predicate evaluator support: =, !=, IN, NOT IN, range, REGEXP_LIKE.
• Filter pushdown: WHERE urnKeyLong(col) OP N rewrites to a direct LONG predicate on the underlying typed column.
• No companion column anywhere: the only place entity index info lives per row is during build, in a transient row attribute that's dropped before metadata write.

---

Polymorphic dictionary (the only URN dictionary)

Every URN column with a dictionary uses UrnPolymorphicDictionary — a list of per-entity sub-dictionaries with cumulative global dict-ID offsets. Each sub-dictionary is a standard Pinot scalar dictionary chosen per entity from its declared valueType.

URN column "actorId"
├── entity: urn:li:memberId         → LongDictionary    (global dict IDs [0, N0))
├── entity: urn:li:corpUser         → StringDictionary  (global dict IDs [N0, N0+N1))
└── entity: urn:li:applicationName  → IntDictionary     (global dict IDs [N0+N1, N0+N1+N2))

The dictionary file on disk is the per-entity sub-buffers concatenated back-to-back. Per-entity ranges + dictClass + numBytesPerValue live in column metadata as urn.entityDictRanges, format urn:li:memberId,N0,LONG,8;urn:li:corpUser,N1,STRING,16;.... The reader slices the file at the byte boundaries derived from numBytesPerValue and wraps each region in the matching sub-dictionary class.

UrnPolymorphicDictionary itself is a thin router:

• getStringValue(globalId) → linear scan over the small per-entity offset table, compute the local ID, ask the sub-dict for its value, prepend the entity prefix.
• indexOf(urnString) → parse the entity prefix, call the matching sub-dict's indexOf(suffix), add the entity's global offset.
• getDictIdsInRange(lo, hi) → same-entity ranges binary-search the sub-dict via insertionIndexOf; cross-entity ranges return empty (semantically undefined).
• isSorted() returns true for single-entity columns (the one sub-dict is sorted, prefix is constant) and false for multi-entity (cross-entity dict-IDs are not lex-sorted in URN string space).

Single-entity URN columns are just polymorphic columns with one entry — no special-case reader, no historical UrnReconstructing*Dictionary wrappers. The prefix is prepended on read by the polymorphic wrapper itself.

---

Schema configuration

Single-entity typed (fastest path, smallest storage)

{
  "dimensionFieldSpecs": [
    { "name": "memberId", "dataType": "URN",
      "urnTypes": [{ "entityType": "urn:li:memberId", "valueType": "LONG" }] }
  ]
}

• Physical storage: a polymorphic dictionary with one LongDictionary sub-dict, keys only — the urn:li:memberId: prefix is stripped from every entry.
• Reads return full URN strings.
• All predicates accept full URN string literals.

Multi-entity typed (numeric — original sorted-LONG case)

{
  "name": "actorId", "dataType": "URN",
  "urnTypes": [
    { "entityType": "urn:li:corpUser",        "valueType": "LONG" },
    { "entityType": "urn:li:corpGroup",       "valueType": "LONG" },
    { "entityType": "urn:li:applicationName", "valueType": "LONG" }
  ]
}

• Three LongDictionary sub-dicts concatenated; per-entity ranges in metadata.
• Same LONG key can occur in multiple ranges (one per owning entity).
• Predicate WHERE col = 'urn:li:corpUser:100' parses the prefix, routes to the corpUser sub-dict, does a binary search for 100, returns a single absolute dict ID.

Multi-entity typed (mixed-type)

{
  "name": "subject", "dataType": "URN",
  "urnTypes": [
    { "entityType": "urn:li:memberId",  "valueType": "LONG"   },
    { "entityType": "urn:li:corpuser",  "valueType": "STRING" }
  ]
}

• One LongDictionary region + one StringDictionary region in the same dictionary file.
• The polymorphic wrapper routes per-row predicates and reads to the appropriate sub-dict.
• No more "fall back to plain STRING storage of the full URN" for mixed columns.

Untyped (auto-detected)

{ "name": "rawUrn", "dataType": "URN" }

• At segment build the creator parses every value as a URN, groups by entity prefix, and chooses a per-entity sub-dict type: LONG when every key for that entity parses as a long, otherwise STRING.
• The synthesized urnTypes are persisted in segment metadata so the column behaves like a declared polymorphic column at load and query time.
• UrnValidationTransformer rejects non-URN values at ingest (configurable fail / skip via continueOnError, mirroring JSON ingest).

valueType defaults to STRING

If a declared UrnType omits valueType, setUrnTypes normalizes it to STRING. Every URN entity ends up with a concrete sub-dictionary type — STRING is the safe default that accepts any key shape.

---

Query usage

Transform functions

SELECT urnEntity(memberId), urnKeyLong(memberId), COUNT(*)
FROM events
GROUP BY 1, 2

• urnEntity(col) → returns the entity prefix string (e.g. "urn:li:memberId"). Three strategies picked at init(); none parse URN strings per row:
  • CONSTANT — single-entity column: Arrays.fill once per block.
  • RANGE_LOOKUP — multi-entity column with per-entity dict-ID ranges in metadata: linear scan over the (typically ≤16) range table to find the owning entity per dict ID.
  • DICT_LOOKUP — defensive fallback for multi-entity STRING-stored columns without entity ranges (legacy segments). Precomputes a dictId → entityPrefix array at init time; O(1) per row.
• urnKeyLong(col) → returns the numeric key as long. Direct dictionary read on typed LONG entities.

Filtering

SELECT * FROM events WHERE memberId = 'urn:li:memberId:12345';
SELECT * FROM events WHERE memberId IN ('urn:li:memberId:1', 'urn:li:memberId:2');
SELECT * FROM events WHERE memberId BETWEEN 'urn:li:memberId:100' AND 'urn:li:memberId:200';
SELECT * FROM events WHERE urnKeyLong(memberId) > 1000;
SELECT * FROM events WHERE urnEntity(actorId) = 'urn:li:corpUser';

EQ / IN / range queries on full URN literals route through standard dictionary-based predicate evaluators — the polymorphic wrapper translates between full URN and per-entity-typed sub-dict lookups transparently.

---

Performance comparison

Dictionary storage (per entry)

Column type	Per-entry width	Dict file for 1M unique values
STRING (full URN as string)	~25 bytes padded to max length	~25 MB
URN(LONG)	8 bytes	8 MB (~3× smaller)
URN(INT)	4 bytes	4 MB (~6× smaller)
URN(STRING) (key suffix only)	length of key suffix	a few bytes less per entry than the full-URN STRING column (prefix is stripped)

The savings come from stripping the constant urn:li:memberId: prefix once into segment metadata instead of repeating it in every dictionary entry. The forward index is identical regardless — it stores dict IDs, not raw values.

For real LinkedIn-scale workloads the gap widens with key length:

Key length         STRING dict / entry (full URN)    LONG dict / entry    Ratio
4-digit            ~20 bytes                          8 bytes              2.5×
10-digit           ~26 bytes                          8 bytes              3.3×
13-digit           ~29 bytes                          8 bytes              3.6×

Lookup latency

Operation	STRING column	URN(LONG / INT)	URN(STRING)
col = 'urn:li:memberId:42'	binary search over UTF-8 strings	strip prefix + parse 42 → binary search 8-byte longs	strip prefix → binary search padded strings (byte compare)
Range col BETWEEN x AND y	string lex compare per probe	long compare per probe	string lex compare per probe
urnEntity(col)	per-row URN parse (or DICT_LOOKUP legacy fallback)	Arrays.fill with the constant prefix	same as LONG
urnKeyLong(col)	per-row substring + Long.parseLong	direct LongDictionary.getLongValue(dictId)	per-row Long.parseLong(suffix)

The LONG/INT path wins biggest on urnKeyLong and on range/equality predicates — primitive comparisons vs UTF-8 string comparisons over longer payloads. urnEntity() is constant-time on any single-entity URN column because the prefix is in metadata.

Multi-entity URN configurations

Configuration	Behavior
All entities LONG/INT	Per-entity LongDictionary/IntDictionary regions; per-entity dict-ID ranges in metadata
Mixed (LONG + STRING)	LongDictionary region back-to-back with StringDictionary region in one dictionary file; per-entity ranges select the sub-dict
Untyped (no schema urnTypes)	Auto-detected at build: per-entity LONG when all keys numeric, else STRING; same wire format as declared

Ingest cost

UrnKeyEncodingTransformer parses each URN row once. For polymorphic columns it replaces the URN string with the per-entity-typed key (Long/Integer/String) and stashes the entity index in a transient row field. The stats collector dedupes on (entityIdx, value) instead of full URN strings — smaller hash keys, faster comparison.

Net summary

For numeric-keyed URN columns (the common LinkedIn case): 3–6× dictionary storage win plus measurably faster predicates and urnKey*() transforms because everything is primitive-typed end to end. For STRING-keyed entities the storage savings are smaller (just the prefix bytes), but urnEntity() becomes O(1) instead of per-row URN parsing.

---

JMH benchmark results

pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkUrnTransform.java — 100K rows, 10K unique keys. Single-entity uses urn:li:memberId. Multi-entity rotates among urn:li:corpUser, urn:li:corpGroup, urn:li:applicationName.

Storage (bytes on disk)

Total segment size:

Layout	STRING baseline (untyped)	URN typed	Reduction
Single-entity	376,736	256,782	32%
Multi-entity	446,692	256,971	42%

Dictionary file alone:

Layout	STRING baseline (untyped)	URN typed	Reduction
Single-entity	40,008	80,008	— (auto-detected single-prefix STRING dict happens to beat fixed-width LONG for 4-digit keys)
Multi-entity	269,981	80,000	70% (3.37×)

So a multi-entity URN column with 1M unique URNs at LinkedIn scale: ~30 MB STRING dict → ~8 MB LONG dict, a ~22 MB-per-segment saving on the dict alone.

End-to-end filter (full FilterPlanNode → execution, µs/op)

Query	STRING baseline	URN typed (single)	URN typed (multi)
WHERE col = 'urn:...'	240	230	234
WHERE urnEntity(col) = 'urn:...'	13,941	1,647 (8.5×)	1,252 (11×)

Microbenchmarks (single-entity, µs/op)

Op	STRING	URN typed	Speedup
urnEntity() transform	1,028	156	6.6×
urnKeyLong() transform	1,028	138	7.5×
Dictionary.indexOf(String)	3,044	911	3.3×
Dictionary.getStringValue(dictId)	443	217	2.0×

Simple EQ filter is essentially a wash because forward-index scan dominates the per-query cost. The dict-lookup speedup is visible in microbenchmarks and in IN-with-many-values queries.

---

Implementation notes

SPI (pinot-spi)

• DataType.URN — logical URN type.
• DimensionFieldSpec.urnTypes, setUrnTypes (normalizes unspecified valueType to STRING).
• Predicates: usesPolymorphicUrnDictionary() (any URN with declared entities + valueTypes), usesSortedUrnDictionary() (all-numeric subset), hasTypedUrnStorage() (numeric stored type), isSingleUrnEntityType(), transientUrnEntityIdxFieldName().
• DimensionFieldSpec.EntityDictRange — data class carrying entityType, endDictIdExclusive, dictClass, numBytesPerValue.
• parseEntityDictRanges / serializeEntityDictRanges — 4-field-per-range wire format.
• Urn.tryParse / Urn.parse utility.

Build (pinot-segment-local)

• UrnValidationTransformer — rejects non-URN-shaped data at ingest (FAIL by default, SKIP when continueOnError=true, mirroring JSON ingest mode). Covers both declared-prefix and untyped columns.
• UrnKeyEncodingTransformer — for polymorphic columns: parses the URN, looks up the entity index, replaces the column value with the per-entity-typed key (Long/Integer/String), stashes the entity index in a transient row field.
• UrnMultiEntityPreIndexStatsCollector — dedupes (entityIdx, value) pairs via EntityValuePair.
• UrnPolymorphicDictionaryCreator — writes per-entity typed sub-dictionaries concatenated into one file; computes per-entity dict-ID ranges + byte ranges; handles empty input (null-only columns) gracefully.
• SegmentColumnarIndexCreator — routes any URN column to the polymorphic creator; tryUrnAutoDetection synthesizes polymorphic urnTypes for untyped columns (per-entity LONG when all keys numeric, else STRING).

Read (pinot-segment-local, pinot-segment-spi, pinot-core)

• UrnPolymorphicDictionary — the only URN dictionary. Routes indexOf, getStringValue, getDictIdsInRange, etc. to the matching per-entity sub-dictionary.
• DictionaryIndexType.read — when entityDictRanges is set, slices the file at per-entity byte offsets and dispatches each region to the declared sub-dict class (LongDictionary, IntDictionary, StringDictionary, ...).
• ColumnMetadataImpl.fromPropertiesConfiguration reads urnTypes and urn.entityDictRanges (the latter using getStringArray + rejoin to survive PropertiesConfiguration's comma split).
• Predicate evaluator factories (PredicateUtils, RangePredicateEvaluatorFactory, RegexpLikePredicateEvaluatorFactory, EQ/IN/NEQ/NIN) — case URN: routes URN columns through the STRING path; the polymorphic wrapper handles prefix/range translation.
• UrnEntityTransformFunction — CONSTANT / RANGE_LOOKUP / DICT_LOOKUP strategies; never parses URN strings per row.
• UrnKeyLongTransformFunction — direct LONG read on typed columns.
• TransformFunctionFactory — registers urnEntity, urnKeyLong, and underscore aliases.
• FilterPlanNode — rewrites urnKeyLong(typed_col) OP literal into a direct LONG-typed predicate on the underlying column.

Common (pinot-common)

• UrnFunctions — scalar UDF helpers.
• Standard case URN: additions in switch statements throughout serdes / encoders so URN flows through the STRING path.

---

Test coverage

98 URN-specific tests pass across pinot-spi, pinot-segment-local, pinot-core. Suites:

Suite	Module	Tests
DimensionFieldSpecUrnTest, UrnTest, FieldSpecTest	pinot-spi	21+
UrnFunctionsTest	pinot-common	29
UrnColumnSegmentTest, UrnPolymorphicDictionaryTest, UrnPolymorphicDictionaryCreatorTest, UrnKeyEncodingTransformerTest, UrnValidationTransformerTest, DictionaryIndexTypeTest	pinot-segment-local	62
UrnTransformFunctionTest, UrnEntityMultiEntityTest, UrnEntityDictLookupTest, FilterPlanNodeUrnRewriteTest	pinot-core	15+

Run locally with:

mvn test -pl pinot-segment-local,pinot-core,pinot-spi -am \
  -Dtest="DimensionFieldSpecUrnTest,UrnColumnSegmentTest,UrnPolymorphicDictionaryTest,UrnPolymorphicDictionaryCreatorTest,UrnKeyEncodingTransformerTest,UrnValidationTransformerTest,UrnTransformFunctionTest,UrnEntityMultiEntityTest,UrnEntityDictLookupTest" \
  -Dsurefire.failIfNoSpecifiedTests=false -Dscan=false -Dcheckstyle.skip

Always run the URN test suite with -am (or build all five modules together). A -pl pinot-core-only invocation will use stale upstream JARs from ~/.m2/.

---

Compatibility

• Untyped URN columns auto-detect their structure at build; no schema change is required to opt in.
• Adding urnTypes to a column is a forward-only schema change; existing segments aren't rewritten, but new segments will use the compact polymorphic storage.
• Each segment is fully self-describing — its urnTypes list, per-entity dict-ID ranges, and per-entity dictClass/numBytesPerValue are persisted in segment metadata and read back at load time.

Adding entity types over time

When you update a table's schema to add a new entity type to a URN column, existing segments are not retroactively rewritten — they keep using the urnTypes ordering and per-entity types they were built with, and continue to query correctly for the entity types they know about. Newly ingested segments pick up the extended schema. Lookups against an old segment for an entity type it doesn't know about return zero matches, which is the semantically correct answer (the segment contains no rows with that entity). Validated by UrnColumnSegmentTest.multiEntitySegmentsWithDifferentUrnTypesBothQueryable and ...ReorderedUrnTypesReconstructCorrectly.

Layout history

Earlier drafts of this PR used: a STRING-encoded dictionary plus a per-row __urnEntityIdx companion column; then a sorted-LONG UrnSortedDictionary; then a polymorphic wrapper over typed sub-dicts. The current implementation is the polymorphic wrapper as the only URN dictionary path, with no companion column and no legacy wire format. Segments built with earlier drafts of this PR need re-ingestion. First-time ingestion against any released version of the URN feature uses the final layout directly.

[siddharthteotia]

@dinoocch - I can spend time reviewing this today if this is somewhat ready / in shape. Let me know if not yet.

[siddharthteotia]

Assuming DictRange here is referring to segment dictionary info -- I don't think this information should be kept here. Dictionary is a property of segment storage format. Abstraction wise, schema / fieldSpec should not be aware of segment storage internals / state.

This belongs in ColumnMetadata or any of it's equivalents that live closer to segment layout. May be create a new UrnSegmentLayout.

[siddharthteotia]

Similar feedback here (as I gave earlier - https://github.com/linkedin/pinot/pull/206/changes#r3343763062).

This should not be in SPI

It should ideally be in pinot-segment-local (and may be exposed via a helper if needed)

[dinoocch]

lol yeah lots to clean up

[siddharthteotia]

Clarifying question - Is it possible to declare an URN without urnTypes ?

[siddharthteotia]

Do we support nested / composite URN ?

Something like "urn:li:dataset: (<another urn)"

How will the urnType be defined in this case? The valueType for the entityType will be URN ? I did not see an example for this in the PR description and yet to see in the code (so far during the review).

[siddharthteotia]

Upon reading more code, I am somewhat more confused about how we are handling these. Suggest clearly calling out our stance on this in the PR description.

• The UrnKeyEncodingTransformer seems to be silently NULLing out composite URN row if it detects URN is not simple.
• The UrnValidationTransformer is explicitly rejecting composite URNs by enforcing urn.isSimple()
• The UrnValidationTransformer is potentially accepting them. It checks only entrityPrefix membership and that will not catch urn:li:dataset( < another urn>)

[dinoocch]

Oops sorry this is not done! It's just a draft that I pointed claude at and steered it away from stupid ideas...

So there's probably a lot of kinda weird things - I wanted to start with a proof of concept to iterate on the savings a little and then invest in some more polish, but don't have much (any) dedicated time on this :)

I figure simple urns are a huge proportion of our current usage. This pr will handle the complex ones by treating the remaining part as a string

[siddharthteotia]

No problem. I happen to read the PR for some time and left feedback as otherwise I would have forgotten.

[siddharthteotia]

What are we referring to as "callers" here ?

Who is making a decision on how the URN is being laid out on disk ?

[siddharthteotia]

Partial Review - At a high level, I think DimensionFieldSpec is being bloated with decision ./ info / state on how the URN is stored on disk. I don't think that's the right abstraction. I recommend revisiting it.

[siddharthteotia]

Is it not possible to sort in lex order at the time of construction ? The current implementation will give incorrect min and max.

I see that for immutable segments, we are simply skipping the min and max for multi-entity URNs.

However, mutable segments cache min and max as rows arrive IIRC. Similarly during segment commit, we may copy wrong min and max ?

[siddharthteotia]

Leaving some more feedback / questions before I context switch out of this PR review for some time. Will review again when you find time to iterate on it.

• Do we need urnKeyLong() as a SQL function ? I feel the same outcome can potentially be achieved if the planner does implicit literal coercion

Example --

WHERE urnKeyLong(memberId) = 1000

works via FilterPlanNode rewrite to

WHERE memberId = 'urn:li:memberId:1000'.

Alternative - Planner sees memberId is a single-entity LONG-typed URN, auto-prepends the prefix.

WHERE memberId = 1000

• What are the semantics (functionality) for a filter on URN column matching multiple entities ? I checked this out locally and ran a test for WHERE actorId = 12345 on a multi-entity URN column and got empty results whereas the data did contain 12345

• In the initial version, do we plan to support for consuming segments as well ?

• Part of me thinks that in the initial version, we should restrict the support to only simple URNs and then treat URN as a first class data type in predicate evaluator like other primitives. My first impression is that the current implementation is trying to cater to both simpler and complex URNs while leaving the latter as standard STRING operations and that is preventing us from truly supporting URN as a first class type in the engine (across all dimensions of storage, query, and user abstraction / SQL construct wise).