Virtual Field Metadata for Semi-Structured (VARIANT) Columns

[jeffbuser]

Proposed Change

Proposed Change

Add a spec-level mechanism for Iceberg tables to declare "virtual fields" - typed metadata about known field paths inside semi-structured columns (VARIANT). Virtual fields let engines resolve types, push down predicates, and transparently redirect queries to extracted physical columns, all without requiring users to manage schema evolution manually.

The Problem

Iceberg v3 adds the VARIANT type where semi-structured data gets first-class storage. But once documents land in a VARIANT column, engines are blind: no type information, no predicate pushdown, no statistics. Every field access is a runtime parse-and-guess.

Meanwhile, in every semi-structured workload, a small set of field paths accounts for a majority of query access. Those hot fields deserve the same treatment as physical columns.

The Idea

Add a virtual fields array to Iceberg table metadata. Each entry declares a known field path inside a VARIANT column, its type, and (optionally) a pointer to a physical column that materializes it.

"virtual-fields": [
  {
    "source-column-id": 2,
    "field-path": "user.email",
    "field-id": 100,
    "type": "string",
    "extracted-column-id": 3
  },
  {
    "source-column-id": 2,
    "field-path": "user.age",
    "field-id": 101,
    "type": "int"
  }
]

That's it. Field path, type, and an optional redirect to a physical column.

What This Enables

Type consistency across engines. The table declares user.age is an int once. Spark, Trino, and Flink all read the same metadata. No more divergent type inference or manual CAST wrappers.

Transparent column extraction. When a hot field is promoted to a physical column, set extracted-column-id. Engines automatically rewrite queries to use the physical column (e.g., predicate pushdown, statistics, bloom filters) with zero query changes.

Progressive optimization. Day 1: store any JSON, all fields are virtual. Day 30: hot fields get extracted to physical columns automatically. Day 90: new fields appear in documents, get discovered, and the cycle continues. No manual schema management.

Catalog introspection. Tools can enumerate every known field path inside VARIANT columns so features such as auto-complete, data catalogs, governance can work without sampling.

How Engines Use It

1. Query references doc.user.age > 25
2. Engine checks virtual-fields for source-column-id=2, field-path=user.age
3. Found with extracted-column-id? Read the physical column directly (fast path)
4. Found without extraction? Use the declared type for casting (correct path)
5. Not found? Fall back to current VARIANT behavior (no regression)

Engines that don't understand virtual fields ignore them entirely. Pure backward compatibility.

Scope

• Targets Iceberg format version 3 (VARIANT type required)
• Adds virtual-fields array to table metadata JSON
• Virtual fields evolve like schema fields: add, widen type, drop, extract
• Changes produce new metadata versions (time-travel auditable)
• Optional discovery metadata in table properties (advisory, not required)

Prior Art

• Snowflake: Automatically extracts frequently-accessed VARIANT paths to columnar storage
• Elasticsearch: Dynamic mapping discovers field types from documents at ingestion
• Iceberg schema evolution: ID-based column tracking, type widening rules. Virtual fields will follow the same patterns

What This Is Not

• Not computed/derived columns (no SQL expressions, just field paths and types)
• Not a query engine feature (spec-level metadata, engine-independent)
• Not a new file format (no data file changes, metadata only)

Proposal document

https://docs.google.com/document/d/1rxX7MgkByVjY-fCXRY9xTlAxo7rVbMMs2V3S8NWCdoA/edit?usp=sharing

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[nssalian]

Thanks for sharing this @jeffbuser. I took a look and adding some thoughts here:

The shredding work in PR #14297 addresses several of the motivating problems here. The VariantShreddingAnalyzer infers types at write time and extracts frequently-occurring fields to typed Parquet columns (typed_value), which enables column statistics and predicate pushdown (#15510). Engines that understand the Parquet variant shredding spec read from the typed columns transparently; engines that don't would fall back to the binary value field.

A few questions on how virtual fields would interact with this:

1. Shredding already creates typed physical columns inside the variant group. How would extracted-column-id (pointing to a top-level schema column) coexist with typed_value columns in the same variant? Would engines need to check both?

2. Type consistency is a real gap in shredding today - the analyzer runs per batch, so different files can have different shredding layouts; I don't think this is incorrect though. Would virtual fields serve as the declaration of intent that the shredding writer then enforces? That would be a useful layering.

3. Catalog introspection is interesting. Surfacing shredded field paths at the metadata level would be a larger conversation - worth discussing whether that belongs in the spec or as a table property convention.

It feels like virtual fields could be a metadata layer on top of shredding rather than a parallel mechanism - declaring which paths should be typed and shredded, with the physical optimization handled at the file format level. That would avoid duplicating the extraction logic at two levels.

@aihuaxu @steveloughran @huaxingao PTAL

[jeffbuser]

@nssalian Appreciate you taking the time. Very thoughtful and relevant questions. My opinions of where I think the concept of virtual fields fits into the existing shredding work:

• "metadata layer on top of shredding" I think is the right way to think about this.

• "declaring which paths should be typed and shredded" Perhaps these are 2 different concepts as I imagine them. Paths being typed regardless of being shredded allows for the catalog introspection regardless of the field being physical/shredded.

1. extracted-column-id vs typed_value

These should be operating at different layers of the stack, not as competing extractions of the same data. typed_value is a physical artifact of the Parquet shredding layout. As I understand it, it lives inside the variant group and is unknown to the Iceberg schema. extracted-column-id would be an Iceberg schema-level object that says "this logical path has a stable identity and type within the table." The physical realization of that could be a shredded typed_value column, but the schema ID persists even if a file was written before shredding was enabled, or by an engine or file format (e.g., ORC) that doesn't implement the shredding spec.

2. Type consistency

I think virtual fields could be the enforcement layer for type consistency at a field level. The shredding writer can make a decision to enforce types in a virtual field, ignore them (backward compatability), or evolve/type-widen the virtual field.

Virtual field definitions would become the source of thru for the VariantShreddingAnalyzer if the virtual field metadata is already present for the table. The shredding writer could also become the one that creates/modifies the virtual field definition for a table as it infers new fields to be shredded that were not previously defined.

3. Catalog introspection

I think this is the larger conversation I wanted this Issue to surface. I believe the implementation would be in table properties regardless because it makes the most sense - but it seems like a spec update that defines the table property (and structure) would be needed to enforce tooling adoption and propagation in the case of the table property modification.