[Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736
• Requires [Variant] Support creating Variants with pre-existing Metadata #8152
• Requires [Variant] Support Shredded Objects in variant_get: typed path access (STEP 1) #8150

Note this is likely one of the most complex parts of implementing Shredded Variants , so it is not a good first task

Please see more commentary on

• [Variant] API to construct Shredded Variant Arrays #7895

We are trying to support the general case of the variant_get function, which allows runtime dynamic access to Variants (either shredded or unshredded).

• We found in [Variant] Support Shredded Objects in variant_get #8083 that supporting variant_get is quite complicated (see here), so we are proposing to brake it down into multiple piece.

This ticket tracks
Support variant_get for Some(DataType::Struct) (nested shredding)

The idea here is that the user would specify a "shredding schema" (similar to what @friendlymatthew is sketching out in #7921) and the variant_get kernel would produce a VariantArray with the defined schema, extracting fields as necessary

Implementing this functionality will likely require the basic representation for shredded Variant arrays along with path traversal in variant_get. However, it does NOT cover the following (which are / will be broken into separate tickets)

• Support for retrieving as a specific non Struct data type (e.g. Some(DataType::Utf8))
• Retrieving any arbitrary path and returning what is there (no type specified)
• Retrieving any arbitrary path as a Variant (aka "unshredding")

Describe the solution you'd like
@scovich sketched out a high level design for Shredded Objects (see Representing Variant In Arrow Proposal: "Shredding an Object" and Variant Shredding::Objects) in this PR

• [WIP][Variant] Very rough pathfinding for variant get/shredding #8122

So roughly that means supporting

// get the named field of variant object as a typed field 
variant_get(array, "$.field_name", DataType::Struct <....>)

Where $.field_name represents some arbitrary VariantPath such as a for field "a", or a.b for field "b" of field "a"

And DataType::Struct is a "shredding schema" that reflects both value and typed_value

This should work for:

1. Variants where the field_name is in a typed_value
2. Variants where the field_name is not in the typed value

Describe alternatives you've considered

1. Add a test that manually constructs a shredded variant array (follow the example in the arrow proposal)
2. Add a test that calls variant_get appropriately
3. Implement the code

I suggest getting this working for non-nested obejcts first, and then working on nesting / pathing as a second pR

Additional context

Reference

• Variant Spec
• Variant Shredding Spec
• Representing Variant In Arrow Proposal

[scovich]

I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?

1. User provides a plain old ordinary struct schema whose non-leaf fields are all normal struct/list and whose leaf fields are all strongly-typed.
   • The top-level does not have a metadata column, for example. This really is just a plain ordinary struct.
   • A variant_get call should attempt to extract exactly that type, and either produce NULL or an error if any of the casting fails.
   • The result is a StructArray whose leaves are e.g. PrimitiveArray
2. User provides an official shredding schema -- a variant-tagged struct schema where the top-level has a metadata column, and each field (both leaf and non-leaf) is a struct with value/typed_value pair that is characteristic of shredding.
   • A variant_get call should attempt to shred the data at each level into its respective typed_value field, falling back to value when necessary.
   • The result is a VariantArray that may have an arbitrarily complex shreding schema.
3. And, just for fun, the user can do both at the same time -- providing a struct schema where some of the leaf fields are shredded variants.
   • The result is a StructArray where some of the leaves are VariantArray
   • Each of those variant leaves needs its own metadata column and has its own shredding schema, because it's now a stand-alone variant value (internal detail that they all can use a (shallow) copy of the original metadata column without having to re-encode anything).

[alamb]

I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?

sounds reasonable -- do you think we should file new tickets for each of the three kinds? Or maybe whoever works on this ticket can figure out the details and let us know if more tickets are needed

[sdf-jkl]

I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?

1. User provides a plain old ordinary struct schema whose non-leaf fields are all normal struct/list and whose leaf fields are all strongly-typed.

   • The top-level does not have a metadata column, for example. This really is just a plain ordinary struct.
   • A variant_get call should attempt to extract exactly that type, and either produce NULL or an error if any of the casting fails.
   • The result is a StructArray whose leaves are e.g. PrimitiveArray

We do have support for shredded Objects in variant_get via plain old ordinary struct schema

2. User provides an official shredding schema -- a variant-tagged struct schema where the top-level has a metadata column, and each field (both leaf and non-leaf) is a struct with value/typed_value pair that is characteristic of shredding.

   • A variant_get call should attempt to shred the data at each level into its respective typed_value field, falling back to value when necessary.
   • The result is a VariantArray that may have an arbitrarily complex shreding schema.

3. And, just for fun, the user can do both at the same time -- providing a struct schema where some of the leaf fields are shredded variants.

   • The result is a StructArray where some of the leaves are VariantArray
   • Each of those variant leaves needs its own metadata column and has its own shredding schema, because it's now a stand-alone variant value (internal detail that they all can use a (shallow) copy of the original metadata column without having to re-encode anything).

These two are not supported yet.

[scovich]

@sdf-jkl your assessment makes sense to me. I think we even had conversations in the past about a utility method to expand a "logical" shredding schema into the equivalent physical shredding schema. As in, caller provides a struct that matches the desired shredding (what paths to shred as struct, int, string, etc vs leave as unshredded variant), and the utility spits out the corresponding arrangement of value and typed_value column pairs.

We didn't pursue it back then because there was no way to know if we had a physical variant schema vs. a normal struct that happened to look variant-ish. Now that we have the variant extension type, a quick metadata check should distinguish the two and we could do what you say. Even better -- given the recursive nature of variant_get, we "should" get the mixed scenario "for free" once we have support for the two base cases.