Skip to content

Convert JSON to VariantArray without copying (8 - 32% faster)#7911

Merged
alamb merged 7 commits intoapache:mainfrom
alamb:alamb/append_variant_builder
Jul 23, 2025
Merged

Convert JSON to VariantArray without copying (8 - 32% faster)#7911
alamb merged 7 commits intoapache:mainfrom
alamb:alamb/append_variant_builder

Conversation

@alamb
Copy link
Copy Markdown
Contributor

@alamb alamb commented Jul 11, 2025

Which issue does this PR close?

Rationale for this change

In a quest to have the fastest and most efficient Variant implementation I would like to avoid copies if at all possible
Right now, to make a VariantArray first requires completing an individual buffer and appending it
to the array.

Let's make that faster by having the VariantBuilder append directly into the buffer

What changes are included in this PR?

  1. Add VariantBuilder::new_from_existing
  2. Add a VariantArrayBuilder::variant_builder that reuses the buffers

Are these changes tested?

  1. New unit tests
  2. Yes by existing tests

Are there any user-facing changes?

Hopefully faster performance

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] Convert JSON to Variant with fewer copies

4 participants