Skip to content

RFC: Encode parquet-format minor_version in thrift metadata #581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alamb wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

RFC: Encode parquet-format minor_version in thrift metadata #581

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bc61314
Encode parquet-format minor_version in thrift metadata
alamb Jun 9, 2026
cf448f1
tweaks
alamb Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 28 additions & 6 deletions src/main/thrift/parquet.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1363,16 +1363,38 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
/** Version of this file
/** Major Parquet Format Version
*
* As of December 2025, there is no agreed upon consensus of what constitutes
* version 2 of the file. For maximum compatibility with readers, writers should
* always populate "1" for version. For maximum compatibility with writers,
* readers should accept "1" and "2" interchangeably. All other versions are
* reserved for potential future use-cases.
* This corresponds to the highest major version of the parquet-format whose
* features the file uses. For example, if a file contains features from parquet-format
* version 2.4, then this field should be set to "2".
*
* Prior to 2026, some readers supported features added in version 2.0 and
* greater, but would reject files with the version set to 2. It was common
* practice for writers to populate "1" for version even if they used version
* 2.0.
*
* For maximum compatibility with writers, readers should accept "1" and "2"
* interchangeably. All other versions are reserved for potential future
* use-cases.
*/
1: required i32 version

/** Minor Parquet Format Version
*
* This corresponds to the highest minor version of the parquet-format whose
* features the file uses. For example, if a file contains features from
* parquet-format version 2.4, then this field should be set to "4".
*
* Note that Parquet does not follow semantic versioning and new
* forward-incompatible features, such as new encodings, can be added in
* minor versions. See the documentation[1] for more details on the versioning
* scheme and the features added in each version.
*
* [1]: http://parquet.apache.org/docs/file-format/versions

@alamb alamb Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This URL would correspond to apache/parquet-site#186 when published

*/
10: optional i32 minor_version

/** Parquet schema for this file. This schema contains metadata for all the columns.
* The schema is represented as a tree with a single root. The nodes of the tree
* are flattened to a list by doing a depth-first traversal.
Expand Down