Core, API, Spec: Extend partition statistics support to unpartitioned tables

[ebyhr]

Proposed Change

Currently, the partition statistics file is only supported for partitioned tables.

iceberg/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java

Line 207 in b987e60

Preconditions.checkArgument(Partitioning.isPartitioned(table), "Table must be partitioned");

While this design makes sense since it’s conceptually a “partition” statistics file, the information it contains is also valuable for unpartitioned tables.

For example, the Trino Iceberg connector currently needs to read manifest files (TableStatisticsReader) to build internal table statistics. If Iceberg provided a statistics file for unpartitioned tables as well, Trino (and other query engines) could leverage it directly, improving planning performance by avoiding expensive manifest reads.

• Fixes Extend partition statistics support to unpartitioned tables #14375

[ebyhr]

I'm happy to post this change to the mailing list if needed.

[ajantha-bhat]

yes. any spec change has to go for vote.

And whats the use case? You want to know data files, delete files, file size for unpartitioned table? We can use metadata tables for that Or maybe the new v4 manifests can solve it (with less I/O)?

Also, since the name itself is partitions stats. It makes odd to generate the file for unpartitioned table.

[ebyhr]

@ajantha-bhat Thanks, I'll post after exploring another approach, since Parquet writer can't write an empty struct.

We want to obtain the record count more quickly and efficiently for CBO. As mentioned in the PR description, Trino currently needs to read manifest files to estimate table size.

Even if v4 manifests address this, migration would take a long time. We need a solution that works for tables with format versions < v4 as well.

[ajantha-bhat]

It is still a behavior change for existing users.
Partition stats was computed only for partition tables. Now you are enabling for non-partition tables with NULL partition. Plus like I mentioned, it is still odd for non partition table to write this file as it is via compute_partition_stats method. Why does user has to call compute_partition_stats on non-partition table?

I can understand that you need a quick table level stats for CBO in Trino without doing I/O of all the manifests.

a) Can we introduce a new table level stats (along with NDV puffin file) in compute_table_stats? People can still refer it for both partition table and non-partition table if they need whole table level info?
b) Or Can we check if snapshot summary already has this table level stats you are looking for? (we don't have to do multiple IO of the files in that case). If not, can we enhance snapshot summary to include it?

[ebyhr]

This PR ran into a known issue in the Parquet writer, which cannot handle writing an empty struct. https://issues.apache.org/jira/browse/SPARK-20593

I'm going to change the value to NULL.

[ajantha-bhat]

I think we should explore other options as It is still a behavior change for existing users.

Partition stats was computed only for partition tables. Now you are enabling for non-partition tables with NULL partition. Plus like I mentioned, it is still odd for non partition table to write this file as it is via 'compute_partition_stats' method. Why does user has to call compute_partition_stats on non-partition table?

I can understand that you need a quick table level stats for CBO in Trino without doing I/O of all the manifests.

a) Can we introduce a new table level stats (along with NDV puffin file) in compute_table_stats? People can still refer it for both partition table and non-partition table if they need whole table level info?
b) Or Can we check if snapshot summary already has this table level stats you are looking for? (we don't have to do multiple IO of the files in that case). If not, can we enhance snapshot summary to include it?