Extend partition statistics support to unpartitioned tables

[ebyhr]

Proposed Change

Currently, the partition statistics file is only supported for partitioned tables.

iceberg/core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java

Line 207 in b987e60

Preconditions.checkArgument(Partitioning.isPartitioned(table), "Table must be partitioned");

While this design makes sense since it’s conceptually a “partition” statistics file, the information it contains is also valuable for unpartitioned tables.

For example, the Trino Iceberg connector currently needs to read manifest files (TableStatisticsReader) to build internal table statistics. If Iceberg provided a statistics file for unpartitioned tables as well, Trino (and other query engines) could leverage it directly, improving planning performance by avoiding expensive manifest reads.

“Extending partition statistics support” is just one possible approach. We are open to other options as well.

Proposal document

No response

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[ebyhr]

cc: @ajantha-bhat @findepi @raunaqmorarka @sopel39

[manuzhang]

If Iceberg provided a statistics file for unpartitioned tables as well, Trino (and other query engines) could leverage it directly, improving planning performance by avoiding expensive manifest reads.

It looks to me this would be better handled in a separate place. Trino needs a stats file rather than a partition stats file.

[raunaqmorarka]

It looks to me this would be better handled in a separate place. Trino needs a stats file rather than a partition stats file.

This is not a Trino only problem, its a problem for any engine that wants to read table statistics for cost based optimizations in the planner. We already have a statistics file for per column NDV stats at the table level. However, we are forced to read manifest files to get all the other statistics (row count, column size, min, max, nulls count). Reading manifests is an expensive operation for large tables, or tables with lots of small manifests.
Now that we already have a concept of partition statistics in iceberg which helps us to avoid reading manifests to build table statistics, it is natural to extend that concept to unpartitioned tables as well. An unpartitioned table can be thought of as a partitioned table with a single partition.
Trino can of course create its own stats file for unpartitioned tables. However, that is an unsatisfactory solution as it becomes an engine specific thing. It also looks like poor design to have one solution for getting statistics for a partitioned table and an entirely different solution for unpartitioned tables.

[moomindani]

I agree with @raunaqmorarka's perspective — an unpartitioned table can be naturally modeled as a partitioned table with a single partition. Reusing the existing partition statistics infrastructure keeps the design consistent and avoids introducing a separate mechanism for table-level statistics.

Looking at the current implementation, the restriction to partitioned tables is enforced by a few precondition checks in PartitionStatsHandler and PartitionStatistics, while the spec itself does not explicitly prohibit unpartitioned tables. The EMPTY_PARTITION_FIELD constant already exists to represent an empty partition struct, which could serve as the single partition entry for unpartitioned tables.

[ebyhr]

#15896 removes the condition for supporting unpartitioned tables in partition stats.