Disabling statistics across multiple columns

[GabrielM98]

Apache Iceberg version

1.10.1

Query engine

None

Please describe the bug 🐞

#12771 added a write table property (write.parquet.stats-enabled.column.<COLUMN_NAME>) to allow statistics to be disabled on a per-column basis. However, it appears that this only seems to work for a single column?

When adding a couple of properties to the table to disable stats across the wire_format_message and json_format_message columns, it appeared that stats were still being written to the Parquet file for the latter column. Here's some output from the Parquet CLI/DuckDB which I used to confirm this...

Row group 0:  count: 1000  117.33 B records  start: 4  total(compressed): 114.576 kB total(uncompressed):675.036 kB
--------------------------------------------------------------------------------
                                                                      type      encodings count     avg size   nulls   min / max
...
wire_format_message                                                   BINARY    Z   _     1000      36.82 B
json_format_message                                                   BINARY    Z   _     1000      36.02 B    0       "{"eventMetadata":{"uuid":..." / "{"eventMetadata":{"uuid":..."
...

➜  duckdb
DuckDB v1.4.4 (Andium) 6ddac802ff
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D ATTACH 'warehouse' AS iceberg_catalog (
      TYPE iceberg,
      ENDPOINT 'http://localhost:8181',
      AUTHORIZATION_TYPE 'none'
    );
D SELECT * FROM iceberg_table_properties(iceberg_catalog.events.entity_events);
┌─────────────────────────────────────────────────────────┬─────────┐
│                           key                           │  value  │
│                         varchar                         │ varchar │
├─────────────────────────────────────────────────────────┼─────────┤
│ write.parquet.compression-codec                         │ zstd    │
│ commit.retry.total-timeout-ms                           │ 120000  │
│ commit.retry.min-wait-ms                                │ 3000    │
│ write.parquet.stats-enabled.column.wire_format_message  │ false   │
│ write.distribution-mode                                 │ hash    │
│ commit.retry.num-retries                                │ 5       │
│ write.parquet.stats-enabled.column.json_format_message  │ false   │
│ commit.retry.max-wait-ms                                │ 60000   │
│ owner                                                   │ root    │
└─────────────────────────────────────────────────────────┴─────────┘

I was able to reproduce this bug in the tests by changing the string here to "false" and flipping the boolean in the assertion here to false. This resulted in the test failing as stats were still being written for the int_field.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Guosmilesmile]

I was able to reproduce this bug in the tests by changing the string here to "false" and flipping the boolean in the assertion here to false. This resulted in the test failing as stats were still being written for the int_field.

@GabrielM98 There’s a small issue with the reproduction steps in the test case. You changed the int_field setting to false, but the assertion you updated is checking string_field.

I tried setting the properties for both fields to false, and the Statistics I got back were all empty.

Could you provide more details, or confirm which Iceberg version you’re using?

Thanks!

[GabrielM98]

I tried setting the properties for both fields to false, and the Statistics I got back were all empty.

@Guosmilesmile apologies you're right re the issue with the reproduction steps. I'm seeing the stats return empty now too.

I'm seeing this behaviour using Iceberg version 1.10.1. This occurs when running a Docker Compose stack locally, using the Iceberg sink connector (again, same version as I've mentioned). So I'm wondering if the issue might be present there somewhere, in that it's not respecting the write table properties?

[moomindani]

I investigated this issue and was unable to reproduce it on main. I wrote the following test that disables stats for two columns (int_field and string_field) and verifies a third column (long_field) still has stats:

@Test
public void testMultipleColumnsStatisticsDisabled() throws Exception {
  Schema schema =
      new Schema(
          optional(1, "int_field", IntegerType.get()),
          optional(2, "string_field", Types.StringType.get()),
          optional(3, "long_field", Types.LongType.get()));

  File file = createTempFile(temp);

  List<GenericData.Record> records = Lists.newArrayListWithCapacity(5);
  org.apache.avro.Schema avroSchema = AvroSchemaUtil.convert(schema.asStruct());
  for (int i = 1; i <= 5; i++) {
    GenericData.Record record = new GenericData.Record(avroSchema);
    record.put("int_field", i);
    record.put("string_field", "test");
    record.put("long_field", (long) i);
    records.add(record);
  }

  write(
      file,
      schema,
      ImmutableMap.<String, String>builder()
          .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "int_field", "false")
          .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "string_field", "false")
          .buildOrThrow(),
      ParquetAvroWriter::buildWriter,
      records.toArray(new GenericData.Record[] {}));

  InputFile inputFile = Files.localInput(file);

  try (ParquetFileReader reader = ParquetFileReader.open(ParquetIO.file(inputFile))) {
    for (BlockMetaData block : reader.getFooter().getBlocks()) {
      for (ColumnChunkMetaData column : block.getColumns()) {
        boolean emptyStats = column.getStatistics().isEmpty();
        if (column.getPath().toDotString().equals("int_field")) {
          assertThat(emptyStats).as("int_field should not have statistics").isTrue();
        } else if (column.getPath().toDotString().equals("string_field")) {
          assertThat(emptyStats).as("string_field should not have statistics").isTrue();
        } else if (column.getPath().toDotString().equals("long_field")) {
          assertThat(emptyStats).as("long_field should have statistics").isFalse();
        }
      }
    }
  }
}

This test passes on main — both columns have stats disabled as expected. Could you share more details about your setup (query engine, catalog type, how the table properties are set)? The issue might be related to how properties are propagated in your specific environment rather than the parquet writer itself.

[mukund-thakur]

I was investigating as well and was thinking if we can add this test in the codebase but found something similar is already present https://github.com/apache/iceberg/blob/main/parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java#L229
So yes seem like some environment specific issue.