Make push_batch_with_filter up to 3x faster

[Dandandan]

Which issue does this PR close?

• Closes Avoid copies for push_batch_with_filter for primitive types #9136

Rationale for this change

filter: primitive, 8192, nulls: 0, selectivity: 0.001
                        time:   [20.430 ms 20.678 ms 21.105 ms]
                        change: [−65.000% −64.516% −63.806%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) high mild
  6 (6.00%) high severe

filter: primitive, 8192, nulls: 0, selectivity: 0.01
                        time:   [3.3275 ms 3.3451 ms 3.3665 ms]
                        change: [−49.062% −48.663% −48.260%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  5 (5.00%) high severe

Benchmarking filter: primitive, 8192, nulls: 0, selectivity: 0.1: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.5s, enable flat sampling, or reduce sample count to 50.
filter: primitive, 8192, nulls: 0, selectivity: 0.1
                        time:   [1.4759 ms 1.4887 ms 1.5105 ms]
                        change: [−26.613% −23.553% −15.842%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild
  6 (6.00%) high severe

Benchmarking filter: primitive, 8192, nulls: 0, selectivity: 0.8: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.9s, enable flat sampling, or reduce sample count to 60.
filter: primitive, 8192, nulls: 0, selectivity: 0.8
                        time:   [1.3569 ms 1.3626 ms 1.3702 ms]
                        change: [−47.225% −46.850% −46.451%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe

filter: primitive, 8192, nulls: 0.1, selectivity: 0.001
                        time:   [23.231 ms 23.295 ms 23.376 ms]
                        change: [−69.694% −69.516% −69.351%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

filter: primitive, 8192, nulls: 0.1, selectivity: 0.01
                        time:   [5.4033 ms 5.4201 ms 5.4424 ms]
                        change: [−49.860% −49.590% −49.325%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

filter: primitive, 8192, nulls: 0.1, selectivity: 0.1
                        time:   [3.6111 ms 3.6270 ms 3.6475 ms]
                        change: [−27.778% −26.284% −25.286%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

filter: primitive, 8192, nulls: 0.1, selectivity: 0.8
                        time:   [3.6298 ms 3.7206 ms 3.8600 ms]
                        change: [−26.637% −24.714% −21.997%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[Dandandan]

@alamb you are probably interested in this

[alamb]

YAAAAASSS -- this is exactly the type of thing I was hoping for with BatchCoalescer. I will check this out shortly

[Dandandan]

Not sure if always better to take into account biggest_coalesce_batch_size

[alamb]

Yeah, I do think it would be good to take into account biggest_coalesce_batch_size but maybe we can do so as a follow on PR

[alamb]

run benchmark filter_kernels

[alamb]

show benchmark queue

[alamb-ghbot]

🤖 Hi @alamb, you asked to view the benchmark queue (#8951 (comment)).

Job	User	Benchmarks	Comment
arrow-8933-3613162300.sh	alamb	default	https://github.com/apache/arrow-rs/pull/8933#issuecomment-3613162300
arrow-8933-3613131981.sh	alamb	filter_kernels	https://github.com/apache/arrow-rs/pull/8933#issuecomment-3613131981
arrow-8951-3613212415.sh	alamb	filter_kernels	https://github.com/apache/arrow-rs/pull/8951#issuecomment-3613212415

[Dandandan]

Hm it seems it contains a bug, probably makes the benchmark results off as well (will take a look tomorrow).

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (0872a9b) to ed9efe7 diff
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=coalesce_batches_filter
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                         coalesce_batches_filter                main
-----                                                                         -----------------------                ----
filter context decimal128 (kept 1/2)                                          1.36     57.5±5.45µs        ? ?/sec    1.00     42.1±1.93µs        ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024)                   1.00     55.7±4.51µs        ? ?/sec    1.09     60.5±0.29µs        ? ?/sec
filter context decimal128 low selectivity (kept 1/1024)                       1.00    242.4±0.35ns        ? ?/sec    1.06    256.0±1.60ns        ? ?/sec
filter context f32 (kept 1/2)                                                 1.00     77.7±1.20µs        ? ?/sec    1.00     78.0±2.52µs        ? ?/sec
filter context f32 high selectivity (kept 1023/1024)                          1.00      9.9±0.32µs        ? ?/sec    1.01     10.1±0.30µs        ? ?/sec
filter context f32 low selectivity (kept 1/1024)                              1.00    444.2±7.59ns        ? ?/sec    1.06   469.4±13.36ns        ? ?/sec
filter context fsb with value length 20 (kept 1/2)                            1.00     60.7±1.16µs        ? ?/sec    1.00     60.7±0.37µs        ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     60.7±0.36µs        ? ?/sec    1.00     60.7±0.56µs        ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     60.6±0.26µs        ? ?/sec    1.00     60.8±1.05µs        ? ?/sec
filter context fsb with value length 5 (kept 1/2)                             1.00     60.8±1.45µs        ? ?/sec    1.00     60.7±1.02µs        ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     60.7±0.71µs        ? ?/sec    1.00     60.8±1.22µs        ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024)          1.01     61.2±3.05µs        ? ?/sec    1.00     60.8±0.90µs        ? ?/sec
filter context fsb with value length 50 (kept 1/2)                            1.00     60.7±0.46µs        ? ?/sec    1.00     60.8±0.46µs        ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     61.0±2.06µs        ? ?/sec    1.00     60.7±0.55µs        ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     60.8±1.25µs        ? ?/sec    1.00     60.8±1.00µs        ? ?/sec
filter context i32 (kept 1/2)                                                 1.01     16.6±0.28µs        ? ?/sec    1.00     16.5±0.30µs        ? ?/sec
filter context i32 high selectivity (kept 1023/1024)                          1.04      6.5±0.20µs        ? ?/sec    1.00      6.2±0.17µs        ? ?/sec
filter context i32 low selectivity (kept 1/1024)                              1.00    236.0±5.78ns        ? ?/sec    1.05    246.9±1.45ns        ? ?/sec
filter context i32 w NULLs (kept 1/2)                                         1.00     77.8±2.17µs        ? ?/sec    1.00     77.9±0.80µs        ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00     10.1±0.52µs        ? ?/sec    1.04     10.5±0.18µs        ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024)                      1.00    446.9±4.94ns        ? ?/sec    1.06    471.6±6.49ns        ? ?/sec
filter context mixed string view (kept 1/2)                                   1.00    109.0±3.21µs        ? ?/sec    1.11    120.7±3.20µs        ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024)            1.00     53.9±2.45µs        ? ?/sec    1.03     55.3±2.41µs        ? ?/sec
filter context mixed string view low selectivity (kept 1/1024)                1.00   654.9±19.57ns        ? ?/sec    1.04   677.9±18.99ns        ? ?/sec
filter context short string view (kept 1/2)                                   1.00    104.2±1.47µs        ? ?/sec    1.08    112.2±3.44µs        ? ?/sec
filter context short string view high selectivity (kept 1023/1024)            1.02     55.5±1.25µs        ? ?/sec    1.00     54.5±0.23µs        ? ?/sec
filter context short string view low selectivity (kept 1/1024)                1.00    464.2±2.70ns        ? ?/sec    1.06    491.4±7.75ns        ? ?/sec
filter context string (kept 1/2)                                              1.03   599.4±17.30µs        ? ?/sec    1.00    582.1±5.14µs        ? ?/sec
filter context string dictionary (kept 1/2)                                   1.00     17.0±0.13µs        ? ?/sec    1.02     17.3±0.27µs        ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024)            1.00      7.0±0.34µs        ? ?/sec    1.02      7.2±0.27µs        ? ?/sec
filter context string dictionary low selectivity (kept 1/1024)                1.02    847.3±9.58ns        ? ?/sec    1.00    829.8±3.84ns        ? ?/sec
filter context string dictionary w NULLs (kept 1/2)                           1.00     78.8±1.05µs        ? ?/sec    1.00     78.9±2.34µs        ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     10.7±0.41µs        ? ?/sec    1.01     10.8±0.35µs        ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.01  1076.9±14.42ns        ? ?/sec    1.00  1067.4±30.14ns        ? ?/sec
filter context string high selectivity (kept 1023/1024)                       1.00   703.0±13.80µs        ? ?/sec    1.00   703.8±19.93µs        ? ?/sec
filter context string low selectivity (kept 1/1024)                           1.00  1016.7±52.17ns        ? ?/sec    1.02  1036.2±34.58ns        ? ?/sec
filter context u8 (kept 1/2)                                                  1.00     14.9±0.05µs        ? ?/sec    1.00     15.0±0.14µs        ? ?/sec
filter context u8 high selectivity (kept 1023/1024)                           1.00  1829.3±23.69ns        ? ?/sec    1.11      2.0±0.01µs        ? ?/sec
filter context u8 low selectivity (kept 1/1024)                               1.00    231.0±5.30ns        ? ?/sec    1.03    238.8±0.83ns        ? ?/sec
filter context u8 w NULLs (kept 1/2)                                          1.00     75.9±0.20µs        ? ?/sec    1.00     76.1±0.78µs        ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      5.1±0.08µs        ? ?/sec    1.05      5.4±0.06µs        ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00   441.3±12.39ns        ? ?/sec    1.06    467.4±2.32ns        ? ?/sec
filter decimal128 (kept 1/2)                                                  1.00     49.5±0.83µs        ? ?/sec    1.18     58.6±2.81µs        ? ?/sec
filter decimal128 high selectivity (kept 1023/1024)                           1.17     61.3±2.70µs        ? ?/sec    1.00     52.6±1.25µs        ? ?/sec
filter decimal128 low selectivity (kept 1/1024)                               1.00      2.9±0.09µs        ? ?/sec    1.13      3.2±0.08µs        ? ?/sec
filter f32 (kept 1/2)                                                         1.07    166.6±7.99µs        ? ?/sec    1.00    156.4±2.84µs        ? ?/sec
filter fsb with value length 20 (kept 1/2)                                    1.12    141.6±1.19µs        ? ?/sec    1.00    126.0±3.73µs        ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024)             1.11     76.6±1.07µs        ? ?/sec    1.00     68.7±1.04µs        ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      2.7±0.09µs        ? ?/sec    1.29      3.5±0.10µs        ? ?/sec
filter fsb with value length 5 (kept 1/2)                                     1.17    141.8±2.30µs        ? ?/sec    1.00    121.1±0.87µs        ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024)              1.00     10.8±0.16µs        ? ?/sec    1.05     11.3±0.33µs        ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      2.6±0.08µs        ? ?/sec    1.28      3.3±0.02µs        ? ?/sec
filter fsb with value length 50 (kept 1/2)                                    1.05    189.3±7.05µs        ? ?/sec    1.00    181.1±9.22µs        ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024)             1.00    255.5±8.77µs        ? ?/sec    1.03    264.3±6.26µs        ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      2.6±0.03µs        ? ?/sec    1.27      3.3±0.10µs        ? ?/sec
filter i32 (kept 1/2)                                                         1.25     53.8±0.68µs        ? ?/sec    1.00     43.2±0.31µs        ? ?/sec
filter i32 high selectivity (kept 1023/1024)                                  1.05      8.9±0.48µs        ? ?/sec    1.00      8.4±0.32µs        ? ?/sec
filter i32 low selectivity (kept 1/1024)                                      1.24      2.9±0.06µs        ? ?/sec    1.00      2.4±0.03µs        ? ?/sec
filter optimize (kept 1/2)                                                    1.01     54.8±2.99µs        ? ?/sec    1.00     54.5±1.51µs        ? ?/sec
filter optimize high selectivity (kept 1023/1024)                             1.18      3.1±0.14µs        ? ?/sec    1.00      2.6±0.01µs        ? ?/sec
filter optimize low selectivity (kept 1/1024)                                 1.00      2.7±0.00µs        ? ?/sec    1.00      2.7±0.02µs        ? ?/sec
filter run array (kept 1/2)                                                   1.03   436.4±17.42µs        ? ?/sec    1.00    422.5±4.27µs        ? ?/sec
filter run array high selectivity (kept 1023/1024)                            1.01    452.6±7.45µs        ? ?/sec    1.00   449.3±12.94µs        ? ?/sec
filter run array low selectivity (kept 1/1024)                                1.01   336.4±10.57µs        ? ?/sec    1.00    334.5±2.82µs        ? ?/sec
filter single record batch                                                    1.23     54.3±2.92µs        ? ?/sec    1.00     44.2±0.07µs        ? ?/sec
filter u8 (kept 1/2)                                                          1.00     45.5±0.99µs        ? ?/sec    1.00     45.7±0.44µs        ? ?/sec
filter u8 high selectivity (kept 1023/1024)                                   1.05      4.0±0.11µs        ? ?/sec    1.00      3.8±0.04µs        ? ?/sec
filter u8 low selectivity (kept 1/1024)                                       1.00      3.0±0.05µs        ? ?/sec    1.12      3.3±0.11µs        ? ?/sec

[Dandandan]

run benchmark coalesce_kernels

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: arrow_reader, concatenate_kernels, filter_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...
Unsupported benchmarks: coalesce_kernels.

[Dandandan]

@alamb I think it's ok now - I called AI (Opus 4.5) for some help on the find_nth_set_bit_position function.

Mainly needs some polish and seeing if we can improve the filter: primitive, 8192, nulls: 0.1, selectivity: 0.8 case.

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[alamb-ghbot]

🤖 Hi @Dandandan, thanks for the request (#8951 (comment)).

scrape_comments.py only supports whitelisted benchmarks.

• Standard: (none)
• Criterion: array_iter, arrow_reader, arrow_reader_clickbench, arrow_reader_row_filter, arrow_statistics, arrow_writer, bitwise_kernel, boolean_kernels, buffer_bit_ops, cast_kernels, coalesce_kernels, comparison_kernels, concatenate_kernel, csv_writer, encoding, filter_kernels, interleave_kernels, json-reader, metadata, row_format, take_kernels, union_array, variant_builder, variant_kernels, variant_validation, view_types, zip_kernels

Please choose one or more of these with run benchmark <name> or run benchmark <name1> <name2>...

[Dandandan]

run benchmark coalesce_kernels

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (0dbd786) to 9c6065c diff
BENCH_NAME=coalesce_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=coalesce_batches_filter
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                                coalesce_batches_filter                main
-----                                                                                -----------------------                ----
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    249.5±2.28ms        ? ?/sec    1.00    250.5±3.57ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.1±0.36ms        ? ?/sec    1.00      8.1±0.11ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.05      4.1±0.13ms        ? ?/sec    1.00      3.9±0.08ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.05ms        ? ?/sec    1.08      3.4±0.03ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    234.1±2.50ms        ? ?/sec    1.00    234.4±2.89ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.07ms        ? ?/sec    1.02      9.0±0.17ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.06      4.5±0.18ms        ? ?/sec    1.00      4.2±0.04ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.10ms        ? ?/sec    1.25      4.4±0.04ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.8±1.12ms        ? ?/sec    1.09     57.5±1.21ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.9±0.18ms        ? ?/sec    1.04     11.3±0.18ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.46ms        ? ?/sec    1.03      9.5±0.37ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.7±0.29ms        ? ?/sec    1.27      9.8±0.13ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.9±1.15ms        ? ?/sec    1.02     67.1±0.55ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.4±0.10ms        ? ?/sec    1.01     12.6±0.31ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.03      9.8±0.34ms        ? ?/sec    1.00      9.5±0.35ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.1±0.13ms        ? ?/sec    1.15      9.3±0.14ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.4±0.37ms        ? ?/sec    1.63     46.3±0.69ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.6±0.06ms        ? ?/sec    1.24      5.7±0.16ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.16      5.0±0.21ms        ? ?/sec    1.00      4.3±0.18ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.8±0.06ms        ? ?/sec    1.02      2.9±0.09ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     47.1±0.78ms        ? ?/sec    1.19     55.9±0.52ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.2±0.11ms        ? ?/sec    1.03      7.5±0.08ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.17      6.2±0.28ms        ? ?/sec    1.00      5.3±0.24ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.1±0.02ms        ? ?/sec    1.66      3.6±0.04ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.2±0.36ms        ? ?/sec    1.53     41.6±0.44ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.03ms        ? ?/sec    1.20      4.4±0.03ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.23      2.7±0.22ms        ? ?/sec    1.00      2.2±0.17ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1184.8±26.03µs        ? ?/sec    1.10  1304.6±12.37µs        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     45.2±0.79ms        ? ?/sec    1.13     51.1±0.60ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.4±0.09ms        ? ?/sec    1.02      6.5±0.05ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.20      4.0±0.20ms        ? ?/sec    1.00      3.4±0.20ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.3±0.06ms        ? ?/sec    1.53      3.6±0.02ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.3±1.40ms        ? ?/sec    1.34     93.0±1.97ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.11ms        ? ?/sec    1.29      8.9±0.31ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.4±0.27ms        ? ?/sec    1.13      3.8±0.37ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1617.3±17.28µs        ? ?/sec    1.73      2.8±0.04ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    106.7±1.41ms        ? ?/sec    1.14    121.3±2.03ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.2±0.46ms        ? ?/sec    1.08     14.2±0.44ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.11      7.0±0.41ms        ? ?/sec    1.00      6.3±0.30ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.08ms        ? ?/sec    1.91      8.5±0.10ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     53.1±1.20ms        ? ?/sec    1.20     63.9±0.39ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.05      7.7±0.06ms        ? ?/sec    1.00      7.3±0.09ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.46      6.1±0.31ms        ? ?/sec    1.00      4.2±0.37ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            2.04      2.6±0.03ms        ? ?/sec    1.00  1291.6±15.36µs        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     74.9±0.41ms        ? ?/sec    1.09     81.4±0.58ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.3±0.33ms        ? ?/sec    1.07     11.0±0.41ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.37      7.1±0.35ms        ? ?/sec    1.00      5.2±0.26ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.02      3.7±0.18ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec

[Dandandan]

Hmm that's weird, my local results are so different.

[Dandandan]

run benchmark coalesce_kernels

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (0a5a575) to 9c6065c diff
BENCH_NAME=coalesce_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=coalesce_batches_filter
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                                coalesce_batches_filter                main
-----                                                                                -----------------------                ----
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    251.5±2.74ms        ? ?/sec    1.01    253.5±2.74ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.9±0.08ms        ? ?/sec    1.05      8.3±0.19ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.01      4.0±0.12ms        ? ?/sec    1.00      4.0±0.11ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.03ms        ? ?/sec    1.08      3.4±0.03ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.00    238.0±2.71ms        ? ?/sec    1.00    238.1±3.54ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.9±0.11ms        ? ?/sec    1.01      9.0±0.12ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.02      4.4±0.09ms        ? ?/sec    1.00      4.3±0.12ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.6±0.03ms        ? ?/sec    1.24      4.4±0.03ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     53.0±1.15ms        ? ?/sec    1.08     57.2±0.57ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.9±0.14ms        ? ?/sec    1.03     11.3±0.15ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.2±0.41ms        ? ?/sec    1.00      9.2±0.38ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.5±0.13ms        ? ?/sec    1.35     10.1±0.20ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.6±0.89ms        ? ?/sec    1.02     67.1±0.95ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.5±0.13ms        ? ?/sec    1.01     12.6±0.17ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.01      9.6±0.28ms        ? ?/sec    1.00      9.5±0.32ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.1±0.14ms        ? ?/sec    1.16      9.4±0.15ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.8±0.13ms        ? ?/sec    1.58     45.5±0.25ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.7±0.25ms        ? ?/sec    1.21      5.7±0.07ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.20      5.3±0.24ms        ? ?/sec    1.00      4.5±0.23ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.01      2.9±0.03ms        ? ?/sec    1.00      2.9±0.05ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     47.7±0.56ms        ? ?/sec    1.17     56.0±0.40ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.4±0.04ms        ? ?/sec    1.02      7.5±0.04ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.14      6.1±0.17ms        ? ?/sec    1.00      5.4±0.24ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.1±0.05ms        ? ?/sec    1.70      3.5±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.6±0.12ms        ? ?/sec    1.49     41.1±0.18ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.04ms        ? ?/sec    1.19      4.4±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.31      2.8±0.22ms        ? ?/sec    1.00      2.2±0.16ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1003.8±66.37µs        ? ?/sec    1.31  1318.7±26.64µs        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     45.8±0.42ms        ? ?/sec    1.11     50.8±0.30ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.5±0.22ms        ? ?/sec    1.00      6.5±0.13ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.20      4.0±0.23ms        ? ?/sec    1.00      3.4±0.22ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.61      3.6±0.02ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     69.7±1.11ms        ? ?/sec    1.32     92.2±1.66ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.15ms        ? ?/sec    1.27      8.8±0.12ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.04      3.5±0.40ms        ? ?/sec    1.00      3.4±0.22ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00   1610.6±7.85µs        ? ?/sec    1.73      2.8±0.02ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    107.1±0.85ms        ? ?/sec    1.13    120.7±1.35ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.1±0.07ms        ? ?/sec    1.08     14.1±0.55ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.7±0.37ms        ? ?/sec    1.01      6.8±0.35ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.11ms        ? ?/sec    1.89      8.4±0.17ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     53.5±1.44ms        ? ?/sec    1.18     63.4±0.28ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.05      7.7±0.16ms        ? ?/sec    1.00      7.3±0.19ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.67      6.1±0.24ms        ? ?/sec    1.00      3.7±0.17ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.63      2.1±0.07ms        ? ?/sec    1.00   1277.3±8.95µs        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     75.7±0.33ms        ? ?/sec    1.07     80.9±0.95ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.5±0.05ms        ? ?/sec    1.06     11.1±0.23ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.42      7.1±0.27ms        ? ?/sec    1.00      5.0±0.12ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.2±0.03ms        ? ?/sec    1.16      3.7±0.03ms        ? ?/sec

[Dandandan]

run benchmark coalesce_kernels

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (3fbd560) to 9c6065c diff
BENCH_NAME=coalesce_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=coalesce_batches_filter
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                                coalesce_batches_filter                main
-----                                                                                -----------------------                ----
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    247.7±3.30ms        ? ?/sec    1.01    250.1±2.54ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      7.8±0.13ms        ? ?/sec    1.04      8.1±0.11ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      3.9±0.12ms        ? ?/sec    1.01      4.0±0.10ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.1±0.03ms        ? ?/sec    1.10      3.4±0.04ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.01    235.7±2.80ms        ? ?/sec    1.00    234.4±3.30ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.7±0.09ms        ? ?/sec    1.01      8.8±0.09ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.4±0.11ms        ? ?/sec    1.00      4.4±0.13ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.5±0.05ms        ? ?/sec    1.25      4.4±0.06ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     52.5±1.43ms        ? ?/sec    1.09     57.4±0.91ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     10.8±0.14ms        ? ?/sec    1.03     11.1±0.12ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.01      8.9±0.38ms        ? ?/sec    1.00      8.8±0.29ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.4±0.12ms        ? ?/sec    1.37     10.1±0.46ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     65.1±0.79ms        ? ?/sec    1.04     67.7±1.28ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.02     12.6±0.21ms        ? ?/sec    1.00     12.4±0.39ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.02      9.4±0.37ms        ? ?/sec    1.00      9.2±0.30ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      7.9±0.14ms        ? ?/sec    1.17      9.2±0.13ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.2±0.49ms        ? ?/sec    1.63     45.8±0.31ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.6±0.05ms        ? ?/sec    1.22      5.6±0.07ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.17      5.1±0.19ms        ? ?/sec    1.00      4.3±0.25ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.7±0.07ms        ? ?/sec    1.03      2.8±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.4±0.40ms        ? ?/sec    1.21     56.1±0.42ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.2±0.13ms        ? ?/sec    1.05      7.6±0.22ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.18      6.0±0.24ms        ? ?/sec    1.00      5.1±0.28ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.1±0.04ms        ? ?/sec    1.74      3.6±0.04ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     26.9±0.37ms        ? ?/sec    1.55     41.5±0.36ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.03ms        ? ?/sec    1.21      4.4±0.03ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.27      2.8±0.18ms        ? ?/sec    1.00      2.2±0.15ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00   998.6±17.38µs        ? ?/sec    1.32  1316.0±19.98µs        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.6±0.38ms        ? ?/sec    1.15     51.1±1.13ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.4±0.23ms        ? ?/sec    1.03      6.6±0.29ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.22      4.1±0.15ms        ? ?/sec    1.00      3.3±0.18ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.65      3.6±0.08ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     68.1±1.18ms        ? ?/sec    1.38     93.9±0.96ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.8±0.04ms        ? ?/sec    1.29      8.8±0.08ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.00      3.5±0.39ms        ? ?/sec    1.00      3.5±0.30ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1514.9±33.93µs        ? ?/sec    1.89      2.9±0.03ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    105.2±0.60ms        ? ?/sec    1.15    121.2±1.38ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     12.9±0.07ms        ? ?/sec    1.09     14.2±0.13ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.04      6.6±0.44ms        ? ?/sec    1.00      6.4±0.37ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.3±0.11ms        ? ?/sec    1.95      8.5±0.06ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     52.1±0.14ms        ? ?/sec    1.23     64.0±0.37ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.03      7.6±0.03ms        ? ?/sec    1.00      7.3±0.11ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.60      6.1±0.11ms        ? ?/sec    1.00      3.8±0.29ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.58      2.1±0.05ms        ? ?/sec    1.00  1301.1±23.53µs        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     74.4±1.68ms        ? ?/sec    1.09     81.5±0.58ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.2±0.19ms        ? ?/sec    1.09     11.2±0.28ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.28      7.2±0.29ms        ? ?/sec    1.00      5.6±0.36ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.1±0.05ms        ? ?/sec    1.18      3.7±0.04ms        ? ?/sec

[Dandandan]

run benchmark coalesce_kernels

[alamb-ghbot]

🤖 ./gh_compare_arrow.sh gh_compare_arrow.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (d3cf7cb) to ebace17 diff
BENCH_NAME=coalesce_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=coalesce_batches_filter
Results will be posted here when complete

[alamb-ghbot]

🤖: Benchmark completed

Details

group                                                                                coalesce_batches_filter                main
-----                                                                                -----------------------                ----
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.00    254.9±2.59ms        ? ?/sec    1.02    259.0±3.61ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      8.0±0.13ms        ? ?/sec    1.04      8.3±0.13ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.00      4.0±0.09ms        ? ?/sec    1.01      4.0±0.06ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.00      3.2±0.05ms        ? ?/sec    1.10      3.5±0.08ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.02    242.7±2.85ms        ? ?/sec    1.00    237.6±3.16ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00      8.9±0.07ms        ? ?/sec    1.02      9.1±0.26ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.03      4.5±0.17ms        ? ?/sec    1.00      4.4±0.14ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      3.6±0.04ms        ? ?/sec    1.28      4.6±0.08ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     53.3±1.27ms        ? ?/sec    1.03     54.9±0.75ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.00     11.1±0.32ms        ? ?/sec    1.03     11.3±0.20ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00      9.3±0.15ms        ? ?/sec    1.00      9.3±0.23ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.00      7.8±0.33ms        ? ?/sec    1.44     11.2±0.89ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.01     66.3±1.66ms        ? ?/sec    1.00     65.8±1.89ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     12.5±0.15ms        ? ?/sec    1.00     12.4±0.29ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00      9.7±0.35ms        ? ?/sec    1.03      9.9±0.34ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.00      8.7±0.53ms        ? ?/sec    1.10      9.5±0.16ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.00     28.3±0.28ms        ? ?/sec    1.52     43.0±0.62ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      4.6±0.05ms        ? ?/sec    1.20      5.6±0.05ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.15      5.2±0.15ms        ? ?/sec    1.00      4.5±0.17ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.00      2.9±0.13ms        ? ?/sec    1.00      2.9±0.07ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.00     46.8±0.24ms        ? ?/sec    1.14     53.3±0.46ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.00      7.3±0.14ms        ? ?/sec    1.04      7.6±0.17ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.14      6.1±0.14ms        ? ?/sec    1.00      5.4±0.17ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.00      2.0±0.03ms        ? ?/sec    1.85      3.7±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.00     27.1±0.25ms        ? ?/sec    1.42     38.6±0.74ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      3.7±0.03ms        ? ?/sec    1.14      4.2±0.05ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.19      2.6±0.06ms        ? ?/sec    1.00      2.2±0.23ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00  1002.1±18.04µs        ? ?/sec    1.32  1318.2±16.16µs        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.00     44.7±0.26ms        ? ?/sec    1.07     47.9±0.35ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.00      6.4±0.07ms        ? ?/sec    1.01      6.4±0.08ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.15      3.9±0.12ms        ? ?/sec    1.00      3.4±0.15ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      2.2±0.02ms        ? ?/sec    1.73      3.8±0.05ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.001                                1.00     68.8±1.94ms        ? ?/sec    1.24     85.1±0.58ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.01                                 1.00      6.9±0.05ms        ? ?/sec    1.22      8.4±0.10ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.1                                  1.04      3.6±0.25ms        ? ?/sec    1.00      3.4±0.37ms        ? ?/sec
filter: primitive, 8192, nulls: 0, selectivity: 0.8                                  1.00  1515.7±17.60µs        ? ?/sec    1.84      2.8±0.03ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.001                              1.00    106.0±1.07ms        ? ?/sec    1.07    113.4±1.28ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.01                               1.00     13.0±0.11ms        ? ?/sec    1.07     13.9±0.27ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.1                                1.00      6.7±0.39ms        ? ?/sec    1.01      6.7±0.34ms        ? ?/sec
filter: primitive, 8192, nulls: 0.1, selectivity: 0.8                                1.00      4.4±0.05ms        ? ?/sec    2.05      8.9±0.07ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     52.6±0.31ms        ? ?/sec    1.21     63.6±0.88ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.02      7.6±0.07ms        ? ?/sec    1.00      7.5±0.06ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.54      6.1±0.26ms        ? ?/sec    1.00      4.0±0.36ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.57      2.1±0.06ms        ? ?/sec    1.00  1307.5±19.67µs        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     75.1±0.93ms        ? ?/sec    1.10     82.2±1.02ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.00     10.4±0.15ms        ? ?/sec    1.07     11.1±0.15ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.27      6.7±0.11ms        ? ?/sec    1.00      5.3±0.30ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.00      3.1±0.08ms        ? ?/sec    1.27      3.9±0.05ms        ? ?/sec

[Dandandan]

The plan is now to finalize #9284 to see it's effects on this kernel.
(I believe non-zero offsets after slicing can have some impact).