[Bug](parquet) SIGSEGV in gen_filter_map when reading nested columns with filter_all=true

[Larborator]

Search before asking

• I had searched in the issues and found no similar issues.

Version

4.x/master

What's Wrong?

BE crashes with SIGSEGV (null pointer dereference at 0x0) when querying Parquet-based external tables (Paimon/Hive/Iceberg) with nested type columns (Struct/Array/Map), if a predicate filters out all rows in a RowGroup.

The crash occurs in ScalarColumnReader::gen_filter_map which dereferences filter_map.filter_map_data() — this is nullptr when filter_all=true.

Root Cause: _read_nested_column only checks has_filter() but not filter_all(). When all rows are filtered out, FilterMap is initialized via init(nullptr, total_rows, true), setting _has_filter=true but _filter_map_data=nullptr. The newer FilterMap::generate_nested_filter_map already has the correct guard (if (!has_filter() || filter_all()) return error), but the inline gen_filter_map lacks this check.

*** SIGSEGV address not mapped to object (@0x0) received by PID 72584 (TID 88200 OR 0x7f319ec10700) from PID 0; stack trace: ***
0# 0x000055EC0721DC35 in /doris/be/lib/doris_be
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/local/jdk-17.0.10/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/local/jdk-17.0.10/lib/server/libjvm.so
 3# 0x00007F783FC78630 in /lib64/libpthread.so.0
 4# doris::ScalarColumnReader<false, true>::gen_filter_map(doris::FilterMap&, unsigned long, unsigned long, unsigned long, std::vector<unsigned char, std::allocator<unsigned char> >&, std::unique_ptr<doris::FilterMap, std::default_delete<doris::FilterMap> >*) in /doris/be/lib/doris_be
 5# doris::ScalarColumnReader<false, true>::_read_nested_column(doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&, std::shared_ptr<doris::IDataType const>&, doris::FilterMap&, unsigned long, unsigned long*, bool*, bool)::{lambda(unsigned long, unsigned long)#1}::operator()(unsigned long, unsigned long) const in /doris/be/lib/doris_be
 6# doris::ScalarColumnReader<false, true>::_read_nested_column(doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&, std::shared_ptr<doris::IDataType const>&, doris::FilterMap&, unsigned long, unsigned long*, bool*, bool) in /doris/be/lib/doris_be
 7# doris::ScalarColumnReader<false, true>::read_column_data(doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&, std::shared_ptr<doris::IDataType const>&, std::shared_ptr<doris::TableSchemaChangeHelper::Node> const&, doris::FilterMap&, unsigned long, unsigned long*, bool*, bool, long) in /doris/be/lib/doris_be
 8# doris::StructColumnReader::read_column_data(doris::COW<doris::IColumn>::immutable_ptr<doris::IColumn>&, std::shared_ptr<doris::IDataType const>&, std::shared_ptr<doris::TableSchemaChangeHelper::Node> const&, doris::FilterMap&, unsigned long, unsigned long*, bool*, bool, long) in /doris/be/lib/doris_be
 9# doris::RowGroupReader::_read_column_data(doris::Block*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, unsigned long, unsigned long*, bool*, doris::FilterMap&) in /doris/be/lib/doris_be
10# doris::RowGroupReader::_do_lazy_read(doris::Block*, unsigned long, unsigned long*, bool*) in /doris/be/lib/doris_be
11# doris::RowGroupReader::next_batch(doris::Block*, unsigned long, unsigned long*, bool*) in /doris/be/lib/doris_be
12# doris::ParquetReader::get_next_block(doris::Block*, unsigned long*, bool*) in /doris/be/lib/doris_be
13# doris::PaimonReader::get_next_block_inner(doris::Block*, unsigned long*, bool*) in /doris/be/lib/doris_be
14# doris::TableFormatReader::get_next_block(doris::Block*, unsigned long*, bool*) in /doris/be/lib/doris_be
15# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) in /doris/be/lib/doris_be
16# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) in /doris/be/lib/doris_be
17# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) in /doris/be/lib/doris_be
18# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /doris/be/lib/doris_be
19# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) in /doris/be/lib/doris_be
20# 0x000055EC0CCBBB55 in /doris/be/lib/doris_be
21# doris::ScannerSplitRunner::process_for(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) in /doris/be/lib/doris_be
22# doris::PrioritizedSplitRunner::process() in /doris/be/lib/doris_be
23# doris::TimeSharingTaskExecutor::_dispatch_thread() in /doris/be/lib/doris_be
24# doris::Thread::supervise_thread(void*) in /doris/be/lib/doris_be
25# start_thread in /lib64/libpthread.so.0
26# __clone in /lib64/libc.so.6

### What You Expected?

The query should return results without crashing. When `filter_all=true`, nested columns should be correctly skipped without dereferencing `nullptr`.


### How to Reproduce?

The following standalone program reproduces the core logic of `gen_filter_map`. Commenting out the `if (filter_all)` guard and running the `else` branch causes SIGSEGV:

```cpp
#include <cassert>
#include <cstdint>
#include <cstdio>
#include <vector>

int main() {
    // Simulate filter_all=true: filter_map_data is nullptr
    const uint8_t* filter_map_data = nullptr;
    bool has_filter = true;
    bool filter_all = true;

    // rep_levels for a nested column: 3 rows with varying element counts
    std::vector<uint16_t> rep_levels = {0, 1, 1, 0, 1, 0};
    std::vector<uint8_t> nested_filter_map_data;

    if (has_filter) {
        if (filter_all) {
            // FIX: skip gen_filter_map, produce all-zero nested filter
            nested_filter_map_data.assign(rep_levels.size(), 0);
            printf("PASS: filter_all path correctly produces all-zero nested filter\n");
        } else {
            // BUG: dereferences nullptr → SIGSEGV
            size_t filter_loc = 0;
            nested_filter_map_data.resize(rep_levels.size());
            for (size_t i = 0; i < rep_levels.size(); i++) {
                if (i != 0 && rep_levels[i] == 0) filter_loc++;
                nested_filter_map_data[i] = filter_map_data[filter_loc]; // CRASH HERE
            }
        }
    }

    for (auto v : nested_filter_map_data) assert(v == 0);
    printf("All elements filtered — correct behavior\n");
    return 0;
}

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_bb619bce1d11

Initial maintainer triage for apache/doris#63887.

I checked the live issue metadata and current upstream/master (6e27f117471f481e13cebabe0454dddc60e5245c). The issue currently has no labels, and the title has already been updated to the concrete Parquet crash path. The report is credible and actionable.

Evidence from the current code:

• FilterMap::init() in be/src/format/parquet/parquet_common.cpp sets _has_filter=true when filter_all=true. If the caller passes nullptr, filter_map_data() remains nullptr.
• RowGroupReader::_rebuild_filter_map() in be/src/format/parquet/vparquet_group_reader.cpp has a direct filter_map.init(nullptr, total_rows, true) path when rebuilding an all-filtered lazy-read filter map.
• ScalarColumnReader::_read_nested_column() in be/src/format/parquet/vparquet_column_reader.cpp calls gen_filter_map() whenever filter_map.has_filter() is true.
• ScalarColumnReader::gen_filter_map() in be/src/format/parquet/vparquet_column_reader.h unconditionally evaluates filter_map.filter_map_data()[filter_loc].

So the important invariant mismatch is real: filter_all() implies has_filter(), but it does not guarantee a non-null filter-map data pointer. For nested Parquet columns, the helper expands the parent row filter into a nested value filter and can dereference null instead of producing an empty result. This is consistent with the submitted stack around _do_lazy_read -> _read_column_data -> StructColumnReader -> ScalarColumnReader::_read_nested_column -> gen_filter_map.

Maintainer judgment:

This should be treated as a BE crash bug in the Parquet nested-column lazy-read path, not as only a standalone reproducer issue. The right fix direction is to handle filter_map.filter_all() before expanding the nested filter map, materializing a valid all-zero nested filter map or otherwise skipping nested decoding safely. Checking only has_filter() is insufficient.

There is already an open candidate fix, #63889. Its direction matches the root cause above: it avoids calling gen_filter_map() when the parent filter map is filter_all() and creates an all-zero nested filter map with a valid backing buffer. Before merge, I would still ask for data-path coverage in addition to helper-level tests:

• A regression with a real Parquet nested column (STRUCT, ARRAY, or MAP) read through the external-table path.
• A predicate/lazy-read query where at least one row group or lazy-read batch is fully filtered before a nested non-predicate column is read.
• Coverage for one of Paimon/Hive/Iceberg is enough if it reaches the shared Parquet reader path; the bug is in shared Parquet code, not obviously format-catalog specific.

Missing information that would make the issue fully reproducible:

• Exact affected build or commit for the crashing BE, not just 4.x/master.
• Minimal DDL/catalog setup and a small Parquet or table-format fixture that contains the nested column and filtered row group.
• The actual SQL, predicate, and selected columns that trigger lazy read.
• Full BE log around the crash with query id/profile if available.

Recommended next steps:

1. Review #63889 as the likely fix for this issue.
2. Add or request a real nested-Parquet lazy-read regression before merging or backporting.
3. Consider branch-pick labels for affected 4.x branches after the regression confirms the shared Parquet path.