Skip to content

GH-49392: [C++][Compute] Fix fixed-width gather byte offset overflow in list filtering#49602

Merged
zanmato1984 merged 2 commits into
apache:mainfrom
zanmato1984:fix/gh-49392-list-filter-overflow
Mar 30, 2026
Merged

GH-49392: [C++][Compute] Fix fixed-width gather byte offset overflow in list filtering#49602
zanmato1984 merged 2 commits into
apache:mainfrom
zanmato1984:fix/gh-49392-list-filter-overflow

Conversation

@zanmato1984

Copy link
Copy Markdown
Contributor

Rationale for this change

Issue #49392 reports a user-visible corruption when filtering a table that contains a list<double> column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values.

What changes are included in this PR?

This patch moves fixed-width gather byte-offset scaling onto an explicit int64_t helper before the memcpy and memset address calculations. That fixes the underlying 32-bit byte-offset overflow when a uint32 gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at 999998000; for double values, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the child Take() call used under list filtering, and the final filtered Table all address the correct bytes.

To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (N=500000, ARRAY_LEN=2000, an id column, and a numbers: list<double> column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup, Slice() returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed child Take() and table Filter() results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary.

Are these changes tested?

Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug.

Are there any user-facing changes?

Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #49392 has been automatically assigned in GitHub to PR creator.


} // namespace

// GH-49392: Fixed-width gather byte offsets must be computed in int64_t.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the test actually useful? It's not really doing an end-to-end test, just testing a trivial helper function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This is not particularly useful for a E2E protection. I think I shall remove it.

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, just a minor question. Feel free to merge when you're content with the PR :)

@github-actions github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 30, 2026
@zanmato1984

Copy link
Copy Markdown
Contributor Author

CI failures seem unrelated. I'm merging this. Thanks @pitrou for reviewing!

@zanmato1984
zanmato1984 merged commit 61ef672 into apache:main Mar 30, 2026
49 of 51 checks passed
@zanmato1984 zanmato1984 removed the awaiting committer review Awaiting committer review label Mar 30, 2026
@zanmato1984
zanmato1984 deleted the fix/gh-49392-list-filter-overflow branch March 30, 2026 19:45
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 61ef672.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.

thisisnic pushed a commit to thisisnic/arrow that referenced this pull request Apr 6, 2026
…rflow in list filtering (apache#49602)

## Rationale for this change

Issue apache#49392 reports a user-visible corruption when filtering a table that contains a `list<double>` column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values.

## What changes are included in this PR?

This patch moves fixed-width gather byte-offset scaling onto an explicit `int64_t` helper before the `memcpy` and `memset` address calculations. That fixes the underlying 32-bit byte-offset overflow when a `uint32` gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at `999998000`; for `double` values, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the child `Take()` call used under list filtering, and the final filtered `Table` all address the correct bytes.

To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (`N=500000`, `ARRAY_LEN=2000`, an `id` column, and a `numbers: list<double>` column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup, `Slice()` returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed child `Take()` and table `Filter()` results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary.

## Are these changes tested?

Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug.

## Are there any user-facing changes?

Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset.

* GitHub Issue: apache#49392

Authored-by: Rossi Sun <zanmato1984@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
Mottl pushed a commit to Mottl/arrow that referenced this pull request May 26, 2026
…rflow in list filtering (apache#49602)

## Rationale for this change

Issue apache#49392 reports a user-visible corruption when filtering a table that contains a `list<double>` column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values.

## What changes are included in this PR?

This patch moves fixed-width gather byte-offset scaling onto an explicit `int64_t` helper before the `memcpy` and `memset` address calculations. That fixes the underlying 32-bit byte-offset overflow when a `uint32` gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at `999998000`; for `double` values, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the child `Take()` call used under list filtering, and the final filtered `Table` all address the correct bytes.

To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (`N=500000`, `ARRAY_LEN=2000`, an `id` column, and a `numbers: list<double>` column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup, `Slice()` returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed child `Take()` and table `Filter()` results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary.

## Are these changes tested?

Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug.

## Are there any user-facing changes?

Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset.

* GitHub Issue: apache#49392

Authored-by: Rossi Sun <zanmato1984@gmail.com>
Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants