GH-49392: [C++][Compute] Fix fixed-width gather byte offset overflow in list filtering#49602
Conversation
|
|
|
|
||
| } // namespace | ||
|
|
||
| // GH-49392: Fixed-width gather byte offsets must be computed in int64_t. |
There was a problem hiding this comment.
Is the test actually useful? It's not really doing an end-to-end test, just testing a trivial helper function.
There was a problem hiding this comment.
Agreed. This is not particularly useful for a E2E protection. I think I shall remove it.
pitrou
left a comment
There was a problem hiding this comment.
+1, just a minor question. Feel free to merge when you're content with the PR :)
|
CI failures seem unrelated. I'm merging this. Thanks @pitrou for reviewing! |
|
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 61ef672. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them. |
…rflow in list filtering (apache#49602) ## Rationale for this change Issue apache#49392 reports a user-visible corruption when filtering a table that contains a `list<double>` column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values. ## What changes are included in this PR? This patch moves fixed-width gather byte-offset scaling onto an explicit `int64_t` helper before the `memcpy` and `memset` address calculations. That fixes the underlying 32-bit byte-offset overflow when a `uint32` gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at `999998000`; for `double` values, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the child `Take()` call used under list filtering, and the final filtered `Table` all address the correct bytes. To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (`N=500000`, `ARRAY_LEN=2000`, an `id` column, and a `numbers: list<double>` column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup, `Slice()` returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed child `Take()` and table `Filter()` results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary. ## Are these changes tested? Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug. ## Are there any user-facing changes? Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset. * GitHub Issue: apache#49392 Authored-by: Rossi Sun <zanmato1984@gmail.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
…rflow in list filtering (apache#49602) ## Rationale for this change Issue apache#49392 reports a user-visible corruption when filtering a table that contains a `list<double>` column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values. ## What changes are included in this PR? This patch moves fixed-width gather byte-offset scaling onto an explicit `int64_t` helper before the `memcpy` and `memset` address calculations. That fixes the underlying 32-bit byte-offset overflow when a `uint32` gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at `999998000`; for `double` values, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the child `Take()` call used under list filtering, and the final filtered `Table` all address the correct bytes. To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (`N=500000`, `ARRAY_LEN=2000`, an `id` column, and a `numbers: list<double>` column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup, `Slice()` returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed child `Take()` and table `Filter()` results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary. ## Are these changes tested? Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug. ## Are there any user-facing changes? Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset. * GitHub Issue: apache#49392 Authored-by: Rossi Sun <zanmato1984@gmail.com> Signed-off-by: Rossi Sun <zanmato1984@gmail.com>
Rationale for this change
Issue #49392 reports a user-visible corruption when filtering a table that contains a
list<double>column: slicing the last row returns the expected values, while filtering the same row returns values from a different child span. The corruption only appears once the selected child-value range is large enough, which points to an overflow in the fixed-width gather path used when list filtering materializes the selected child values.What changes are included in this PR?
This patch moves fixed-width gather byte-offset scaling onto an explicit
int64_thelper before thememcpyandmemsetaddress calculations. That fixes the underlying 32-bit byte-offset overflow when auint32gather index is multiplied by the fixed value width. In the source issue's last-row case, the selected child span starts at999998000; fordoublevalues, scaling that index by 8 bytes wrapped in 32-bit arithmetic and redirected the gather to the wrong child span. Keeping the byte-offset arithmetic in 64 bits makes the fixed-width gather path, the childTake()call used under list filtering, and the final filteredTableall address the correct bytes.To validate that this is the same bug reported in the issue, I also used a local near-e2e C++ reproduction that keeps the issue's logical shape (
N=500000,ARRAY_LEN=2000, anidcolumn, and anumbers: list<double>column), filters the last row, and seeds both the true target child span and the pre-fix wrapped span with distinct sentinels. In that setup,Slice()returns the expected row, a replay of the pre-fix gather arithmetic returns the wrapped sentinel span instead, and the fixed childTake()and tableFilter()results both match the sliced row. That ties the user-visible issue and this root-cause fix back to the same overflow boundary.Are these changes tested?
Yes. The patch adds a targeted unit test that checks fixed-width gather byte offsets are computed with 64-bit arithmetic. This is intentionally smaller than an end-to-end filter regression: the original user-visible failure only shows up at very large logical offsets, while the new unit test isolates the exact overflow boundary directly without constructing a huge table or depending on a PyArrow-level reproduction. That makes it smaller, more stable, and more appropriate for regular C++ CI, while the near-e2e local reproduction was used separately to confirm that this root-cause regression test and the reported filtering corruption are exercising the same bug.
Are there any user-facing changes?
Yes. Filtering tables with large list columns backed by fixed-width child values no longer risks returning data from a wrapped byte offset.