GH-17211: [C++] Add hash32 and hash64 scalar compute functions

[kszucs]

Rationale for this change

Support for calculating elementwise hashes.

The PR adds to scalar functions hash32() and hash64() using the existing internal hashing machinery.

What changes are included in this PR?

Continuation of #39836 with the following changes:

• Use column oriented hash-combine rather than flattening nested elements
• Support arbitrary nesting levels with an optimization that only hash child arrays if they are also nested

Are these changes tested?

Partially, working on a proper testing suite.

Are there any user-facing changes?

There is a new compute kernel hash_64 available.

• GitHub Issue: [C++][Compute] Add scalar_hash function #17211

[kszucs]

Seems like we generate the same hash for both NULL and 0 which is not ideal.

In [1]: import pyarrow as pa

In [2]: import pyarrow.compute as pc

In [3]: pc.hash_64([None])
Out[3]:
<pyarrow.lib.UInt64Array object at 0x124247be0>
[
  0
]

In [4]: pc.hash_64([0])
Out[4]:
<pyarrow.lib.UInt64Array object at 0x1033027a0>
[
  0
]

[zanmato1984]

Some first glance comments. I'll look into more details later.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

cpp/src/arrow/compute/key_hash_benchmark.cc:89

• Same TempVectorStack sizing issue as above: the 64-bit hashing path should size the stack using Hashing64::kHashBatchTempStackUsage to ensure sufficient scratch space.

  ASSERT_OK(
      stack_memallocator.Init(compute::default_exec_context()->memory_pool(),
                              3 * sizeof(int32_t) * util::MiniBatch::kMiniBatchLength));

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (5)

cpp/src/arrow/compute/kernels/scalar_hash_test.cc:135

• In CheckDeterministic(), the second slicing assertion is unreachable because it's in an else if that only runs when arr->length() < 1. This reduces coverage of slice invariance for longer arrays.

    if (arr->length() >= 1) {
      auto in1 = arr->Slice(1);
      ASSERT_OK_AND_ASSIGN(Datum out1, CallFunction(func, {in1}));
      ValidateOutput(out1);
      AssertArraysEqual(*out1.make_array(), *hashes->Slice(1));
    } else if (arr->length() >= 4) {
      auto in2 = arr->Slice(2, 2);
      ASSERT_OK_AND_ASSIGN(Datum out2, CallFunction(func, {in2}));
      ValidateOutput(out2);
      AssertArraysEqual(*out2.make_array(), *hashes->Slice(2, 2));
    }

cpp/src/arrow/util/hashing_benchmark.cc:40

• kSeed and hashing_rng are currently unused in this benchmark. Some builds treat unused variables as errors (e.g., -Werror), which would break the benchmark build.

// copied from scalar_string_benchmark
constexpr auto kSeed = 0x94378165;

static random::RandomArrayGenerator hashing_rng(kSeed);
}  // namespace

cpp/src/arrow/compute/key_hash_benchmark.cc:53

• TempVectorStack is initialized with a hard-coded size (based on int32) but Hashing32 expects callers to reserve at least Hashing32::kHashBatchTempStackUsage. Using too small a stack can lead to OOB writes/corruption inside HashMultiColumn()'s temp allocations.

  // initialize the stack allocator
  util::TempVectorStack stack_memallocator;
  ASSERT_OK(
      stack_memallocator.Init(compute::default_exec_context()->memory_pool(),
                              3 * sizeof(int32_t) * util::MiniBatch::kMiniBatchLength));

cpp/src/arrow/compute/kernels/scalar_hash.cc:83

• KeyColumnArray variable-length hashing expects byte offsets into the concatenated values buffer (see Hashing32::HashVarLenImp using key_length = offsets[i + 1] - offsets[i]). For LIST/LARGE_LIST/MAP the offsets buffer is element offsets, and for FIXED_SIZE_LIST list_size() is a count, not a byte width. As written, list/map/fixed_size_list hashes will be incorrect (and may read the wrong number of bytes) when list_values_buffer contains 4/8-byte per-element hashes.

  } else if (type_id == Type::MAP) {
    metadata = KeyColumnMetadata(false, sizeof(uint32_t));
    var_length_buffer = list_values_buffer;
  } else if (type_id == Type::LIST) {
    metadata = KeyColumnMetadata(false, sizeof(uint32_t));
    var_length_buffer = list_values_buffer;
  } else if (type_id == Type::LARGE_LIST) {
    metadata = KeyColumnMetadata(false, sizeof(uint64_t));
    var_length_buffer = list_values_buffer;
  } else if (type_id == Type::FIXED_SIZE_LIST) {
    auto list_type = checked_cast<const FixedSizeListType*>(type);
    metadata = KeyColumnMetadata(true, list_type->list_size());
    fixed_length_buffer = list_values_buffer;

cpp/src/arrow/compute/kernels/scalar_hash.cc:139

• For STRUCT inputs, the struct's own validity bitmap is not applied. If a struct row is null but its children have non-null values, HashMultiColumn() will still incorporate the children and the output hash for that row can be non-zero, contradicting the stated behavior that nulls hash to a fixed sentinel value (0).

    if (type_id == Type::STRUCT) {
      std::vector<std::shared_ptr<ArrayData>> child_hashes(array.child_data.size());
      columns.resize(array.child_data.size());
      for (size_t i = 0; i < array.child_data.size(); i++) {
        auto child = array.child_data[i];
        if (is_nested(child.type->id())) {
          ARROW_ASSIGN_OR_RAISE(child_hashes[i],
                                HashChild(array, child, hash_ctx, memory_pool));
          ARROW_ASSIGN_OR_RAISE(column, ToColumnArray(*child_hashes[i]));
        } else {
          ARROW_ASSIGN_OR_RAISE(column, ToColumnArray(child));
        }
        columns[i] = column.Slice(array.offset, array.length);
      }
      Hasher::HashMultiColumn(columns, hash_ctx, out);
    } else if (is_list_like(type_id)) {

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

cpp/src/arrow/compute/kernels/scalar_hash.cc:108

• HashChild() builds the hashed child ArrayData using the parent array's validity buffer and null_count. For STRUCT fields and especially LIST/MAP values (different lengths), this can ignore child nulls and can also produce an invalid/null bitmap length mismatch when the hashed child is later consumed, leading to incorrect hashes or out-of-bounds reads.

  static Result<std::shared_ptr<ArrayData>> HashChild(const ArraySpan& array,
                                                      const ArraySpan& child,
                                                      LightContext* hash_ctx,
                                                      MemoryPool* memory_pool) {
    auto arrow_type = TypeTraits<ArrowType>::type_singleton();
    auto buffer_size = child.length * sizeof(c_type);
    ARROW_ASSIGN_OR_RAISE(auto buffer, AllocateBuffer(buffer_size, memory_pool));
    ARROW_RETURN_NOT_OK(
        HashArray(child, hash_ctx, memory_pool, buffer->mutable_data_as<c_type>()));
    return ArrayData::Make(arrow_type, child.length,
                           {array.GetBuffer(0), std::move(buffer)}, array.null_count);
  }

cpp/src/arrow/compute/kernels/scalar_hash.cc:146

• LIST/LARGE_LIST/MAP/FIXED_SIZE_LIST handling is currently passing Arrow list/map offsets (which are element indices) into Hashing{32,64}::HashVarLen, which expects byte offsets into the concatenated values buffer. Additionally, FIXED_SIZE_LIST is treated as fixed-length with fixed_length=list_size (elements) rather than bytes. This makes hashes for list-like types incorrect (and fixed-size lists misaligned).

    } else if (is_list_like(type_id)) {
      auto values = array.child_data[0];
      ARROW_ASSIGN_OR_RAISE(auto value_hashes,
                            HashChild(array, values, hash_ctx, memory_pool));
      ARROW_ASSIGN_OR_RAISE(column,
                            ToColumnArray(array, value_hashes->buffers[1]->data()));
      columns[0] = column.Slice(array.offset, array.length);
      Hasher::HashMultiColumn(columns, hash_ctx, out);