Add a preprocessor in Turbo#548
Conversation
…efactor/turbo_preprocessor
…rocessor # Conflicts: # .gitignore
…c into refactor/turbo_quantizer
…nto refactor/turbo_preprocessor
…nto refactor/turbo_preprocessor
|
|
||
| #pragma once | ||
|
|
||
| #include <turbo/quantizer/quantizer.h> |
There was a problem hiding this comment.
framework的头文件不应该依赖turbo的头文件，zvec::turbo::Quantizer应该使用前向声明来规避这个include
| int serialize(std::string *out) const override; | ||
| int deserialize(const void *data, size_t len) override; | ||
|
|
||
| //! Rotator type tag (kFht = 1). |
| void FhtRotator::apply_inverse(const float *in, float *out) const { | ||
| const size_t dim = static_cast<size_t>(in_dim_); | ||
| // Copy input into working buffer | ||
| std::vector<float> data(in, in + dim); |
There was a problem hiding this comment.
既然最后都要memcpy到out上，为何不一开始就在out上操作？这个data有点冗余。
There was a problem hiding this comment.
apply改了，此处忘记改了，已解决
…c into refactor/turbo_quantizer
…nto refactor/turbo_preprocessor
| #include "scalar/rotate/fht/fht.h" | ||
|
|
||
| #if defined(__SSE2__) | ||
| #include "sse/rotate/fht/fht.h" |
There was a problem hiding this comment.
把 defined 判断放每个simd实现的内部头文件？这样include方比较简单，虽然当前pr没差别，只有一处include。
| #if defined(__AVX512F__) | ||
| if (zvec::ailego::internal::CpuFeatures::static_flags_.AVX512F && | ||
| zvec::ailego::internal::CpuFeatures::static_flags_.AVX512DQ && | ||
| (cpu_arch_type == CpuArchType::kAuto || |
There was a problem hiding this comment.
加个cpu arch type match的helper？
| switch (rotate_type) { | ||
| case RotateType::kFht: { | ||
| RotatorKernels k; | ||
| // Default: scalar |
There was a problem hiding this comment.
把scalar定义放最后吧，实现和使用间隔比较远
| zvec::ailego::internal::CpuFeatures::static_flags_.AVX512DQ && | ||
| (cpu_arch_type == CpuArchType::kAuto || | ||
| cpu_arch_type == CpuArchType::kAVX512)) { | ||
| k.rotate = avx512::fht_rotate_avx512; |
There was a problem hiding this comment.
可以用aggregation初始化，其他类似
return {avx512::fht_rotate_avx512, avx512::fht_unrotate_avx512};
| #if defined(__AVX2__) | ||
| #include "avx2/rotate/fht/fht.h" | ||
| #endif | ||
| #if defined(__AVX512F__) |
There was a problem hiding this comment.
turbo.cc编译时没开simd开关，这些if define实际不起作用？
| static constexpr FhtPrimitives kPrim = { | ||
| fht_flip_sign_avx2, fht_inplace_avx2, fht_kacs_walk_avx2, | ||
| fht_inv_kacs_walk_avx2, fht_vec_rescale_avx2}; | ||
| fht_rotate_impl(out, in_dim, ctx, kPrim); |
There was a problem hiding this comment.
in参数完全没用到，要求输入已经在out了，这个接口有问题吧？
| using FhtVecRescaleFunc = void (*)(float *data, size_t n, float factor); | ||
|
|
||
| // Aggregate of all FHT kernels needed by FhtRotator, dispatched by ISA. | ||
| struct FhtKernels { |
There was a problem hiding this comment.
这些函数是旋转的通用函数还是Fht特有的？
FhtFlipSignFunc flip_sign;
FhtKacsWalkFunc kacs_walk;
FhtKacsWalkFunc inv_kacs_walk;
FhtInplaceFunc inplace;
FhtVecRescaleFunc rescale;
There was a problem hiding this comment.
一个方案是，把rotate和inverse rotate封装成march相关的函数，距离细节封装在对应的fht文件实现里
| }; | ||
|
|
||
| enum class RotateType : uint16_t { | ||
| kFht = 0, //!< O(d log d) FHT-based Kac random rotation |
There was a problem hiding this comment.
和rorator header里保持一致：
static uint16_t type_to_ser(RotatorType t) {
return t == RotatorType::Matrix ? 0 : 1;
}
4 participants
Description: A preprocessor is an optional internal component of the Turbo quantizer, primarily responsible for preprocessing data prior to retrieval. Its main functionalities include rotation and dimensionality reduction. Currently, only FHT rotation has been implemented, with support for OPQ rotation, dimensionality reduction, and other features planned for future releases.
Benefits: A module developed for Turbo