feat: static dispatch

[Nelsi11120]

This PR tries to eliminate the dynamic dispatch costing (a tiny) lookup at runtime, by using static dispatch at compile time.

Hopefully we got rid of it properly and did not introduce anything slower, defeating the purpose of this PR.

[Nelsi11120]

Actually we could still explore the static dispatch by using an Enum with our chunkers and match statements inside the methods.

The Pros:

• It's faster because it is direct and no indirection with the vtable
• It's also faster because Vectors can store their elements in continous regions of memory, which is not guaranteed with the vtable

The Cons:

• It's more code.
• Less flexible, for instance if a user wants to add a new type, we would not be able to do so because he would need to modify the enum itself. But do we need this functionality ?

Overall I would say that this is a tradeoff between speed and flexibility.

But because here we know all the variants, will probably not allow users to define their own chunkers and because we focus on high performance, I would say that trying the enum approach is fine, but maybe I am missing something.

One another thing to consider is that maybe the bottleneck is all the IO stuff and the vtable + pointer chasing is noise.

What do you think @jimzer ?

[jimzer]

Actually we could still explore the static dispatch by using an Enum with our chunkers and match statements inside the methods.

The Pros:

• It's faster because it is direct and no indirection with the vtable
• It's also faster because Vectors can store their elements in continous regions of memory, which is not guaranteed with the vtable

The Cons:

• It's more code.
• Less flexible, for instance if a user wants to add a new type, we would not be able to do so because he would need to modify the enum itself. But do we need this functionality ?

Overall I would say that this is a tradeoff between speed and flexibility.

But because here we know all the variants, will probably not allow users to define their own chunkers and because we focus on high performance, I would say that trying the enum approach is fine, but maybe I am missing something.

One another thing to consider is that maybe the bottleneck is all the IO stuff and the vtable + pointer chasing is noise.

What do you think @jimzer ?

Thanks again for the PR :)
I finally could take all the time to check it, sorry for the delay!
It works great and I love the static dispatch version.

These are good points, and I think for now we can tolerate the dynamic dispatch in the Python iterator bindings, since we now have static dispatch for the pure Rust version, and Python overhead is already significantly more important.

Let's go shipping this new version!