Support for “real” batch processing

[talbaumel]

Hi, thank you for your contribution! Great work!

Currently batch processing just iterates over the inputs and send them separately to process.

Any plans to support parallel batch processing?

Thanks!

[yochail]

I have been trying to implement batch inference directly with the torch model, but surprisingly I got Significantly different results for the batch inference when comparing output of the exact same input (to the padding level) when running it in a batch or 1 by 1 example.
The results for the 1 example in the batch are identical, but the results for the second one are different.

I wonder if this is a known issue of the ModernBert model.

outputs_ids_debug_1.txt
outputs_ids_debug_2.txt
inputs_ids_debug_1.txt
inputs_ids_debug_2.txt