It seems like this package provides the same functionality as the native julia implementations provided by Transformers.jl. https://chengchingwen.github.io/Transformers.jl/stable/huggingface/#Transformers.HuggingFace.load_tokenizer
In view of that, could you offer some motivation for this wrapper? It may also be helpful to add such a discussion to the README to allow users to make an informed decision.
It seems like this package provides the same functionality as the native julia implementations provided by
Transformers.jl. https://chengchingwen.github.io/Transformers.jl/stable/huggingface/#Transformers.HuggingFace.load_tokenizerIn view of that, could you offer some motivation for this wrapper? It may also be helpful to add such a discussion to the README to allow users to make an informed decision.