Code for weight conversion and inference

[bigwatergun]

Thanks for sharing your work!

I notice that under /examples there are few MoE models, config.json indicates the expert placement for different expert parallelism degree. However, I wonder how can I use these these configs to inference models. I presume HF transformers can be used to load model config and ModelForCausalLM, but how should I load model weights from .safetensors? Since moe layers for Occult are self-modified nn.Module. So I am wondering whether you can provide inference codes for models under /examples with data parallelism and expert parallelism supported.

Looking forward to your reply.

[luoshuqing2001]

Thanks for your attention. In this paper, we converted the HF safetensors into .pt files for each rank involved in expert parallelism before the training and inference tasks, following the config.json files. Currently, I'm working on directly loading the checkpoints from the safetensors. I will update this repository once I have finished.