multilingual-evaluation

Evaluation of the Multilingual BabyLM models

Test run pipeline

VLLM_WORKER_MULTIPROC_METHOD=spawn lm_eval --model vllm --model_args pretrained=Qwen/Qwen3-0.6B,tensor_parallel_size=2,gpu_memory_utilization=0.6 --tasks babybabellm_uk --batch_size auto --output_path ./eval-results --log_samples --include_path ./tasks --num_fewshot 3 --limit 100


VLLM_WORKER_MULTIPROC_METHOD=spawn lm_eval --model vllm --model_args pretrained=google/gemma-3-4b-pt,tensor_parallel_size=2,dtype=bfloat16,gpu_memory_utilization=0.6,max_model_len=6000 --tasks babybabellm_uk --batch_size auto --output_path ./eval-results --log_samples --include_path ./tasks --num_fewshot 3 --limit 100