What are your sampling settings?

[dejian-lc]

Thank you for your outstanding work. Looking at the code, it seems you are using vllm as the inference backend. For the reasoning model, what are your sampling parameters? In theory, should greedy sampling be used, and should the Transformer library serve as the backend?

[zli12321]

Hello

For standard GRPO training, since each query samples N responses, we use vLLM to speed up sampling N responses so the training time can be reasonable. The sampling parameters can be found in the training bash scripts.

Check the following arguments for our sampling parameters:
--num_generations 8
--temperature 1.0

If you want to modify the maximum response token length in the training code, check the file and line 380 to adjust other sampling parameters.
self.sampling_params = SamplingParams( temperature=1.0, top_p=0.95, max_tokens=self.max_completion_length, )

If you would like to use greedy decoding, you can simply change adjust sampling parameters temperature=0, top_k=1, but that defeats the purpose of GRPO training since the algorithm want to sample N diverse responses. If temperature is 0, the advantage computation would be near 0 for each rollout, causing the model to not learn anything.

[dejian-lc]

Hello, I realize my question wasn't phrased clearly. What I meant was: when evaluating benchmarks, should all models under test use the Transformer library as the backend with greedy sampling? I saw your eval_bench.py. Is this the script used for evaluation? I noticed that it uses vLLM.

[zli12321]

Oh yes. We used to use the regular transformer library to run the evaluation. But we use vLLM and set temperature to 0 or 0.1 to speed up evaluation. The transformer library takes 7 hours to run Qwen inference while takes an hour if we use vLLM.