Have you conducted experimental comparisons on DeepSeek-R1-Distill-Qwen-32B?

[biandh]

In Table 1 of the paper, the results for OREAL-7B, OREAL-DSR1-Distill-Qwen-7B, and OREAL-32B are provided, but there are no results for OREAL-DSR1-Distill-Qwen-32B. Is the RL performance on this model not good?

[RangiLyu]

We did some experiments with R1-Distill-Qwen-32B for OREAL and observed a notable performance improvement. However, we chose not to include these results in our paper for some reasons: First, we were unable to reproduce the scores reported in the DeepSeek-R1 paper. Using our evaluation method, R1-Distill-Qwen-32B achieved MATH500 = 93% and AIME2024 = 60%. After applying OREAL, the scores improved to MATH500 = 95.6% and AIME2024 = 66.7%. That's a clear performance gain. However, the AIME score remained lower than the one reported in the DeepSeek-R1 paper.

This is why we chose to cite the original paper's scores in our table. We did not want to present a lower baseline score and then show an increase after RL, as this could lead to concerns that we deliberately set a weaker baseline—potentially leading to unnecessary debate.

Moreover, this experiment is not directly related to the core content of our paper. For these reasons, we decided not to include the results in the paper.

[biandh]

I am confused about not being able to reproduce the results of the R1-Distill-Qwen-32B model at AIME2024. I evaluated the deepseek-ai/DeepSeek-R1-Distill-Qwen-32B model on SiliconFlow. I set the Temperature=0.6; Max Tokens=16k, ran it only once(excluding cases where the maximum token limit is exceeded), and achieved an accuracy of 22/30=73.3%.

The input source for AIME_2024 can be found at https://github.com/agentica-project/deepscaler/blob/main/deepscaler/data/test/aime.json.
output: aime_2024-32b.xlsx

[RangiLyu]

For a fair comparison, all our experiments were conducted using lmdeploy and opencompass for inference and evaluation, and we used greedy decoding to avoid randomness, as reported in our paper. The difference in AIME accuracy may be due to differences in the inference engines and decoding strategies.

[RangiLyu]

Additionally, I noticed that your testing method re-evaluates the questions that hit the max_token_limit. In my opinion, it is not a strict pass@1 evaluation. Under our evaluation setting, if a single inference reaches the max_token_limit and does not generate a final answer, it is considered incorrect.

[biandh]

Additionally, I noticed that your testing method re-evaluates the questions that hit the max_token_limit. In my opinion, it is not a strict pass@1 evaluation. Under our evaluation setting, if a single inference reaches the max_token_limit and does not generate a final answer, it is considered incorrect.

Thank you for your reply. By the way, the 22 correct results do not include retries (if retries are included, the accurate count would be 23). So the score of 73.3% is the result of pass@1.

[biandh]

For a fair comparison, all our experiments were conducted using lmdeploy and opencompass for inference and evaluation, and we used greedy decoding to avoid randomness, as reported in our paper. The difference in AIME accuracy may be due to differences in the inference engines and decoding strategies.

Thank you for your answer. I just verified it according to the best configuration guidelines provided by the official DeepSeek documentation, and it seems there are no issues. Could you run the experiment again based on the official configuration? I believe the results of OREAL-DSR1-Distill-Qwen-32B are very valuable, as they fill in some content that was not covered in the R1 paper.