Questions regarding metric reproduction

[rOtking]

Dear Authors,
Thank you very much for your high-quality open-source contribution. I have currently completed the training for the first 2 stages and have a few questions regarding the reproduction process:

1. In the code, both Stage 1 and Stage 2 are set to train for only 1 epoch. Is this the correct configuration as intended?
2. My achieved Meta Action Accuracy in Stage 1 is approximately 0.6, whereas the provided Drivema-2B model achieves around 0.7. Is this performance gap considered normal?
3. My RFS score in Stage 2 is 7.28, which shows a discrepancy compared to the 7.89 reported in the paper.
   I would appreciate any insights or suggestions you could provide.

Best regards

[wczheng]

Thank you for your interest in our work and for sharing your reproduction results.
Regarding your questions:
Yes, the configuration is correct. Both Stage 1 and Stage 2 are intended to be trained for only one epoch.

For the lower Meta Action Accuracy in Stage 1 and the RFS discrepancy in Stage 2, there may be an issue with the training configuration or a mismatch between the prompt templates used during training and inference. Could you please share your training and inference scripts so that we can investigate further?

In the meantime, we suggest checking the following points carefully:
Whether the training data has been prepared completely and correctly, especially whether the image data is properly loaded and provided to the model.
Whether the image resolution used during training is consistent with that used during inference.
Whether the inference prompts, including their formatting and templates, are fully aligned with those in the training data. Even a small mismatch may significantly degrade the model’s performance.
Whether the model produces any malformed, unexpected, or otherwise abnormal outputs during inference. Manually inspecting several prediction samples may help identify the issue.
We hope these suggestions are helpful, and we would be happy to take a closer look once you share the relevant scripts and configurations.
Best regards,
Weicheng Zheng