Bug Report and Evaluation Feedback for CheXone

[amir333222111]

Hello CheXone team,

First of all, thank you for developing and sharing CheXone. I really appreciate your work on building an open model for chest X-ray interpretation.

I tested the model using several random chest X-ray images from the internet, and overall it was interesting to evaluate. However, I noticed two issues that may need attention:

1. The model frequently mentions comparisons such as “compared to prior views” or “no new changes compared to prior views,” even when only a single chest X-ray image was provided and there was no prior study available. This seems like a systematic reporting bug.

2. In my informal testing, the diagnostic accuracy appears lower compared to CheXagent, especially for detailed radiology report generation.

Here is one example of the generated report:

“Assessing Lung Fields: The bilateral lung fields appear diffusely affected by small, discrete opacities distributed symmetrically without new changes compared to prior views...

Final Answer:
The cardiomediastinal silhouette is normal. There is persistent diffuse bilateral micronodular opacities seen. There is no pleural effusion or pneumothorax identified. The lungs are clear.”

As you can see, the report refers to prior views despite only a single first-time image being input. Also, there is some internal inconsistency (mentioning diffuse bilateral micronodular opacities while also concluding the lungs are clear).

I hope this feedback is helpful. Thank you again for making the model available.

[YBZh]

Thank you very much for your detailed feedback and for testing CheXOne. We really appreciate you taking the time to report these observations.

Regarding the first issue, yes, we have also observed that the model may sometimes mention prior studies or comparisons such as “compared to prior views” even when only a single image is provided. We discussed this artifact in the paper (Page 16): this behavior likely comes from the training reports, where longitudinal comparison with prior studies is common. As a result, the model may retain this reporting style even in single-study contexts.

We agree that this is an important issue to improve. A reasonable next step would be to further preprocess the training reports, especially for single-image/single-study cases, by removing or normalizing comparative descriptions related to prior studies. This should help reduce such unsupported comparison statements in future versions.

Regarding the comparison with CheXagent, thank you for sharing your observation. Since your test was based on several random images from the internet, the results may vary depending on image quality, distribution shift, and prompting format. In our large-scale evaluation on ReXRank, CheXOne achieves stronger report-generation quality than CheXagent under the standardized benchmark setting. We would suggest referring to the ReXRank results for a more systematic comparison.

Thanks again for your helpful feedback.