Hello,
thank you for sharing your paper and code.
I am currently studying your paper and trying to reproduce the Bayesian Optimization based fine-tuning pipeline for Stable Diffusion v1.5, following the settings described in the paper and the released code.
In my reproduction, I fine-tune the diffusion model with BO and without BO (baseline), using the same dataset and training configuration as reported. During training, I save checkpoints every 1000 steps. For each setting (with BO / without BO), for each checkpoint, I generate synthetic images and train a downstream classifier, which is evaluated on the real test set. I then report the best result across checkpoints, following the same evaluation protocol as described in the paper.
However, in my experiments, I do not observe a clear improvement when using BO compared to the baseline. The results obtained with BO are very similar to those without BO, both of these results are stronger than the performance reported in the paper.
I strictly follow the training parameters and pipeline described in the paper and codebase, and the main difference is that I explicitly select the best checkpoint for each method, and the model I used is SD1.5
I would like to ask whether:
I select the best checkpoint based on downstream classification performance for both BO and the baseline. Is this how you evaluated the models, or did you use a fixed checkpoint?
Any suggestions or clarifications would be greatly appreciated.
Thank you very much for your time.
Hello,
thank you for sharing your paper and code.
I am currently studying your paper and trying to reproduce the Bayesian Optimization based fine-tuning pipeline for Stable Diffusion v1.5, following the settings described in the paper and the released code.
In my reproduction, I fine-tune the diffusion model with BO and without BO (baseline), using the same dataset and training configuration as reported. During training, I save checkpoints every 1000 steps. For each setting (with BO / without BO), for each checkpoint, I generate synthetic images and train a downstream classifier, which is evaluated on the real test set. I then report the best result across checkpoints, following the same evaluation protocol as described in the paper.
However, in my experiments, I do not observe a clear improvement when using BO compared to the baseline. The results obtained with BO are very similar to those without BO, both of these results are stronger than the performance reported in the paper.
I strictly follow the training parameters and pipeline described in the paper and codebase, and the main difference is that I explicitly select the best checkpoint for each method, and the model I used is SD1.5
I would like to ask whether:
I select the best checkpoint based on downstream classification performance for both BO and the baseline. Is this how you evaluated the models, or did you use a fixed checkpoint?
Any suggestions or clarifications would be greatly appreciated.
Thank you very much for your time.