Fine-Tuning on Schubert's lieder compositions #33
Description
Has anyone run the fine-tuning example on Schubert’s works as suggested in the README? I’m running with the large pre-trained model and the default config training parameters (batch size: 1, learning_rate: 1e-5, num_epochs: 64) on A40 (48GB), but I don’t observe any improvement as the epochs progress. Specifically, the best epoch is the 3rd (out of 64) with a minimum eval loss of 0.131. I want to perform fine-tuning on a separate dataset and I’m trying to figure out whether it’s my mistake or this is normal for this specific dataset.