Can performance on a small dataset become worse after fine-tuning?

[TPF2017]

After fine-tuning on a small data set, the performance on this data set is worse. Is this normal? What are the possible reasons?

[abdulfatir]

Hi @TPF2017!

Fine-tuning performance depends on a variety of factors, including hyperparameters such as the learning rate and training steps. If your dataset is small, it is possible that a specific selection of these hyperparameters (e.g., high learning rate or large number of step) worsens the performance. There are other potential causes such as distribution shifts in the time series.