NBME - Score Clinical Patient Notes
A NLP competition for NER (Named Entity Recognition) task.
Ranked on 117th (117 out of 1501) - bronze medal
Focal loss was basically invented for object detection task, to overcome the class imbalance issue. However, this method is also perfectly suitable for NER tasks.
Also, we could improve the focal loss by using it with Label smoothing.
As the provided dataset in this competition has a huge number of unlabeled data with imbalanced classes, training the model with focal loss and pseudo labeling worked perfectly fine.
Sample notebook for training MLM model with unlabeled data
Training the DeBERTa v3 large model with the unlabeled data as a masked language model is one of the key point of getting a high score for this competition. By pre-training the masked language model with unlabeled text data, the fine-tuned models were able to understand the distribution of the words in the patient notes.
Notebook for CommonLit competition (1)
Notebook for CommonLit competition (2)
Sample notebook for meta pseudo labeling
Related papers: - Meta Pseudo Labels - Fine-Tuning Pre-trained Language Model with Weak Supervision - Can Students outperform Teacher models
Basically, the notebook above for the MPL (Meta Pseudo Labeling) is designed on the hypothethis that the student model could outperform the teacher model when additional data is involved.
The Adversarial Weight Perturbation (AWP) was used in the first place solution of the Feedback Prize competition, and it is showen to be effective in the NBME competition as well.