Welcome to the repoitory of the final project for the Algorithms for Speech and Natural Language Processing class.
This section contains various datasets utilized for training our models.
- Outputs: Explore our project's code and outputs:
Our project is inspired by "It’s not just size that matters" by Schick and Schütze (2020), introducing Pattern Exploitation Training (PET). PET reframes tasks as language modeling problems, using fine-tuned language models to label unlabeled data. This enables classical classifiers with small training datasets and supports few-shot learning. Our project explores PET, replicates its results on various datasets, and compares different masked language models.
Language models like GPT-3 and GPT-4 excel in natural language processing but require substantial computational resources. PET offers an efficient alternative by transforming tasks into language modeling challenges. It uses predefined patterns and verbalizers to convert features into sentences and labels into words. PET facilitates few-shot learning and works well with small training datasets, leveraging easily accessible unlabeled data.
Our project replicates Schick and Schütze's study and compares masked language models. While some SuperGLUE tasks showed lower accuracies, sentiment classification tasks performed well. We faced challenges such as labeling errors for specific tasks. Additionally, we created new tasks and explored MLM performance within a few-shot learning paradigm.
In conclusion, our project provides insights into the effectiveness of Pattern Exploitation Training (PET) for natural language tasks. Despite limitations due to hardware and time constraints, our findings contribute to the discussion on efficient language model training and real-world applications.