Entity Tracking in Language Models

Najoung Kim and Sebastian Schuster

This is the repository accompanying the ACL 2023 paper Entity Tracking in Language Models.

Dataset

We provide the dataset in a password-protected ZIP file to ideally prevent leakage of the dataset into the training data of future language models. Please do not include the uncompressed files in any repositories if you use the data.

• Download: data/boxes-dataset-v1.zip
• Password: iamnotaLM

Model Outputs

We provide the predictions of our model runs in a password-protected ZIP file to prevent leakage of the dataset into the training data of future language models. Please do not include the uncompressed files in any repositories if you use the predictions.

• Download: model-outputs/model-outputs.zip
• Password: iamnotaLM

Code

The directory code contains code for computing evaluation metrics and for generating new datasets.

Citation

If you use the dataset or the dataset generation script, please cite the following paper:

@inproceedings{kim-schuster-2023-entity,
    title = "Entity Tracking in Language Models",
    author = "Kim, Najoung  and
      Schuster, Sebastian",
    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023)",
    year = "2023",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.acl-long.213",
    pages = "3835--3855"
}