Usage with BigBird-Roberta-Base

[jordanparker6]

Would it be possible to use LiLT with BigBird-Roberta-Base models?

If so, any feedback on the best approach of doing so? What might need changing in the LiLT repository to do so?

https://huggingface.co/google/bigbird-roberta-base

[jordanparker6]

I was able to use the provided script to create a lilt-roberta-base-en using the following: https://huggingface.co/google/bigbird-roberta-base. If I can get this working, I will post up to HuggingfaceHub.

BigBird uses the same tokenizer as roberta so no issue with tokenizationgoogle/bigbird-roberta-base.

However, the following error occurs when loading the model.

RuntimeError: Error(s) in loading state_dict for LiltForTokenClassification: size mismatch for lilt.layout_embeddings.box_position_embeddings.weight: copying a param with shape torch.Size([514, 192]) from checkpoint, the shape in current model is torch.Size([4096, 192]). You may consider adding ignore_mismatched_sizes=Truein the modelfrom_pretrained method.

I think this error is created when the pytorch state dicts are fused with the following line.

total_model = {**text_model, **lilt_model}

The lilt_model dim changes the incoming bigbird dim.

Would it be problematic to switch this:

total_model = {**lilt_model, **text_model }

Or would this break the architecture?