BERT

[jopetty]

Would be nice to have BERT an option for the encoder. Some issues are:

• BERT uses its own tokenizer, which probably doesn't play nicely with the Fields we've been using.
• How do we get access to the source vocabulary
• Do we always need to have the transformation token included in the target vocabulary, since BERT's tokenizer might do weird things to it?

[jopetty]

aedca6f adds a "working" (i.e., doesn't error) BERT model, but it doesn't seem to learn very well. Among the design considerations:

• wiring up the actual config parameters in the YAML file so they do something. Not sure how much flexibility we have with the pre-trained HuggingFace models, but at the very least we shouldn't have extraneous options
• it seems best (for training time) to freeze the layers of the BERT encoder, but maybe this should be a user-configurable option as well?

[jopetty]

It seems that the positional encodings built into the HuggingFace BERT models are not useful in a sequence to sequence context. I'm not really sure why this is, but it is fixable if we add our own positional encodings to the embedding layer of the pretrained models.