Training a model using multiple corpora

[thiippal]

Hey!

I'm looking into training a single Transformer-based model for Finnish that would perform tagging, parsing, morphological analysis and named entity recognition.

For tagging, parsing and morphological analysis I would naturally use the UD-compliant Turku Dependency Treebank (TDT).

For NER, I would like to use the Turku NER corpus which is provided in a CoNLL-like format.

Can I use a spaCy config file to train different components of a model using different corpora? If so, is there an example of how to do this?

[thiippal]

@svlandeg, @adrianeboyd, anyone? :-)

Should I just train UD + NER separately, combine them into a single pipeline and serialize to disk?

[honnibal]

@thiippal Training them separately and "collating" them is what we do for our pipelines that have multiple corpora.

I think we should have some workflow for this though, maybe a loader that can read multiple corpora or something.

[Tadusko]

Hi @honnibal!

Expanding on the collating part, now that we've trained the pipelines separately (the general linguistic UD component 'transformer', 'tagger', 'morphologizer', 'parser' and the NER part 'transformer', 'ner'), merging them was simple.

However, the embedding layers can't seem to be shared between pipelines. Thus, I added both embedding layers like this ['transformer', 'tagger', 'morphologizer', 'parser', 'transformer_ner', 'ner']. This seems to work, but the separate embedding layers bloats the pipeline to about twice the size.

Is there a way to share the transformer layer while training on multiple corpora? If we train the UD part first and then try to source the transformer from it for NER training, the other components' performance will suffer if they're included and frozen (as described in the docs). Freezing the transformer raises an error.

Having them separately might be the optimal solution based on previous discussion (#6649 #6548) but I wanted to make sure if the're other ways.

[thiippal]

^ hi @honnibal @adrianeboyd @svlandeg any takes on the post above?

It would also be cool to have the "recipes" for small, medium and large models for English so we could try to replicate those for Finnish as well, and maybe eventually incorporate the models into spaCy.