Boosting tagger and lemmatizer using additional data

[Branislava]

Info about spaCy

• spaCy version: 2.2.2
• Platform: Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid
• Python version: 3.6.5

Dear spaCy developers,

I have two questions regarding the inclusion of the external resources in the tagger training procedure.
I saw that a tagger model can be improved using the word vectors. But:

1. Is it possible to somehow "plug-in" a morphological dictionary to the tagger training procedure?
   and
2. Is it possible to train a lemmatizer that is not only based on lookup tables, but relies on information that contains part-of-speech, as well?

Put precisely, we would like to include external resources, besides our tagger training corpora, that contain information about a word, its lemma, universal part of speech tag and grammatical categories.

Many thanks

[adrianeboyd]

These are all reasonable things to want to include and I hope most or all of this is possible in the future, but right now none of this is very easy with spacy, sorry! We would like to develop a neural lemmatizer, but first we're working on a morphological analyzer, and it's still very much under development.

The options for lemmatization are a bit limited, since spacy only supports lookup-based and rule-based lemmatizers. The rule-based lemmatizer can currently use the UD POS tag and suffix rules along with a table of exceptions, but it doesn't have any other features or other kinds of rules. It's enough for English and languages with a similar amount of suffix-based inflection, but not great for many other languages.

You might be able to rely heavily on the exceptions table in order to have a more POS-based lookup than with the plain lookup lemmatizer, but you might still run into some issues because the supported POS tags are still a bit English-centric. What language are you working on?

[Branislava]

Thank you very much for your prompt response and good luck with the further development! The language I use spaCy for is Serbian, that is highly inflectional: e.g. nouns can change in gender, number and case. Therefore, the lookup table would be highly repetitive and some heuristic at least is desired. For now, I think we will try to figure out some rules. Thank you once again!