Implementation on bigdata

[St-mlengineer]

I am trying to implement symspell on a large dataset consisting of text strings. The udf is running infinitely on databricks cluster. Is there a way to make it faster?

[wolfgarbe]

SymSpell can process about 10,000 words per second, on a single processor core. With multithreading up to 100,000 words per second.

Of course it depends on

• the average word length,
• the maximum edit distance,
• the dictionary size,
• the error probability per word,
• the specific hardware you run it,
• and the programming language of the specific SymSpell port used ( https://github.com/wolfgarbe/SymSpell#ports ).

Which SymSpell port/language you are using? Python is known for its slow execution times as an interpreted language. For big data you should use C#, C++, Java or Rust instead of Python.

Also, make sure that LoadDictionary() is called only once, when you are initializing SymSpell - but NOT every single time before you do a Lookup() spelling correction.

[St-mlengineer]

I am calling load_dictionary twice. One for English and another for my domain specific words. But it is during initialisation of symspell object.
In my function, I am using this object to first call lookup_compound and then word_segmentation.
I am using symspellpy.