v2.0.4

Bugfix: Prevent race conditions between tokenizer and sentence splitter in parallel processing (--parallel > 1).

v2.0.3

• Skip tests for unimplemented features (some builds will fail if any of the unit tests fail).

v2.0.2

• Bugfix: Parallel tokenization (--parallel > 1) works again.
• Support for musical notes (sharps).

v2.0.1

Bugfix. As always, there is a miniscule detail that causes things to go wrong… 🙄

v2.0.0

New features and improvements

• New API: Use new class SoMaJo instead of Tokenizer and SentenceSplitter. Currently, the old API is still supported but will issue deprecation warnings.
• Speed-up: Due to a new internal representation of the input text during processing (as a doubly linked list of Token objects), tokenization is now two to three times faster.
• Incremental and parallel processing of XML: If a sensible set of eos_tags is specified, the XML input will be processed incrementally (allowing for arbitrarily large XML input). In addition, if a sensible set of eos_tags is specified, processing can also be parallelized.
• New option --strip-tags to suppress the output of XML tags.
• Support for textual representations of emojis (:smile:, :stuck_out_tongue_winking_eye:, etc.).
• Support for textfaces (༼ʘ̚ل͜ʘ̚༽, ╚(ಠ_ಠ)=┐, etc.).

Breaking changes

• Removed the tokenizer script (deprecated since version 1.5.0 released in October 2017). Use somajo-tokenizer instead.
• Language codes contain the tokenization guideline: "de_CMC" instead of "de" and "en_PTB" instead of "en".

v1.11.0

• XML sentence splitting: Added hr tag to default sentence breaks
• Recognize Reddit links in shorthand notation
• Improved robustness of XML processing

v1.10.7

• Make recognition of gender star case insensitive
• Fix problem with “nasty” character as last character of text unit

v1.10.6

• Added support for gender star (Mitarbeiter*innen)
• Improvements regarding lists of numbers (1,2,3,4,5,6,7), section numbers (1.1.4) and IPv4 addresses (192.0.2.42)

v1.10.5

A few small improvements regarding the emoji variation selector character.

v1.10.4

Bugfix related to the --version option.