Skip to content

README.md

Latest commit

 

History

History
9 lines (7 loc) · 544 Bytes

README.md

File metadata and controls

9 lines (7 loc) · 544 Bytes

vibrato/docs

Here provides several documents for advanced usages.

  • compile.md describes how to compile system dictionaries from language resources.
  • map.md describes how to achieve faster tokenization through cache-efficient id mapping.
  • train.md describes how to train dictionaries from your own corpus.
  • small-dic.md describes how to generate smaller system dictionaries.
  • benchmark describes how to benchmark the tokenization speed of Vibrato.