v1.0.3
added the mode parameter in the save and load methods to help developers, save and load their vocab and the merges of the tokenizer in their desired format .
Full Changelog: v1.0.21...v1.0.3
added the mode parameter in the save and load methods to help developers, save and load their vocab and the merges of the tokenizer in their desired format .
Full Changelog: v1.0.21...v1.0.3