Align tokenizer pipeline and terminology with Hugging Face tokenizers #427
Labels
Comments
robertknight
added a commit
that referenced
this issue
Dec 2, 2024
Add a `Normalizer` trait which provides the common interface for normalizers and rename the previous `Normalizer` struct to `BertNormalizer`, to align with the name in tokenizer.json files. Change other parts of the tokenization process to use the normalizer via a trait object. Part of #427
robertknight
added a commit
that referenced
this issue
Dec 3, 2024
Rename `Encoder` to `Model` to align with the terminology used in Hugging Face tokenizer.json files. Part of #427.
robertknight
added a commit
that referenced
this issue
Dec 3, 2024
robertknight
added a commit
that referenced
this issue
Dec 3, 2024
This was referenced Dec 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The main function of the rten-text crate is to provide a relatively lightweight Rust-only implementation of popular tokenizers (WordPiece, BPE etc.).
The separation of concerns is currently a little muddled. Since most projects and examples instantiate tokenizers from Hugging Face
tokenizer.jsonfiles, it would make sense to align the crate with the tokenization pipeline that these files capture.The sub-tasks are not well-defined yet, but the outcome should be that rten-text's tokenizers are a fairly straightforward implementation of this pipeline.
The text was updated successfully, but these errors were encountered: