Commit f18227a
committed
feat: add comprehensive benchmark scripts for tokenizers
Add benchmark suites for cl100k_base, Llama 3, and o200k_base tokenizers.
Each script compares splintr performance against reference implementations
(tiktoken, HuggingFace) across single/batch encoding and decoding operations.
- benchmark_cl100k.py: GPT-4/GPT-3.5-turbo tokenizer benchmarks
- benchmark_llama3.py: Llama 3 family tokenizer benchmarks
- benchmark_o200k.py: GPT-4o tokenizer benchmarks
Benchmarks measure throughput (MB/s, tokens/s) and latency across various
text types (short, medium, long, code, multilingual) with visualization
support via matplotlib charts.1 parent 1c8a7cd commit f18227a
1 file changed
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
95 | | - | |
| 95 | + | |
0 commit comments