Skip to content

Commit f18227a

Browse files
committed
feat: add comprehensive benchmark scripts for tokenizers
Add benchmark suites for cl100k_base, Llama 3, and o200k_base tokenizers. Each script compares splintr performance against reference implementations (tiktoken, HuggingFace) across single/batch encoding and decoding operations. - benchmark_cl100k.py: GPT-4/GPT-3.5-turbo tokenizer benchmarks - benchmark_llama3.py: Llama 3 family tokenizer benchmarks - benchmark_o200k.py: GPT-4o tokenizer benchmarks Benchmarks measure throughput (MB/s, tokens/s) and latency across various text types (short, medium, long, code, multilingual) with visualization support via matplotlib charts.
1 parent 1c8a7cd commit f18227a

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

python/splintr/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,4 +92,4 @@
9292
"O200K_AGENT_TOKENS",
9393
"LLAMA3_AGENT_TOKENS",
9494
]
95-
__version__ = "0.4.0"
95+
__version__ = "0.5.0"

0 commit comments

Comments
 (0)