fix(deps): update rust crate tokenizers to 0.22.0 by renovate[bot] · Pull Request #6 · Danite/pancakeLLM

renovate · 2024-11-27T13:17:53Z

This PR contains the following updates:

Package	Type	Update	Change
tokenizers	dependencies	minor	`0.20.0` → `0.22.0`

Release Notes

huggingface/tokenizers (tokenizers)

`v0.22.2`

Compare Source

What's Changed

Okay mostly doing the release for these PR:

Update deserialize of added tokens by @ArthurZucker in #1891
update stub for typing by @ArthurZucker in #1896
bump PyO3 to 0.26 by @davidhewitt in #1901

Basically good typing with at least ty, and a lot fast (from 4 to 8x faster) loading vocab with a lot of added tokens and GIL free !?

ci: add support for building Win-ARM64 wheels by @MugundanMCW in #1869
Add cargo-semver-checks to Rust CI workflow by @haixuanTao in #1875
Update indicatif dependency by @gordonmessmer in #1867
Bump node-forge from 1.3.1 to 1.3.2 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1889
Bump js-yaml from 3.14.1 to 3.14.2 in /bindings/node by @dependabot[bot] in #1892
fix: used normalize_str in BaseTokenizer.normalize by @ishitab02 in #1884
[MINOR:TYPO] Update mod.rs by @cakiki in #1883
Remove runtime stderr warning from Python bindings by @Copilot in #1898
Mark immutable pyclasses as frozen by @ngoldbaum in #1861
DOCS: add add_prefix_space to processors.ByteLevel by @CloseChoice in #1878
Bump express from 4.21.2 to 4.22.1 in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1903

New Contributors

@MugundanMCW made their first contribution in #1869
@haixuanTao made their first contribution in #1875
@gordonmessmer made their first contribution in #1867
@ishitab02 made their first contribution in #1884
@Copilot made their first contribution in #1898
@ngoldbaum made their first contribution in #1861
@CloseChoice made their first contribution in #1878

Full Changelog: huggingface/tokenizers@v0.22.1...v0.22.2

`v0.22.1`

Compare Source

Release v0.22.1

Main change:

Bump huggingface_hub upper version (#1866) from @Wauplin
chore(trainer): add and improve trainer signature (#1838) from @shenxiangzhuang
Some doc updates: c91d76a, 7b02178, 57eb8d7

`v0.22.0`

Compare Source

What's Changed

Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www by @dependabot[bot] in #1827
Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility by @sondalex in #1758
fix: use AHashMap to fix compile error by @b00f in #1840
New stream by @ArthurZucker in #1856
[docs] Add more decoders by @pcuenca in #1849
Fix missing parenthesis in EncodingVisualizer.calculate_label_colors by @Liam-DeVoe in #1853
Update quicktour.mdx re: Issue #1625 by @WilliamPLaCroix in #1846
remove stray comment by @sanderland in #1831
Fix typo in README by @aisk in #1808
RUSTSEC-2024-0436 - replace paste with pastey by @nystromjd in #1834
Tokenizer: Add native async bindings, via py03-async-runtimes. by @michaelfeil in #1843

New Contributors

@b00f made their first contribution in #1840
@pcuenca made their first contribution in #1849
@Liam-DeVoe made their first contribution in #1853
@WilliamPLaCroix made their first contribution in #1846
@sanderland made their first contribution in #1831
@aisk made their first contribution in #1808
@nystromjd made their first contribution in #1834
@michaelfeil made their first contribution in #1843

Full Changelog: huggingface/tokenizers@v0.21.3...v0.22.0rc0

`v0.21.4`

Compare Source

Full Changelog: huggingface/tokenizers@v0.21.3...v0.21.4

No change, the 0.21.3 release failed, this is just a re-release.

https://github.com/huggingface/tokenizers/releases/tag/v0.21.3

`v0.21.2`

Compare Source

What's Changed

This release if focused around some performance optimization, enabling broader python no gil support, and fixing some onig issues!

Update the release builds following 0.21.1. by @Narsil in #1746
replace lazy_static with stabilized std::sync::LazyLock in 1.80 by @sftse in #1739
Fix no-onig no-wasm builds by @414owen in #1772
Fix typos in strings and comments by @co63oc in #1770
Fix type notation of merges in BPE Python binding by @Coqueue in #1766
Bump http-proxy-middleware from 2.0.6 to 2.0.9 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1762
Fix data path in test_continuing_prefix_trainer_mismatch by @GaetanLepage in #1747
clippy by @ArthurZucker in #1781
Update pyo3 and rust-numpy depends for no-gil/free-threading compat by @Qubitium in #1774
Use ApiBuilder::from_env() in from_pretrained function by @BenLocal in #1737
Upgrade onig, to get it compiling with GCC 15 by @414owen in #1771
Itertools upgrade by @sftse in #1756
Bump webpack-dev-server from 4.10.0 to 5.2.1 in /tokenizers/examples/unstable_wasm/www by @dependabot in #1792
Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node by @dependabot in #1796
Fix features blending into a paragraph by @bionicles in #1798
Adding throughput to benches to have a more consistent measure across by @Narsil in #1800
Upgrading dependencies. by @Narsil in #1801
[docs] Whitespace by @stevhliu in #1785
Hotfixing the stub. by @Narsil in #1802
Bpe clones by @sftse in #1707
Fixed Length Pre-Tokenizer by @jonvet in #1713
Consolidated optimization ahash dary compact str by @Narsil in #1799
🚨 breaking: Fix training with special tokens by @ArthurZucker in #1617

New Contributors

@414owen made their first contribution in #1772
@co63oc made their first contribution in #1770
@Coqueue made their first contribution in #1766
@GaetanLepage made their first contribution in #1747
@Qubitium made their first contribution in #1774
@BenLocal made their first contribution in #1737
@bionicles made their first contribution in #1798
@stevhliu made their first contribution in #1785
@jonvet made their first contribution in #1713

Full Changelog: huggingface/tokenizers@v0.21.1...v0.21.2rc0

`v0.21.1`

Compare Source

What's Changed

Update dev version and pyproject.toml by @ArthurZucker in #1693
Add feature flag hint to README.md, fixes #1633 by @sftse in #1709
Upgrade to PyO3 0.23 by @Narsil in #1708
Fixing the README. by @Narsil in #1714
Fix typo in Split docstrings by @Dylan-Harden3 in #1701
Fix typos by @tinyboxvk in #1715
Update documentation of Rust feature by @sondalex in #1711
Fix panic in DecodeStream::step due to incorrect index usage by @n0gu-furiosa in #1699
Fixing the stream by removing the read_index altogether. by @Narsil in #1716
Fixing NormalizedString append when normalized is empty. by @Narsil in #1717
🚨 Support updating template processors by @ArthurZucker in #1652. Removed in this release to keep backware compatibility temporarily.
Update metadata as Python3.7 and Python3.8 support was dropped by @earlytobed in #1724
Add rustls-tls feature by @torymur in #1732

New Contributors

@Dylan-Harden3 made their first contribution in #1701
@sondalex made their first contribution in #1711
@n0gu-furiosa made their first contribution in #1699
@earlytobed made their first contribution in #1724
@torymur made their first contribution in #1732

Full Changelog: huggingface/tokenizers@v0.21.0...v0.21.1

`v0.21.0`

Compare Source

Release v0.20.4 v0.21.0

More cache options. by @Narsil in #1675
Disable caching for long strings. by @Narsil in #1676
Testing ABI3 wheels to reduce number of wheels by @Narsil in #1674
Adding an API for decode streaming. by @Narsil in #1677
Decode stream python by @Narsil in #1678
Fix encode_batch and encode_batch_fast to accept ndarrays again by @diliop in #1679

We also no longer support python 3.7 or 3.8 (similar to transformers) as they are deprecated.

Full Changelog: huggingface/tokenizers@v0.20.3...v0.21.0

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

fix(deps): update rust crate tokenizers to 0.22.0

8f051b7

renovate bot force-pushed the renovate/tokenizers-0.x branch from 33a6027 to 8f051b7 Compare August 31, 2025 09:25

renovate bot changed the title ~~fix(deps): update rust crate tokenizers to 0.21.0~~ fix(deps): update rust crate tokenizers to 0.22.0 Aug 31, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(deps): update rust crate tokenizers to 0.22.0#6

fix(deps): update rust crate tokenizers to 0.22.0#6
renovate[bot] wants to merge 1 commit intomainfrom
renovate/tokenizers-0.x

renovate bot commented Nov 27, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

Conversation

renovate bot commented Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Notes

v0.22.2

What's Changed

New Contributors

v0.22.1

Release v0.22.1

v0.22.0

What's Changed

New Contributors

v0.21.4

v0.21.2

What's Changed

New Contributors

v0.21.1

What's Changed

New Contributors

v0.21.0

Release v0.20.4 v0.21.0

Configuration

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

0 participants

renovate bot commented Nov 27, 2024 •

edited

Loading

`v0.22.2`

`v0.22.1`

`v0.22.0`

`v0.21.4`

`v0.21.2`

`v0.21.1`

`v0.21.0`