Support Llama 3 tokenizer (implement ignore_merges behavior)

[robertknight]

Attempting to load the tokenizer.json file from Llama 3.2 fails with an error processing the BPE merge entries:

Error: BpeError(InvalidMergeEntry("Ġ ĠĠĠ"))

If rten-text is modified to ignore this error, then the qwen2_chat example works with Llama 3.2, after a minor modification to the special token IDs.

Edit: I have just noticed the ignore_merges: true in the tokenizer.json file. This seems relevant.

[robertknight]

ignore_merges was added in huggingface/tokenizers@914576f. See also https://github.com/huggingface/tokenizers/pull/1493/files.

The documentation says:

ignore_merges (bool, optional) — Whether or not to match tokens with the vocab before using merges.