• Type issue: Namespace '".../transformers.js/node_modules/@huggingface/tokenizers/types/index"' has no exported member 'Encoding'.ts(2694)
• We should expose the different components. Maybe with separate exports like it is in the rust/python library: from tokenizers.pre_tokenizers import Metaspace. This is necessary because... for example... certain llama tokenizers need to handle a legacy mode to ensure backwards compatibility (see here)

For example, these python imports/exports:

from tokenizers import Tokenizer, Encoding
from tokenizers.pre_tokenizers import Metaspace, Whitespace
from tokenizers.models import BPE
...

should be importable in JS like this:

import { Tokenizer, Encoding } from "@huggingface/tokenizers";
import { Metaspace, Whitespace } from "@huggingface/tokenizers/pre-tokenizers";
import { BPE } from "@huggingface/tokenizers/models";