spm_precompiled (which tokenizers depends on) contains multi-thousand-line macro invocations that confuse rust analyzer. This is normally not an issue because the macros exist only in tests and so are usually not built, but can lead to confusing issues in certain build setups that do build downstream crate tests.
I have a PR out to spm_precompiled (huggingface/spm_precompiled#5) to fix the issue - if/when that is merged tokenizers will need to be updated to use the new version.