Skip to content

spm_precompiled dependency bricks rust-analyzer in certain build setups #1917

@gvilums

Description

@gvilums

spm_precompiled (which tokenizers depends on) contains multi-thousand-line macro invocations that confuse rust analyzer. This is normally not an issue because the macros exist only in tests and so are usually not built, but can lead to confusing issues in certain build setups that do build downstream crate tests.

I have a PR out to spm_precompiled (huggingface/spm_precompiled#5) to fix the issue - if/when that is merged tokenizers will need to be updated to use the new version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions