[Tests] Model revision parameter not used consistently

[ckadner]

While debugging draft PR #497, we found the unit tests are not consistently using the model revision parameter.

From PR #499:

E huggingface_hub.errors.LocalEntryNotFoundError: Cannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, pass 'local_files_only=False' as input.

head_call_error = OfflineModeIsEnabled("Cannot access file since 'local_files_only=True' as been set. (repo_id: ibm-ai-platform/micro-g3.3-8b-instruct-1b, repo_type: model, revision: main, filename: config.json)") force_download = False, local_files_only = True

[prashantgupta24]

Thanks @ckadner for finding this bug! Interestingly, the errors on the CB test job are for the tokenizer:

tests/spyre_util.py:336: in create_random_request
    tokenizer = AutoTokenizer.from_pretrained(model)
.venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py:1069: in from_pretrained

and if I run

❯ HF_HUB_OFFLINE=0 python -c "from transformers import AutoTokenizer; tokenizer=AutoTokenizer.from_pretrained('ibm-ai-platform/micro-g3.3-8b-instruct-1b')"

it downloads the 6e9c6465a9d7e5e9fa35004a29f0c90befa7d23f revision. That's why if we remove that revision the tests are complaining.

It sounds like we somehow need a revision in the tokenizer too??

[prashantgupta24]

Checking out AutoTokenizer.from_pretrained:
It has an argument called revision:

   
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
                identifier allowed by git.

[prashantgupta24]

Yep this works

    tokenizer = AutoTokenizer.from_pretrained(model, revision="2714578f54cfb744ece40df9326ee0b47e879e03")

[prashantgupta24]

Which is what @joerunde tackled for tests/output_util.py:

https://github.com/vllm-project/vllm-spyre/pull/456/files#diff-ba17e02b8821fcd6250d626231859beb557749a4f6c1d6f0b0129094f721cc0eR65

but we need to do it in other places like spyre_util

[prashantgupta24]

And files like tests/precompilation/test_disable_compilation.py need to use the model argument to get the right revision

[prashantgupta24]

Maybe we also need to change places where we're using

tokenizer=model

to instead be

    tokenizer = AutoTokenizer.from_pretrained(model.name, revision=model.revision)