[FEATURE] Show only available models for fine-tuning

[nicolette-tml]

🚀 Feature

Hide or remove models from the fine-tuning model selection list if they are not currently available in the Models Hub. Only display models that are available for fine-tuning.

Motivation

When users select models that are not actually available in the Models Hub, they encounter the issue of dependencies not available for those models, resulting in failed experiments.

[orendain]

Hiya! @pascal-pfeiffer suggested that this may be a ModelHub request. ModelHub supports HFHub's model listing endpoint. For example:

import huggingface_hub as hf

models = hf.list_models()
for m in models:
    print(m)

or

from huggingface_hub import HfApi

hf_api = HfApi()
for m in hf_api.list_models():
    print(m)

MH's implementation doesn't support filtering options (e.g., hf.list_models(model_name=...)) - it will always return a basic list of all models within in.

For example, here's the return value of the above commands for a MH deployment in cloud-dev (note that all models are labeled as private because technically all MH models must be auth'd in order to reach):

ModelInfo(id='albert/albert-base-v2',
          private=True,
          disabled=None,
          ...
          spaces=None,
          safetensors=None,
          security_repo_status=None)

ModelInfo(id='microsoft/Phi-3-medium-4k-instruct',
          private=True,
          disabled=None,
          ...
          spaces=None,
          safetensors=None,
          security_repo_status=None)

This endpoint could satisfy the requirement from MH's end. LMK if there's something missing needed to support this feature.

[pascal-pfeiffer]

Thank you @orendain, I wasn't aware that this feature already was implemented. We can add it then on our side.

[nicolette-tml]

Thanks @orendain, can I confirm that this implementation works for on-prem customers too?

[orendain]

Thanks @orendain, can I confirm that this implementation works for on-prem customers too?

Yes, indeed.