feat(transformers): Transformers 4.54 base

[wcrzlh]

What does this PR do?

This pr is proposed to upgrade modeling_XXX/generation/cache_utils/activations/processing_XXX module to align with hf transformers 4.54.1.

Highlighted Features

modeing_XXX

1. EmbeddingUtilsMixin have been extracted from BaseClass PretrainedClass as the independent part for dealing with input and output embeddings and tie_weights.
2. The process of getting config have been extracted to init__subclass function. Both config and config_class could be supported for transformers config.
3. attn_implementation part has been refactored. In transformers 4.50, there are "autoset_attn_implementation" in from_pretrained and from_config func so that attn_implementation could still be modified after setting attn_implementation in your scripts. In transformers 4.54, check_and_adjust_attn_implementation has been set to repalce autoset_attn_implementation so that default attn_implementation would be set during init process, then in from_pretrained or from_config func, attn_implementation could only be modified by setting in user's scripts.
4. get_mindspore_dtype has been extracted from from_pretrained basic logic to clarify the code.
5. "load_pretrained_model" has been refactored. Firstly, find_missing_and unexpected keys has been extracted as an independant func. Then key_rename_mapping is added to minimize the model_weight gap between pytorch and hf transformers.
6. get_compiled_call is added for initial exploration of substitution for "torch.compile".

generation

1. In previous mindone.transformers, we use _supports_dynamic_input as the label to control if padding inputs and compilable cache need to be implemented. But right now the logic is converted to use _supports_jit as the label so that we do not need to supplement any variables in modeling_xxx.py.
2. load_custom_generate has been added so that custom generate.py could been used for inference.
3. chunk_prefill and compile has been added.

cache

1. CacheLayerMixin has been extracted as the basic class for HybridCache.

Models

1. Qwen3_VL/Qwen3_VL_MoE model support
2. VLM model refactor. XXXModel has been extracted from XXXForConditionalGeneration to deal with visual model and text model. XXXForConditionalGeneration only keep the general func and is consisted of XXXModel and lm_head

Validation

1. ALL Fast UT passed except ModernBert:

pytest tests/transformers_tests/models

ModernBert UT could pass if running single model like pytest tests/transformers_tests/models/ModernBert. But it would raise an OSError if running above commands.

OSError: [Errno 7] Argument list too long: '/home/wcr/miniconda3/envs/transformers_4.54/bin/python'

3. Vital models (bert/qwen2.5/glm/gpt2/qwen3/glm4v/qwen3_vl) has also been validated with real model weights.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes? E.g. record bug fixes or new features in What's New. Here are the
  documentation guidelines
• Did you build and run the code without any errors?
• Did you report the running environment (NPU type/MS version) and performance in the doc? (better record it for data loading, model inference, or training tasks)
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@xxx

[gemini-code-assist]

Summary of Changes

Hello @wcrzlh, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the Qwen3-VL model series, encompassing both standard and Mixture-of-Experts architectures, into the MindOne Transformers library. This integration is accompanied by new inference examples and a specialized module for MoE handling. Concurrently, the PR features a comprehensive refactoring effort across the library's foundational components. This includes updating activation functions to leverage mint operations, overhauling audio and image processing, streamlining attention mask creation, and standardizing model input/output structures. These architectural improvements are designed to boost code maintainability, enhance flexibility, and lay the groundwork for future model developments and performance optimizations.

Highlights

• Qwen3-VL Model Integration: Introduced support for Qwen3-VL models, including both dense and Mixture-of-Experts (MoE) variants, with new inference scripts and a dedicated MoE text expert module.
• Core Library Refactoring: Undertook significant refactoring across the mindone.transformers library, modernizing activation functions, enhancing audio and image processing utilities, and refining attention mask generation.
• Attention Mechanism Standardization: Standardized attention mechanisms and caching, including the introduction of new mask functions (chunked, padding, packed sequence) and a unified create_masks_for_generate function.
• New Base Classes and Utilities: Added new modules like MoeTextExperts for parallel processing and modeling_layers.py with generic base classes for sequence classification, question answering, and token classification, improving code reusability and maintainability.
• Multimodal Processing Enhancements: Improved multimodal processing with new audio loading/processing functions, refined image handling (including image grouping and channel dimension formats), and updated placeholder mask logic for VLMs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request upgrades several modules to align with Hugging Face Transformers v4.54.1, including significant refactoring for MoE support, attention mechanisms, and generation utilities. Overall, the changes are a good step forward, introducing better structure and new features. I've identified a few areas for improvement, mainly concerning code robustness and documentation clarity.