|
| 1 | +<!--Copyright 2026 the HuggingFace Inc. team. All rights reserved. |
| 2 | +
|
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with |
| 4 | +the License. You may obtain a copy of the License at |
| 5 | +
|
| 6 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 7 | +
|
| 8 | +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on |
| 9 | +an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the |
| 10 | +specific language governing permissions and limitations under the License. |
| 11 | +
|
| 12 | +⚠️ Note that this file is in Markdown but contains specific syntax for our doc-builder (similar to MDX) that may not be |
| 13 | +rendered properly in your Markdown viewer. |
| 14 | +
|
| 15 | +--> |
| 16 | +*This model was published in HF papers on 2025-10-30 and contributed to Hugging Face Transformers on 2026-09-04.* |
| 17 | + |
| 18 | +## Overview |
| 19 | + |
| 20 | +Kimi Linear is a hybrid linear attention architecture from Moonshot AI, introduced in |
| 21 | +[Kimi Linear: An Expressive, Efficient Attention Architecture](https://huggingface.co/papers/2510.26692). |
| 22 | + |
| 23 | +At its core is **Kimi Delta Attention (KDA)**, a refinement of [Gated DeltaNet](https://huggingface.co/papers/2412.06464) |
| 24 | +that gives each key channel its own forget gate, so the recurrent state decays per channel instead of per head. KDA is |
| 25 | +used in most layers; every fourth layer keeps a full-attention block that reuses DeepSeek-V3's Multi-head Latent |
| 26 | +Attention (MLA), and the feed-forward blocks are DeepSeek-V3-style MoE with a shared expert. |
| 27 | + |
| 28 | +The abstract from the paper is the following: |
| 29 | + |
| 30 | +*We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention |
| 31 | +under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning |
| 32 | +(RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends |
| 33 | +Gated DeltaNet with a finer-grained gating mechanism.* |
| 34 | + |
| 35 | +Two things are worth knowing when reading the modeling code: |
| 36 | + |
| 37 | +- **The model is NoPE.** Every released checkpoint sets `mla_use_nope=True`, so no rotary embedding is applied |
| 38 | + anywhere: the KDA layers encode position through their recurrence, and the full-attention layers are left without |
| 39 | + positional encoding. The `qk_rope_head_dim` slice still exists in the projections, it is simply never rotated. |
| 40 | +- **The layer pattern comes from the checkpoint.** `linear_attn_config` lists `kda_layers` / `full_attn_layers` with |
| 41 | + 1-based indices; the config converts them into the standard `layer_types` list. |
| 42 | + |
| 43 | +This model was contributed by [Remi Ouazan](https://huggingface.co/ror). |
| 44 | +The original code can be found [here](https://github.com/MoonshotAI/Kimi-Linear). |
| 45 | + |
| 46 | +## Usage examples |
| 47 | + |
| 48 | +```python |
| 49 | +from transformers import AutoModelForCausalLM, AutoTokenizer |
| 50 | + |
| 51 | + |
| 52 | +model_name = "moonshotai/Kimi-Linear-48B-A3B-Instruct" |
| 53 | + |
| 54 | +model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto") |
| 55 | +tokenizer = AutoTokenizer.from_pretrained(model_name) |
| 56 | + |
| 57 | +messages = [{"role": "user", "content": "Tell me about the french revolution."}] |
| 58 | +model_inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device) |
| 59 | + |
| 60 | +generated_ids = model.generate(**model_inputs, max_new_tokens=128) |
| 61 | +output_ids = generated_ids[0][len(model_inputs.input_ids[0]) :] |
| 62 | + |
| 63 | +print(tokenizer.decode(output_ids, skip_special_tokens=True)) |
| 64 | +``` |
| 65 | + |
| 66 | +The KDA layers run on a pure PyTorch implementation by default. Installing |
| 67 | +[`kernels`](https://github.com/huggingface/kernels) (`pip install -U kernels`) and passing `use_kernels=True` |
| 68 | +in `from_pretrained` makes them dispatch to custom kernels instead, which is considerably faster for long sequences. |
| 69 | + |
| 70 | +## KimiLinearConfig |
| 71 | + |
| 72 | +[[autodoc]] KimiLinearConfig |
| 73 | + |
| 74 | +## KimiLinearModel |
| 75 | + |
| 76 | +[[autodoc]] KimiLinearModel |
| 77 | + - forward |
| 78 | + |
| 79 | +## KimiLinearForCausalLM |
| 80 | + |
| 81 | +[[autodoc]] KimiLinearForCausalLM |
| 82 | + - forward |
0 commit comments