Let your LLM think twice, inside the network.
Training-free layer looping for open-source LLMs: one line of code, zero extra weights, zero extra KV memory.
Quick start · Results · How it works · Adaptive mode · Supported models · Configuration
ThinkMore passes a model's hidden state through a block of its own middle layers several times before moving on. The model gets extra thinking per token, with no fine-tuning, no adapters, and no new parameters. It wraps any 🤗 Transformers causal LM in place, works with model.generate(), and comes back off with one call.
from thinkmore import loopify, RECIPES
handle = loopify(model, RECIPES["qwen3-4b-instruct"]) # that's it
model.generate(**inputs)- 🪄 One line, zero training. No fine-tuning, no adapters, no calibration data. The model's own weights are reused.
- 📈 Measured accuracy gains on the full MMLU, SciQ and GSM8K test sets, with per-question paired significance tests. SciQ matches the original paper to the decimal.
- 🧮 No extra memory. No weights are copied and the KV cache keeps its exact size, thanks to a cache-consistent loop design.
- ⚡ Adaptive mode, about 5% overhead. An entropy gate loops only the tokens the model is unsure about. Decoding runs at 1.05× the plain model's cost; on GSM8K it scores as well as looping every token.
- 🎛️ Compute budget control. Say "at most 5% extra compute" and ThinkMore tunes itself online to hit it.
- 🆕 Ready for the newest Qwen. Supports Qwen3.8 and the Qwen4 architecture out of the box, including hybrid linear attention (Gated DeltaNet), sparse indexed attention, MoE and hyper-connections.
- 🧱 Drop-in. Works with
generate(), batching and left padding.handle.unloopify()restores the original model. - ✅ Rigorously tested. 117 unit tests, including bit-exact identity checks, equivalence to explicitly duplicated layers, and cache-state checks on every supported architecture.
pip install git+https://github.com/Zihao-Wu06/LLM-ThinkMore.git # PyTorch >= 2.4, transformers >= 5import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from thinkmore import loopify, LoopConfig
model_id = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda") # or "mps" / "cpu"
# Loop layers 15-18 three times (damped Euler), and only on the tokens the model is unsure about
handle = loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", tau=1.0))
inputs = tok("A train leaves at 3:40 pm and arrives at 6:15 pm. How long is the trip?", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0], skip_special_tokens=True))
print(f"looped {handle.stats.gated_fraction:.0%} of decode tokens, "
f"+{handle.stats.decode_overhead:.1%} decode compute")
handle.unloopify() # back to the original modelMore examples are in examples/.
Every number below comes from a full benchmark test set in lm-eval-harness, compared question by question against the same model without looping. Each row lists the exact McNemar p-value of that paired comparison.
| Model | Benchmark | Plain | ThinkMore | Gain | p |
|---|---|---|---|---|---|
| Qwen3-4B | SciQ, 0-shot | 93.30 | 95.10 | +1.80 | < 0.001 |
| Qwen3-4B | MMLU, 0-shot (14,042 questions) | 68.35 | 69.34 | +0.99 | < 0.001 |
| Qwen3-4B-Instruct-2507 | SciQ, 0-shot | 92.80 | 94.40 | +1.60 | < 0.001 |
| Qwen3-4B-Instruct-2507 | GSM8K, 5-shot (adaptive mode) | 85.90 | 87.34 | +1.44 | 0.05 |
On Qwen3-4B, SciQ (93.30 → 95.10) matches the result reported by Training-Free Looped Transformers exactly, an independent replication of the method ThinkMore builds on.
Decode latency at batch size 1 on an Apple M4 (Qwen3-4B-Instruct-2507, bf16, real GSM8K prompts):
| Mode | Decode cost | Extra KV memory | Extra weights |
|---|---|---|---|
| Plain model | 1.00× | — | — |
| ThinkMore, loop every token | 1.22× | 0 | 0 |
ThinkMore, adaptive (prev_entropy gate) |
1.05× | 0 | 0 |
| ThinkMore, adaptive (look-ahead gate) | 1.07× | 0 | 0 |
The adaptive gate loops only about 5% of decode tokens on these prompts, and it is the configuration behind the GSM8K result above.
embeddings
│
layers 0 … 14 (run once)
│
┌───────▼───────┐
│ layers 15–18 │ ◄──┐ x ← x + (g(x) − x) / K damped Euler refinement,
└───────┬───────┘ │ K passes over the same weights
└────────────┘
│
layers 19 … 35 (run once)
│
logits
- Damped Euler looping. Instead of re-applying the block naively, which tends to hurt, each pass takes a small step toward the block's output. The first pass is reused as the first step, so K passes cost only K − 1 extra block evaluations.
- Cache-consistent by design. Extra passes read the KV cache but never write it. Prefill and scoring are exactly the full-sequence computation, and the cache keeps its original size. Tokens can also be looped a different number of times without breaking the cache, which is what makes the adaptive mode possible.
- Index-preserving wrappers. Layers are wrapped in place, so masks, rotary embeddings and caches keep flowing exactly as in the original model.
Most tokens are easy, and looping them is wasted compute. ThinkMore reads the model's own uncertainty, the entropy of its next-token distribution, and loops only when it is high. No labels, no task knowledge, one global threshold.
# loop a decode step only when the model is uncertain (entropy > tau nats)
loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", gate="prev_entropy", tau=1.0))
# or set a compute budget and let ThinkMore adjust tau online
loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", budget=0.05)) # ≈5% extra decode compute| Gate | Signal | Cost when triggered | Best for |
|---|---|---|---|
prev_entropy (default) |
entropy of the step that produced the current token | +22% of a step | everything, including batched decoding |
entropy (look-ahead) |
entropy of the current step, recomputed with the loop if high | ≈ +80% of a step | batch size 1, full-attention models |
With budget=0.05, the controller hit 4.8% measured extra decode compute against a 5% target.
LoopConfig(
window=(15, 19), # half-open range of decoder layers to loop
K=3, # window evaluations per looped token (K=1 = original model)
strategy="euler", # "euler" (damped, recommended) or "naive"
beta=0.0, # anchor: out = beta*g(x0) + (1-beta)*x_K
mode="block", # "block" loops the window; "layer" loops each layer (for MoE)
decode="full", # "full" | "bypass" (prompt only) | "gate" (adaptive)
gate="prev_entropy", # adaptive signal: "prev_entropy" | "entropy"
tau=1.0, # entropy threshold in nats
budget=None, # target extra decode compute, e.g. 0.05
)Recipes and defaults. RECIPES["qwen3-4b-instruct"] (layers 15–18, K=3) and RECIPES["llama-3.2-3b-instruct"] (layers 12–15, K=2) come from published configurations. default_config(num_layers) picks a 4-layer window starting at round(0.43 · L) with zero calibration; for 28- and 36-layer models it lands on the published windows.
Stats. handle.stats reports compute_overhead, decode_overhead, gated_fraction, looped_tokens, mean_tau and more.
pip install -e ".[eval]"
# accuracy: plain vs looped, paired per question
python scripts/eval_lm.py --model Qwen/Qwen3-4B --tasks sciq --num-fewshot 0 \
--configs base,qwen3-4b-instruct --out results/sciq.json
# latency at batch size 1, including the adaptive gate
python scripts/bench_gate.py --model Qwen/Qwen3-4B-Instruct-2507 \
--configs "base,qwen3-4b-instruct,qwen3-4b-instruct+decode=gate+tau=1.0"Config specs accept a recipe name, default, a:b:K, and +field=value overrides, for example qwen3-4b-instruct+decode=gate+budget=0.05.
| Model | Architecture | Highlights |
|---|---|---|
| Qwen3.8-27B | qwen3_5 |
hybrid Gated DeltaNet linear attention + gated full attention |
| Qwen3.8-Flash-Next (Qwen4 architecture) | qwen4_exp |
hybrid attention + sparse indexed attention, 512-expert MoE, hyper-connections, per-layer embeddings |
| Qwen3.5 family, dense and MoE | qwen3_5, qwen3_5_moe |
hybrid linear / full attention |
| Qwen3 | qwen3 |
benchmarked above |
| Llama 3.x | llama |
covered by the test suite |
Hybrid models loop through linear-attention and full-attention layers alike. ThinkMore replays each cache layer type with the right semantics (KV, recurrent state, conv state, indexer keys): prefill stays identical to the cache-less computation, and the cache keeps its size. Released Qwen3.8 checkpoints load as *ForConditionalGeneration, which works as is. For adaptive decoding on hybrid models, use the default prev_entropy gate.
from thinkmore import loopify, default_config, find_decoder_layers
cfg = default_config(len(find_decoder_layers(model))) # e.g. layers 28-31 of Qwen3.8-27B, K=3
handle = loopify(model, cfg)ThinkMore builds on the damped-Euler looping operator introduced in Training-Free Looped Transformers (arXiv:2605.23872); ThinkMore reproduces its Qwen3-4B SciQ result exactly. On top of it, ThinkMore adds the cache-consistent implementation, adaptive entropy gating, the compute budget controller and batched row gating. Ideas from The Tell-Tale Norm (adaptive, norm-triggered looping) and loopgraft (looping with KV caches) informed the design.
@software{thinkmore2026,
title = {ThinkMore: Training-Free Adaptive Layer Looping for Open-Source LLMs},
year = {2026},
url = {https://github.com/Zihao-Wu06/LLM-ThinkMore}
}Please also cite Training-Free Looped Transformers (arXiv:2605.23872) if you use the looping operator.