Skip to content

About

🧠🔁 Let your LLM think twice, inside the network. Training-free, adaptive layer looping for open-source LLMs (Qwen3.8 / Qwen4, Qwen3, Llama): one line of code, zero extra weights, zero extra KV memory.

Topics

Resources

Stars

1 star

Watchers

2 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

🧠🔁 ThinkMore

Let your LLM think twice, inside the network.

Training-free layer looping for open-source LLMs: one line of code, zero extra weights, zero extra KV memory.

License Python Transformers Tests Qwen3.8

Quick start · Results · How it works · Adaptive mode · Supported models · Configuration


ThinkMore passes a model's hidden state through a block of its own middle layers several times before moving on. The model gets extra thinking per token, with no fine-tuning, no adapters, and no new parameters. It wraps any 🤗 Transformers causal LM in place, works with model.generate(), and comes back off with one call.

from thinkmore import loopify, RECIPES

handle = loopify(model, RECIPES["qwen3-4b-instruct"])   # that's it
model.generate(**inputs)

✨ Highlights

  • 🪄 One line, zero training. No fine-tuning, no adapters, no calibration data. The model's own weights are reused.
  • 📈 Measured accuracy gains on the full MMLU, SciQ and GSM8K test sets, with per-question paired significance tests. SciQ matches the original paper to the decimal.
  • 🧮 No extra memory. No weights are copied and the KV cache keeps its exact size, thanks to a cache-consistent loop design.
  • ⚡ Adaptive mode, about 5% overhead. An entropy gate loops only the tokens the model is unsure about. Decoding runs at 1.05× the plain model's cost; on GSM8K it scores as well as looping every token.
  • 🎛️ Compute budget control. Say "at most 5% extra compute" and ThinkMore tunes itself online to hit it.
  • 🆕 Ready for the newest Qwen. Supports Qwen3.8 and the Qwen4 architecture out of the box, including hybrid linear attention (Gated DeltaNet), sparse indexed attention, MoE and hyper-connections.
  • 🧱 Drop-in. Works with generate(), batching and left padding. handle.unloopify() restores the original model.
  • ✅ Rigorously tested. 117 unit tests, including bit-exact identity checks, equivalence to explicitly duplicated layers, and cache-state checks on every supported architecture.

🚀 Quick start

pip install git+https://github.com/Zihao-Wu06/LLM-ThinkMore.git        # PyTorch >= 2.4, transformers >= 5
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from thinkmore import loopify, LoopConfig

model_id = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda")  # or "mps" / "cpu"

# Loop layers 15-18 three times (damped Euler), and only on the tokens the model is unsure about
handle = loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", tau=1.0))

inputs = tok("A train leaves at 3:40 pm and arrives at 6:15 pm. How long is the trip?", return_tensors="pt").to(model.device)
print(tok.decode(model.generate(**inputs, max_new_tokens=128)[0], skip_special_tokens=True))

print(f"looped {handle.stats.gated_fraction:.0%} of decode tokens, "
      f"+{handle.stats.decode_overhead:.1%} decode compute")
handle.unloopify()   # back to the original model

More examples are in examples/.

📊 Results

Every number below comes from a full benchmark test set in lm-eval-harness, compared question by question against the same model without looping. Each row lists the exact McNemar p-value of that paired comparison.

Accuracy

Model Benchmark Plain ThinkMore Gain p
Qwen3-4B SciQ, 0-shot 93.30 95.10 +1.80 < 0.001
Qwen3-4B MMLU, 0-shot (14,042 questions) 68.35 69.34 +0.99 < 0.001
Qwen3-4B-Instruct-2507 SciQ, 0-shot 92.80 94.40 +1.60 < 0.001
Qwen3-4B-Instruct-2507 GSM8K, 5-shot (adaptive mode) 85.90 87.34 +1.44 0.05

On Qwen3-4B, SciQ (93.30 → 95.10) matches the result reported by Training-Free Looped Transformers exactly, an independent replication of the method ThinkMore builds on.

Speed and memory

Decode latency at batch size 1 on an Apple M4 (Qwen3-4B-Instruct-2507, bf16, real GSM8K prompts):

Mode Decode cost Extra KV memory Extra weights
Plain model 1.00× — —
ThinkMore, loop every token 1.22× 0 0
ThinkMore, adaptive (prev_entropy gate) 1.05× 0 0
ThinkMore, adaptive (look-ahead gate) 1.07× 0 0

The adaptive gate loops only about 5% of decode tokens on these prompts, and it is the configuration behind the GSM8K result above.

🔍 How it works

          embeddings
              │
      layers 0 … 14        (run once)
              │
      ┌───────▼───────┐
      │ layers 15–18  │ ◄──┐   x ← x + (g(x) − x) / K    damped Euler refinement,
      └───────┬───────┘    │                             K passes over the same weights
              └────────────┘
              │
      layers 19 … 35       (run once)
              │
            logits
  • Damped Euler looping. Instead of re-applying the block naively, which tends to hurt, each pass takes a small step toward the block's output. The first pass is reused as the first step, so K passes cost only K − 1 extra block evaluations.
  • Cache-consistent by design. Extra passes read the KV cache but never write it. Prefill and scoring are exactly the full-sequence computation, and the cache keeps its original size. Tokens can also be looped a different number of times without breaking the cache, which is what makes the adaptive mode possible.
  • Index-preserving wrappers. Layers are wrapped in place, so masks, rotary embeddings and caches keep flowing exactly as in the original model.

🎯 Adaptive mode: think only when it matters

Most tokens are easy, and looping them is wasted compute. ThinkMore reads the model's own uncertainty, the entropy of its next-token distribution, and loops only when it is high. No labels, no task knowledge, one global threshold.

# loop a decode step only when the model is uncertain (entropy > tau nats)
loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", gate="prev_entropy", tau=1.0))

# or set a compute budget and let ThinkMore adjust tau online
loopify(model, LoopConfig(window=(15, 19), K=3, decode="gate", budget=0.05))   # ≈5% extra decode compute
Gate Signal Cost when triggered Best for
prev_entropy (default) entropy of the step that produced the current token +22% of a step everything, including batched decoding
entropy (look-ahead) entropy of the current step, recomputed with the loop if high ≈ +80% of a step batch size 1, full-attention models

With budget=0.05, the controller hit 4.8% measured extra decode compute against a 5% target.

⚙️ Configuration

LoopConfig(
    window=(15, 19),      # half-open range of decoder layers to loop
    K=3,                  # window evaluations per looped token (K=1 = original model)
    strategy="euler",     # "euler" (damped, recommended) or "naive"
    beta=0.0,             # anchor: out = beta*g(x0) + (1-beta)*x_K
    mode="block",         # "block" loops the window; "layer" loops each layer (for MoE)
    decode="full",        # "full" | "bypass" (prompt only) | "gate" (adaptive)
    gate="prev_entropy",  # adaptive signal: "prev_entropy" | "entropy"
    tau=1.0,              # entropy threshold in nats
    budget=None,          # target extra decode compute, e.g. 0.05
)

Recipes and defaults. RECIPES["qwen3-4b-instruct"] (layers 15–18, K=3) and RECIPES["llama-3.2-3b-instruct"] (layers 12–15, K=2) come from published configurations. default_config(num_layers) picks a 4-layer window starting at round(0.43 · L) with zero calibration; for 28- and 36-layer models it lands on the published windows.

Stats. handle.stats reports compute_overhead, decode_overhead, gated_fraction, looped_tokens, mean_tau and more.

🧪 Reproduce the benchmarks

pip install -e ".[eval]"

# accuracy: plain vs looped, paired per question
python scripts/eval_lm.py --model Qwen/Qwen3-4B --tasks sciq --num-fewshot 0 \
    --configs base,qwen3-4b-instruct --out results/sciq.json

# latency at batch size 1, including the adaptive gate
python scripts/bench_gate.py --model Qwen/Qwen3-4B-Instruct-2507 \
    --configs "base,qwen3-4b-instruct,qwen3-4b-instruct+decode=gate+tau=1.0"

Config specs accept a recipe name, default, a:b:K, and +field=value overrides, for example qwen3-4b-instruct+decode=gate+budget=0.05.

🧩 Supported models

Model Architecture Highlights
Qwen3.8-27B qwen3_5 hybrid Gated DeltaNet linear attention + gated full attention
Qwen3.8-Flash-Next (Qwen4 architecture) qwen4_exp hybrid attention + sparse indexed attention, 512-expert MoE, hyper-connections, per-layer embeddings
Qwen3.5 family, dense and MoE qwen3_5, qwen3_5_moe hybrid linear / full attention
Qwen3 qwen3 benchmarked above
Llama 3.x llama covered by the test suite

Hybrid models loop through linear-attention and full-attention layers alike. ThinkMore replays each cache layer type with the right semantics (KV, recurrent state, conv state, indexer keys): prefill stays identical to the cache-less computation, and the cache keeps its size. Released Qwen3.8 checkpoints load as *ForConditionalGeneration, which works as is. For adaptive decoding on hybrid models, use the default prev_entropy gate.

from thinkmore import loopify, default_config, find_decoder_layers

cfg = default_config(len(find_decoder_layers(model)))   # e.g. layers 28-31 of Qwen3.8-27B, K=3
handle = loopify(model, cfg)

🙏 Acknowledgements

ThinkMore builds on the damped-Euler looping operator introduced in Training-Free Looped Transformers (arXiv:2605.23872); ThinkMore reproduces its Qwen3-4B SciQ result exactly. On top of it, ThinkMore adds the cache-consistent implementation, adaptive entropy gating, the compute budget controller and batched row gating. Ideas from The Tell-Tale Norm (adaptive, norm-triggered looping) and loopgraft (looping with KV caches) informed the design.

📄 Citation

@software{thinkmore2026,
  title  = {ThinkMore: Training-Free Adaptive Layer Looping for Open-Source LLMs},
  year   = {2026},
  url    = {https://github.com/Zihao-Wu06/LLM-ThinkMore}
}

Please also cite Training-Free Looped Transformers (arXiv:2605.23872) if you use the looping operator.

📜 License

Apache 2.0

About

🧠🔁 Let your LLM think twice, inside the network. Training-free, adaptive layer looping for open-source LLMs (Qwen3.8 / Qwen4, Qwen3, Llama): one line of code, zero extra weights, zero extra KV memory.

Topics

Resources

Stars

1 star

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages