Add Hierarchical Reasoning Model #41271
Description
Model description
The Hierarchical Reasoning Model (HRM) is showing promise in reasoning tasks. I have been working on cleaning up the original codebase and decided I would also integrate it into the Transformers library so a larger audience could benefit from this project.
Below is a snippet from the team's paper which best describes the model.
... Inspired by the hierarchical and multi-timescale processing in the human brain, we propose the Hierarchical Reasoning Model (HRM), a novel recurrent architecture that attains significant computational depth while maintaining both training stability and efficiency.
HRM executes sequential reasoning tasks in a single forward pass without explicit supervision of the intermediate process, through two interdependent recurrent modules: a high-level module responsible for slow, abstract planning, and a low-level module handling rapid, detailed computations.
With only 27 million parameters, HRM achieves exceptional performance on complex reasoning tasks using only 1000 training samples. The model operates without pre-training or CoT data, yet achieves nearly perfect performance on challenging tasks including complex Sudoku puzzles and optimal path finding in large mazes.
Furthermore, HRM outperforms much larger models with significantly longer context windows on the Abstraction and Reasoning Corpus (ARC), a key benchmark for measuring artificial general intelligence capabilities. These results underscore HRM’s potential as a transformative advancement toward universal computation and general-purpose reasoning systems.
I have a PR ready to close out this issue.
Open source status
- The model implementation is available
- The model weights are available
Provide useful links for the implementation
Authors
- @imoneoi
- @btoo
- @raincchio
- @liamnorm
- @WC-william
- @zbloss - not an original author but I've implemented a refactor to the original codebase.
Weights
- sapientinc/HRM-checkpoint-ARC-2
- sapientinc/HRM-checkpoint-sudoku-extreme
- sapientinc/HRM-checkpoint-maze-30x30-hard
Original Implementation
Relevant Incoming Changes
- I've got a PR currently open in their repo that improves readability and organizes the code. This may be helpful to read through if it has not been merged into main yet: feat: v0.1.0 Refactor & Packaging sapientinc/HRM#89