Learning to predict, reason, and plan in latent space.
A curated reading map for latent world models, latent reasoning, JEPA-style predictive architectures, and latent planning/control. The focus is work where latent representations are used as the substrate for prediction, imagination, planning, control, action-conditioned modeling, or reasoning. Generic embedding papers are intentionally out of scope unless the latent space itself does useful dynamics, planning, or decision-making work.
Last curated: 2026-06-13.
- Start Here
- Latent Meme Corner
- Surveys and Position Papers
- Yann LeCun / JEPA Series
- Classical Latent World Models
- Latent Planning and Control
- Latent Reasoning in Language Models
- Latent Reasoning in VLMs
- Latent Action Models
- Video World Models
- Embodied AI and Robotics
- Evaluation, Interpretability, and Benchmarks
- Open-Source Codebases
- Related Awesome Lists
- Contributing
- License
- Why latent world models? Read LeCun's position paper, Ha and Schmidhuber's World Models, and PlaNet.
- How do agents plan in latent space? Follow Dreamer, TD-MPC, TD-MPC2, MuZero, and EfficientZero.
- Why JEPA? Read I-JEPA, V-JEPA, V-JEPA 2, V-JEPA 2.1, and LeWorldModel.
- What is new in latent reasoning? Compare COCONUT, recurrent-depth latent reasoning, concept-space models, and visual latent reasoning in VLMs.
- Where does embodied control enter? Study latent action policies, LAPA/LAWM, V-JEPA 2-AC, WAMs, and robotics world models.
The model said it was thinking, but all we found was a beautifully compressed hidden state.
- A Path Towards Autonomous Machine Intelligence — LeCun's position paper frames JEPA-style predictive world models as a route from perception to planning and common-sense reasoning.
- Foundation Models for Decision Making: Problems, Methods, and Opportunities — Places world models, planning, and control inside the broader foundation-model-for-agents agenda.
- A Tutorial on World Models and Physical AI — A recent physical-AI tutorial that distinguishes explicit rollout models from implicit predictive representations.
- Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture — Introduces I-JEPA, predicting target block representations from context blocks without pixel reconstruction.
- Learning and Leveraging World Models in Visual Representation Learning — Extends JEPA-style prediction beyond masking by learning latent effects of image transformations.
- Revisiting Feature Prediction for Learning Visual Representations from Video — Introduces V-JEPA and shows that feature prediction from video can learn strong motion-aware representations.
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning — Scales video JEPA and includes V-JEPA 2-AC, a latent action-conditioned world model for robot planning.
- V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning — Improves dense spatial-temporal features while retaining video understanding and robotics transfer.
- LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — Trains a compact JEPA world model end-to-end from pixels for fast latent planning and surprise detection.
- A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures — Provides small, reproducible JEPA examples that connect representation learning, temporal prediction, and action-conditioned world models.
- Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images — An early control-from-pixels model that constrains dynamics to be locally linear in latent space.
- World Models — Compresses observations with a VAE, models temporal dynamics with an RNN, and trains compact policies inside imagined rollouts.
- Recurrent World Models Facilitate Policy Evolution — Shows that compact policies can exploit recurrent latent world models trained separately from controllers.
- Learning Latent Dynamics for Planning from Pixels — PlaNet learns stochastic latent dynamics from images and uses online planning directly in latent space.
- Dream to Control: Learning Behaviors by Latent Imagination — Dreamer learns behavior by backpropagating value gradients through imagined latent trajectories.
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model — SLAC separates representation learning from task learning by performing RL over learned stochastic latent states.
- Mastering Atari with Discrete World Models — DreamerV2 uses discrete latent world states to train policies by imagination at Atari scale.
- Mastering Diverse Domains through World Models — DreamerV3 demonstrates a robust general world-model agent across control, Atari, Minecraft, and more.
- Temporal Difference Learning for Model Predictive Control — TD-MPC learns a task-oriented latent dynamics model and plans with value-guided trajectory optimization.
- TD-MPC2: Scalable, Robust World Models for Continuous Control — Scales decoder-free latent world models to many tasks, embodiments, and action spaces.
- Hierarchical Planning with Latent World Models — Learns latent world models at multiple temporal scales so long-horizon embodied planning can use coarser abstractions before fine control.
- Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model — MuZero learns hidden dynamics for search without reconstructing observations or knowing environment rules.
- Learning and Planning in Complex Action Spaces — Sampled MuZero extends latent planning to large and continuous action spaces by sampling candidate actions.
- Mastering Atari Games with Limited Data — EfficientZero improves MuZero-style latent planning with better sample efficiency in visual RL.
- EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data — Extends EfficientZero-style model-based planning across visual, low-dimensional, discrete, and continuous domains.
- Training Large Language Models to Reason in a Continuous Latent Space — COCONUT replaces some verbal chain-of-thought tokens with continuous hidden-state reasoning steps.
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Studies recurrent-depth language models that spend extra inference compute by iterating hidden states rather than emitting more tokens.
- Large Concept Models: Language Modeling in a Sentence Representation Space — Models sequences at a higher-level concept embedding scale instead of only next-token space.
- Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space — Allocates computation between tokens and compressed concept latents through learned semantic boundaries.
- Spatiotemporal Hidden-State Dynamics as a Signature of Internal Reasoning in Large Language Models — Proposes hidden-state transition statistics for probing when reasoning-like computation is happening internally.
- Latent Chain-of-Thought for Visual Reasoning — Treats LVLM reasoning as posterior inference and learns latent visual CoT with amortized variational inference rather than only supervised textual rationales.
- Multimodal Latent Reasoning via Hierarchical Visual Cues Injection — HIVE injects global-to-regional visual cues into recurrent latent reasoning loops for grounded slow thinking in MLLMs.
- Multimodal Latent Reasoning via Predictive Embeddings — Pearl uses JEPA-inspired predictive embedding alignment to internalize multimodal tool-use trajectories in latent space.
- Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model — SCOLAR addresses information-gain collapse in long visual latent CoT by anchoring auxiliary visual tokens to the original visual space.
- ReGuLaR: Relation-Grounded Latent Reasoning for Large Vision-Language Models — Grounds latent reasoning states in question-relevant objects and relations so LVLMs can reason beyond unstructured continuous traces.
- Learning Latent Plans from Play — Play-LMP learns a latent plan space from unlabeled play data and reuses those plans for goal-conditioned control.
- Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills — Learns goal-reaching skills from offline robot data by turning a functional world understanding into action.
- Learning to Act without Actions — LAPO recovers latent actions from videos so policies and world models can be pretrained without labeled actions.
- Latent Action Pretraining from Videos — LAPA quantizes frame-to-frame changes into latent actions for VLA pretraining from web and robot videos.
- Latent Action Pretraining Through World Modeling — LAWM learns latent actions through world modeling to reduce dependence on teleoperation labels.
- OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation — Introduces slot-addressable world-action latents so robot policies can bind instructions to persistent objects.
- Light-WAM: Efficient World Action Models with State-Fusion Action Decoding — Makes WAM-style future-video supervision cheaper by decoding actions from compact fused latent states.
- RepWAM: World Action Modeling with Representation Visual-Action Tokenizers — Learns semantic visual-action tokens for instruction-following robot world-action modeling.
- Learning Interactive Real-World Simulators — UniSim combines heterogeneous interaction datasets to learn controllable real-world visual simulation.
- Genie: Generative Interactive Environments — Learns action-controllable environments from unlabeled videos using a tokenizer, latent action model, and dynamics model.
- Learning Interactive Real-Robot Action Simulators — IRASim generates realistic robot-action videos from initial frames and action trajectories for scalable robot learning.
- Diffusion Models Are Real-Time Game Engines — GameNGen shows a neural video model can act as an interactive game simulator conditioned on actions.
- Learning Generative Interactive Environments By Trained Agent Exploration — GenieRedux studies open implementations of Genie-style interactive environments using trained-agent exploration data.
- Exploration-Driven Generative Interactive Environments — Uses world-model uncertainty to collect better interaction data for Genie-like simulators.
- XIRL: Cross-embodiment Inverse Reinforcement Learning — Learns task-progress embeddings from cross-embodiment videos and uses latent distances as rewards.
- RoboDreamer: Learning Compositional World Models for Robot Imagination — Factorizes language-conditioned video imagination to improve compositional robot planning.
- ManipDreamer: Boosting Robotic Manipulation World Model with Action Tree and Visual Guidance — Adds action-tree structure and visual guidance to robotic manipulation world-model videos.
- VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model — Brings leakage-free JEPA-style future state prediction to VLA pretraining and robot manipulation.
- PlayWorld: Learning Robot World Models from Autonomous Play — Trains robot video world simulators from unsupervised self-play and uses them for policy evaluation and RL.
- Grounded World Model for Semantically Generalizable Planning — Scores imagined visuomotor futures in a vision-language latent space rather than against fixed goal images.
- Toward Safe Autonomous Robotic Endovascular Interventions using World Models — Applies TD-MPC2-style world models to safety-critical robotic navigation under visual feedback.
- PHYRE: A New Benchmark for Physical Reasoning — Tests whether agents can solve mechanics puzzles that require compact physical models and intervention planning.
- CLEVRER: CoLlision Events for Video REpresentation and Reasoning — Evaluates causal, predictive, and counterfactual video reasoning over simple physical scenes.
- IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments — Uses violation-of-expectation videos to test intuitive physics in richer scenes.
- A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs — MVPBench pairs minimally changed videos to expose shortcut-based physical reasoning.
- PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models — Tests whether VLMs and video world models obey mechanics principles such as center of mass, lever equilibrium, and Newtonian inertia.
- WISER Benchmark — Introduced with Grounded World Models to test semantically generalizable visuomotor planning in latent language-vision space.
- facebookresearch/ijepa — Official I-JEPA implementation for image-based joint-embedding predictive learning.
- facebookresearch/jepa — Official Meta JEPA codebase for video feature prediction experiments.
- facebookresearch/eb_jepa — Lightweight educational JEPA library covering image, video, and action-conditioned examples.
- danijar/dreamerv3 — Reference DreamerV3 implementation for world-model reinforcement learning across domains.
- nicklashansen/tdmpc2 — Official TD-MPC2 implementation for scalable latent model-predictive control.
- YeWR/EfficientZero — Official EfficientZero code for sample-efficient MuZero-style visual planning.
- werner-duvaud/muzero-general — Popular open-source MuZero implementation for studying learned hidden dynamics and MCTS.
- insait-institute/GenieRedux — Open implementation and dataset work for Genie-style generative interactive environments.
- wdrink/RepWAM — Code and weights location announced for representation-centric world-action modeling.
- Awesome World Models — A broad world-model collection; use this repo as the narrower latent-reasoning and latent-control companion.
- Awesome Robotics Foundation Models — A robotics foundation-model reading list useful for embodied and VLA context.
Contributions are welcome when they keep the latent-world-model focus sharp. Please read CONTRIBUTING.md before opening a PR or use the paper suggestion issue template for candidates that need discussion.
Short version: explain why the latent representation matters, prefer primary links, avoid dumping generic embedding papers, and add one useful sentence per entry.
This list is released under CC0-1.0. Public-domain curation is the easiest latent space to share.