A unified library for diffusion / flow policy RL post-training, iterative offline data flywheels, and real-world robot RL.
Kun Lei*† ·
Huanyu Li* ·
Dongjie Yu* ·
Zhenyu Wei*
Lingxiao Guo ·
Zhennan Jiang ·
Ziyu Wang ·
Shiyu Liang ·
Huazhe Xu
* Equal contribution · † Project Lead
RL-100 is a real-world reinforcement learning framework for robotic manipulation built on top of diffusion and flow visuomotor policies. It targets deployment-level reliability, efficiency, and robustness through a unified imitation-to-reinforcement learning pipeline.
RL-100 aims to provide one of the most complete codebases for diffusion-policy RL and real-world robot RL post-training. The framework covers the following combinations in a unified repo.
The supported options below are designed to be freely combined across axes for different robot post-training setups.
All combinations are implemented through a compact offline RL / online RL training interface.
| Axis | Supported options |
|---|---|
| Policy backbone |
✓ Diffusion policy ✓ Flow policy |
| One-step deployment |
✓ Diffusion-to-CM one-step distillation ✓ Flow-to-flow one-step on-policy distillation |
| Observation modality |
✓ 3D point cloud ✓ 2D RGB image |
| Control mode |
✓ Action chunking ✓ Single-action high-frequency control |
| Policy extraction strategy |
✓ Policy gradient ✓ IDQL-style extraction with reject sampling ✓ Hybrid variants |
| RL stage |
✓ Offline policy gradient ✓ Online policy gradient |
| Training data regime |
✓ Teleoperation datasets ✓ Iterative offline rollout datasets ✓ Online rollouts ✓ Offline-to-online mixed training |
Starting from human teleoperation demonstrations, RL-100 iteratively refines policies with offline RL, deploys them to collect new real-robot rollouts, merges those rollouts back into the offline dataset, and optionally performs lightweight online RL fine-tuning.
- 0. Quick Install Pointer
- 1. Project Overview
- 2. RL Post-Training Framework
- 3. Real-Robot Training and Data Flywheel
- Installation
- Citation
- Acknowledgements
- License
For environment setup and dependency installation, see the Installation section and INSTALL.md. We recommend using adroit_door_medium as the first smoke-test task.
The recommended adroit_door_medium zarr dataset is hosted on Hugging Face Datasets: leokk/RL-100-adroit-door-medium. From the repository root, download it with the Hugging Face CLI:
python -m pip install -U huggingface_hub
mkdir -p RL-100/data
hf download leokk/RL-100-adroit-door-medium \
adroit_door_medium.zarr.tar.gz \
--repo-type dataset \
--local-dir /tmp
tar -xzf /tmp/adroit_door_medium.zarr.tar.gz -C RL-100/dataIf direct URL downloads are more convenient in your environment, use:
mkdir -p RL-100/data
wget -O /tmp/adroit_door_medium.zarr.tar.gz \
https://huggingface.co/datasets/leokk/RL-100-adroit-door-medium/resolve/main/adroit_door_medium.zarr.tar.gz
tar -xzf /tmp/adroit_door_medium.zarr.tar.gz -C RL-100/dataOptional checksum verification:
echo "73a7cd510a0715492e8f8041843ac51edf1a0feb6d30706a95a5e5857addef37 /tmp/adroit_door_medium.zarr.tar.gz" | sha256sum -c -After extraction, the dataset should be available at:
Repo/RL-100/data/adroit_door_medium.zarr
The dataset follows the RL-100 repository/data release terms. The formal dataset license will be finalized together with the public repository license.
RL-100 is designed as a full-stack real-world robot learning system, not only a collection of training scripts.
It combines
- Behavior cloning
- Offline policy-gradient RL
- Iterative offline rollout data expansion
- Online policy-gradient fine-tuning
- One-step policy distillation
in one codebase. The goal is to make real-world robot policy post-training a repeatable pipeline rather than a set of disconnected scripts.
RL-100/
├── rl_100/
│ ├── policy/ # 2D and 3D visuomotor policy implementations
│ ├── unidpg/ # offline / online RL training logic
│ └── config/ # policy, task, and training configs
├── scripts/
│ ├── Diffusion/ # diffusion policy training and online scripts
│ └── Flow/ # flow policy training and online scripts
├── tools/
│ └── teleop_off2off_data/ # teleoperation and iterative offline data flywheel tools
├── third_party/ # external environments and dependencies
└── visualizer/ # lightweight point-cloud visualizer
RL-100 turns the capability matrix above into a practical post-training pipeline for adapting large visuomotor policies from logged data and online interaction. This section focuses on the algorithmic stages and launchers; the real-robot iterative offline data flywheel is described in Section 3.
Use the interactive selector to generate the launcher command from a few choices:
python scripts/select_recipe.pyIt asks for:
| # | Choice |
|---|---|
| 1 | DDIM / Flow |
| 2 | Offline / Online |
| 3 | 3D / 2D |
| 4 | Chunk action / Single action |
| 5 | PG / PG + IDQL |
| 6 | One-step distill: Yes / No |
The first four choices select the launcher path. The PG/IDQL and distillation choices print launcher-level suggestions, because those settings are usually controlled inside the bash sweep. For example, the selector will remind you to use launcher flags for PG or PG + IDQL-style extraction; for offline distillation, first finish the offline sweep and parameter selection, then set distill_phase='after_offline'; for online distillation, set distill_phase='online'.
Example output
Command:
bash scripts/Flow/Online/3D/train_policy_online_flow_distill_online.sh rl100 adroit_door_medium 0112 100 8
The full algorithmic pipeline is organized around the following stages.
Train an initial visuomotor policy from demonstrations. This stage provides a stable policy prior before RL post-training.
Improve the initialized policy with offline policy-gradient updates on logged trajectories. This stage supports diffusion and flow policies, 2D and 3D observation backbones, chunk-action and single-action control, and optional IDQL-style extraction configured through the launcher sweep.
Example offline RL launchers:
# Single-GPU 3D diffusion offline RL.
bash scripts/Diffusion/Offline/3D/train_policy.sh rl100 adroit_door_medium 0112 100
# DDP two-stage 3D diffusion offline RL.
# The last argument is the number of GPUs.
bash scripts/Diffusion/Offline/3D/train_policy_two_stage.sh rl100 adroit_door_medium 0112 100 4The 2D and chunk-action offline recipes also provide DDP-style launchers, including:
scripts/Diffusion/Offline/2D/train_policy_image_unet_two_stage.shscripts/Diffusion/Offline/2D/train_policy_image_unet_chunk_two_stage.shscripts/Diffusion/Offline/3D/train_policy_chunk_two_stage.sh
Continue improving the policy using real or simulated online rollouts. The current rollout policy collects transitions, and PPO-style updates reuse the existing rollout and log-probability flow.
This stage supports:
- Online policy gradient.
- Diffusion and flow online fine-tuning.
- 2D and 3D online runners.
- Chunked and high-frequency single-action control.
- Lightweight online adaptation after offline RL.
For deployment efficiency, RL-100 includes one-step policy extraction and distillation paths:
- Diffusion-to-CM — distill a multi-step diffusion policy into a consistency-model style one-step policy.
- Flow-to-flow — distill or extract a one-step flow policy through on-policy training.
These paths are intended for fast real-robot inference while retaining the behavior improved by RL post-training.
RL-100 supports policy extraction strategies used in robot policy post-training:
- Policy-gradient extraction from logged or online transitions.
- IDQL-style extraction with reject sampling from higher-quality sampled trajectories.
- Hybrid strategies that combine policy-gradient updates with filtered or selected data.
The main implementation and configuration entry points are:
RL-100/rl_100/policy/rl100_3d.py
RL-100/rl_100/policy/rl100_2d.py
RL-100/rl_100/unidpg/
RL-100/rl_100/config/
scripts/Diffusion/
scripts/Flow/
Detailed command lines and recommended configs are organized around these entry points and will be expanded as the release scripts are finalized.
RL-100 includes a real-robot data flywheel for iterative offline reinforcement learning and final online improvement.
| Stage | Input | Output |
|---|---|---|
| Teleoperation | Human demonstrations | Raw robot episodes |
| Data preparation | Raw teleop data and rollout data | Training zarr dataset |
| Offline RL | Merged offline dataset | Improved policy checkpoint |
| Real-robot rollout | Offline RL checkpoint | New policy rollout dataset |
| Iterative offline dataset merge | Base zarr and new rollout h5 files | Next-round zarr dataset |
| Online RL | Offline RL checkpoint and live rollouts | Final deployment checkpoint |
human teleoperation
└─▶ raw teleop data
└─▶ processed offline dataset
└─▶ BC / offline RL training
└─▶ real-robot policy rollout
└─▶ rollout dataset
└─▶ merge with previous offline data
└─▶ next offline RL round
└─▶ online RL fine-tuning
This section tracks the hardware and runtime assumptions needed for real-robot deployment:
- Robot platform and controller requirements.
- Camera and point-cloud setup.
- End-effector / hand / gripper setup.
- Workspace limits and reset protocol.
- Safety checks before running learned policies.
Teleoperation and data processing utilities live in:
tools/teleop_off2off_data/
The planned workflow is:
cd tools/teleop_off2off_data
# Collect teleoperation data
python teleop.py
# Or use the swapped teleoperation entry if required by the robot setup
python teleop_swapped.pyHardware-specific command-line arguments should be documented next to the corresponding robot setup.
The data preparation entry point is:
cd tools/teleop_off2off_data
python data_prepare.py --config configs/data_prepare.yamlFor the full teleoperation and iterative offline dataset workflow, see tools/teleop_off2off_data/DATA_PREPARE.md.
The tool supports three modes:
| Mode | Purpose |
|---|---|
raw_to_npy | Convert raw teleoperation episodes into processed .npy data. |
build_zarr | Build a new training zarr from processed teleop data and selected rollout sources. |
extend_zarr | Append new rollout sources to a base zarr and write a new zarr without modifying the base dataset in place. |
Iterative offline training is the main data flywheel used by the real-robot pipeline. A policy trained by offline RL is rolled out on the robot, the resulting trajectories are stored as rollout datasets, and the next offline RL round trains on the merged dataset.
Policy rollouts collected on the real robot are treated as named data sources in YAML:
rollout_sources:
- name: off2off_004
round: "004"
enabled: true
type: h5_rollout_dir
path: /path/to/policy_rollouts/004/online_ftTypical usage:
# Build a dataset from teleop data plus all enabled rollout sources.
python data_prepare.py --config configs/data_prepare.yaml --mode build_zarr --include-rollouts
# Extend an existing zarr with one new rollout source.
python data_prepare.py \
--config configs/data_prepare.yaml \
--mode extend_zarr \
--rollout-source off2off_004 \
--base-zarr-path /path/to/base.zarr \
--zarr-output-path /path/to/new.zarrThe merge output preserves the training schema expected by RL-100:
data/
├── point_cloud, next_point_cloud
├── state, next_state
├── action, next_action
└── reward, return, done, timeout
meta/
└── episode_ends
After generating a zarr dataset, point the RL-100 task config to the dataset path and run the selected offline RL script.
This section is reserved for the concrete real-robot training recipes:
- Real-robot task config examples.
- 3D point-cloud offline training commands.
- 2D image offline training commands.
- Recommended checkpoint selection and evaluation workflow.
Iterative offline rollout collection reuses the online training launcher as a real-robot data collection wrapper. In practice, load the offline-RL-finetuned policy checkpoint, enter the online script, and use the evaluation call before or during online fine-tuning to roll out the policy on the robot and save .h5 trajectories.
The real-robot offline and online training bash scripts are the same launchers introduced in Section 2. The only extra setting for iterative offline rollout collection is enabling data_collect=True in the collection run.
The offline and online launchers must describe the same policy network so that checkpoint weights load cleanly. Keep architecture-related options consistent between the offline training bash and the rollout-collection bash, including the policy family, observation modality, chunk/single-action setting, policy.model, policy.encoder_type, policy.use_vib, policy.use_recon, horizon, n_action_steps, n_obs_steps, encoder output size, and diffusion/flow scheduler settings. If the online bash changes these fields, the offline checkpoint may partially fail to load or silently run with a mismatched policy.
For RL data collection, the rollout policy should be stochastic rather than a deterministic evaluation policy. Set data_collect=True in the bash/config path used for collection. This enables two noise sources:
- 🔴 Denoising/SDE action noise: the real-robot runner switches
deterministic=Falsewhendata_collect=True, sopolicy.predict_action(...)samples stochastic denoising transitions. - 🔴 VIB latent noise: with
policy.use_vib=Trueandppo.force_stochastic_online=True,train_real.pykeeps the observation encoder stochastic duringeval(..., data_collect=True).
Typical real-robot collection settings:
data_collect=True
policy.use_vib=True
ppo.force_stochastic_online=TrueUse offline_cp_timestamp / offline_cp_timestep or the corresponding checkpoint-loading fields in the online launcher to point to the selected offline RL checkpoint. The generated rollout .h5 files should then be added as a new rollout_sources entry in tools/teleop_off2off_data/configs/data_prepare.yaml.
This section covers the final online RL stage:
- Set
data_collect=Falsebefore starting actual online RL fine-tuning.data_collect=Trueis only for iterative offline rollout dataset collection; leaving it enabled during online fine-tuning will make periodic eval calls run in collection mode and write rollout.h5files. - Start from a BC or offline RL checkpoint.
- Collect short online rollouts on the real robot.
- Run online policy-gradient updates.
- Evaluate and save deployment checkpoints.
⚠ Before running policies on hardware:
- Verify workspace bounds and action scaling.
- Test controller latency and emergency stop.
- Start with low speed and conservative action limits.
- Run short supervised rollouts before autonomous long runs.
- Do not deploy a checkpoint on hardware before validating observation normalization, action dimensions, and reset behavior.
See INSTALL.md for the full environment setup. We recommend using adroit_door_medium as the first smoke-test task.
The verified server setup uses:
| Component | Version |
|---|---|
| NVIDIA driver | 550.54.15 |
| System CUDA | 12.4 |
| Python | 3.8.20 |
| PyTorch | 2.4.0+cu121 |
Minimal workflow:
conda create -n rl100 --clone dp3 -y
conda activate rl100
export REPO_ROOT=$(pwd)
export PYTHONPATH=${REPO_ROOT}/RL-100:${PYTHONPATH}
export LD_LIBRARY_PATH=${HOME}/.mujoco/mujoco210/bin:/usr/lib/nvidia:/usr/local/cuda/lib64:${LD_LIBRARY_PATH}
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
python -m pip install -e third_party/dexart-release
python -m pip install -e third_party/gym-0.21.0
python -m pip install -e third_party/Metaworld
python -m pip install -e third_party/rrl-dependencies/mj_envs/.
python -m pip install -e third_party/rrl-dependencies/mjrl/.
python -m pip install -e third_party/mujoco-py-2.1.2.14
python -m pip install -e third_party/pytorch3d_simplified
python -m pip install -e visualizerSanity check:
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"
python -c "import rl_100, zarr, hydra, einops, metaworld, mujoco_py, open3d as o3d; print('imports ok', o3d.__version__)"Verified flow online distillation entry:
./scripts/Flow/Online/3D/train_policy_online_flow_distill_online.sh rl100 adroit_door_medium 0112 100 8If you find this work useful, please cite:
@article{rl100,
title = {RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning},
author = {Lei, Kun and Li, Huanyu and Yu, Dongjie and Wei, Zhenyu and Guo, Lingxiao and Jiang, Zhennan and Wang, Ziyu and Liang, Shiyu and Xu, Huazhe},
journal = {arXiv preprint arXiv:2510.14830},
year = {2025}
}Also consider citing the related RL post-training methods used by this project
@inproceedings{lei2024unio,
title = {Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization},
author = {Kun LEI and Zhengmao He and Chenhao Lu and Kaizhe Hu and Yang Gao and Huazhe Xu},
booktitle = {The Twelfth International Conference on Learning Representations},
year = {2024},
url = {https://openreview.net/forum?id=tbFBh3LMKi}
}
@inproceedings{zhuang2023behavior,
title = {Behavior Proximal Policy Optimization},
author = {Zifeng Zhuang and Kun LEI and Jinxin Liu and Donglin Wang and Yilang Guo},
booktitle = {The Eleventh International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=3c13LptpIph}
}RL-100 builds on prior work in Uni-O4, BPPO, DP3, Diffusion Policy, robotic manipulation environments, and reinforcement learning infrastructure. Detailed third-party acknowledgements and license notes will be maintained with the corresponding code and dependencies.
This project is released under the Apache License 2.0.