Skip to content

Commit 7871100

Browse files
committed
fix: move Unity lab into PPO chapter
1 parent a1c2803 commit 7871100

4 files changed

Lines changed: 4 additions & 8 deletions

File tree

docs/chapter10_ppo/ppo-game-benchmark.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ Mario 是经典项目,但**奖励很容易写歪**。只奖励向右走,策
195195

196196
SoccerTwos 是 2v2 足球,引入了一个全新维度:**多智能体**。PPO 在这里需要和 **self-play** 配合使用——智能体与历史版本的自己对战,通过 **ELO 评分**追踪进步。**Unity ML-Agents 内置 PPO** 是最直接的实现路径,batch_size **2048**、buffer_size **20480**、学习率 **$3 \times 10^{-4}$**、熵系数 **0.005**
197197

198+
<OnlineTraining studios="unity" compact />
199+
198200
训练过程能看到明显的**策略演化**。初期所有智能体追着球跑,场面混乱;ELO 从初始 **1200** 慢慢爬到约 **1600** 后,开始出现简单的**前锋-后卫角色分化****SAC** 在这个环境上表现差很多,基本停留在 **1200-1250**,说明 **PPO 的稳定性对多智能体 competitive 场景很重要**
199201

200202
**课程学习**的效果很显著。设计良好的课程(逐步调整初始位置、球速)能让 PPO 在 **250k iterations** 内平均奖励超过 **1.8**,节省约 **40%** 训练时间;但设计不好的课程反而降低性能(**0.35 vs 0.45**),这提醒我们**课程设计本身需要调优**

docs/en/chapter10_ppo/ppo-game-benchmark.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,8 @@ PPO often fails on unfamiliar terrain hazards rather than because the low-level
195195

196196
SoccerTwos is 2v2 soccer, so it introduces **multi-agent learning**. PPO is usually paired with **self-play**: agents play against historical versions of themselves, and progress is tracked with ELO. Unity ML-Agents provides the most direct PPO implementation path.
197197

198+
<OnlineTraining studios="unity" compact />
199+
198200
The training process often shows strategy evolution. Early agents chase the ball chaotically. As ELO rises from **1200** toward **1600**, simple attacker-defender role separation can appear. SAC tends to perform much worse in this setting, which highlights PPO's stability in competitive multi-agent training.
199201

200202
Curriculum learning matters. Gradually changing start positions and ball speed can reduce training time, but a poorly designed curriculum can hurt final performance. Multi-agent reward design is also delicate: rewarding only goals is sparse; over-rewarding distance to the ball makes every agent crowd the same location.

docs/en/preface/env-setup.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,6 @@ pip install numpy scipy matplotlib seaborn pandas
135135
pip install tqdm tensorboard wandb
136136
```
137137

138-
<OnlineTraining studios="gymnasium" compact />
139-
140138
## Simulator Environments
141139

142140
Different chapters use different simulators. Install them as needed.
@@ -211,8 +209,6 @@ Using ML-Agents also requires downloading or building Unity environments (`.exe`
211209
The unique value of ML-Agents is **3D spatial reasoning**: Atari uses 2D pixels, CartPole uses low-dimensional vectors, while ML-Agents provides a complete 3D physics environment (gravity, collisions, occlusion). If your research involves visual navigation, spatial reasoning, or multi-agent 3D coordination, ML-Agents is a strong complement to Gymnasium/PyBullet.
212210
:::
213211

214-
<OnlineTraining studios="unity" compact />
215-
216212
## LLM Training Frameworks
217213

218214
The second half of the course (Chapters 7-10) involves large model alignment training.

docs/preface/env-setup.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -137,8 +137,6 @@ pip install numpy scipy matplotlib seaborn pandas
137137
pip install tqdm tensorboard wandb
138138
```
139139

140-
<OnlineTraining studios="gymnasium" compact />
141-
142140
## 仿真环境安装
143141

144142
不同章节用到不同的仿真环境,可以按需安装。
@@ -213,8 +211,6 @@ python -c "from mlagents_envs.environment import UnityEnvironment; print('ML-Age
213211
ML-Agents 的独特价值在于**3D 空间推理**:Atari 是 2D 像素,CartPole 是低维向量,而 ML-Agents 提供完整的 3D 物理环境(重力、碰撞、遮挡)。如果你的研究涉及视觉导航、空间推理或多智能体 3D 协作,ML-Agents 是 Gymnasium/PyBullet 之外的有力补充。
214212
:::
215213

216-
<OnlineTraining studios="unity" compact />
217-
218214
## LLM 训练框架
219215

220216
课程后半部分(Ch7-Ch10)涉及大模型对齐训练。

0 commit comments

Comments
 (0)