docs(chapter22): refine agentic rl intro prose

Author: sanbuphy

AGENTS.md

@@ -0,0 +1,75 @@
+# Repository Guidelines
+
+## Project Structure
+
+- `docs/`: VitePress course site content.
+- `docs/.vitepress/`: site configuration and theme overrides.
+- `docs/public/`: static assets copied to the final site.
+- `docs/en/`: English translation stubs (mirror of Chinese structure).
+- `code/`: runnable Python experiments aligned with chapters.
+- `scripts/`: repo maintenance scripts such as sitemap, verification, and asset optimization.
+- `.github/workflows/`: deployment workflow for GitHub Pages.
+
+## Commands
+
+```bash
+npm install
+npm run dev            # dev server with asset optimization
+npm run dev:fast       # dev server skipping asset optimization (~5s startup)
+npm run build
+npm run preview
+npm run verify
+npm run assets:optimize
+```
+
+## Editing Rules
+
+- Keep changes scoped. Do not rewrite unrelated docs or config.
+- Prefer adding new course material under `docs/` and updating sidebar/nav in `docs/.vitepress/config.mjs`.
+- Run `npm run verify` before pushing if you touched config, theme, or build scripts.
+- When adding/moving/renaming pages, update both Chinese and English sidebars if applicable.
+
+## TikZ → SVG Workflow
+
+For academic-style diagrams with proper LaTeX math rendering:
+
+1. Write the diagram as a `standalone` TikZ document (see `docs/preface/images/actor-critic.tex` for style reference)
+2. Compile: `xelatex <file>.tex && pdf2svg <file>.pdf <file>.svg`
+3. Keep both `.tex` (source) and `.svg` (output) in the same `images/` directory
+4. Reference the `.svg` from markdown with `<img src="./images/<name>.svg">`
+
+Style: rounded-corner boxes (`draw=black!70, fill=black!3`), Stealth arrows, Songti SC font, `$...$` for math.
+
+## Writing Conventions
+
+- Write docs as tutorial and lecture material, not as terse reference notes.
+- **Direct and concise**: No guiding filler like "用...做解释会更直观". Present the example or formula directly.
+- **No redundancy**: Avoid saying the same thing twice in different words.
+- **No vague qualifiers**: Replace metaphors ("相当于...") and hedging ("由于...") with precise causal statements.
+- **Follow the causal line**: Do not circle around the same point with "first summarize, then restate, then contrast". Write in the order the idea actually changes: new capability or setting appears → it changes the training/problem object → this enables or forces a new formulation.
+- When opening a new concept, prefer the pattern "有了新的 X，系统会 Y，于是可以/必须 Z". Avoid unnecessary callbacks to previous chapters unless they are needed for the next sentence.
+- Avoid slogan-like summary phrases such as "一问一答一打分", "骨架始终未变", "系统地扩展", or "完全不涉及". State the concrete mechanism instead.
+- Avoid AI-like error lists. Instead of listing many parallel mistakes ("query 太宽、没比价、库存错、参数错"), describe the process and show how an early wrong state propagates into later actions.
+- Prefer Chinese terms in Chinese prose. Use "一段回答" instead of "completion", "奖励" instead of "reward", "动作掩码" instead of "action mask", "逐步记录/逐步优势" instead of "step-level record/advantage". Keep English only when it is a standard acronym or the local chapter title requires it, and explain it in Chinese.
+- Match the narrative style of Chapter 3.1 and 3.2: conversational, patient, and problem-driven.
+- Open sections by connecting to what the learner already knows, then name the new problem the section solves.
+- Use concrete scenes before abstractions: examples like CartPole, multi-armed bandits, LLM generation, game tasks, and everyday choices should make the concept feel necessary.
+- Prefer a guided lecture voice. Use phrases like "Let's start with the intuition", "Look at it from another angle", "What this step means is…", "The real issue is…", while keeping the tone serious and precise.
+- Maintain textbook prose for course material. Avoid chatty, assistant-like, slogan-like, or casual internet phrasing; write as a polished Chinese textbook chapter.
+- Ensure every conceptual step has a textbook-style transition. Do not jump directly from a definition to a formula, term, algorithm, or conclusion; explain why the next idea is needed before introducing it.
+- Avoid encyclopedia-style exposition that only lists definitions. Definitions should appear after motivation and should be followed by interpretation.
+- Keep formulas close to their meaning: explain every important symbol, then restate the formula in plain English.
+- Avoid the "不是……而是……" / "not A, but B" contrast pattern. It makes the prose feel indirect. State the positive claim directly, then explain the reason.
+- Let paragraphs flow from question to answer. Good sections often follow this rhythm: familiar example → confusion or tension → formal tool → worked example → takeaway.
+- Use bold sparingly to mark conceptual pivots, not decoration.
+- Explain ideas progressively: intuition first, then formal definitions, then code or practice.
+- Prefer learner-facing prose in Chinese, with concrete examples, small derivations, and explicit transitions.
+- When introducing formulas or algorithms, state what problem they solve before showing the notation.
+- Keep paragraphs teachable and paced for reading aloud or self-study.
+- Use short sections and explicit headings.
+- Use kebab-case for paths under `docs/`.
+- Prefer directory-based routes with `index.md`.
+
+## Commit Style
+
+- Use Conventional Commits such as `feat:`, `fix:`, `docs:`, `chore:`.

docs/chapter22_agentic/intro.md

@@ -29,13 +29,47 @@ $$
 
 **本章公式的作用**
 
-第 22 章把强化学习的训练对象从一段回答扩展到一条完整交互轨迹。轨迹 $\tau$ 定义了多轮交互的数据形态；POMDP 六元组刻画了"模型只能看到部分状态"这一 Agentic 场景的本质；轨迹期望 $J(\theta)$ 把单轮的 $\mathbb{E}_{a \sim \pi_\theta}[r(a)]$ 推广到多步累积奖励；step-level advantage $A_t$ 则是信用分配的核心工具，决定最终 reward 如何回拆到每一步。后续小节中的 ORM/PRM、turn-level discounting、group-based advantage 都建立在这组对象之上。
+模型一旦接入工具，就不再只是写完一段回答。它会先行动，看到环境返回，再决定下一步。强化学习要训练的对象也随之改变：**从生成一段回答，变成完成一条多步交互过程**。
 
-第 21 章讨论的 Constitutional AI 与 RLAIF 解决的是"标注信号从哪来"——让 AI 当裁判替代人类标注。但无论是人类标注还是 AI 标注，前面所有章节的 RL 训练都共享同一个隐含假设：**模型接收一个 prompt，输出一段完整回答，奖励模型给出一个分数，策略据此更新一次**。"一问一答一打分"的骨架始终未变。
+订机票 Agent 就是这样的任务。用户说"帮我订一张明天北京到上海最便宜的早班机票"，模型不能直接写一句"已为你订好"。它要先把航班搜出来，再按时间和价格筛选，确认目标航班还有余票，最后把正确的乘机人、航班号和舱位信息传给下单接口。前面某一步如果带入了错误信息，后面的动作就会沿着错误状态继续执行，直到出票失败。
 
-但真实的智能体不这样工作。考虑一个订机票 Agent：用户说"帮我订一张明天北京到上海最便宜的早班机票"，Agent 必须分步行动——先搜索航班、对比价格和时间、确认座位库存、调用下单 API、等待出票确认。中间任何一步出错（搜索 query 太宽、没比价直接选第一条、库存判断失误、下单参数错误），整个任务就失败。环境只在最后给出一个二元信号：出票成功或失败。
+因此，训练时首先要保存的不是最终回答，而是完整过程：模型在什么状态下采取了什么动作，环境又返回了什么观测。这个过程称为**轨迹**，记作：
 
-这种从"一问一答"到"多步与环境交互"的转变，正是 Agentic RL 要解决的核心问题。本章把前面建立的 RL 工具——MDP、策略梯度、GRPO、可验证奖励——系统地扩展到多轮交互场景，并补充单轮 RL 完全不涉及的工程议题：环境异步、沙箱管理、异构轨迹、长时程信用分配。
+$$
+\tau = (s_0, a_0, o_1, a_1, o_2, \ldots, a_T)
+$$
+
+这里的 $s_t$ 表示第 $t$ 步前的状态，$a_t$ 表示模型采取的动作，$o_{t+1}$ 表示环境随后返回的观测。它把对话、工具调用和环境返回统一放进同一条序列里。后面讨论轨迹概率、动作掩码和逐步记录，都建立在这个表示之上。
+
+轨迹记录了发生过什么，但它还没有说明模型能看到什么。订机票时，模型能读到搜索结果页的航班列表，却读不到航空公司的完整数据库；它能看到工具返回的库存提示，却不知道价格下一分钟会不会变化。真实环境有完整状态，模型只能看到其中一部分。这个设定用 **POMDP** 表示：
+
+$$
+\langle S_{\text{agent}},\ A_{\text{agent}},\ P_{\text{agent}},\ R_{\text{agent}},\ \gamma,\ O \rangle
+$$
+
+其中 $S_{\text{agent}}$ 是环境和上下文共同构成的状态空间，$A_{\text{agent}}$ 是模型可采取的文本动作或工具动作，$P_{\text{agent}}$ 描述动作如何改变环境和后续观测，$R_{\text{agent}}$ 给出任务奖励，$O$ 表示模型实际能看到的观测。POMDP 在这里强调一件事：**策略不是在完整世界状态上决策，而是在有限观测和历史上下文上决策**。
+
+有了轨迹和观测设定，优化目标也要跟着改变。单轮 RL 最大化的是一段输出的期望奖励：
+
+$$\mathbb{E}_{a \sim \pi_\theta}[r(a)]$$
+
+Agentic RL 最大化的是整条轨迹的累积回报。模型在第 $0$ 步搜什么，会影响第 $1$ 步看到哪些航班；第 $1$ 步选哪趟航班，又会影响第 $2$ 步需要检查什么库存。目标函数因此写成：
+
+$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^{T} \gamma^t R(s_t, a_t)\right]$$
+
+这里的 $\tau \sim \pi_\theta$ 表示轨迹由当前策略与环境交互采样得到，$\gamma^t$ 用来折扣远期奖励，$R(s_t,a_t)$ 表示第 $t$ 步动作带来的奖励或最终奖励回传后的贡献。这个公式把训练目标从"让回答得高分"改成了**让交互过程的总回报更高**。
+
+轨迹回报解决了"优化什么"，还没有解决"每一步怎么学"。订票成功，不表示每个动作都值得强化；模型可能前面绕了远路，最后才补救回来。订票失败，也不表示每个动作都错；前几步可能已经找到正确航班，只是在最后下单时填错了参数。最终奖励必须拆回每一步，训练才知道该强化哪个动作、修正哪个动作。
+
+这就是**逐步优势**要解决的问题：
+
+$$
+A_t = R(\tau) - \bar{R}(s_t)
+$$
+
+它把整条轨迹的结果 $R(\tau)$ 和同一状态附近的基准回报 $\bar{R}(s_t)$ 作比较，判断第 $t$ 步之后的表现是否高于通常水平。后续小节里的结果奖励、过程奖励、轮次折扣、组内优势，都是在构造更可靠的 $A_t$，让最终奖励能更准确地影响每一步动作。
+
+本章沿用前面已经建立的 MDP、策略梯度、GRPO 和可验证奖励，把它们放进多轮交互任务里重新看。到了 Agent 场景，训练不只要更新策略，还要处理环境异步、沙箱管理、轨迹长度不一致和长时程信用分配等工程问题。
 
 ## 章节安排