Skip to content

Commit 3dfa415

Browse files
kajalj22claude
kajalj22
and
claude
authored
docs(fern): add v0.3.0 version snapshot for GA release (#1521)
## Summary - Snapshot the bleeding-edge docs tree (`fern/versions/latest/`) into `fern/versions/v0.3.0/` as the frozen GA copy - Create `v0.3.0.yml` nav tree (copied from `main.yml` with path prefixes rewritten to `./v0.3.0/`) - Retarget the `latest.yml` symlink from `v0.2.1` → `v0.3.0` - Register the new `0.3.0` version in `fern/docs.yml` with `availability: stable` After merge, docs will auto-publish to `docs.nvidia.com/nemo/gym` via the `publish-fern-docs` workflow. ## Test plan - [ ] Verify `fern check` passes in CI (`fern-docs-ci` workflow) - [x] Verify Fern preview link renders correctly with the new v0.3.0 version in the dropdown - [ ] Confirm `latest.yml` symlink resolves to `v0.3.0.yml` - [ ] Confirm `main.yml` still points to `./latest/pages/` (bleeding-edge unchanged) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 9f59272 commit 3dfa415

74 files changed

Lines changed: 9986 additions & 1 deletion

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎fern/docs.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ versions:
3030
- display-name: Main
3131
path: versions/main.yml
3232
slug: main
33+
- display-name: "0.3.0"
34+
path: versions/v0.3.0.yml
35+
slug: v0.3.0
36+
availability: stable
3337
- display-name: "0.2.1"
3438
path: versions/v0.2.1.yml
3539
slug: v0.2.1

‎fern/versions/latest.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
v0.2.1.yml
1+
v0.3.0.yml

‎fern/versions/v0.3.0.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
tabs:
2+
docs:
3+
display-name: Documentation
4+
slug: ""
5+
api:
6+
display-name: API Reference
7+
slug: api
8+
9+
navigation:
10+
- tab: docs
11+
layout:
12+
- folder: ./v0.3.0/pages/about
13+
title: "About"
14+
title-source: frontmatter
15+
- folder: ./v0.3.0/pages/get-started
16+
title: "Get Started"
17+
title-source: frontmatter
18+
- folder: ./v0.3.0/pages/agent-server
19+
title: "Agent Server"
20+
title-source: frontmatter
21+
- folder: ./v0.3.0/pages/model-server
22+
title: "Model Server"
23+
title-source: frontmatter
24+
- folder: ./v0.3.0/pages/resources-server
25+
title: "Resources Server"
26+
title-source: frontmatter
27+
- folder: ./v0.3.0/pages/data
28+
title: "Data"
29+
title-source: frontmatter
30+
- folder: ./v0.3.0/pages/environment-tutorials
31+
title: "Environment Tutorials"
32+
title-source: frontmatter
33+
- folder: ./v0.3.0/pages/training-tutorials
34+
title: "Training Tutorials"
35+
title-source: frontmatter
36+
- folder: ./v0.3.0/pages/model-recipes
37+
title: "Model Recipes"
38+
title-source: frontmatter
39+
- folder: ./v0.3.0/pages/infrastructure
40+
title: "Infrastructure"
41+
title-source: frontmatter
42+
- folder: ./v0.3.0/pages/reference
43+
title: "Reference"
44+
title-source: frontmatter
45+
- folder: ./v0.3.0/pages/troubleshooting
46+
title: "Troubleshooting"
47+
title-source: frontmatter
48+
- folder: ./v0.3.0/pages/contribute
49+
title: "Contribute"
50+
title-source: frontmatter
51+
- tab: api
52+
layout:
53+
- section: API Reference
54+
slug: reference/api-reference
55+
contents:
56+
- page: Overview
57+
path: ./v0.3.0/pages/api-reference/index.mdx
58+
slug: ""
59+
- folder: ../product-docs/nemo-gym/Full-Library-Reference

‎fern/versions/v0.3.0/pages/about/architecture.mdx‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
title: "Architecture"
3+
description: "Understand how NeMo Gym implements environments as composable server components."
4+
position: 2.5
5+
---
6+
7+
<Info>
8+
9+
**Goal**: Understand how NeMo Gym implements environments as composable server components.
10+
11+
**Read first**: [Environments](/about/concepts/environments) — what an environment is and how it decomposes into dataset, agent harness, verifier, and state.
12+
13+
</Info>
14+
15+
## How NeMo Gym Implements Environments
16+
17+
NeMo Gym implements environments as composable FastAPI servers that communicate over async HTTP:
18+
19+
| Concept | NeMo Gym Component Implementation |
20+
|---------|---------------|
21+
| Dataset | JSONL: Responses API input — each row is a task (a single problem or challenge) for the agent to solve |
22+
| Agent Harness | FastAPI Agent Server |
23+
| Verifier + State | FastAPI Resources Server |
24+
| Model | FastAPI Model Server or managed by your own agent harness |
25+
26+
All components are composable. Use different datasets with the same resources server, the same resources server with different models, or the same model with different agent harnesses.
27+
28+
This gives you flexibility to integrate with your existing models and agents:
29+
30+
- **Bring your own agent** — Integrate your existing agent to use it with any Gym environment components.
31+
- **Use a built-in agent** — NeMo Gym includes some native agents, e.g. general-purpose multi-step tool calling, as well as built-in integrations with external harnesses like OpenHands.
32+
- **Train with any model endpoint** — The Model Server standardizes different LLM endpoints behind the Responses API and provides token IDs and log probabilities needed for RL training.
33+
34+
## How an Agent Runs a Task in NeMo Gym Environments
35+
36+
Each task attempt flows through three steps. The resulting trajectory is called a *rollout*:
37+
38+
1. **Initialize** — The agent receives a task row from the dataset and initializes a session on the Resources Server, which sets up isolated state for this task.
39+
2. **Agent Loop** — The agent calls a model for inference, then routes any tool calls to either its own tools or the Resources Server. This repeats until the agent decides the task is complete.
40+
3. **Verify** — The agent asks the Resources Server to score the attempt. The verifier inspects the final state and returns a reward signal.
41+
42+
43+
```
44+
Dataset (JSONL - one row per task)
45+
│
46+
▼
47+
┌──────────────────────────────────────────┐
48+
│ Agent Server │
49+
│ │
50+
│ run(): │
51+
│ 1. resources.seed_session() ─────────────► Resources Server
52+
│ 2. agent loop: │
53+
│ model.responses() ─────────────► Model Server
54+
│ resources.my_tool() ─────────────► Resources Server
55+
│ 3. resources.verify() ─────────────► Resources Server
56+
└──────────────────────────────────────────┘
57+
58+
┌───────────────────────────┐ ┌────────────────────────────────────┐
59+
│ Model Server │ │ Resources Server │
60+
│ │ │ │
61+
│ responses(): │ │ seed_session(): init env state │
62+
│ → text, tool calls, │ │ my_tool(): execute action │
63+
│ or code │ │ verify(): evaluate → reward │
64+
└───────────────────────────┘ └────────────────────────────────────┘
65+
```
66+
67+
68+
## Server Types
69+
70+
### Agent Server
71+
72+
Hosts agent harnesses that orchestrate rollouts. Use your own harness, or use built-in harnesses such as OpenHands or NeMo Gym's native harnesses such as Simple Agent.
73+
74+
### Model Server
75+
76+
A stateless LLM inference endpoint that standardizes different model providers behind the Responses API. Supports local inference and inference providers.
77+
78+
### Resources Server
79+
80+
Manages environment-specific tools, per-task state isolation, and verification:
81+
82+
- **Environment-Specific Tools** — capabilities the environment provides to any agent (e.g., code execution, database queries, API calls)
83+
- **State Isolation** — each rollout gets its own session, so attempts never interfere with each other. Environments range from lightweight (verify a math answer, no setup needed) to heavyweight (provision a Docker container with a specific repo checkout for SWE-Bench-style tasks).
84+
- **Verification** — scoring logic that evaluates the agent's output and returns a reward
85+
86+
## Where Tools Live
87+
88+
Tools exist on a spectrum — some belong to the agent and can be used with any environment, some belong to the environment and can be used with any agent:
89+
90+
- **Agent-specific tools** are part of the agent harness. They're capabilities the agent brings regardless of which environment it runs in (e.g., OpenHands brings file editing and terminal tools).
91+
- **Environment-specific tools** are part of the Resources Server. They're capabilities the environment provides to any agent that connects (e.g., a `run_tests` endpoint, a database query tool, a sandbox execution API).
92+
93+
An agent can use both simultaneously — its own tools and the environment's tools in the same task. NeMo Gym's server split reflects this: agent-specific logic in the harness, environment-specific logic in the Resources Server.
94+
95+
## Communication
96+
97+
Servers communicate over async HTTP (aiohttp) with:
98+
- **Session cookies** propagated through the call stack for stateful environments
99+
- **Retry logic** with exponential backoff (3 attempts)
100+
- **Connection pooling** via a singleton aiohttp client for high-concurrency workloads
101+
102+
## Next Steps
103+
104+
<Cards>
105+
106+
<Card title="Concepts" href="/about/concepts">
107+
Understand environments, evaluation, and training before diving into implementation.
108+
</Card>
109+
110+
<Card title="Browse Environments" href="https://github.com/NVIDIA-NeMo/Gym#-available-environments">
111+
Browse available environments for evaluation and training.
112+
</Card>
113+
114+
<Card title="Agents" href="/agent-server">
115+
Explore available agent harnesses and learn how to integrate your own agent.
116+
</Card>
117+
118+
<Card title="Training" href="/training-tutorials">
119+
Improve your agent or model with RL or fine-tuning.
120+
</Card>
121+
122+
<Card title="Build Custom Environments" href="/environment-tutorials">
123+
Create your own evaluation or training environments.
124+
</Card>
125+
126+
</Cards>

‎fern/versions/v0.3.0/pages/about/concepts/environments.mdx‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
---
2+
title: "Environments"
3+
description: "Environments decompose into dataset, agent harness, verifier, and state — unifying evaluation, agent optimization, and training."
4+
position: 1.5
5+
---
6+
7+
<Info>
8+
9+
**Goal**: Understand what an environment is, where the concept comes from, and how it decomposes into components that serve evaluation, agent optimization, and training.
10+
11+
</Info>
12+
13+
## Environment Origins
14+
15+
The term *environment* comes from reinforcement learning, where it describes the world an agent interacts with. The agent takes actions, the environment returns observations and a reward, and the cycle repeats. Historically, the word "Gym" refers to a collection of RL training environments, which inspired the naming of NeMo Gym.
16+
17+
The concept is the same today but the usage of environments has expanded beyond RL to include much more, such as model evaluation, agent evaluation, agent harness optimization, and synthetic data generation.
18+
19+
Accordingly, an environment contains everything required for an agent to complete a task:
20+
- Dataset
21+
- Agent Harness
22+
- Verifier
23+
- State
24+
25+
The model powering the agent is external to the environment. It generates a response (actions), which the environment processes. The environment updates its internal state, returns observations (tool results, error messages, state changes), and produces a reward (a numerical score of how well the agent performed).
26+
27+
## Anatomy of an Environment
28+
29+
### Dataset
30+
31+
A collection of tasks (prompts for the agent to solve), along with metadata and privileged information necessary for scoring each task attempt. Each task defines a problem for the model: e.g. a coding issue to fix, a math problem to solve, a tool-calling scenario to navigate.
32+
33+
Tasks can vary structurally: single-turn question answering, multi-step tool use, multi-turn dialogue, agentic workflows with sandboxed execution, and more.
34+
35+
### Agent Harness
36+
37+
Agent = model + agent harness. The agent harness defines how the model interacts with the environment. Language models perform stateless inference; the harness is what turns a model into an agent — it loops model calls, routes tool use, manages context, and decides when the task is done.
38+
39+
Harnesses exist on a spectrum. A simple harness just loops model calls until the task is complete. Other harnesses like Claude Code or OpenClaw include tools, planning, memory, self-correction, skills and more.
40+
41+
### Verifier
42+
43+
The verifier (sometimes referred to as a scorer or grader) scores a task attempt to calculate a reward, typically between 0 and 1. It defines what "good" means for this environment by scoring the model's output against task-specific metadata — e.g. expected answers, test cases, unit tests, ground truth, rubrics.
44+
45+
Common patterns include exact match, code execution (run tests and check if they pass), state matching, LLM-as-Judge, and reward models.
46+
47+
The verifier is used for both evaluation and training:
48+
49+
- **For evaluation**: verifier scores become benchmark metrics (e.g. accuracy, pass@1, pass@k).
50+
- **For training**: the same scores become the reward signal that drives reinforcement learning.
51+
52+
### State
53+
54+
Per-task state that changes as the agent takes actions. Each task attempt starts from a clean state, this ensures attempts are independent and rewards are attributable to the agent's actions on that task.
55+
56+
Some environments have minimal state (e.g., math — just input and output). Others have rich state that evolves across turns: file systems being modified, databases being updated, code repositories being patched, game boards advancing.
57+
58+
Runtimes and sandboxes host and run the environment. A runtime is the execution infrastructure (e.g. local process, Docker, Apptainer). A sandbox is a runtime with isolated execution — for example, a container per episode — providing security boundaries when the agent can execute arbitrary code.
59+
60+
61+
## Environments Unify Agent Improvement
62+
63+
![Evaluate-improve loop](/assets/images/eval-improve-loop.png)
64+
65+
- **Evaluation:** scores become metrics (e.g. accuracy, pass@1, pass@k).
66+
- **Harness optimization:** scores guide harness-level changes: e.g. prompt rewrites, tool changes, context management, orchestration tuning.
67+
- **Training:** scores become the reward signal that drives reinforcement learning. A training framework consumes rewards and updates model weights, improving the capabilities of the underlying model.
68+
69+
## Next Steps
70+
71+
Continue with the other concept pages:
72+
73+
<Cards>
74+
75+
<Card title="Evaluation" href="/about/concepts/evaluation">
76+
How environments are used to measure model and agent performance.
77+
</Card>
78+
79+
<Card title="Training" href="/about/concepts/training">
80+
Post-training techniques and how environments are used for training.
81+
</Card>
82+
83+
</Cards>

‎fern/versions/v0.3.0/pages/about/concepts/evaluation.mdx‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
title: "Evaluation"
3+
description: "What evaluation is, how to evaluate agents and models, and what makes a good eval."
4+
position: 1.6
5+
---
6+
7+
<Info>
8+
9+
**Goal**: Understand what evaluation is, how to evaluate agents and models, and what makes a good eval.
10+
11+
**Read first**: [Environments](/about/concepts/environments) — what an environment is and how it decomposes into dataset, agent harness, verifier, and state.
12+
13+
</Info>
14+
15+
## What is Evaluation?
16+
17+
Run an agent on tasks, score the results, measure performance. Evaluation exists to guide improvement — scores tell you what's working, what's not, and where to focus next.
18+
19+
Accuracy is the most common metric, but it's not the only one that matters. Token usage, latency, and number of tool calls are often just as important — two agents might solve the same tasks but one uses 3x fewer tokens or completes in half the time. A good eval captures the dimensions most important for your use case.
20+
21+
For example: you want to know how accurate your coding agent is at fixing bugs. You need:
22+
- A set of real bugs to test against (**dataset**)
23+
- A sandbox with the code repository for the agent to work in (**state**)
24+
- A test suite to check if the fix actually works (**verifier**)
25+
26+
## How to Evaluate Agents and Models
27+
28+
You can evaluate the model, the agent harness, or both together depending on what you're trying to improve.
29+
30+
![Model vs Agent Evaluation](/assets/images/model-vs-agent-eval.png)
31+
32+
### Model Capability
33+
34+
Swap the model, keep everything else the same.
35+
36+
> E.g. compare Qwen vs Kimi on SWE-Bench with OpenHands
37+
38+
The agent harness (e.g. OpenHands) is held constant so the scores reflect model capability. Evaluation metrics can span accuracy, token usage, latency, and cost across models. The environment is the same for both runs — only the model changes.
39+
40+
### Agent Capability
41+
42+
Swap the agent harness, keep everything else the same.
43+
44+
> E.g. you added a new tool, rewrote the system prompt, or changed the retry logic — run the same tasks with the same model to see if your agent improved.
45+
46+
The model is held constant so the scores reflect harness quality. A new tool might improve accuracy while tripling token usage; the eval should capture both. The environment and model are the same — only the harness changes.
47+
48+
## Evaluation Data
49+
50+
The most important property of evaluation data is that it represents your actual use case. High scores on a public benchmark don't guarantee performance on your tasks if the distribution is different. When possible, combine public benchmarks with custom data drawn from your domain. Tasks can come from multiple sources:
51+
52+
- **Public benchmarks** — standardized task sets for community comparison (e.g. SWE-Bench, AIME, HLE, Tau2).
53+
- **Custom datasets** — tasks specific to your use case, domain, or deployment.
54+
- **Production traces** — real agent interactions captured from deployed systems.
55+
56+
How much data you need depends on what you're measuring. A few hundred tasks can surface major capability gaps. Distinguishing small improvements (e.g. 2-3% accuracy gains) requires larger datasets and multiple repeats per task to separate signal from noise.
57+
58+
## What Makes a Good Eval
59+
60+
### Coverage
61+
62+
Tasks should be diverse and representative of real usage. A narrow eval suite gives false confidence — high scores on one task type can mask failures elsewhere.
63+
64+
### Verifier Design
65+
66+
The verifier defines what "good" means — not just correctness, but potentially efficiency, format adherence, or safety. A noisy or biased verifier gives misleading scores — you'll optimize for the wrong thing. Programmatic checks (exact match, test suites, code execution) are more consistent than LLM-as-Judge; use them when possible.
67+
68+
### Realistic Interaction
69+
70+
The evaluation should reflect how the agent actually operates: multi-turn conversation, tool use, code execution, stateful environments. Simplifying the interaction pattern during eval means your scores won't predict real-world performance.
71+
72+
### Statistical Confidence
73+
74+
A single run is noisy. Multiple repeats on each task let you distinguish real improvements from variance. Increase repeats until scores stabilize across runs.
75+
76+
### Reproducibility
77+
78+
Fixed components and clean state per task ensure that the same input produces the same score. Without this, you can't tell whether a score difference reflects a real improvement or just environmental variation.
79+
80+
## Next Steps
81+
82+
Evaluation scores guide what to improve next.
83+
84+
<Cards>
85+
86+
<Card title="Training" href="/about/concepts/training">
87+
Post-training techniques and how environments are used for training.
88+
</Card>
89+
90+
<Card title="Data" href="/data">
91+
Prepare datasets for environments.
92+
</Card>
93+
94+
</Cards>

0 commit comments

Comments
 (0)