You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -131,25 +131,96 @@ This auto-detects which agent backends (Claude Code, Codex, etc.) are installed
131
131
132
132
### 3. Initialize a workspace
133
133
134
+
`ph init` sets up two things:
135
+
136
+
1.**Who optimizes** (`--agent`) — which AI does the thinking: a CLI tool like `claude-code`, or an API like `api` / `openai`.
137
+
2.**What to optimize** (`--template` or `--base-harness` + `--task-dir`) — your harness code, test cases, and evaluation script. These three are always needed for `ph run` to work.
138
+
139
+
#### Option A: Use a bundled template (recommended for first run)
140
+
141
+
PolyHarness ships with ready-to-run templates. One command sets up everything:
142
+
143
+
```bash
144
+
ph init --agent api --template text-classification
145
+
```
146
+
147
+
This copies a complete set of harness + tasks + evaluate script into the workspace automatically:
148
+
149
+
```
150
+
.ph_workspace/
151
+
├── base_harness/
152
+
│ └── harness.py # starting code to optimize
153
+
├── tasks/
154
+
│ └── test_cases.json # test inputs + expected outputs
155
+
├── evaluate.py # scoring script
156
+
└── config.yaml # auto-generated
157
+
```
158
+
159
+
That's it — skip to [step 4](#4-run-the-optimization-loop).
160
+
161
+
> Available templates: `text-classification`, `math-word-problems`, `code-generation`, `rag-qa`, `api-calling`.
162
+
163
+
#### Option B: Use your own project
164
+
165
+
You need three files: `harness.py` (code to optimize), `tasks/test_cases.json` (test data), and `evaluate.py` (scoring script). Generate them all with one command:
166
+
167
+
```bash
168
+
ph new my-project
169
+
```
170
+
171
+
This creates:
172
+
173
+
```
174
+
my-project/
175
+
├── base_harness/
176
+
│ └── harness.py # ← edit: your starting logic
177
+
├── tasks/
178
+
│ └── test_cases.json # ← edit: your test inputs + expected outputs
179
+
└── evaluate.py # ← edit if needed: scoring logic
180
+
```
181
+
182
+
Edit the generated files for your task. For example, if you're building a text classifier:
183
+
184
+
```python
185
+
# my-project/base_harness/harness.py
186
+
defsolve(input_data: str) -> str:
187
+
# A simple starting point — the agent will improve this
188
+
if"good"in input_data.lower():
189
+
return"positive"
190
+
return"negative"
191
+
```
192
+
193
+
```json
194
+
// my-project/tasks/test_cases.json
195
+
[
196
+
{"input": "This product is good", "expected": "positive"},
{"input": "The meeting is at 3pm", "expected": "neutral"}
199
+
]
200
+
```
201
+
202
+
> `evaluate.py` works out of the box — it calls `harness.solve(case["input"])`, compares with `case["expected"]`, and reports accuracy. Only edit it if your scoring needs custom logic.
203
+
204
+
Then initialize:
205
+
134
206
```bash
135
207
ph init \
136
208
--agent claude-code \
137
-
--base-harness ./my_harness/ \
138
-
--task-dir ./my_tasks/ \
139
-
--eval-script ./evaluate.py
209
+
--base-harness ./my-project/base_harness \
210
+
--task-dir ./my-project
140
211
```
141
212
142
-
This copies your harness code, test cases, and evaluation script into an isolated **optimization workspace** (by default `.ph_workspace` in the current directory, or the folder specified by `--workspace`).
143
-
144
-
#### What you need to prepare
213
+
| Flag | What to pass | Required? |
214
+
|------|-------------|:---------:|
215
+
|`--agent`| Who optimizes: `claude-code`, `codex`, `api`, `openai`, etc. | Yes (default `api`) |
216
+
|`--base-harness`| Directory with your starting harness code (at least `harness.py`) | Yes*|
217
+
|`--task-dir`| Directory with `tasks/test_cases.json` and optionally `evaluate.py`| Yes*|
218
+
|`--eval-script`| Path to `evaluate.py`, if it lives outside `--task-dir`| Only if not in task-dir |
219
+
|`--workspace`| Where to create the workspace (default `.ph_workspace`) | No |
145
220
146
-
| Item | What it is | Example |
147
-
|------|-----------|---------|
148
-
|**base-harness/**| A directory with your starting harness code. Must contain at least one entry file (default: `harness.py`). This is the code PolyHarness will iteratively optimize. | A Python file with a `classify(text)` or `solve(question)` function |
149
-
|**tasks/**| A directory with `test_cases.json` — a JSON array of test inputs and expected outputs. Each item should have an `"input"` and `"expected"` field. |`[{"input": "2+3", "expected": "5"}, ...]`|
150
-
|**evaluate.py**| A script that runs the harness against each task and outputs a JSON score. Must print `{"overall_score": 0.85, "task_scores": {...}}` to stdout. | See any `examples/*/evaluate.py` for a working template |
221
+
\* Technically optional at `init` time, but `ph run` will fail without harness code and test data.
151
222
152
-
> **Tip:** If you're unsure about the format, start with one of the bundled `examples/` directories — each one is a complete, working template you can copy and adapt.
223
+
`ph init` copies everything into an isolated **optimization workspace** — your original code is never modified.
153
224
154
225
**Configure Your Agent**
155
226
@@ -190,14 +261,9 @@ ph clean --keep-best # remove candidates to free disk space
190
261
### Try it now (no API key needed)
191
262
192
263
```bash
193
-
cd examples/math-word-problems
194
-
195
-
ph init --agent local \
196
-
--base-harness ./base_harness \
197
-
--task-dir . \
198
-
--workspace .ph_workspace
199
-
200
-
ph log --workspace .ph_workspace
264
+
ph init --agent local --template math-word-problems
265
+
ph run --max-iterations 5
266
+
ph log
201
267
202
268
# Search Tree
203
269
# └── iter_0 0.3500
@@ -277,10 +343,8 @@ When you run `ph init --agent claude-code`, PolyHarness automatically generates
277
343
If you're running a local model (Ollama, vLLM, LM Studio, or any OpenAI-compatible server), use the `openai` backend:
278
344
279
345
```bash
280
-
# 1. Initialize
281
-
ph init --agent openai \
282
-
--base-harness ./my_harness/ \
283
-
--task-dir ./my_tasks/
346
+
# 1. Initialize (use a template, or --base-harness + --task-dir for your own project)
0 commit comments