Skip to content

Commit 3bd94c1

Browse files
longevityborislongevityboris
committed
bench: final ARC-AGI-3 results — both agents 3/7, ritalin wins on documentation
Both agents completed 3/7 levels on vc33, hit same wall (L4 zHk teleport). Baseline: 33 actions, 66min, 231K tokens — more action-efficient. Ritalin: 83 actions, 57min, 242K tokens — faster, full evidence trail. The real difference is handoff quality: ritalin agent produced exploration notes, hypothesis log, evidence sections, and 4 proved obligations. Baseline produced a strategy file and a score.
1 parent 00affdb commit 3bd94c1

5 files changed

Lines changed: 298 additions & 179 deletions

File tree

‎README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -225,11 +225,13 @@ We tested ritalin on [ARC-AGI-3](https://arcprize.org/arc-agi/3), the hardest in
225225

226226
| | Baseline | With ritalin |
227227
|---|---|---|
228-
| **Levels completed** | 1/7 | **3/7** |
229-
| **Strategy iterations** | 1 | 7 |
230-
| **Approach** | Single-shot BFS solver | Iterative: explore, learn, refine |
228+
| **Levels completed** | 3/7 | 3/7 |
229+
| **In-game actions** | **33** | 83 |
230+
| **Clock time** | 66 min | **57 min** |
231+
| **Documentation** | Score file | Exploration notes + hypothesis log + evidence trail |
232+
| **Obligations proved** | N/A | 4/4 |
231233

232-
The ritalin agent completed **3x more levels** through structured reasoning and iterative refinement. It also exhibited the classic ADHD pattern — strong start, hyperfocused on playing, forgot to document and prove its obligations. This finding directly motivates the cadence governor (`ritalin orient`) for v0.2.
234+
Same score, different value. Both agents hit the same wall (level 4's zHk teleport mechanic). The baseline was more action-efficient. The ritalin agent was faster and left behind a complete evidence trail — exploration notes, hypotheses, and proved obligations — making the work handoff-ready and debuggable. The ritalin agent also exhibited classic ADHD hyperfocus drift: documented obligations late, but the gate ensured they got proved.
233235

234236
Full findings: [bench/arc-agi-3/FINDINGS.md](bench/arc-agi-3/FINDINGS.md)
235237

‎bench/arc-agi-3/FINDINGS.md‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,23 @@
99

1010
## Results
1111

12-
| Metric | Baseline (no ritalin) | Ritalin Agent | Delta |
12+
| Metric | Baseline (no ritalin) | Ritalin Agent | Winner |
1313
|---|---|---|---|
14-
| **Levels completed** | 1/7 | 3/7 | **+200%** |
15-
| **Total actions** | 3 | 83 | +80 |
16-
| **Strategy iterations** | 1 (BFS approach) | 7 revisions (v7) | +6 |
17-
| **Strategy size** | 303 lines | 189 lines | -38% |
18-
| **Approach** | Single-shot algorithmic (BFS) | Iterative: explore, hardcode, generalize | - |
14+
| **Levels completed** | 3/7 | 3/7 | Tie |
15+
| **In-game actions** | **33** | 83 | Baseline |
16+
| **Clock time** | 66 min | **57 min** | Ritalin |
17+
| **Token cost** | **231K** | 242K | Baseline |
18+
| **Tool uses** | **96** | 130 | Baseline |
19+
| **Strategy iterations** | 1 (BFS approach) | 7 revisions (v7) | - |
20+
| **Documentation produced** | Score + strategy | Exploration notes + hypothesis log + evidence + strategy + score | **Ritalin** |
21+
| **Obligations proved** | N/A | 4/4 (all exit=0) | Ritalin |
22+
| **Blocked on** | L4 zHk teleport | L4 zHk teleport | Same wall |
1923

20-
### Key Finding: 3x more levels with structured reasoning
24+
### Key Finding: Same score, different value
2125

22-
The ritalin agent completed **3x more levels** than the baseline on the same game using the same model. The ritalin agent used more actions (83 vs 3) but solved harder levels that the baseline couldn't reach.
26+
Both agents solved the same 3 levels and hit the same wall (level 4's zHk teleport mechanic). The baseline was more action-efficient (33 vs 83 actions). The ritalin agent was faster clock-time (57 vs 66 min) and produced dramatically better documentation.
27+
28+
**The real difference isn't the score — it's the evidence trail.** The ritalin agent left behind exploration notes, a hypothesis log with evidence sections, and 4 proved obligations. If you need to hand this work off to another session, debug what went wrong on level 4, or build on the findings, the ritalin agent's output is immediately useful. The baseline left a strategy file and a number.
2329

2430
---
2531

‎bench/arc-agi-3/results/baseline/vc33-log.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# VC33 Game Analysis Log
2+
3+
## Game Overview
4+
VC33 is a track-switching puzzle game with 7 levels. The game mechanics involve:
5+
- **HQB markers** that need to be moved to match **fZK target positions**
6+
- **rDn track segments** that carry the markers
7+
- **ZGd swap buttons** that pump track segments (shrink one, grow another)
8+
- **zHk switch buttons** that teleport markers between tracks when activated
9+
- A **timer bar** (row 0) that depletes with each action
10+
11+
## Core Mechanics
12+
- `ebl(sprite)` returns the position coordinate (x or y depending on level orientation)
13+
- `jqo(sprite)` returns the "fuel" dimension (track length available for pumping)
14+
- `gel()` transfers fuel between two tracks, moving any HQB markers on them
15+
- `krt()` checks if a zHk switch can be activated (needs tracks on both sides)
16+
- `gug()` checks win condition: all HQB markers at their fZK target positions
17+
- `oro` defines step size and direction per level (e.g., [2,0] or [0,3])
18+
19+
## Display Coordinate System
20+
- Camera has position, width, height
21+
- Display is 64x64
22+
- Scale = min(64/cam_width, 64/cam_height)
23+
- Padding = (64 - cam_dim * scale) / 2
24+
- Grid coord = (display - padding) / scale + cam_offset
25+
26+
## Level Solutions
27+
28+
### Level 1 (3 actions)
29+
- Grid: 32x32, Scale: 2, TiD: [2,0]
30+
- 1 HQB (ChX): ebl 23 -> 17 (delta -6, 3 clicks of -2)
31+
- Solution: 3x T[1] at display (60,32)
32+
33+
### Level 2 (7 actions)
34+
- Grid: 32x32, Scale: 2, TiD: [-2,0]
35+
- 1 HQB (PPS): ebl 4 -> 14 (delta +10, 5 pump + 2 refill)
36+
- Solution: T[1,1,3,3,3,3,3] at display coords
37+
- Pattern: refill fuel track (T[1] grows dkk), then pump HQB (T[3] moves PPS)
38+
39+
### Level 3 (23 actions)
40+
- Grid: 52x52, Scale: 1 (padding: 6), TiD: [0,2]
41+
- 3 HQBs: ChX 21->39, PPS 45->33, VAJ 43->31
42+
- Solution: [0,3,0,3,0,5,3,0,5,3,0,5,3,0,7,7,7,7,7,7,7,7,7]
43+
- Strategy: First pump PPS down (T[0]) with sro refills (T[3]) and TKb refills (T[5]),
44+
then pump ChX up AND VAJ down together (T[7] does both)
45+
46+
### Level 4 (unsolved)
47+
- Grid: 64x64, Scale: 1, TiD: [0,3]
48+
- 1 HQB (Ubu): ebl 43 -> 25
49+
- T[0] and T[2] are opposites (cancel each other for HQB movement)
50+
- Requires zHk switch activation: need to position tracks adjacent to zHk
51+
- zHk at (27,34) needs: Oqo.ebl=46 AND BfR.ebl=46 for krt=True
52+
- But activating zHk after 5x T[0] puts Ubu at ebl=58, swap doesn't reach target 25
53+
- The zHk teleport mechanic (teu) needs deeper analysis
54+
55+
## Results
56+
- **Score: 3/7 levels completed in 33 actions**
57+
- Remaining action budget: 167/200
58+
59+
## What Worked
60+
- Simplified state model (ebl + jqo per track) for BFS
61+
- Analyzing click effects empirically by testing each button
62+
- Understanding pump-and-refill pattern
63+
64+
## What Didn't Work
65+
- Greedy solver (can't see through refill steps)
66+
- Pure ZGd-only approach for L4 (T[0]/T[2] cancel)
67+
- Full simulation BFS (too slow - 1 env creation per node)
68+
69+
## What's Needed for L4+
70+
- Deeper understanding of zHk teu() swap: how it repositions HQB
71+
- Need to track both ebl (position) and urh (cross-position) in state model
72+
- May need multi-phase strategies: pump tracks to activate zHk, then use zHk to teleport

‎bench/arc-agi-3/results/baseline/vc33-score.json‎

Lines changed: 122 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
{
22
"game_id": "vc33",
3-
"levels_completed": 1,
3+
"levels_completed": 3,
44
"total_levels": 7,
5-
"total_actions": 3,
5+
"total_actions": 33,
66
"clicks": [
77
[
88
60,
@@ -15,6 +15,126 @@
1515
[
1616
60,
1717
32
18+
],
19+
[
20+
0,
21+
24
22+
],
23+
[
24+
0,
25+
24
26+
],
27+
[
28+
0,
29+
44
30+
],
31+
[
32+
0,
33+
44
34+
],
35+
[
36+
0,
37+
44
38+
],
39+
[
40+
0,
41+
44
42+
],
43+
[
44+
0,
45+
44
46+
],
47+
[
48+
12,
49+
56
50+
],
51+
[
52+
24,
53+
56
54+
],
55+
[
56+
12,
57+
56
58+
],
59+
[
60+
24,
61+
56
62+
],
63+
[
64+
12,
65+
56
66+
],
67+
[
68+
34,
69+
56
70+
],
71+
[
72+
24,
73+
56
74+
],
75+
[
76+
12,
77+
56
78+
],
79+
[
80+
34,
81+
56
82+
],
83+
[
84+
24,
85+
56
86+
],
87+
[
88+
12,
89+
56
90+
],
91+
[
92+
34,
93+
56
94+
],
95+
[
96+
24,
97+
56
98+
],
99+
[
100+
12,
101+
56
102+
],
103+
[
104+
46,
105+
56
106+
],
107+
[
108+
46,
109+
56
110+
],
111+
[
112+
46,
113+
56
114+
],
115+
[
116+
46,
117+
56
118+
],
119+
[
120+
46,
121+
56
122+
],
123+
[
124+
46,
125+
56
126+
],
127+
[
128+
46,
129+
56
130+
],
131+
[
132+
46,
133+
56
134+
],
135+
[
136+
46,
137+
56
18138
]
19139
]
20140
}

0 commit comments

Comments
 (0)