Skip to content

Commit abcf172

Browse files
rsasaki0109rsasaki0109
committed
Add closed-loop sequence harness and bounded degraded-acceptance variant
The sequence harness derives gap and rejection streak from the policy's own decisions, so the self-confirming unbounded acceptance that broke the 2026-06-12 replay now fails offline: correction_conditioned accepts all 20 degraded steps of the distilled Koide window, while the new bounded_degraded variant (non-resetting degraded-accept budget plus a cumulative-correction bound, reset only by a genuinely below-threshold measurement) bridges the onset with 3 accepts and then keeps rejecting. Replay validation remains mandatory before any runtime gate change.
1 parent 95b1113 commit abcf172

8 files changed

Lines changed: 391 additions & 15 deletions

File tree

‎docs/decisions.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,11 +32,11 @@
3232

3333
- adopted variant for this problem: `correction_conditioned`
3434
- design family: fitness threshold + correction/staleness cross-check
35-
- rationale: highest combined score (`83.84`) under the shared fixture/evaluation contract
36-
- benchmark score: `100.0`
35+
- rationale: highest combined score (`75.27`) under the shared fixture/evaluation contract
36+
- benchmark score: `85.7`
3737
- readability score: `25.2`
3838
- extensibility score: `94.0`
39-
- nearest alternative: `score_ratio_budget` at `78.72`
39+
- nearest alternative: `bounded_degraded` at `75.03`
4040
- generated_from: `scripts/run_measurement_acceptance_experiments.py`
4141

4242
## Recovery Action Selection

‎docs/experiments.md‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,10 @@ Accept degraded-but-consistent registration results that the scalar fitness gate
7373

7474
| Variant | Design | Benchmark | Readability | Extensibility | Overall |
7575
|---|---|---:|---:|---:|---:|
76-
| `correction_conditioned` | fitness threshold + correction/staleness cross-check | 100.0 | 25.2 | 94.0 | 83.84 |
77-
| `score_ratio_budget` | relative score cap with gap/streak budget | 83.3 | 49.6 | 94.0 | 78.72 |
78-
| `fixed_threshold` | scalar fitness threshold (runtime baseline) | 50.0 | 79.0 | 85.0 | 62.80 |
76+
| `correction_conditioned` | fitness threshold + correction/staleness cross-check | 85.7 | 25.2 | 94.0 | 75.27 |
77+
| `bounded_degraded` | correction cross-check with non-resetting degraded-accept budget | 85.7 | 34.0 | 84.0 | 75.03 |
78+
| `score_ratio_budget` | relative score cap with gap/streak budget | 71.4 | 49.6 | 94.0 | 71.58 |
79+
| `fixed_threshold` | scalar fitness threshold (runtime baseline) | 42.9 | 79.0 | 85.0 | 58.51 |
7980

8081
### Fixture Outcomes
8182

@@ -87,6 +88,14 @@ Accept degraded-but-consistent registration results that the scalar fitness gate
8788
- `lost_huge_score_should_reject`: pass=`True` decision=`reject` reason=`score_over_threshold_unsupported`
8889
- `stale_prediction_should_reject`: pass=`True` decision=`reject` reason=`score_over_threshold_unsupported`
8990

91+
#### `bounded_degraded`
92+
- `degraded_onset_small_correction_should_accept`: pass=`True` decision=`accept` reason=`degraded_accept_within_budget`
93+
- `degraded_streak_small_correction_should_accept`: pass=`True` decision=`accept` reason=`degraded_accept_within_budget`
94+
- `fresh_jump_good_score_should_reject`: pass=`False` decision=`accept` reason=`score_within_threshold`
95+
- `healthy_tracking_should_accept`: pass=`True` decision=`accept` reason=`score_within_threshold`
96+
- `lost_huge_score_should_reject`: pass=`True` decision=`reject` reason=`degraded_budget_exhausted_or_unsupported`
97+
- `stale_prediction_should_reject`: pass=`True` decision=`reject` reason=`degraded_budget_exhausted_or_unsupported`
98+
9099
#### `score_ratio_budget`
91100
- `degraded_onset_small_correction_should_accept`: pass=`True` decision=`accept` reason=`score_within_ratio_budget`
92101
- `degraded_streak_small_correction_should_accept`: pass=`True` decision=`accept` reason=`score_within_ratio_budget`

‎docs/interfaces.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,7 @@ Accept degraded-but-consistent registration results that the scalar fitness gate
123123
### Candidate Families
124124

125125
- `correction_conditioned`: fitness threshold + correction/staleness cross-check
126+
- `bounded_degraded`: correction cross-check with non-resetting degraded-accept budget
126127
- `score_ratio_budget`: relative score cap with gap/streak budget
127128
- `fixed_threshold`: scalar fitness threshold (runtime baseline)
128129

‎experiments/measurement_acceptance/fixtures_sequences/koide_degraded_window_sequence.json‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
{
2+
"name": "koide_degraded_window_sequence",
3+
"description": "Closed-loop-style sequence distilled from the Koide outdoor_hard_01a failure window. The harness derives gap and rejection streak from the policy's own decisions, so a policy that accepts degraded measurements without a non-resetting budget exhibits the unbounded self-confirming acceptance that broke the 2026-06-12 replay.",
4+
"source": "distilled from koide outdoor_hard_01a 180s window rows 139-160 (score/correction ranges)",
5+
"effective_score_threshold": 6.0,
6+
"steps": [
7+
{
8+
"fitness_score": 0.3,
9+
"correction_translation_m": 0.01,
10+
"correction_yaw_deg": 0.1
11+
},
12+
{
13+
"fitness_score": 13.59,
14+
"correction_translation_m": 1.032,
15+
"correction_yaw_deg": 2.0
16+
},
17+
{
18+
"fitness_score": 26.56,
19+
"correction_translation_m": 0.754,
20+
"correction_yaw_deg": 2.0
21+
},
22+
{
23+
"fitness_score": 18.0,
24+
"correction_translation_m": 1.5,
25+
"correction_yaw_deg": 2.0
26+
},
27+
{
28+
"fitness_score": 14.2,
29+
"correction_translation_m": 0.9,
30+
"correction_yaw_deg": 2.0
31+
},
32+
{
33+
"fitness_score": 22.1,
34+
"correction_translation_m": 1.2,
35+
"correction_yaw_deg": 2.0
36+
},
37+
{
38+
"fitness_score": 16.4,
39+
"correction_translation_m": 2.0,
40+
"correction_yaw_deg": 2.0
41+
},
42+
{
43+
"fitness_score": 19.8,
44+
"correction_translation_m": 0.8,
45+
"correction_yaw_deg": 2.0
46+
},
47+
{
48+
"fitness_score": 25.0,
49+
"correction_translation_m": 1.1,
50+
"correction_yaw_deg": 2.0
51+
},
52+
{
53+
"fitness_score": 15.5,
54+
"correction_translation_m": 1.7,
55+
"correction_yaw_deg": 2.0
56+
},
57+
{
58+
"fitness_score": 21.3,
59+
"correction_translation_m": 0.95,
60+
"correction_yaw_deg": 2.0
61+
},
62+
{
63+
"fitness_score": 17.6,
64+
"correction_translation_m": 1.3,
65+
"correction_yaw_deg": 2.0
66+
},
67+
{
68+
"fitness_score": 23.9,
69+
"correction_translation_m": 0.85,
70+
"correction_yaw_deg": 2.0
71+
},
72+
{
73+
"fitness_score": 14.8,
74+
"correction_translation_m": 1.6,
75+
"correction_yaw_deg": 2.0
76+
},
77+
{
78+
"fitness_score": 20.2,
79+
"correction_translation_m": 1.05,
80+
"correction_yaw_deg": 2.0
81+
},
82+
{
83+
"fitness_score": 26.1,
84+
"correction_translation_m": 0.9,
85+
"correction_yaw_deg": 2.0
86+
},
87+
{
88+
"fitness_score": 13.9,
89+
"correction_translation_m": 1.4,
90+
"correction_yaw_deg": 2.0
91+
},
92+
{
93+
"fitness_score": 18.7,
94+
"correction_translation_m": 1.15,
95+
"correction_yaw_deg": 2.0
96+
},
97+
{
98+
"fitness_score": 24.4,
99+
"correction_translation_m": 0.75,
100+
"correction_yaw_deg": 2.0
101+
},
102+
{
103+
"fitness_score": 16.0,
104+
"correction_translation_m": 1.9,
105+
"correction_yaw_deg": 2.0
106+
},
107+
{
108+
"fitness_score": 22.8,
109+
"correction_translation_m": 1.0,
110+
"correction_yaw_deg": 2.0
111+
}
112+
],
113+
"expectations": {
114+
"max_consecutive_degraded_accepts": 4,
115+
"min_degraded_accepts": 1,
116+
"must_reject_final_step": true
117+
}
118+
}

‎experiments/measurement_acceptance/results.json‎

Lines changed: 135 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,20 @@
123123
"expected_reject": true
124124
}
125125
],
126-
"benchmark_score": 100.0,
126+
"sequence_results": [
127+
{
128+
"fixture": "koide_degraded_window_sequence",
129+
"passed": false,
130+
"checks": {
131+
"bounded_degraded_run": false,
132+
"bridges_onset": true,
133+
"rejects_when_exhausted": false
134+
},
135+
"degraded_accepts": 20,
136+
"max_consecutive_degraded_accepts": 20
137+
}
138+
],
139+
"benchmark_score": 85.71428571428571,
127140
"readability_score": 25.200000000000003,
128141
"extensibility_score": 94.0,
129142
"static_metrics": {
@@ -138,7 +151,96 @@
138151
"readability_score": 25.200000000000003,
139152
"extensibility_score": 94.0
140153
},
141-
"overall_score": 83.84
154+
"overall_score": 75.27
155+
},
156+
{
157+
"name": "bounded_degraded",
158+
"design": "correction cross-check with non-resetting degraded-accept budget",
159+
"fixture_results": [
160+
{
161+
"fixture": "degraded_onset_small_correction_should_accept",
162+
"passed": true,
163+
"outcome": "matched_expected_reject",
164+
"decision": "accept",
165+
"decision_reason": "degraded_accept_within_budget",
166+
"decision_score": 13.59,
167+
"expected_reject": false
168+
},
169+
{
170+
"fixture": "degraded_streak_small_correction_should_accept",
171+
"passed": true,
172+
"outcome": "matched_expected_reject",
173+
"decision": "accept",
174+
"decision_reason": "degraded_accept_within_budget",
175+
"decision_score": 26.56,
176+
"expected_reject": false
177+
},
178+
{
179+
"fixture": "fresh_jump_good_score_should_reject",
180+
"passed": false,
181+
"outcome": "mismatched_expected_reject",
182+
"decision": "accept",
183+
"decision_reason": "score_within_threshold",
184+
"decision_score": 1.5,
185+
"expected_reject": true
186+
},
187+
{
188+
"fixture": "healthy_tracking_should_accept",
189+
"passed": true,
190+
"outcome": "matched_expected_reject",
191+
"decision": "accept",
192+
"decision_reason": "score_within_threshold",
193+
"decision_score": 0.3,
194+
"expected_reject": false
195+
},
196+
{
197+
"fixture": "lost_huge_score_should_reject",
198+
"passed": true,
199+
"outcome": "matched_expected_reject",
200+
"decision": "reject",
201+
"decision_reason": "degraded_budget_exhausted_or_unsupported",
202+
"decision_score": 4353.99,
203+
"expected_reject": true
204+
},
205+
{
206+
"fixture": "stale_prediction_should_reject",
207+
"passed": true,
208+
"outcome": "matched_expected_reject",
209+
"decision": "reject",
210+
"decision_reason": "degraded_budget_exhausted_or_unsupported",
211+
"decision_score": 18.72,
212+
"expected_reject": true
213+
}
214+
],
215+
"sequence_results": [
216+
{
217+
"fixture": "koide_degraded_window_sequence",
218+
"passed": true,
219+
"checks": {
220+
"bounded_degraded_run": true,
221+
"bridges_onset": true,
222+
"rejects_when_exhausted": true
223+
},
224+
"degraded_accepts": 3,
225+
"max_consecutive_degraded_accepts": 3
226+
}
227+
],
228+
"benchmark_score": 85.71428571428571,
229+
"readability_score": 34.0,
230+
"extensibility_score": 84.0,
231+
"static_metrics": {
232+
"implementation_file": "experiments/measurement_acceptance/variants/bounded_degraded.py",
233+
"loc": 35,
234+
"branch_count": 4,
235+
"max_nesting_depth": 1,
236+
"state_field_count": 5,
237+
"public_method_count": 2,
238+
"import_count": 4,
239+
"has_config_dataclass": true,
240+
"readability_score": 34.0,
241+
"extensibility_score": 84.0
242+
},
243+
"overall_score": 75.03
142244
},
143245
{
144246
"name": "score_ratio_budget",
@@ -199,7 +301,20 @@
199301
"expected_reject": true
200302
}
201303
],
202-
"benchmark_score": 83.33333333333334,
304+
"sequence_results": [
305+
{
306+
"fixture": "koide_degraded_window_sequence",
307+
"passed": false,
308+
"checks": {
309+
"bounded_degraded_run": false,
310+
"bridges_onset": true,
311+
"rejects_when_exhausted": false
312+
},
313+
"degraded_accepts": 20,
314+
"max_consecutive_degraded_accepts": 20
315+
}
316+
],
317+
"benchmark_score": 71.42857142857143,
203318
"readability_score": 49.599999999999994,
204319
"extensibility_score": 94.0,
205320
"static_metrics": {
@@ -214,7 +329,7 @@
214329
"readability_score": 49.599999999999994,
215330
"extensibility_score": 94.0
216331
},
217-
"overall_score": 78.72
332+
"overall_score": 71.58
218333
},
219334
{
220335
"name": "fixed_threshold",
@@ -275,7 +390,20 @@
275390
"expected_reject": true
276391
}
277392
],
278-
"benchmark_score": 50.0,
393+
"sequence_results": [
394+
{
395+
"fixture": "koide_degraded_window_sequence",
396+
"passed": false,
397+
"checks": {
398+
"bounded_degraded_run": true,
399+
"bridges_onset": false,
400+
"rejects_when_exhausted": true
401+
},
402+
"degraded_accepts": 0,
403+
"max_consecutive_degraded_accepts": 0
404+
}
405+
],
406+
"benchmark_score": 42.857142857142854,
279407
"readability_score": 79.0,
280408
"extensibility_score": 85.0,
281409
"static_metrics": {
@@ -290,8 +418,8 @@
290418
"readability_score": 79.0,
291419
"extensibility_score": 85.0
292420
},
293-
"overall_score": 62.8
421+
"overall_score": 58.51
294422
}
295423
],
296-
"benchmark_score_mean": 77.77777777777779
424+
"benchmark_score_mean": 71.42857142857143
297425
}

‎experiments/measurement_acceptance/variants/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1+
from .bounded_degraded import BoundedDegradedAcceptance
12
from .correction_conditioned import CorrectionConditionedAcceptance
23
from .fixed_threshold import FixedThresholdAcceptance
34
from .score_ratio_budget import ScoreRatioBudgetAcceptance
45

56
__all__ = [
7+
"BoundedDegradedAcceptance",
68
"CorrectionConditionedAcceptance",
79
"FixedThresholdAcceptance",
810
"ScoreRatioBudgetAcceptance",

0 commit comments

Comments
 (0)