Skip to content

Commit 0232cc9

Browse files
committed
[update] benchmark docs
1 parent ada132d commit 0232cc9

1 file changed

Lines changed: 10 additions & 6 deletions

File tree

‎BENCHMARKS.md‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -240,8 +240,6 @@ need not have followed the same route in the checked lane.
240240
| Curated | Alethe | `term-gap` | 1 |
241241
| Cashmere | Alethe | `certificate-error` | 23 |
242242
| Velvet | Alethe | `certificate-error` | 236 |
243-
| Velvet | Alethe | `rule-gap` | 22 |
244-
| Velvet | Alethe | `term-gap` | 9 |
245243
| Velvet | Portfolio | `certificate-error+core-failed` | 4 |
246244
| Velvet | Portfolio | `timeout` | 2 |
247245
| PLean | Alethe | `certificate-error` | 172 |
@@ -252,10 +250,16 @@ need not have followed the same route in the checked lane.
252250
These counts reproduce `reconstruction-failures.tsv` and use the same
253251
profiler filter as the table above. In particular, PLean's four portfolio
254252
`solver-unknown` records include the four passing VCs discussed above; they
255-
should not be read as four failed whole-VC attempts. Certificate errors in
256-
this run involve cvc5's `DUMMY_SKOLEM` proof-output limitation. Rule and term
257-
gaps are replay coverage limitations. Every measured lane attempted all
258-
754 VCs.
253+
should not be read as four failed whole-VC attempts. Every measured lane
254+
attempted all 754 VCs.
255+
256+
Almost every remaining failure is a `certificate-error`: cvc5's `DUMMY_SKOLEM`
257+
proof-output limitation, which no amount of replay coverage can reach because
258+
no usable certificate is emitted. Replay coverage itself now accounts for a
259+
single VC in the whole dataset -- one `term-gap` on Curated. Velvet's earlier
260+
22 `rule-gap` and 9 `term-gap` records are gone: the replay fixes closed them,
261+
and this run measures the Alethe lane once rather than in two studies whose
262+
copies had drifted apart.
259263

260264
### Reconstruction Comparison
261265

0 commit comments

Comments
 (0)