Skip to content

test: grammar-conformance differential (transcribe docs/grammar.md, diff against the parser) - #615

Open
enomado wants to merge 1 commit into
ron-rs:masterfrom
enomado:test/grammar-conformance
Open

test: grammar-conformance differential (transcribe docs/grammar.md, diff against the parser)#615
enomado wants to merge 1 commit into
ron-rs:masterfrom
enomado:test/grammar-conformance

Conversation

@enomado

@enomado enomado commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What

ron has ~450 correctness tests but nothing checks that the parser agrees with the
grammar it documents in
docs/grammar.md
there is no grammar-conformance test.

This adds one. conformance/ is an isolated sub-crate — the same trick as
fuzz/ (its own [workspace]), so pest never enters ron's own dependency
graph or the MSRV job
. It transcribes the EBNF verbatim into a PEG acceptor
(conformance/src/ron.pest) and diffs it, input by input, against the parser. A
disagreement is either a parser bug or a grammar-doc bug.

The oracle for "does ron accept this as one syntactically-valid value?" is
Deserializer::from_str -> IgnoredAny::deserialize -> end(): pure syntactic
validation, whole input consumed, no data model imposed. (Value can't be the
oracle — it lacks struct/enum/range/byte variants and would reject or mangle them.)

What it found

Building the differential surfaced seven places where docs/grammar.md
disagrees with the parser — in every case the parser is right and the doc is
wrong. They are filed as #614 and pinned here in KNOWN_DOC_BUGS:

# input doc parser gap
A "a\c" accept reject a lone \ must start a valid escape
B '\n' reject accept char takes the full string escape set
C "�" accept reject \u requires { 1..=6 hex }
D 1 ..2 accept reject no whitespace around range operators
E ''' reject accept a bare ' is accepted as char content
F 1e_+0 reject accept _ allowed before the exponent sign
G #![enable(..)] reject accept leading whitespace before extensions

Each row is a self-checking assertion: the transcription is faithful to the
doc, the parser behaves as claimed, and the two disagree — a wrong expectation
fails loudly instead of passing silently. Everywhere else (the CONFORMANT
battery, spanning every production family) the transcription and the parser agree.

Inputs ron rejects for semantic reasons (911u8, -1u8, Some()) are pinned
separately in SEMANTIC_NOT_GRAMMAR, so the differential does not mistake them for
conformance gaps.

CI

A dedicated Conformance job runs cd conformance && cargo test, isolated from the
main test / MSRV matrix (pest is pulled into this job only).

Note

This is the executable evidence behind #614. If useful, I'm happy to also open the
grammar.md fix PR; once the doc is corrected the KNOWN_DOC_BUGS entries fold
into the conformant battery and the divergence set goes to zero, leaving the test
as a guard against any new drift.

ron has ~450 correctness tests but nothing checks that the parser agrees with
docs/grammar.md — there is no grammar-conformance test. This adds a conformance/
sub-crate (isolated like fuzz/, so pest never enters ron's own dependency graph
or MSRV job) that transcribes the EBNF verbatim into a PEG acceptor and diffs it,
input by input, against the parser. A disagreement is a parser bug or a doc bug.

The oracle is Deserializer::from_str -> IgnoredAny -> end() (pure syntactic
validation of one value; Value can't be the oracle, it lacks struct/enum/range/
byte variants). Running the differential over a wide battery, hardened by
mutation-fuzzing valid seeds, surfaced seven places where docs/grammar.md
disagrees with the parser (reported in ron-rs#614). All are pinned in KNOWN_DOC_BUGS
with a self-checking assertion: the transcription is faithful to the doc, the
parser behaves as claimed, and the two disagree. Everywhere else they agree.

A dedicated 'Conformance' CI job runs 'cd conformance && cargo test'.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@juntyr

juntyr commented Jul 16, 2026

Copy link
Copy Markdown
Member

@enomado Thank you for all of your PRs today! I will have a look at them over the coming days once I have some free time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants