test: grammar-conformance differential (transcribe docs/grammar.md, diff against the parser) - #615
Open
enomado wants to merge 1 commit into
Open
test: grammar-conformance differential (transcribe docs/grammar.md, diff against the parser)#615enomado wants to merge 1 commit into
enomado wants to merge 1 commit into
Conversation
ron has ~450 correctness tests but nothing checks that the parser agrees with docs/grammar.md — there is no grammar-conformance test. This adds a conformance/ sub-crate (isolated like fuzz/, so pest never enters ron's own dependency graph or MSRV job) that transcribes the EBNF verbatim into a PEG acceptor and diffs it, input by input, against the parser. A disagreement is a parser bug or a doc bug. The oracle is Deserializer::from_str -> IgnoredAny -> end() (pure syntactic validation of one value; Value can't be the oracle, it lacks struct/enum/range/ byte variants). Running the differential over a wide battery, hardened by mutation-fuzzing valid seeds, surfaced seven places where docs/grammar.md disagrees with the parser (reported in ron-rs#614). All are pinned in KNOWN_DOC_BUGS with a self-checking assertion: the transcription is faithful to the doc, the parser behaves as claimed, and the two disagree. Everywhere else they agree. A dedicated 'Conformance' CI job runs 'cd conformance && cargo test'. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
@enomado Thank you for all of your PRs today! I will have a look at them over the coming days once I have some free time |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ron has ~450 correctness tests but nothing checks that the parser agrees with the
grammar it documents in
docs/grammar.md—there is no grammar-conformance test.
This adds one.
conformance/is an isolated sub-crate — the same trick asfuzz/(its own[workspace]), sopestnever enters ron's own dependencygraph or the MSRV job. It transcribes the EBNF verbatim into a PEG acceptor
(
conformance/src/ron.pest) and diffs it, input by input, against the parser. Adisagreement is either a parser bug or a grammar-doc bug.
The oracle for "does ron accept this as one syntactically-valid value?" is
Deserializer::from_str -> IgnoredAny::deserialize -> end(): pure syntacticvalidation, whole input consumed, no data model imposed. (
Valuecan't be theoracle — it lacks struct/enum/range/byte variants and would reject or mangle them.)
What it found
Building the differential surfaced seven places where
docs/grammar.mddisagrees with the parser — in every case the parser is right and the doc is
wrong. They are filed as #614 and pinned here in
KNOWN_DOC_BUGS:"a\c"\must start a valid escape'\n'"�"\urequires{ 1..=6 hex }1 ..2''''is accepted as char content1e_+0_allowed before the exponent sign#![enable(..)]Each row is a self-checking assertion: the transcription is faithful to the
doc, the parser behaves as claimed, and the two disagree — a wrong expectation
fails loudly instead of passing silently. Everywhere else (the
CONFORMANTbattery, spanning every production family) the transcription and the parser agree.
Inputs ron rejects for semantic reasons (
911u8,-1u8,Some()) are pinnedseparately in
SEMANTIC_NOT_GRAMMAR, so the differential does not mistake them forconformance gaps.
CI
A dedicated
Conformancejob runscd conformance && cargo test, isolated from themain test / MSRV matrix (
pestis pulled into this job only).Note
This is the executable evidence behind #614. If useful, I'm happy to also open the
grammar.mdfix PR; once the doc is corrected theKNOWN_DOC_BUGSentries foldinto the conformant battery and the divergence set goes to zero, leaving the test
as a guard against any new drift.