Reject the Unicode noncharacters, and record suiteRevision 6 at 153/153 - #4
Merged
Conversation
suiteRevision 6 makes the RFC 7493 section 2.1 exclusion normative: the sixty-six noncharacters, U+FDD0 through U+FDEF and U+nFFFE and U+nFFFF in each of the seventeen planes, are malformed wherever a string literal appears, at any depth and in member-name as well as value position. This checker admitted them, and the reason is worth recording because it was not an oversight. The earlier text scoped its MUST to string literals being well-formed sequences of Unicode scalar values, and a noncharacter is a scalar value, so the narrower rule was implemented faithfully. What the revision changes is the rule, not the reading: the strict-I-JSON label above that MUST had always implied the wider RFC 7493 exclusion, and the revision closes the gap between the label and the rule underneath it. Both routes into a string body are covered, because the exclusion is over code points and a producer reaches them either way: the raw UTF-8 byte, and the escape including a surrogate pair resolving into a plane-end noncharacter. The plane-end pairs differ only in their lowest bit, so one mask covers all thirty-four, and a test walks the whole code space to confirm the predicate selects exactly sixty-six. Refs in-toto/attestation#570. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The unchanged revision-5 checker scored 151/153. The depth-boundary pair ok-036 and bad-742 passed on the container-branch counter already in place; bad-743 and bad-744 did not, and those two are what the previous commit fixes. The record says plainly that this run is directed, and more so than revision 2 was: the rule was written and the vectors named before this checker ran, so what it shows is that the corrected rule is implementable from the text, not that an outside reader found something. Revision 5 is retired from continuous verification and keeps its checkerCommit, 88c37d1, so its provenance stays checkable after the pin moves. The workflow follows: suite pin, spec pin, the compared report, and the parity string the corpus step greps for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughJSON parsing now rejects Unicode noncharacters from raw UTF-8 and escaped inputs. Conformance metadata, reports, CI pins, parity requirements, and reproducibility documentation are updated for suite revision 6. ChangesUnicode validation and conformance evidence
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant JSONInput
participant parse_string
participant push_scalar
participant is_noncharacter
JSONInput->>parse_string: provide escaped or raw string
parse_string->>push_scalar: pass decoded scalar
push_scalar->>is_noncharacter: check scalar membership
is_noncharacter-->>push_scalar: return classification
push_scalar-->>parse_string: append or reject scalar
Suggested reviewers: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
suiteRevision 6 adds four vectors. Two are the depth-boundary pair,
ok-036andbad-742, and the container-branch counter from #3 already handled both. The other two,bad-743andbad-744, carry Unicode noncharacters in a vocabulary label and in a payload value, and this checker admitted them. The unchanged revision-5 build scored 151/153; this branch scores 153/153 (36/36 accepts, 117/117 rejects).Why it admitted them
Not an oversight, which is worth recording. The earlier text scoped its MUST to string literals being well-formed sequences of Unicode scalar values, and a noncharacter is a scalar value, so the narrower rule was implemented faithfully. The revision changes the rule rather than the reading: the strict-I-JSON label above that MUST had always implied the wider RFC 7493 section 2.1 exclusion, and revision 6 closes the gap between the label and the rule underneath it.
The fix
The sixty-six code points are U+FDD0 through U+FDEF plus U+nFFFE and U+nFFFF in each of the seventeen planes. Both routes into a string body are covered, because the exclusion is over code points and a producer reaches them either way: the raw UTF-8 byte, and the escape, including a surrogate pair resolving into a plane-end noncharacter. The plane-end pairs differ only in their lowest bit, so one mask covers all thirty-four, and a test walks the entire code space to confirm the predicate selects exactly sixty-six and no more.
This run is directed, and the record says so
More so than revision 2 was: the rule was written and the vectors were named before this checker ran. What it demonstrates is that the corrected rule is implementable from the text, not that an independent reader found something. The blind evidence remains where it was, at 125/125 on revision 1 with no vector-driven fixes, and 140/140 first-run unchanged at revision 3.
Verification
astrogilda/aee-conformance@7098f4e, with exactly two verdicts moved from the revision-5 build,bad-743andbad-744, comparing all per-vector fields.reports/suite-revision-5.jsonexactly against suiteea25a1e.checkerCommit(88c37d1), so its provenance stays checkable after the pin moves.Refs in-toto/attestation#570.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests