scanner: accept octal and multi-byte escapes in rune literals (fix #28880)

[quaesitor-scientiam]

Fixes #28880.

validate_char_literal (added in 3b7b5eec98) splits a rune literal into characters to check that it holds only one. It knew the length of \x, \u and \U escapes, but gave every other escape one character after the backslash, and it counted every byte escape as a character of its own:

`\141`          => ([`\1`, `4`, `1`])
`\342\230\205`  => ([`\3`, `4`, `2`, `\2`, `3`, `0`, `\2`, `0`, `5`])
`\xe2\x98\x85`  => ([`\xe2`, `\x98`, `\x85`])

Each of these is one character, strings accept all of them, and doc/docs.md documents both forms for runes — its example is one of the two check-markdown failures on master.

The change (vlib/v/scanner/scanner.v)

• An octal escape is exactly three octal digits, the same rule as the parser's string literal decoder (vlib/v/parser/parser.v:16396). With fewer, the backslash still escapes only the next character, as before.
• A \xHH or octal byte escape holding a UTF-8 lead byte (0xc2–0xf4) takes the continuation byte escapes that complete its character, and the run counts as one character — but only when the bytes are one well-formed UTF-8 sequence (Unicode Table 3-7): every continuation byte in 0x80–0xbf, and the decoded code point not overlong, not a surrogate and not above U+10FFFF (the same range check_string_escape already enforces for \u/\U). Otherwise nothing is merged and the literal is still rejected as more than one character, e.g. \xe0\x80\x80 and \xf0\x80\x80\x80 (overlong), \xed\xa0\x80 (surrogate), \xf4\x90\x80\x80 (above U+10FFFF), \xc0\x80 (overlong lead).
  The first version (9c4c196) only checked the continuation-byte range and accepted those first four — found in review, fixed in cf23b5a.

Nothing else changes: the \x/\u/\U checks, unknown-escape errors and the existing .vv fixtures (`\n\t`, `\nb`) behave as before.

Code generation

No change needed. With only the validator fixed, the whole docs example compiles and every assertion passes under both tcc and gcc, including `\141` == `a` and `\342\230\205`.bytes() == [u8(0xe2), 0x98, 0x85].

Tests

vlib/v/scanner/scanner_test.v:

• test_char_literal_escapes_that_spell_one_character_are_accepted — octal, \x, \u, \U, \0, and 2-, 3- and 4-byte UTF-8 sequences spelled in hex, octal and a mix of both.
• test_char_literal_with_more_than_one_character_is_still_rejected — `\141b`, two ASCII byte escapes, an incomplete sequence, an invalid continuation byte, an overlong lead, and a complete sequence followed by an extra byte.
• test_char_literal_byte_escapes_are_one_character_only_when_well_formed_utf8 — the rule itself rather than examples: every lead byte C0–F7 × every boundary of the second-byte ranges × a valid and an invalid final byte, in hex and in octal spelling, checked against encoding.utf8.validate_str. It fails on 9c4c196 (first on \xe0\x80\x80), and removing any one of the four checks (shortest form, surrogate, maximum, continuation byte) makes it fail on that class.

Phase-R verified: with scanner.v reverted to master, the first test fails on assert char_literal_diagnostics(source) == []; with the fix, both pass.

Verified on this box (Windows 11, tcc + gcc 16.2.0)

With cf23b5a:

• v test vlib/v/scanner/scanner_test.v → OK
• v -silent vlib/v/compiler_errors_test.v → 29 failed / 1689 passed / 7 skipped — exactly the same 29 failures (all under vlib/v/checker/tests/) as a compiler built from the same tree with master's scanner.v.
• The four inputs from review (\xe0\x80\x80, \xf0\x80\x80\x80, \xed\xa0\x80, \xf4\x90\x80\x80) are rejected; the boundary characters \xe0\xa0\x80, \xed\x9f\xbf and \xf4\x8f\xbf\xbf compile to U+0800, U+D7FF and U+10FFFF; the docs rune example still runs and passes.
• v fmt -verify on both touched files → clean

With 9c4c196: v check-md doc/docs.md no longer reports the rune example. What remains is the sql example, fixed separately in #28876, and one example that only fails on Windows (it passes in CI).

Open: #28897. In v -silent test vlib/v/scanner vlib/v/parser vlib/v/gen/c vlib/v/tests (VJOBS=20) the failure lists match master's scanner.v except vlib/v/tests/comptime/comptime_on_generics_func_test.v, which crashed in driver.clone_int_string_map (#28897) in both runs with this branch's scanner (9c4c196: full run; cf23b5a: run stopped at 2204/2326) and in neither the one run with master's scanner nor 130 standalone compiles (60 under 20-way load and 5 alone, with each compiler). The file has no rune literals. Two against zero is not enough to say this change causes it, but it is not ruled out either.

🧙 Built with WOZCODE

[quaesitor-scientiam]

Additional local verification (Windows 11, tcc), comparing a compiler built from this branch against one built from the same tree with master's scanner.v, run one after the other:

v -silent test vlib/v/scanner vlib/v/parser vlib/v/gen/c vlib/v/tests — 2326 test files:

	failed	passed	skipped
this branch	67	2238	21
master scanner.v	66	2239	21

The failure lists are identical except for vlib/v/tests/comptime/comptime_on_generics_func_test.v, which failed once on this branch with a compiler crash in driver__clone_int_string_map under 20 parallel jobs. It contains no rune literals, and run on its own it passes 5 out of 5 times with each compiler.

Notes:

• vlib/v/tests/fns/closure_lifetime_api_test.v hangs on Windows with both compilers (the program it builds with -gc boehm_leak never finishes); it was killed after 5 minutes in both runs.
• The in-module tests under vlib/v/scanner, vlib/v/parser and vlib/v/gen/c compile the scanner from source, so both runs used the fixed scanner there. The only one of them that failed, vlib/v/parser/generic_type_alias_test.v, fails on "generic type aliases are not yet implemented" for a source with no rune literals.
• I did not run the rest of vlib/v/ locally: 250 of the 311 test files in vlib/v/compiler_tests build a compiler from vlib/v/v.v (20–37 CPU-minutes each here), so that part is left to CI.

[JalonSolov]

Local AI Findings:

P1: Invalid UTF-8 sequences are accepted as valid rune literals.

The new continuation check only verifies that continuation bytes are in 0x80..0xBF (vlib/v/scanner/scanner.v:305-323). It does not enforce UTF-8 boundary rules:

\xe0\x80\x80 is overlong
\xed\xa0\x80 encodes a surrogate
\xf0\x80\x80\x80 is overlong
\xf4\x90\x80\x80 exceeds U+10FFFF

All four compile successfully with the PR compiler, despite the PR description stating that overlong and invalid sequences remain rejected. The lead-byte logic needs special-case bounds for E0, ED, F0, and F4, or a shared UTF-8 validator.

The added tests cover \xc0\x80 but miss these structurally valid-looking invalid sequences. The focused scanner tests pass.

[quaesitor-scientiam]

Thanks — confirmed and fixed in cf23b5a. You were right: the merge only checked that each continuation byte was 0x80–0xBF, so all four of your examples compiled, and the PR description's "overlong and invalid sequences are still rejected" was only true for the C0/C1 leads.

The fix decodes the code point and applies the range check_string_escape already enforces for \u/\U (at most U+10FFFF, no surrogates), plus the smallest code point each sequence length may encode, which rules out overlong forms. With leads limited to C2–F4 and continuation bytes to 80–BF, that is exactly Unicode Table 3-7.

The new test checks the rule rather than examples: every lead byte C0–F7 × every boundary of the second-byte ranges × a valid and an invalid final byte, in hex and octal spelling, against encoding.utf8.validate_str. It fails on 9c4c196, and removing any one of the four checks (shortest form, surrogate, maximum, continuation byte) makes it fail on that class. End to end, your four inputs are now rejected, and U+0800 / U+D7FF / U+10FFFF spelled as byte escapes still compile to the right values.

While fixing this I also found that the accept-list test had lost its `\u0061` case (it had been written into the file as a plain a); it is restored, and the PR description is updated.

One thing is still open: #28897. The comptime_on_generics_func_test.v driver crash has now happened in both vlib/v/tests runs with this branch's scanner and in neither the run with master's scanner nor 130 standalone compiles. That is not enough to blame this change, and not enough to rule it out.

[quaesitor-scientiam]

If no further issues can we merge this fix?

[JalonSolov]

Local AI Findings: after most recent changes, no actionable issues found.