Revise escape sequence scanning #126
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@mmatera There is still a lot to do regarding writing tests and ensuring we handle errors and everything correctly. But since this is getting large, this is a heads-up as to what's on the horizon. |
Escape sequences other than named characters have been removed from the prescanner and put in the scanner.
handle syntax errors in mathics3-tokens.
Tokenizer.code -> Tokenizer.source_text Tokenizer.incomplete -> Tokenizer.get_more_input
Start to show syntax errors.
In particular errors with octal digits and incomplete named errors. Go over docstrings in escape_sequences.py
and add more tests.
named-characters.yml: \[Mu] is letterlike tokeniser.py: Correct identifier or pattern for those having letterlike escape sequences
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
feeb068 to
d1d409f
Compare
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
5a0ec02 to
a5b13d0
Compare
rocky
force-pushed
the
revise-escape-sequence-scanning
branch
from
a5b13d0 to
1ab77fa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor the scanner. Remove prescanner. Handling escape sequences is not a separate phase, but instead is integrated into the scanning phase.