Skip to content

gh-153569: consolidate tokenizer reader ownership and source views - #138

Closed
pablogsal wants to merge 2 commits into
gh-153569-tokenizer-validation-tools-cleanfrom
gh-153569-tokenizer-reader-state
Closed

gh-153569: consolidate tokenizer reader ownership and source views#138
pablogsal wants to merge 2 commits into
gh-153569-tokenizer-validation-tools-cleanfrom
gh-153569-tokenizer-reader-state

Conversation

@pablogsal

Copy link
Copy Markdown
Owner

Move input-specific state, buffer relocation, and retained-source ownership into the common reader. Borrow diagnostic lines through the source API, including its line-clamping rules, instead of walking storage in pegen.

Depends on python#156484.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant