This bonus notebook implements a whitespace-preserving variation of the chapter's SimpleTokenizerV2. In hindsight, this SimpleTokenizerV3 here may be a more intuitive simple tokenizer for the main chapter. However, it is kept separate from the chapter notebook so the book's main code to avoid diverging from the actual book.
- simple-tokenizer-v3.ipynb: a simple tokenizer that preserves delimiter and whitespace round trips