docs: document state-machine tokenizer architecture#13

imrim12 merged 2 commits into

mainimrim12/vietscript:mainfrom

claude/docs-fsm-tokenizerimrim12/vietscript:claude/docs-fsm-tokenizerCopy head branch name to clipboard