Fix tokenizer EOF handling for trailing delimiter tokens

[DPS0340]

Summary

Follow-up from CodeRabbit nitpick review on PR #231.

create_index_without_semicolon documents an existing tokenizer EOF workaround: the tokenizer can drop the final ) token near EOF because is_eof() only checks buffer_index and the identifier/read loop does not always push the terminating character back.

Proposed work

• Fix tokenizer EOF handling so tokens immediately before EOF are not lost.
• Consider restoring the terminating character with unread_char() before ending tokenization.
• Keep parser tests focused on parser behavior; add tokenizer-level regression tests for inputs ending with ) and no semicolon.

Acceptance criteria

• CREATE INDEX idx ON tbl (col) parses without parser-side workaround comments.
• Tokenizer tests cover EOF after right parenthesis and other delimiter tokens.
• Existing parser tests pass.

Related: PR #231 review pullrequestreview-4651936004.