Fix tokenization of qualified identifiers with numeric prefix.

[romanb]

This PR is a follow-up to #856. The remaining problem is that queries with qualified identifiers having numeric prefixes currently fail to parse due to incorrect tokenization. For example:

SELECT t.123abc FROM my_table t 

This is currently tokenized as

...
t (Word)
.123abc (Word)
...

whereas it should be tokenized as

...
t (Word)
. (Period)
123abc (Word)
...

Of course, the potential ambiguity of identifiers of the form 12e34, i.e. that on their own could be seen as number tokens, also needs to be taken into account. If 12e34 is unqualified, it should be tokenized as a number (this is already the case) but in SELECT t.12e34 FROM my_table t, it should be tokenized as a word as well, to be able to successfully parse it as a compound identifier in MySQL.

The only option I saw to solve these problems unambiguously was to give the private next_token function in the Tokenizer as context a reference to the previous token in the second argument, which can then be used to disambiguate these cases and correctly decide what type of token to produce for dialects that support numeric prefixes. I included commentary and tests to help further clarify the situation.

[iffyio]

Suggested change

	mysql().verified_stmt("SELECT t.15to29 FROM my_table AS t");
	match mysql()
	.parse_sql_statements("SELECT t.15to29 FROM my_table AS t")
	.unwrap()
	.pop()
	{
	match mysql().verified_stmt("SELECT t.15to29 FROM my_table AS t") {

does this format work the same to remove the second parse statement?

[romanb]

Yes, that simplifies the test, I was simply ignorant of the return value. Thanks for the suggestion.

[iffyio]

Can we implement instead in the parser as a special for self.dialect.supports_numeric_prefix()? somewhat similar to what we do for BigQuery here. In that when its time to combine identifiers into a compound identifier we check if each subsequent identifier part is unquoted and prefixed by ., and if so we drop the prefix. I imagine that would be done here

Thinking that could be a smaller change since the behavior needs a bit of extra context around the tokens which the tokenizer isn't so good at handling cleanly

[romanb]

Thank you for the review and the suggestion. The latest commit implements an alternative approach within Parser#parse_compound_expr. However, I have to admit that I personally prefer the previous fix in the tokenizer, for a few reasons:

1. The Tokenizer is part of the public API of the crate, and without fixing this in the tokenizer, it will continue to exhibit the following behavior with the Mysql dialect:

   • t.1to2 tokenizes to t (Word), .1to2 (Word)

   • t.1e2 tokenizes to t (Word), .1e2 (Number)

   • t.`1to2` tokenizes to t (Word), . (Period), 1to2 (Word)

   • t.`1e2` tokenizes to t (Word), . (Period) 1e2 (Word)

     This could be very surprising for users and arguably be considered incorrect (when using the Mysql dialect).

2. The handling of Word and Number tokens in parse_compound_expr is not the most elegant with having to split off the . and having to create the correct off-by-one spans accordingly.

3. Unqualified identifiers that start with digits are already handled in the tokenizer - and I think have to be (see Support identifiers beginning with digits in MySQL #856). Handling the same problem of misinterpreted tokens for qualified identifiers in the parser seems a bit disconnected and a like a downstream solution to the tokenizer producing incorrect tokens, at least that is how I currently perceive it (see point 1).

Let me know which solution you prefer (with or without the latest commit) or if you have another idea.

[iffyio]

I think that's fair, we can revert back to the tokenizer version in that case? to keep the tokenizer behavior non-surprising

[romanb]

Done!

[iffyio]

could we add a test case for the behavior of multiple accesses e.g. t.15to29.16to30?

[romanb]

Good idea, I added such a test.

[iffyio]

LGTM! Thanks @romanb!
cc @alamb