reject hex jump and repeat lengths that overflow an int

[munzzyy]

Fixes #1791.

Both pattern lexers parse numbers with atoi, which is undefined for input that
doesn't fit in an int. On glibc you get the low 32 bits back. A value that wraps
to a small positive number then passes every bounds check, and the rule compiles
into something the author did not write.

On master at 604822d:

$ cat over.yar
rule a { strings: $a = /a{4294967296}/ condition: $a }
rule b { strings: $b = /ab{1,4294967297}c/ condition: $b }
rule c { strings: $c = { 61 [0-4294967297] 62 } condition: $c }

$ printf 'abbbbc' > data1
$ ./yara -s over.yar data1
warning: rule "a" in over.yar(1): string "$a" may slow down scanning
warning: rule "c" in over.yar(3): string "$c" may slow down scanning
a data1
0x0:$a:
0x1:$a:
0x2:$a:
0x3:$a:
0x4:$a:
0x5:$a:
c data1
0x0:$c: 61 62

a{4294967296} became a{0} and hit at every offset. The hex jump became
[0-1] and matched ab. Rule b is the quiet one. It doesn't appear above
because b{1,4294967297} became b{1,1}, so it misses abbbbc and matches
abc instead:

$ printf 'abc' > data2
$ ./yara -s b.yar data2
b data2
0x0:$b: abc
$ ./yara -s b.yar data1
$

Nothing warned in any of these cases. Values one order of magnitude smaller are
rejected properly, which is what makes this easy to miss:

$ cat d.yar
rule d { strings: $d = /a{40000}/ condition: $d }
$ ./yara d.yar data1
error: rule "d" in d.yar(1): invalid regular expression "$d": repeat interval too large

The rule from #1791 lands on the other side of the wrap. It does error out, but
with a message describing the truncation rather than the input, because atoi
came back negative:

$ ./yara issue1791.yar data1
error: rule "hex_range" in issue1791.yar(1): invalid hex string "$": invalid negative jump length

MSVC saturates atoi at INT_MAX rather than wrapping, so the same rule file
compiles differently there. That is the platform split reported in the issue.

The fix parses with strtoll in both lexers. Numbers past LLONG_MAX get clamped
there and still fail the range check, so repeat interval too large now fires
for anything above RE_MAX_RANGE no matter what the platform does. Hex jumps get
an explicit INT_MAX check, since RE_NODE holds start and end as ints and the
grammar already casts with $$->start = (int) $2. I dropped the hi_bound < 0
and lo_bound < 0 checks because they only existed to catch atoi wrapping
negative, and the lexer patterns match digits only.

After:

$ ./yara -s over.yar data1
error: rule "a" in over.yar(1): invalid regular expression "$a": repeat interval too large
error: rule "b" in over.yar(2): invalid regular expression "$b": repeat interval too large
error: rule "c" in over.yar(3): invalid hex string "$c": jump length too large

Valid patterns are untouched. { 61 [2-4] 62 }, { 61 [0-] 62 },
{ 61 [0-2147483647] 62 }, /a{1,5}/ and /ab{32767}c/ all still compile, and
/ab{5,2}c/ still gives bad repeat interval. The only inputs whose behaviour
changes are the ones that used to wrap.

Tests go next to the existing a{2977952116} overflow test and the hex string
error tests. Both halves are load bearing, confirmed by reverting one lexer at a
time and rebuilding:

$ git checkout master -- libyara/hex_lexer.l libyara/hex_lexer.c
$ make test-rules && ./test-rules
tests/test-rules.c:1708: expecting error 10 but returned 0

$ git checkout master -- libyara/re_lexer.l libyara/re_lexer.c
$ make test-rules && ./test-rules
tests/test-rules.c:2786: expecting error 9 but returned 0

Full suite on Linux, gcc 16.1.1, configured with --enable-dotnet --enable-magic --enable-cuckoo --disable-shared:

# TOTAL: 19
# PASS:  19
# FAIL:  0

The generated hex_lexer.c and re_lexer.c are checked in, so they are
regenerated here with flex 2.6.4, the same version that produced the current
ones. Both .l files set %option outfile, so regeneration has to go through
the build instead of a direct flex -o call:

$ touch libyara/hex_lexer.l libyara/re_lexer.l
$ make
$ git status --porcelain
$

The checked-in files come back byte for byte identical.

This does not add the INT16_MAX cap on hex jumps that the issue floats as an
option. That would reject rules people are running today, and it is a separate
call from fixing the overflow.