gfx_RotateScaleSprite breaks when scale >= 128

[ZERICO2005]

both gfx_RotateScaleSprite and gfx_RotatedScaledSprite_NoClip break when scale is >= 128 (>= 200%). The documentation says that range is about 1% to 400% scale or 1 to 255.

When the scale is >= 128, the sprite is rotated in the opposite direction + 180 degrees, and will start decreasing in size.

Shown below is the scale sweeping from 4 to 255:

I've been able to fix the rotation when scale is >= 128. But fixing the scaling for a scale of [129, 255] is more complicated as it is probably an 8bit/16bit overflow/underflow.

[ZERICO2005]

With larger sprites, a different bug occurs when sprite_width * scale exceeds 16384, preventing (sprite_width * sprite_height / 64) from fitting inside 8 bits. This makes it so only the top-left corner of the sprite is sampled.

This bug doesn't occur for the example shown above, since 48 * 255 is less than 16384.

[ZERICO2005]

Everything has been fixed except for the case that the output is 256x256 or larger. This is not worth fixing however, since a 256x256 sprite doesn't fit onto a 320x240 screen anyways, and would require changing 8 bit counters to be 9 or 16 bits.