Automatically resize chunks to prevent splitting UTF-8 characters

[ManBearTM]

I took a stab at fixing #751 (and maybe the source of a few other issues). You could probably make a similar fix for UTF-16 files, but I opted to not look into that right now.

EDIT: I realize now that this fix maybe only applies to FileStreamer

[dboskovic]

🤔 This logic makes sense but almost definitely adds a major performance hit. It seems the right solution would be to make sure the left over bytes properly re-assemble into valid utf8 bytecode. If you could add a regression test that validates this I'll include the fix in my #1100 refactor

[ManBearTM]

Purely anecdotal, but I tested it with a 50 MB file and did not notice any performance hit. I can try to find the time to add tests, but I can't promise anything - life is a bit busy atm, sorry.

It seems the right solution would be to make sure the left over bytes properly re-assemble into valid utf8 bytecode.

Yeah, this is what it does, but before decoding the bytes. I initially tried to implement a fix AFTER the decoding to UTF-8, but the problem is that the byte "information" is lost when decoded to "�", so there is no way to reverse the broken bytes into its proper form, if that makes sense 🙂