Open
Description
As it was pointed out in #290 (comment), kotlinx-io converts different ill-formed UTF-8 subsequences differently: either the whole multi-code-point subsequence replaced with a single replacement character, or each code points is converted separately:
0xf0 0x89 0x89 <EOF>->�0xf0 0x89 0x89 0x89 <EOF>->�0xf0 0xf0 0xf0 <EOF>->���
The UTF-8 spec allows handling these ill-formed sequences whatever way we want as long as errors are somehow reported. However, such behavior looks a bit inconsistent and it's hard to reason about how an arbitrary byte sequences will be converted.
We should improve the way ill-formed sequences are handled and stick to an approach adopted by other languages/libraries: convert only ill-formed subsequences consisting of a single byte.
That's how it's done in:
- Java:
jshell> new String(new byte[]{(byte)0xf0,(byte)0x89,(byte)0x89,(byte)0x89})
$5 ==> "����"
- Python 3:
>>> b'\xf0\x89\x89\x89'.decode("utf-8", errors='replace')
'����
- Go:
fmt.Println(string([]byte{0xf0, 0x89, 0x89, 0x89}))
...
����