utf-8 in log messages breaks qemu's twister test

[MarcFinetRtone]

Describe the bug

I was writing tests for twister, that passed successfully on native-sim, however when running integration tests within qemu, the test fails with unexpected byte:

FAILED unexpected byte (qemu 0.008s )

My log strings have unicode codepoints (in utf-8) such as δ, which is U+03b4 and encoded as 0xce 0xb4 in utf-8. The 0xce cannot be decoded as a utf-8 string.

The problem comes from QEMUHandler that decodes qemu's output byte by byte, and expect to have correct utf-8 (https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/pylib/twister/twisterlib/handlers.py#L986):

                    c = in_fp.read(1).decode("utf-8")

This code comes from 08ce5a5 (that massively changed the string handling due to python3 support) and from 2968b36, titled "non utf-8 characters are tolerated", but in practice, it stops the test.

I can work on a fix and I intend to mimic other handlers behavior, that is:

• do the utf-8 decoding on full string (i.e. after b'\n')
• 'ignore' decoding errors

That will change the behavior, i.e. no longer detect "weird" output from qemu, but I do not know which case it was addressing (i.e. in which case we need to abort the test on non utf-8 string). I do not know if someone still has access to the Jira issue ZEP-673.

The culprit code has been copy/pasted w/o refactor on 3bf6152 in QEMUWinHandler. I do not know if refactoring is planned (cc @michal-smola), either qemu handlers only or all the handlers (since I think that the output handling could be factorized (and is currently inconsistent: strip_ansi_sequences(), encode()…) but it's probably an other issue).

Regarding the tests (which have not been done for QEMUWinHandler), I'm a bit lost (cc @LukaszMrugala):

• should I just remove the unsupported byte occurrences ?
• I don't know how to add a 'valid' utf-8 string decoding; TESTDATA_25 passed test does not mention the last string 5\n 🤔

Regression

• This is a regression.

Steps to reproduce

Add unicode codepint (in utf-8) in log strings and run integration test w/ qemu (e.g. qemu_cortex_m3)

Relevant log output

INFO    - 1/2 qemu_cortex_m3/ti_lm3s6965 data                                           FAILED unexpected byte (qemu 0.010s <zephyr>)
INFO    -                                    data.two_packet_lost                                           BLOCKED      unexpected byte
INFO    -                                    data.one_packet_lost                                           BLOCKED      unexpected byte
INFO    -                                    data.simple                                         BLOCKED      unexpected byte

Impact

Functional Limitation – Some features not working as expected, but system usable.

Environment

• linux debian 12
• zephyr 4.1.0 (7823374)
• 'zephyr' toolchain (0.16.0)

Additional Context

No response