frontend: parse every error message to utf-8, ignore non-parsable chars

[nikromen]

If user sends to API some weird non-utf-8 character and we then
try to log this, the logging freaks out with logging error (see [1]).
[2] fixes this, but only for one endpoint and one type of user input,
but there are many ways how user can do this. On the other hand, [2]
caused bad formating and now we got messages like b'Project a/b does not exist'.

This patch tries to parse every error message we want to store, and maybe
later log it, to utf-8 and store it as a string. If some non-utf-8
character appears in the message output, its byte won't be pares and it
will be stored/printed as the string representation of that byte.

Example:

User calls https://copr.fedorainfracloud.org/coprs/g/some<nasty_characters>string

the string value in CoprHttpException.message will be some\x01\x02string

where the \x01\x02 is example of the <nasty_characters>

[1] - #3072
[2] - #3076

Fix #3644

[github-actions]

Pull Request validation

Failed

🔴 Review - Missing review from a member (2 required)

Success

🟢 CI - All checks have passed

[nikromen]

I updated the parsing - it doesn't store the bytes, but it rather parses the string right away so we don't work with bytes. I also updated the commit message/PR description, PTAL

[FrostyX]

Strangely enough, when I use code from main and try this URL:

http://127.0.0.1:5000/coprs/frostyx/nvid…

The error on the page is:

Error 404: Page Not Found
b'Project frostyx/nvid\xc3\xa2\xe2\x82\xac\xc2\xa6 does not exist.' 

When I simply revert the code from PR #3076, I get:

Error 404: Page Not Found
Project frostyx/nvid… does not exist. 

and not the original #3072. So either we can simply revert it, or I need to find a better reproducer for the original issue.

[nikromen]

aha, it is true that utf-8 characters like "…" work now, you are right.

I've been researching why this is, and I've concluded that we must have had the locale set to US.US-ASCII or something similar on the old F39 machines at the time. Now we have US.UTF-8. That's probably why this error used to occur, because logging uses by default io.text_encoding, which defaults to Python's UTF-8 mode and returns either locale or UTF-8 if the locale is not set. In that case it was trying to log to ASCII instead of UTF-8.

In any case, it just reduced the set of characters on which this error occurs. Non-UTF-8 characters (such as UTF-16) will fall with the same error.

So there are two possibilities:

• accept this PR as a solution that reverses convert eror message to binary text before emitting #3076 while outputting all non-UTF-8 characters as a pure string of their byte value
• remove the parsing part from this PR and just revert convert eror message to binary text before emitting #3076 . If someone will send us requests with non-utf8 characters and we get errors, then do the parsing afterwards.

what do you think about it?

[praiskup]

The old reproducer was:

b'Project t0xic0der/nvid\xe2\x80\xa6 does not exist.'

That is in utf:

'Project t0xic0der/nvid… does not exist.'

This doesn't fail the logger on F41 (current prod). It is ugly, yes, the logging code still dumps the message using the \xe2\x80... instead of \u2026, not sure why (does logging want ascii? if yes, where/why? because they don't want to ignore the encoding errors?)

But at least it doesn't try to encode elpisis into ASCII.

When I simply revert the code from PR #3076, I get:

Yes, please, revert. I tested with str=$(dd if=/dev/urandom bs=8 count=1) and I got:

[remote 77.92.220.242:48662] ERROR:coprs:Response error: 404 Project frostyx/\xef\xbf\xbd|\xef\xbf\xbd\xef\xbf\xbd|\xef\xbf\xbdO\x02 does not exist.

We should be safe; no tracebacks. But if we miss something, we'll at least get the reproducer.

[nikromen]

not sure why (does logging want ascii? if yes, where/why? because they don't want to ignore the encoding errors?)

I've concluded that we must have had the locale set to US.US-ASCII or something similar on the old F39 machines at the time. Now we have US.UTF-8. That's probably why this error used to occur, because logging uses by default io.text_encoding, which defaults to Python's UTF-8 mode and returns either locale or UTF-8 if the locale is not set. In that case it was trying to log to ASCII instead of UTF-8.

We should be safe; no tracebacks. But if we miss something, we'll at least get the reproducer.

this will still mean that we got logging error when someone sends e.g. utf32 only characters