Allow to set non-UTF8 exception messages

[picnixz]

Feature or enhancement

Proposal:

This is a follow-up to the discussion in #126555 (comment).

dlerror() may return non-UTF-8 messages, possibly translated. We should be able to set exception messages according to the current locale. To that end, we'll expose some internal helper:

extern void 
_PyErr_SetLocaleStringTstate(PyThreadState *tstate, PyObject *exception, const char *string);

extern void
_PyErr_SetLocaleString(PyObject *exception, const char *string);

cc @ZeroIntensity @encukou

For now, both functions would be only declared as extern and not part of the public C API.

Linked PRs

• gh-126742: allow to use non-UTF8 exception messages #126746

[picnixz]

Let's actually mark it as a bugfix since we'll need to backport it in order to use it later.

[ZeroIntensity]

Hmm, so is PyErr_SetLocaleString supposed to be implicitly private? I'm not the biggest fan of that--if something is supposed to be private, then we need to prefix it with _, not only so users don't touch it, but so people reading the source don't mistake it as public. If we want a variation of it that takes a thread state, we could have something like _PyErr_SetLocaleStringTstate.

[picnixz]

Hmm, so is PyErr_SetLocaleString supposed to be implicitly private

For now, but maybe we can directly expose to the C API? I want to be consistent with the naming of each function.

[picnixz]

Nevermind, there is a precedent: _PyErr_FormatFromCauseTstate. Let's have
_PyErr_SetLocaleStringTstate and _PyErr_SetLocaleString and both private.

[encukou]

How many places in the code would use this helper? I don't see the advantage over calling PyUnicode_DecodeLocale and PyErr_SetObject.

[picnixz]

There are a few occurrences of dlerror() in _ctypes.c and callproc.c and dynload_shlib as well.

NVM: the dynload_shlib uses its own logic.

[encukou]

errors.c uses PyUnicode_DecodeLocale (with surrogateescape) on strerror()'s result. That looks like a good precedent.

I don't think it's worth it to add a project-wide helper here. Calling the 2 functions should be fine.

[ZeroIntensity]

My main arguments for adding a new internal function would be that a) it helps remove the risk of reference leaks (which in my experience, are all too common when dealing with string mutations) and b) makes it easier to deal with locale in the future (we have incorrect uses of dlerror() internally that skip the locale check).

[encukou]

As far as I know, this is not a bug on any supported platforms, and so can't be tested in our CI. It's also not been reported/patched on any other platform I know of.

I don't know if it should be called a “bug” (that's more a question about the definition of the term), but I don't think a fix for this issue should be backported. Let's just improve main.

As for the helper: IMO, it's better to first fix the code, and then, if it makes sense, refactor to add a helper. In clear-cut cases you'd combine the steps in one commit, but let's not do that here.

[ZeroIntensity]

As far as I know, this is not a bug on any supported platforms, and so can't be tested in our CI. It's also not been reported/patched on any other platform I know of.

FWIW, I did find BPO-41894 when looking through the places where we use PyUnicode_DecodeLocale.