How to handle encoding issues with TEMPFILE contents

[bdbaddog]

Continuation of: #4749 and #4746

[mwichmann]

In TL;DR form I think these are the most notable : (1) encoding the data for the response file could fail, (2) we could be friendlier if it does

[jcbrill]

A strawman implementation is offered for discussion purposes.

Personal note: In the short term, I have very little available time; extreme response latencies likely.

Tempfile encoding failures could be problematic based on where the encoding value is set:

• Inside of SCons, any value in env['TEMPFILEENCODING'] is considered a user value.
• A library (e.g., PlatformIO) may specify the value of env['TEMPFILEENCODING']. The SCons user is PlatformIO, the end-user is using PlatformIO and may be unaware of the tempfile encoding configuration.

Eventually, the availability of TEMPFILEENCODING should probably be pointed out to the PlatformIO project as they are definining their own TempFileMunge configuration:

• https://github.com/platformio/platformio-core/blob/31d630b7660863dde56b17f3a560a1961aced037/platformio/builder/tools/piomaxlen.py#L76-L96
• https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/piomaxlen.py

The strawman implementation:

• Adds a TempFileEncodeError exception type.
• Adds a file-level default encoding value. This allows clients to retrieve the default value and possibly assign it directly via the construction variable (e.g., always assign a value). The default value may be present in encoding exception messages when necessary.
• Moves the tempfile contents encoding before the creation of temporary file (i.e., fail before opening the temporary file).
• Raises a TempFileEncodeError exception as the result of python exceptions UnicodeError, LookupError, and TypeError raised from the encoding value passed to the bytes method. The original python exception text is used in the TempFileEncodeError error message. The source and value of the encoding argument passed to the bytes method is included in the error message.
• Explicitly protects closing the temporary file handle should the write fail.

Not addressed:

• Handling os.remove exceptions in the atexit handler.

Code:

• https://github.com/jcbrill/scons/blob/jbrill-tempfile-encoding/SCons/Platform/__init__.py
• https://github.com/jcbrill/scons/blob/2dae19ce87bdfec5da7fd68af03393d91ac3235a/SCons/Platform/__init__.py#L57-L64
• https://github.com/jcbrill/scons/blob/2dae19ce87bdfec5da7fd68af03393d91ac3235a/SCons/Platform/__init__.py#L150-L156
• https://github.com/jcbrill/scons/blob/2dae19ce87bdfec5da7fd68af03393d91ac3235a/SCons/Platform/__init__.py#L251-L296

Test script:

import sys
import traceback
import SCons.Platform

DefaultEnvironment()

print(f"\nSCons.Platform.TEMPFILE_DEFAULT_ENCODING={SCons.Platform.TEMPFILE_DEFAULT_ENCODING!r}")
print("\nBegin encoding tests...")

command = 'xyz "/test€/file.c"'

for cmd, encoding, skip_encoding, force_default in [

    # expect success
    (command, 'utf-8', False, None),
    (command, 'utf-8-sig', False, None),
    (command, 'utf-16', False, None),

    # expect failure
    (command, 'ascii', False, None),    # UnicodeEncodeError
    (command, 'missing', False, None),  # LookupError
    (command, '', False, None),         # LookupError
    (command, None, False, None),       # TypeError 

    # expect failure
    (command, None, True, 'ascii'),  # default encoding = ascii

    # expect success
    (command, None, True, None),     # default encoding (make sure restored)

]:

    tfm = SCons.Platform.TempFileMunge(cmd)

    env = Environment(
        tools=[],
        MAXLINELENGTH=2,
    )

    if not skip_encoding:
        env['TEMPFILEENCODING'] = encoding

    print(f"\n===== test: cmd={cmd!r}, encoding={encoding!r})")

    if not force_default:
        reset_default_encoding = None
    else:
        reset_default_encoding = SCons.Platform.TEMPFILE_DEFAULT_ENCODING
        SCons.Platform.TEMPFILE_DEFAULT_ENCODING = force_default
        print(f"+++> force SCons.Platform.TEMPFILE_DEFAULT_ENCODING = {SCons.Platform.TEMPFILE_DEFAULT_ENCODING}\n")

    try:
        _ = tfm(None, None, env, 0)
    except SCons.Platform.TempFileEncodeError as exc:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        print(f"{exc_type.__name__}: {str(exc_value)}")
        traceback.print_tb(exc_traceback)
    except Exception as exc:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        print(f"UNEXPECTED {exc_type.__name__}: {str(exc_value)}")
        traceback.print_tb(exc_traceback)

    if reset_default_encoding:
        SCons.Platform.TEMPFILE_DEFAULT_ENCODING = reset_default_encoding
        print(f"\n+++> reset SCons.Platform.TEMPFILE_DEFAULT_ENCODING = {SCons.Platform.TEMPFILE_DEFAULT_ENCODING}")

    print(f"-----")

print("\nEnd encoding tests.\n")

Test output:

Python 3.13.2
scons: Reading SConscript files ...

SCons.Platform.TEMPFILE_DEFAULT_ENCODING='utf-8'

Begin encoding tests...

===== test: cmd='xyz "/test€/file.c"', encoding='utf-8')
Using tempfile S:\SCons\test-cp936\scons-tempfile\tmpjc8cbc4_.lnk for command line:
xyz "/test€/file.c"
-----

===== test: cmd='xyz "/test€/file.c"', encoding='utf-8-sig')
Using tempfile S:\SCons\test-cp936\scons-tempfile\tmp5gl2gu5h.lnk for command line:
xyz "/test€/file.c"
-----

===== test: cmd='xyz "/test€/file.c"', encoding='utf-16')
Using tempfile S:\SCons\test-cp936\scons-tempfile\tmpqgyacnkb.lnk for command line:
xyz "/test€/file.c"
-----

===== test: cmd='xyz "/test€/file.c"', encoding='ascii')
TempFileEncodeError: [UnicodeEncodeError] 'ascii' codec can't encode character '\u20ac' in position 6: ordinal not in range(128)
  TempFileMunge encoding: env['TEMPFILEENCODING'] = 'ascii'
  File "S:\SCons\test-cp936\scons-tempfile\SConstruct-scons", line 53, in <module>
    _ = tfm(None, None, env, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None
-----

===== test: cmd='xyz "/test€/file.c"', encoding='missing')
TempFileEncodeError: [LookupError] unknown encoding: missing
  TempFileMunge encoding: env['TEMPFILEENCODING'] = 'missing'
  File "S:\SCons\test-cp936\scons-tempfile\SConstruct-scons", line 53, in <module>
    _ = tfm(None, None, env, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None
-----

===== test: cmd='xyz "/test€/file.c"', encoding='')
TempFileEncodeError: [LookupError] unknown encoding: 
  TempFileMunge encoding: env['TEMPFILEENCODING'] = ''
  File "S:\SCons\test-cp936\scons-tempfile\SConstruct-scons", line 53, in <module>
    _ = tfm(None, None, env, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None
-----

===== test: cmd='xyz "/test€/file.c"', encoding=None)
TempFileEncodeError: [TypeError] bytes() argument 'encoding' must be str, not None
  TempFileMunge encoding: env['TEMPFILEENCODING'] = None
  File "S:\SCons\test-cp936\scons-tempfile\SConstruct-scons", line 53, in <module>
    _ = tfm(None, None, env, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None
-----

===== test: cmd='xyz "/test€/file.c"', encoding=None)
+++> force SCons.Platform.TEMPFILE_DEFAULT_ENCODING = ascii

TempFileEncodeError: [UnicodeEncodeError] 'ascii' codec can't encode character '\u20ac' in position 6: ordinal not in range(128)
  TempFileMunge encoding: default = 'ascii'
  File "S:\SCons\test-cp936\scons-tempfile\SConstruct-scons", line 53, in <module>
    _ = tfm(None, None, env, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None

+++> reset SCons.Platform.TEMPFILE_DEFAULT_ENCODING = utf-8
-----

===== test: cmd='xyz "/test€/file.c"', encoding=None)
Using tempfile S:\SCons\test-cp936\scons-tempfile\tmpsj_upu23.lnk for command line:
xyz "/test€/file.c"
-----

End encoding tests.

scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

Example msvc build error:

scons: Building targets ...
scons: *** [src\main.obj] TempFileEncodeError : [UnicodeEncodeError] 'charmap' codec can't encode characters in position 95-96: character maps to <undefined>
  TempFileMunge encoding: env['TEMPFILEENCODING'] = 'cp1252'
Traceback (most recent call last):
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Taskmaster\__init__.py", line 246, in execute
    self.targets[0].build()
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Node\__init__.py", line 774, in build
    self.get_executor()(self, **kw)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Executor.py", line 386, in __call__
    return _do_execute_map[self._do_execute](self, target, kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Executor.py", line 119, in execute_action_list
    status = act(*args, **kw)
             ^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Action.py", line 1273, in __call__
    return act(
           ^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Action.py", line 1342, in __call__
    return c.__call__(self, target, source, env, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Action.py", line 706, in __call__
    cmd = self.strfunction(target, source, env, executor)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Action.py", line 1037, in strfunction
    cmd_list, ignore, silent = self.process(target, source, env, executor, overrides=overrides)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Action.py", line 1008, in process
    result = env.subst_list(self.cmd_list, SUBST_CMD, executor=executor, overrides=overrides)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Environment.py", line 741, in subst_list
    return SCons.Subst.scons_subst_list(string, self, raw, target, source, gvars, lvars, conv, overrides=overrides)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 932, in scons_subst_list
    ls.substitute(strSubst, lvars, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 645, in substitute
    self.expand(a, lvars, within_list)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 593, in expand
    self.substitute(s, lv, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 645, in substitute
    self.expand(a, lvars, within_list)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 593, in expand
    self.substitute(s, lv, 0)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 647, in substitute
    self.expand(args, lvars, within_list)
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Subst.py", line 608, in expand
    s = s(target=lvars['TARGETS'],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "S:\SCons\jbrill-tempfile-encoding\scripts\..\SCons\Platform\__init__.py", line 272, in __call__
    raise TempFileEncodeError(err_msg) from None
SCons.Platform.TempFileEncodeError: [UnicodeEncodeError] 'charmap' codec can't encode characters in position 95-96: character maps to <undefined>
  TempFileMunge encoding: env['TEMPFILEENCODING'] = 'cp1252'
scons: building terminated because of errors.

Revisions:

• 2025-08-06 11:00 EST: Added TempFileEncodeError exception type and streamlined encoding exception handling.
• 2025-08-06 11:17 EST: Removed comments that no longer apply.

[jcbrill]

Streamlined strawman implementation to minimize changes and raise TempFileEncodeError exception type when bytes function fails (see revised comment above: #4752 (comment))