"UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 393: invalid continuation byte" from codecs.py

[hfuchiue-rocket]

Test environment
We use pytest v8.0.2 with Python 3.9.13 environment on Ubuntu 18.04 Docker image.

Here is a part of metadata from pytest output:
platform linux -- Python 3.9.13, pytest-8.0.2, pluggy-1.5.0 -- /home/autouser/venv/bin/python metadata: {'Python': '3.9.13', 'Platform': 'Linux-3.10.0-957.21.3.el7.x86_64-x86_64-with-glibc2.27', 'Packages': {'pytest': '8.0.2', 'pluggy': '1.5.0'}, 'Plugins': {'variables': '3.1.0', 'forked': '1.6.0', 'timeout': '2.3.1', 'parallel': '0.1.1', 'json-report': '1.5.0', 'json': '0.4.0', 'metadata': '3.1.1', 'xdist': '1.24.1', 'html': '4.1.1', 'bdd': '7.2.0', 'ordering': '0.6', 'instafail': '0.4.1'}

Here is the output from pip list:

Package            Version
------------------ ----------
bcrypt             4.2.0
certifi            2024.7.4
cffi               1.17.0
charset-normalizer 3.3.2
cryptography       43.0.0
decorator          5.1.1
defusedxml         0.7.1
Deprecated         1.2.14
distlib            0.3.8
ebcdic             1.0.0
exceptiongroup     1.2.2
execnet            2.1.1
fabric             3.2.2
filelock           3.15.4
idna               3.7
iniconfig          2.0.0
invoke             2.2.0
JayDeBeApi         1.2.3
Jinja2             3.1.4
jira               3.8.0
JPype1             1.5.0
Mako               1.3.5
MarkupSafe         2.1.5
oauthlib           3.2.2
packaging          24.1
paramiko           3.4.0
parse              1.20.2
parse-type         0.6.2
pillow             10.4.0
pip                24.1.2
pluggy             1.5.0
plumbum            1.8.3
py                 1.11.0
pycparser          2.22
PyNaCl             1.5.0
pyodbc             5.1.0
pyspnego           0.11.1
pytest             8.0.2
pytest-bdd         7.2.0
pytest-forked      1.6.0
pytest-html        4.1.1
pytest-json        0.4.0
pytest-json-report 1.5.0
pytest-metadata    3.1.1
pytest-ordering    0.6
pytest-parallel    0.1.1
pytest-timeout     2.3.1
pytest-variables   3.1.0
pytest-xdist       1.24.1
pywinrm            0.5.0
requests           2.32.3
requests_ntlm      1.3.0
requests-oauthlib  2.0.0
requests-toolbelt  1.0.0
selenium           3.141.0
setuptools         70.1.0
six                1.16.0
taf                1!2024.181
taf-zephyr         1.2.2
tblib              3.0.0
tomli              2.0.1
typing_extensions  4.12.2
urllib3            1.26.13
wheel              0.43.0
winapps            0.2.0
wrapt              1.16.0
xmltodict          0.13.0

Test pytest command
We run the testcases with the following command:
pytest --skip-library-install --instafail -rs --tb=short -m "xxxx" --timeout 86400 --disable-pytest-warnings --log-level INFO

Issue
During testing, pytest issues INTERNALERROR as follows:

10:50:36 INTERNALERROR> Traceback (most recent call last):
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/main.py", line 273, in wrap_session
10:50:36 INTERNALERROR>     session.exitstatus = doit(config, session) or 0
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/main.py", line 327, in _main
10:50:36 INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
10:50:36 INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
10:50:36 INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 139, in _multicall
10:50:36 INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 122, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
10:50:36 INTERNALERROR>     return (yield)  # Run all the tests.
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
10:50:36 INTERNALERROR>     res = hook_impl.function(*args)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/main.py", line 352, in pytest_runtestloop
10:50:36 INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
10:50:36 INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
10:50:36 INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
10:50:36 INTERNALERROR>     return outcome.get_result()
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
10:50:36 INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(outcome._exception)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/warnings.py", line 109, in pytest_runtest_protocol
10:50:36 INTERNALERROR>     return (yield)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(outcome._exception)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/assertion/__init__.py", line 175, in pytest_runtest_protocol
10:50:36 INTERNALERROR>     return (yield)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(outcome._exception)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/unittest.py", line 411, in pytest_runtest_protocol
10:50:36 INTERNALERROR>     res = yield
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(outcome._exception)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/faulthandler.py", line 85, in pytest_runtest_protocol
10:50:36 INTERNALERROR>     return (yield)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
10:50:36 INTERNALERROR>     res = hook_impl.function(*args)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/runner.py", line 115, in pytest_runtest_protocol
10:50:36 INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/runner.py", line 134, in runtestprotocol
10:50:36 INTERNALERROR>     reports.append(call_and_report(item, "call", log))
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/runner.py", line 225, in call_and_report
10:50:36 INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
10:50:36 INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
10:50:36 INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 156, in _multicall
10:50:36 INTERNALERROR>     teardown[0].send(outcome)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pytest_bdd/plugin.py", line 87, in pytest_runtest_makereport
10:50:36 INTERNALERROR>     reporting.runtest_makereport(item, call, outcome.get_result())
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
10:50:36 INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 167, in _multicall
10:50:36 INTERNALERROR>     teardown.throw(outcome._exception)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_makereport
10:50:36 INTERNALERROR>     rep = yield
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
10:50:36 INTERNALERROR>     res = hook_impl.function(*args)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/runner.py", line 369, in pytest_runtest_makereport
10:50:36 INTERNALERROR>     return TestReport.from_item_and_call(item, call)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/reports.py", line 363, in from_item_and_call
10:50:36 INTERNALERROR>     longrepr = item.repr_failure(excinfo)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/python.py", line 1874, in repr_failure
10:50:36 INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/nodes.py", line 496, in _repr_failure_py
10:50:36 INTERNALERROR>     return excinfo.getrepr(
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 698, in getrepr
10:50:36 INTERNALERROR>     return fmt.repr_excinfo(self)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 1063, in repr_excinfo
10:50:36 INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo_)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 992, in repr_traceback
10:50:36 INTERNALERROR>     entries = [
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 993, in <listcomp>
10:50:36 INTERNALERROR>     self.repr_traceback_entry(entry, excinfo if last == entry else None)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 939, in repr_traceback_entry
10:50:36 INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 873, in get_source
10:50:36 INTERNALERROR>     lines.extend(self.get_exconly(excinfo, indent=indent, markall=True))
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 885, in get_exconly
10:50:36 INTERNALERROR>     exlines = excinfo.exconly(tryshort=True).split("\n")
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 602, in exconly
10:50:36 INTERNALERROR>     lines = format_exception_only(self.type, self.value)
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/traceback.py", line 140, in format_exception_only
10:50:36 INTERNALERROR>     return list(TracebackException(etype, value, None).format_exception_only())
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 179, in __init__
10:50:36 INTERNALERROR>     context = PatchedTracebackException(
10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/exceptiongroup/_formatting.py", line 96, in __init__
10:50:36 INTERNALERROR>     self.stack = traceback.StackSummary.extract(
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/traceback.py", line 366, in extract
10:50:36 INTERNALERROR>     f.line
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/traceback.py", line 288, in line
10:50:36 INTERNALERROR>     self._line = linecache.getline(self.filename, self.lineno).strip()
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/linecache.py", line 30, in getline
10:50:36 INTERNALERROR>     lines = getlines(filename, module_globals)
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/linecache.py", line 46, in getlines
10:50:36 INTERNALERROR>     return updatecache(filename, module_globals)
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/linecache.py", line 137, in updatecache
10:50:36 INTERNALERROR>     lines = fp.readlines()
10:50:36 INTERNALERROR>   File "/usr/local/lib/python3.9/codecs.py", line 322, in decode
10:50:36 INTERNALERROR>     (result, consumed) = self._buffer_decode(data, self.errors, final)
10:50:36 INTERNALERROR> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 393: invalid continuation byte

We have no idea which part of our test cases caused this issue. We are wondering if this is a bug from pytest?

[pysifu]

Same issue on pytest-django

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Projects\Trade_IT\.venv\Scripts\pytest.exe\__main__.py", line 7, in <module>
    sys.exit(console_main())
             ~~~~~~~~~~~~^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 201, in console_main
    code = main()
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 156, in main
    config = _prepareconfig(args, plugins)
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 341, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
        pluginmanager=pluginmanager, args=args
    )
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\pluggy\_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\pluggy\_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\pluggy\_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\pluggy\_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ~~~~~~~~~~~~~~^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\pluggy\_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 1140, in pytest_cmdline_parse
    self.parse(args)
    ~~~~~~~~~~^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 1494, in parse
    self._preparse(args, addopts=addopts)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 1366, in _preparse
    self._initini(args)
    ~~~~~~~~~~~~~^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\__init__.py", line 1239, in _initini
    rootpath, inipath, inicfg = determine_setup(
                                ~~~~~~~~~~~~~~~^
        inifile=ns.inifilename,
        ^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        invocation_dir=self.invocation_params.dir,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\findpaths.py", line 198, in determine_setup
    rootdir, inipath, inicfg = locate_config(invocation_dir, [ancestor])
                               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\findpaths.py", line 110, in locate_config
    ini_config = load_config_dict_from_file(p)
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\findpaths.py", line 39, in load_config_dict_from_file
    iniconfig = _parse_ini_config(filepath)
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\_pytest\config\findpaths.py", line 25, in _parse_ini_config
    return iniconfig.IniConfig(str(path))
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "D:\Projects\Trade_IT\.venv\Lib\site-packages\iniconfig\__init__.py", line 118, in __init__
    data = fp.read()
  File "<frozen codecs>", line 325, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

[hfuchiue-rocket]

We used —log-level command line option but once we removed this, the issue is not happened again. I’m not sure why logger caused the issue…

[Zac-HD]

Without a reproducing example, unfortunately we're unlikely to be able to help.

[pysifu]

Issue was due to current working directory. I had to move back to previous one.

[Zac-HD]

Unfortunately I can't help much, without some sample code that I can run to see the same problem (and then run again to confirm that a fix makes the problem stop). https://stackoverflow.com/help/minimal-reproducible-example explains the idea style.

I suspect that this is because you have a file which is not in utf-8 encoding - that wouldn't be pytest's fault but we might be able to improve the error message.

[pysifu]

No, encoding was fine. Case is that I have to run pytest from outside django project directory.

[Zac-HD]

Telling me that does not help; I need sample code that I can execute to see the problem for myself.

[pysifu]

Sorry, my english is not good. I solved the problem, this is solution: "I have to run pytest from outside django project directory.". Now is everything alright.

[pysifu]

Thank you so much for trying to help me!

[hfuchiue-rocket]

I removed --log-level and it is working fine now.

I suspect that this is because you have a file which is not in utf-8 encoding - that wouldn't be pytest's fault but we might be able to improve the error message.

My pytest script calls another script via importlib.machinery.SourceFileLoader. Let's call pytest script test_A and original script (callee) testcase_A. testcase_A contains non-UTF-8 characters so I'm sure this is the issue. But, I'm not sure why pytest issues this error because test_A does not contains such characters.

So, in the following error message:

10:50:36 INTERNALERROR>   File "/home/autouser/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 939, in repr_traceback_entry
10:50:36 INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)

It seems pytest tries to capture the whole source code, but test_A does not contain the original testcase.

My observation might be incorrect but just in case leave it as it is for now.

[Zac-HD]

It looks like we might have a pytest issue when trying to construct a custom traceback repr involving source lines from a file which is not utf-8 encoded.

[hfuchiue-rocket]

I tried to reproduce the issue with simple codes but couldn't do that... In case of someone reporting the smilar issue, I will keep this open.