Unclear how to support subclasses of pathlib.Path

[ofek]

They don't work out-of-the-box and I'm uncertain how to get them to work at all without using lazy imports everywhere or defining the class itself within a function.

[mrbean-bremen]

Can you elaborate a bit what you are trying to do and what you are expecting?

I guess that this is just not supported out of the box, given that pyfakefs relies on module and function names, but maybe we can add something to support some scenarios.

[ofek]

It appears to be broken only sometimes, in the following example you can see that the touch method works properly but write_text causes infinite recursion:

import pathlib
import sys
from functools import cached_property

if sys.platform == 'win32':
    _PathBase = pathlib.WindowsPath
else:
    _PathBase = pathlib.PosixPath

class Path(_PathBase):
    @cached_property
    def long_id(self):
        from base64 import urlsafe_b64encode
        from hashlib import sha256

        path = str(self)
        if sys.platform == 'win32' or sys.platform == 'darwin':
            path = path.casefold()

        digest = sha256(path.encode('utf-8')).digest()
        return urlsafe_b64encode(digest).decode('utf-8')

def create_with_touch(path):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.touch()

def create_with_write_text(path):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text('', encoding='utf-8')

import pytest

from p import Path, create_with_touch, create_with_write_text

@pytest.mark.usefixtures("fs")
def test_create_with_touch():
    path = Path.home() / 'Desktop' / 'test_touch.txt'
    create_with_touch(path)
    assert path.exists()
    assert path.is_file()

@pytest.mark.usefixtures("fs")
def test_create_with_write_text():
    path = Path.home() / 'Desktop' / 'test_write_text.txt'
    create_with_write_text(path)
    assert path.exists()
    assert path.is_file()

[mrbean-bremen]

Which os and Python version are you using? With Python 3.13 under Windows I could not reproduce this.

[ofek]

Windows 11

❯ python --version; pip --version; pytest --version
Python 3.12.6
pip 24.2 from C:\Users\ofek\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip (python 3.12)
pytest 8.3.3

[mrbean-bremen]

Hm, that's odd. Under Windows, with Python 3.8 both tests fail, with 3.9 the second (but with no recursion), with 3.10 the first and with 3.11-3.13 none fails.

[mrbean-bremen]

I cannot reproduce this locally with Windows 10, Python 3.12.7, pytest 8.3.3 and pyfakefs 3.7.1 (though as I wrote, I got failures with Python < 3.11). There is obviously a problem with the older Python versions, but I have no idea why it behaves differently for 3.12. You are using the latest pyfakefs version, right?

[ofek]

Yes I am using the latest version. Interesting! Indeed I cannot reproduce when I bump from Python 3.12.6 to 3.12.7. This doesn't exactly fix my situation however because we must support Python 3.9+ for now.

[ofek]

cc @barneygale the pathlib expert for any insight/hunch you may have about this 😄

[mrbean-bremen]

Ok, so the problem appears with 3.12.6, but not with 3.12.7? That's strange... anyway, I will try to understand the problem at least with the older Python versions, though this may take a while.

[barneygale]

I think this is the only pathlib change that landed in that timeframe: python/cpython#125156

That code runs if you pass a PurePath object as an initialiser argument to PurePath. If the parsers don't match, we now call as_posix() to stringify the path (rather than extending with its _raw_paths)

[mrbean-bremen]

@barneygale - thanks, indeed the check

if arg.parser is ntpath and self.parser is posixpath:

also might have been the problem - it will not work with the fake filesystem. I had to overwrite a similar check in another place, but this is all a bit fragile.

[barneygale]

Let me know if there's an alternative formulation that would work better for you!

[ofek]

Please forgive me, I was actually totally incorrect regarding my test of a newer Python. I installed the Python to a particular path so was invoking path\to\python.exe -m pytest. It turns out that also works on 3.12.6 and on both versions running the pytest script directly from an activated virtual environment fails. Here is a more complete picture of tests I just ran:

• 3.8.20
  • touch
    • ✅ python -m pytest
    • ✅ pytest
  • write_text
    • ❌ python -m pytest
    • ❌ pytest
• 3.9.20
  • touch
    • ✅ python -m pytest
    • ✅ pytest
  • write_text
    • ❌ python -m pytest
    • ❌ pytest
• 3.10.15
  • touch
    • ❌ python -m pytest
    • ❌ pytest
  • write_text
    • ✅ python -m pytest
    • ✅ pytest
• 3.11.10
  • touch
    • ✅ python -m pytest
    • ❌ pytest
  • write_text
    • ✅ python -m pytest
    • ❌ pytest
• 3.12.7
  • touch
    • ✅ python -m pytest
    • ❌ pytest
  • write_text
    • ✅ python -m pytest
    • ❌ pytest
• 3.13.0
  • touch
    • ✅ python -m pytest
    • ❌ pytest
  • write_text
    • ✅ python -m pytest
    • ❌ pytest

[ofek]

Something odd and temporary happened in Python 3.10 with respect to this library. Since 3.11 everything works but only if you invoke the Python executable (which we couldn't do as it's difficult to change internal tooling for everyone).