Unexpectedly large memory requirements after test teardown

[konnerthg]

Describe the bug

In the test suite of a django project, memory seems to not always be freed after each test execution. I expect the memory occupied by the fake file system to go down drastically during each teardown. If this is wrong, I'm looking forward to learn the correct behavior.

I use docker container stats to check my container's memory requirements. I know this isn't a proper benchmark, but the results are easy to see. During the test suite, the memory fills up and the container crashes. I narrowed this down to the behavior in the following example.

On my machine, it quickly reaches 3+ GiB of memory. I recommend stopping the execution before it fills, as this produces a long and annoying freeze on my computer. When I manually run garbace collection though (see commented code), the memory usage jumps between 1-2 GiB and goes back to near 0 after each test. Note that you might have to play around with NUM_TESTS and FILE_SIZE.

I also tried running the same example without fake filesystem and the memory usage is negligible.

How To Reproduce

import random
import uuid

import pytest
import os


PATH = os.path.dirname(os.path.abspath(__file__))
NUM_TESTS = 10
FILE_SIZE = int(10e6)


@pytest.fixture()
def use_fake_fs(fs):
    fs.add_real_directory(PATH)
    yield

    # uncomment the following to reduce memory requirements

    # fs.reset()  # Clear filesystem contents (FakeFile/FakeDirectory objects)

    # import linecache
    # linecache.clearcache()  # Clear cached source code for tracebacks

    # import gc
    # gc.collect()  # Force cyclic GC to reclaim circular references


@pytest.mark.parametrize("counter", range(NUM_TESTS))
def test_fake_fs(use_fake_fs, counter):
    content = ''.join([str(random.randint(0, 10))] * FILE_SIZE)
    for _ in range(100):
        unique_file_path = f"{PATH}/file_{uuid.uuid4()}.txt"
        with open(unique_file_path, "w") as fh:
            fh.write(content)
        with open(unique_file_path, "r") as fh:
            assert fh.read() == content

Your environment

Linux-6.12.54-linuxkit-x86_64-with-glibc2.41
Python 3.12.12 (main, Dec 30 2025, 00:43:35) [GCC 14.2.0]
pyfakefs 6.0.0
pytest 9.0.2

requirements.txt:
django==6.0
pytest-django==4.11.1
pyfakefs==6.0.0

Note that django might not be relevant to the problem. it was just easy for me to reproduce it inside a docker container which uses django.

[mrbean-bremen]

Thank you - that indeed doesn't sound good.

I can think of a few things that increase memory usage with pyfakefs (mainly related to caching), but none of these should accumulate given that the same tests are run in a cycle.
Obviously there must be some (or a lot of) circular references involved, given that gc.collect() helps.

That this code takes more memory (compared to the test without pyfakefs) is obvious due to the large files created in memory - in this example at least 1 GB - but these should be freed after each test. If this is not done (which seems to be the case here), it is of course a serious problem. It may not have been noticed so far as most tests don't deal with large files in the fake fs.

[mrbean-bremen]

I can confirm that the problem is reproducible even on my Windows notebook. The memory usage increases for about 1 GB each test run (which is expected, as this is the size of the files), but only decreases about every 15 tests (e.g. after it has reached 15 GB), when the GC runs. A GC run should not be needed to free the memory, as far as I understand.
Most likely there is some unwanted cyclic relationship hiding somewhere - I will let you know if I find something out.

[mrbean-bremen]

Actually the cyclic relationship is quite obvious - a file has a backlink to the filesystem, which is not a weak reference (but should be). Somehow I have never seen this...

[mrbean-bremen]

Note to self: making all file system references weakrefs does not resolve the problem.
With pytest, 2 references remain, that must somehow be related to the pytest plugin. With the patchfs decorator, 1 reference remains (unclear where from). With pyfakefs TestCase, all references are freed (according to the ref count and gc.get_referents, but the memory is still not freed, so something else causes ref cycles. May take a while...

[mrbean-bremen]

@konnerthg - I have replaced all backlinks by weak references in main branch, as far as I can see. This should fix the problem, though I would like you to test it if possible. I may have introduced some potential crashes due to life time issues.
I couldn't find any problem myself, but would like to have some tests in a realistic environment first.

[konnerthg]

@mrbean-bremen thank you very much for handling this. on the above minimal example, the memory requirements on my machine indeed go down from 8GiB to 2GiB. that looks very good. However, on our real application, I have not yet managed to see an improvement. I hope there is a mistake on my end. I will get back to you as soon as I figure our more details.

[mrbean-bremen]

Thanks - I may have missed something, as I used the minimal example for reference testing.

[mrbean-bremen]

@konnerthg - did you find out something?

[konnerthg]

i am sorry, i have not been able to reproduce anything in a minimal example after your fix.
all i can observe is that while manually calling fs.reset() after each test:

@pytest.fixture
def fake_fs(fs):
    yield fs
    fs.reset()

the memory consumption is about 30% lower. no need to call gc.collect(). i don't know if this is meaningful for this discussion.

[mrbean-bremen]

Thank you, that's interesting! That basically means that the fake fs itself is still hold by someone.
I haven't been able to find any more cyclic references that could cause this, so it is difficult to find the problem without having a reproducible example.
Even so, this helped to fix most of these problems even if it didn't fix yours. I'm reopening this as it is clearly not fixed, and will think if there is anything else that can be done.

[davidlbaird]

I added a comment to #1273 of a failure I have found in large scale testing.