Running a sudo command in a fakefs reults in RecursionError

[MaKaNu]

Describe the bug
I am not sure if it is approachable what I am trying.

I have a function, I provided below, which allows sudo calls from inside my python module. I've tested this function multiple times in realfs without any issue. I am now at a point where I utilize the function to copy via rsync file inside the fakefs.

This results in following recursion error:

.
.
.
.venv/lib/python3.11/site-packages/pyfakefs/fake_io.py:124: in open
    return fake_open(
.venv/lib/python3.11/site-packages/pyfakefs/fake_open.py:97: in __call__
    return self.call(*args, **kwargs)
.venv/lib/python3.11/site-packages/pyfakefs/fake_open.py:173: in call
    wrapper = FakePipeWrapper(
.venv/lib/python3.11/site-packages/pyfakefs/fake_file.py:1256: in __init__
    self.real_file = open(fd, mode)
.venv/lib/python3.11/site-packages/pyfakefs/fake_io.py:124: in open
    return fake_open(
.venv/lib/python3.11/site-packages/pyfakefs/fake_open.py:97: in __call__
    return self.call(*args, **kwargs)
.venv/lib/python3.11/site-packages/pyfakefs/fake_open.py:173: in call
    wrapper = FakePipeWrapper(
.venv/lib/python3.11/site-packages/pyfakefs/fake_file.py:1256: in __init__
    self.real_file = open(fd, mode)
.venv/lib/python3.11/site-packages/pyfakefs/fake_io.py:97: in open
    stack = traceback.extract_stack(limit=2)
/usr/lib/python3.11/traceback.py:231: in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
/usr/lib/python3.11/traceback.py:393: in extract
    return klass._extract_from_extended_frame_gen(
/usr/lib/python3.11/traceback.py:397: in _extract_from_extended_frame_gen
    @classmethod
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

klass = <class 'traceback.StackSummary'>
frame_gen = <generator object StackSummary.extract.<locals>.extended_frame_gen at 0x7f0220946890>

>   @classmethod
E   RecursionError: maximum recursion depth exceeded

How To Reproduce
This is my function which calls system sudo calls:

class SysStreamer:
    def __init__(self) -> None:
        self.flush_stdout_cache()
        self.logger = load_logger(__name__)

    def flush_stdout_cache(self):
        self.stdout_lines = []

    def stream_command(
        self,
        args,
        *,
        stdout_handler=logging.info,
        check=True,
        text=True,
        stdin=None,
        stdout=PIPE,
        stderr=STDOUT,
        password=None,
        **kwargs,
    ):
        """Mimic subprocess.run, while processing the command output in real time."""
        with Popen(args, text=text, stdin=stdin, stdout=stdout, stderr=stderr, **kwargs) as process:
            if password:
                process.stdin.write(f"{password}\n")
                process.stdin.flush()
            for line in process.stdout:
                stdout_handler(line[:-1])
        retcode = process.poll()
        if check and retcode:
            raise CalledProcessError(retcode, f"{process.args}:Check Logs!")
        return CompletedProcess(process.args, retcode)

streamer = SysStreamer()

def _logable_sudo(cmds: List[str], password: str) -> List[str]:
        sudo_cmds = ["sudo", "-S"]
        sudo_cmds.extend(cmds)

        streamer.stream_command(
            sudo_cmds,
            stdin=PIPE,
            password=password,
        )

        return streamer.stdout_lines

If I use this function inside a simple test scenario with pyfakefs the recursion error happens:

def test_simple(fs):
    _logabale_sudo(["ls"], "password1234")

Your environment
Please run the following in the environment where the problem happened and
paste the output.

python -c "import platform; print(platform.platform())"
python -c "import sys; print('Python', sys.version)"
python -c "from pyfakefs import __version__; print('pyfakefs', __version__)"
python -c "import pytest; print('pytest', pytest.__version__)"

Linux-6.6.4-arch1-1-x86_64-with-glibc2.38
Python 3.11.6 (main, Nov 14 2023, 09:36:21) [GCC 13.2.1 20230801]
pyfakefs 5.3.0
pytest 7.4.3

[mrbean-bremen]

Without looking at the concrete problem I suspect that this is due to a limitation in pyfakefs with modules using the C file interface, specifically subprocess is known to not play nice with pyfakefs.
I will have a closer look later.

[davidlbaird]

Yes, I have seen these errors with subprocess in unit tests. In those tests, I have also had to mock.patch.object the subprocess calls with the unittest.mock module. This worked well in those unit tests, since pyfakefs cannot provide its file system to the environment created by subprocess.

[mrbean-bremen]

As I suspected and @davidlbaird mentioned, this will indeed not work with pyfakefs, both due to used system functions that are not patched, and the subprocess that will not know about pyfakefs. Without patching subprocess functionality, this is not going to work.