Any calls to subprocess after Task.start() results in segfault

[jasonpfi]

System:
OS: Ubuntu 20.04, 64-bit
Kernel: 5.13.0-28-generic
nidaqmx: 0.6.1
Chassis: cDAQ9189
Module: NI 9477

Details:
I am currently using nidaqmx for some automated functional testing on one of my systems, and I encountered the strangest problem. After calling the start() method on the task that I have set up, any subsequent calls to subprocess result in a segfault. Even if the task is not started but a line is written to in the DO task, the subprocess calls will fail. I am not sure if this is an issue with nidaqmx or subprocess, but it's relatively simple to replicate. Obviously, the line name has to be accurate to your setup.

Code:

#!/usr/bin/env python3

"""Simple NI DAQ Test Script."""

import subprocess

import nidaqmx


def main():
    task = nidaqmx.Task('test')
    task.do_channels.add_do_chan('cDAQ9189-2005203Mod1/port0/line10')
    subprocess.check_output(['ls'])
    task.start()
    subprocess.check_output(['ls'])


if __name__ == '__main__':
    main()

Output:

Traceback (most recent call last):
  File "./github.py", line 19, in <module>
    main()
  File "./github.py", line 15, in main
    subprocess.check_output(['ls'])
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ls']' died with <Signals.SIGSEGV: 11>.

AB#3410281

[jasonpfi]

Python Version:
3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]

numpy Version:
1.22.2

six Version:
1.16.0

NI-DAQmx Driver Version:
2022Q1

[jasonpfi]

The segfault behavior starts after this line executes:
https://github.com/ni/nidaqmx-python/blob/master/nidaqmx/task.py#L1020

[jasonpfi]

Here's the core dump, if this is helpful at all:

[451653.981614] python3[1322315]: segfault at 2d13f00 ip 00007f6850dcc5bf sp 00007fffcbaf80c0 error 4 in ld-2.31.so[7f6850dc2000+23000]
[451653.981622] Code: fa 48 d3 ea 48 89 d1 48 89 c2 48 d3 ea 89 f9 48 d3 e8 48 21 d0 a8 01 0f 85 fe 01 00 00 48 83 c5 01 49 39 eb 0f 86 e1 02 00 00 <49> 8b 44 ed 00 48 8b 58 28 4c 39 e3 74 e5 45 85 ff 74 09 f6 83 1c

[zhindes]

Strange! I know DAQmx can't handle fork without exec, but it looks like Python uses either posix_spawn or fork/exec per the source which should be OK. Can you run your python invocation under strace and share that log?

I'll see if I can reproduce at the office in the next couple of days.

[jasonpfi]

Here's what I got.
strace_output.txt

Edit: This was using python3.8-dbg

[zhindes]

I was able to reproduce and dig in a bit. Here is the relevant bit of the strace log (you need -f param to follow children):

[pid  6163] pipe2([24, 25], O_CLOEXEC)  = 0
[pid  6163] fstat(24, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid  6163] ioctl(24, TCGETS, 0x7ffcfafb5ed0) = -1 ENOTTY (Inappropriate ioctl for device)
[pid  6163] lseek(24, 0, SEEK_CUR)      = -1 ESPIPE (Illegal seek)
[pid  6163] pipe2([26, 27], O_CLOEXEC)  = 0
[pid  6163] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 6180 attached
, child_tidptr=0x7f9a871c3a10) = 6180
[pid  6180] set_robust_list(0x7f9a871c3a20, 24) = 0
[pid  6163] close(27 <unfinished ...>
[pid  6180] close(24 <unfinished ...>
[pid  6163] <... close resumed>)        = 0
[pid  6180] <... close resumed>)        = 0
[pid  6180] close(26)                   = 0
[pid  6180] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x216d440} ---

So you can see the last things that happened in the child process are closing handles 24 and 26. I believe that is this code deep in Python's fork/exec code:

    /* Close parent's pipe ends. */
    if (p2cwrite != -1)
        POSIX_CALL(close(p2cwrite));
    if (c2pread != -1)
        POSIX_CALL(close(c2pread));
    if (errread != -1)
        POSIX_CALL(close(errread));
    POSIX_CALL(close(errpipe_read));

Where 24 is the c2pread and 26 is errpipe_read. Those are the parents file handles for communication between the primary Python process and the fork doing the ls. errpipe is to report errors if something goes wrong on fork/exec, and I'm not sure what c2p pipe is for other than "child to parent". Anyways, based on the log it is segfaulting after that code and before exec. I am able to reproduce it manually using this Python code that is a minimum copy of what the subprocess call is doing (without a ton of error handling and cleanup code):

def not_posix_spawn():
    import _posixsubprocess
    c2pread, c2pwrite = os.pipe()
    print(f"c2pread: {c2pread}, c2pwrite: {c2pwrite}")

    errpipe_read, errpipe_write = os.pipe()
    print(f"errpipe_read: {errpipe_read}, errpipe_write: {errpipe_write}")

    low_fds_to_close = []
    while errpipe_write < 3:
        low_fds_to_close.append(errpipe_write)
        errpipe_write = os.dup(errpipe_write)
    for low_fd in low_fds_to_close:
        os.close(low_fd)

    fds_to_keep = set()
    fds_to_keep.add(errpipe_write)
    executable = os.fsencode("/usr/bin/ls")
    _posixsubprocess.fork_exec(
        ["/usr/bin/ls"], (executable,),
        False, tuple(sorted(map(int, fds_to_keep))),
        None, None,
        -1, -1, c2pread, c2pwrite,
        -1, -1,
        errpipe_read, errpipe_write,
        True, False, None)

However, I am not able to reproduce it with a C program that I wrote to try to reproduce what is happening the CPython code.

#include <stdio.h>
#include <stdlib.h>
#include <NIDAQmx.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <dirent.h>
#include <string.h>
#define DAQmxErrChk(functionCall) if( DAQmxFailed(error=(functionCall)) ) goto Error; else


struct linux_dirent {
   unsigned long  d_ino;
   off_t          d_off;
   unsigned short d_reclen;
   char           d_name[];
};

int close_open_fds(int start_fd, int except_this_fd)
{
   int fd_dir;
   char buffer[1024];
   int bytes;

   fd_dir = open("/proc/self/fd", O_RDONLY);
   while (bytes = syscall(SYS_getdents, fd_dir, buffer, sizeof(buffer)) > 0) {
      struct linux_dirent* entry;
      int offset;
      for (offset = 0; offset < bytes;) {
         int fd = 0;
	 entry = (struct linux_dirent*)(buffer + offset);
	 offset += entry->d_reclen;
	 fd = atoi(entry->d_name);
	 if (fd != fd_dir && fd > start_fd && fd != except_this_fd)
	 {
            close(fd);
	 }
      }
   }
   close(fd_dir);
}

// returns 1 if child
int do_fork_exec(int setup_pipes)
{
   int c2p_pipe[2];
   int err_pipe[2];
   char* argv[2];
   argv[0] = "/usr/bin/ls";
   argv[1] = NULL;

   if (setup_pipes) {
      pipe(c2p_pipe);
      pipe(err_pipe);
      printf("c2p read: %d, write: %d\n", c2p_pipe[0], c2p_pipe[1]);
      printf("err read: %d, write: %d\n", err_pipe[0], err_pipe[1]);
   }
   // parent gets the child pid
   // child gets 0
   if (fork() != 0) return 0;

   if (setup_pipes) {
      // close parent end of the pipes
      close(c2p_pipe[0]);
      close(err_pipe[0]);
      dup2(c2p_pipe[1], 1);
      close_open_fds(3, err_pipe[1]);
   }

   execv("/usr/bin/ls", argv);
   // in theory we don't get here.
   printf("child process after exec, you shouldn't be seeing this!\n");
   return 1;
}

int main(int argc, char* argv[])
{
   int32      error=0;
   TaskHandle taskHandle=0;
   int32      read;
   float64    data[1000];
   char       errBuff[2048]={'\0'};
   int        setup_pipes = 0;

   if (argc > 1 && strcmp(argv[1], "pipes") == 0) {
      setup_pipes = 1;
   }

   DAQmxErrChk(DAQmxCreateTask("", &taskHandle));

   DAQmxErrChk(DAQmxCreateDIChan(taskHandle, "Dev1/port0/line10", "", DAQmx_Val_ChanPerLine));

   DAQmxErrChk(DAQmxStartTask(taskHandle));
   if (do_fork_exec(setup_pipes)) {
      /* child */
      return 0;
   }


Error:
   if( DAQmxFailed(error) )
      DAQmxGetExtendedErrorInfo(errBuff,2048);
   if( taskHandle!=0 )  {
      DAQmxStopTask(taskHandle);
      DAQmxClearTask(taskHandle);
   }
   if( DAQmxFailed(error) )
      printf("DAQmx Error: %s\n",errBuff);

   return 0;
}

That being said, I do know NI-DAQmx does have some oddities with fork (and especially without exec). I'll ping a couple experts to see if they can give me an overview of what that is that could explain this.

[jasonpfi]

Hey there, any updates on how this is going?

[zhindes]

Nothing yet, unfortunately. I did confirm that DAQmx does not support fork-without-exec. However, the Python library does a fork-exec. It appears that some internal implementation detail is doing "too much" between the fork and the exec which is causing issues. I haven't had the time to try to further my attempt to reproduce with a pure C program. Perhaps that is a fools errand? Not sure when I will have that time. For now, I was able to get the basic flow to work in Python by directly using some of the fork/exec calls available in Python without some of the bells and whistles that the library adds on for error handling. If this is critical, you can give that a shot.

[zhindes]

I suspect this a deep issue within the DAQmx driver itself not playing nicely with Python in some rare circumstances. This isn't a priority to fix, but if others run into this issue we can guide them through an escalation process with NI.

[BBSeward]

@zhindes Im running into this as well, how can I escalate this?

[zhindes]

@BBSeward - I would work with NI support to create an escalation. That's the only way we'll have the bandwidth to dig into this any deeper. https://www.ni.com/en/support.html