It seems that the system() and system2() functions don't close file descriptors between the fork() and exec() (on Unix platforms, of course). This means that the child processes inherit open files and socket connections. Running this (from a terminal) will result in the child process writing to a file that was opened by R: R f <- file('foo.txt', 'w') system('echo "abc" >&3') You can also see the open files if you run the following: f <- file('foo.txt', 'w') system2('sleep', '100', wait=F) And then in another terminal: lsof -c R -c sleep it will show that both the R and sleep processes have the file open: ... R 324 root 3w REG 0,48 0 4259 /foo.txt ... sleep 327 root 3w REG 0,48 0 4259 /foo.txt This behavior can cause problems if R spawns a child process that outlives the R process, but keeps open some resources. Would it be possible to add an option to close file descriptors for child processes? It would be nice if that were the default, but I suspect that making that change would break a lot of existing code. To take an example from the Python world, subprocess.Popen() has an option, close_fds, which closes all file descriptors except 0, 1, and 2. https://docs.python.org/2/library/subprocess.html#popen-constructor -Winston [[alternative HTML version deleted]]
In addition to the issue of a child process holding onto open files, the child process can also manipulate a file descriptor in a way that affects the parent process. For example, calling lseek() in the child process will move the file offset in the parent process. Here is a set of commands that demonstrates it. They can be copied and pasted in a terminal. What it does: - Creates C program that seeks to the beginning of a file descriptor, and compiles it to a program named "lseek". - Creates a file with some text in it. - Starts R. In R: - Opens the text file and reads the first line. - Runs lseek in a child process. - Reads the rest of the lines. echo "#include <unistd.h> int main(void) { lseek(3, 0, SEEK_SET); }" > lseek.c gcc lseek.c -o lseek echo "line 1 line 2 line 3" > lines.txt R f <- file('lines.txt', 'r') cat(readLines(f, n = 1), sep = "\n") system('./lseek') cat(readLines(f), sep = "\n") Here's what it outputs:> f <- file('lines.txt', 'r') > cat(readLines(f, n = 1), sep = "\n")line 1> system('./lseek') > cat(readLines(f), sep = "\n")line 2 line 3 line 1 line 2 line 3 The child process has changed what the parent process reads from the file. (I'm guessing that the reason readLines() prints out "line 2" and "line 3" before starting over is because it has already buffered the whole file before lseek is executed.) This is obviously a highly contrived case, but it illustrates what's possible. The other issue I mentioned, with child processes holding open files after the R process exits, is more likely to cause problems in the real world. That's actually how I encountered this issue in the first place: when restarting R inside of RStudio on a Mac, if there are any extant child processes started by system(), they keep some files open, and this causes RStudio to hang. (There's a fix in progress for RStudio for this particular issue.) -Winston On Tue, Apr 18, 2017 at 3:20 PM, Winston Chang <winstonchang1 at gmail.com> wrote:> It seems that the system() and system2() functions don't close file > descriptors between the fork() and exec() (on Unix platforms, of course). > This means that the child processes inherit open files and socket > connections. > > Running this (from a terminal) will result in the child process writing to > a file that was opened by R: > > R > f <- file('foo.txt', 'w') > system('echo "abc" >&3') > > > > You can also see the open files if you run the following: > f <- file('foo.txt', 'w') > system2('sleep', '100', wait=F) > > And then in another terminal: > lsof -c R -c sleep > it will show that both the R and sleep processes have the file open: > ... > R 324 root 3w REG 0,48 0 4259 /foo.txt > ... > sleep 327 root 3w REG 0,48 0 4259 /foo.txt > > > This behavior can cause problems if R spawns a child process that outlives > the R process, but keeps open some resources. > > Would it be possible to add an option to close file descriptors for child > processes? It would be nice if that were the default, but I suspect that > making that change would break a lot of existing code. > > To take an example from the Python world, subprocess.Popen() has an > option, close_fds, which closes all file descriptors except 0, 1, and 2. > https://docs.python.org/2/library/subprocess.html#popen-constructor > > > -Winston >[[alternative HTML version deleted]]
In S+ on Unix-alikes we dealt with this issue by using fcntl(fd, F_SETFD, 1) to set the close-on-exec flag on a file descriptor as soon as we opened it. Bill Dunlap TIBCO Software wdunlap tibco.com On Wed, Apr 19, 2017 at 8:40 PM, Winston Chang <winstonchang1 at gmail.com> wrote:> In addition to the issue of a child process holding onto open files, the > child process can also manipulate a file descriptor in a way that affects > the parent process. For example, calling lseek() in the child process will > move the file offset in the parent process. > > Here is a set of commands that demonstrates it. They can be copied and > pasted in a terminal. What it does: > - Creates C program that seeks to the beginning of a file descriptor, and > compiles it to a program named "lseek". > - Creates a file with some text in it. > - Starts R. In R: > - Opens the text file and reads the first line. > - Runs lseek in a child process. > - Reads the rest of the lines. > > > echo "#include <unistd.h> > int main(void) { > lseek(3, 0, SEEK_SET); > }" > lseek.c > > gcc lseek.c -o lseek > > echo "line 1 > line 2 > line 3" > lines.txt > > R > f <- file('lines.txt', 'r') > cat(readLines(f, n = 1), sep = "\n") > system('./lseek') > cat(readLines(f), sep = "\n") > > > Here's what it outputs: >> f <- file('lines.txt', 'r') >> cat(readLines(f, n = 1), sep = "\n") > line 1 >> system('./lseek') >> cat(readLines(f), sep = "\n") > line 2 > line 3 > line 1 > line 2 > line 3 > > The child process has changed what the parent process reads from the file. > (I'm guessing that the reason readLines() prints out "line 2" and "line 3" > before starting over is because it has already buffered the whole file > before lseek is executed.) > > This is obviously a highly contrived case, but it illustrates what's > possible. The other issue I mentioned, with child processes holding open > files after the R process exits, is more likely to cause problems in the > real world. That's actually how I encountered this issue in the first > place: when restarting R inside of RStudio on a Mac, if there are any > extant child processes started by system(), they keep some files open, and > this causes RStudio to hang. (There's a fix in progress for RStudio for > this particular issue.) > > -Winston > > > > On Tue, Apr 18, 2017 at 3:20 PM, Winston Chang <winstonchang1 at gmail.com> > wrote: > >> It seems that the system() and system2() functions don't close file >> descriptors between the fork() and exec() (on Unix platforms, of course). >> This means that the child processes inherit open files and socket >> connections. >> >> Running this (from a terminal) will result in the child process writing to >> a file that was opened by R: >> >> R >> f <- file('foo.txt', 'w') >> system('echo "abc" >&3') >> >> >> >> You can also see the open files if you run the following: >> f <- file('foo.txt', 'w') >> system2('sleep', '100', wait=F) >> >> And then in another terminal: >> lsof -c R -c sleep >> it will show that both the R and sleep processes have the file open: >> ... >> R 324 root 3w REG 0,48 0 4259 /foo.txt >> ... >> sleep 327 root 3w REG 0,48 0 4259 /foo.txt >> >> >> This behavior can cause problems if R spawns a child process that outlives >> the R process, but keeps open some resources. >> >> Would it be possible to add an option to close file descriptors for child >> processes? It would be nice if that were the default, but I suspect that >> making that change would break a lot of existing code. >> >> To take an example from the Python world, subprocess.Popen() has an >> option, close_fds, which closes all file descriptors except 0, 1, and 2. >> https://docs.python.org/2/library/subprocess.html#popen-constructor >> >> >> -Winston >> > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
Possibly Parallel Threads
- system/system2 and open file descriptors
- Bug when calling system/system2 (and request for Bugzilla account)
- Problem with system2(), directing STDERR to a file
- system2() shortens R.home() to 8.3 form on windows
- system2 fails with quiet=TRUE, but runs through with quiet=FALSE