R devel - Mar 2020 - pipe(): input to, and output from, a single process

hi.  i'd like to instantiate sed(1), send it some input, and retrieve
its output, all via pipes (rather than an intermediate file).

my sense from pipe and looking at the sources (sys-unix.c) is that is
not possible.  is that true?  are there any thoughts of providing such a
facility?

cheers, Greg

I am not sure if `pipe()` works for this, but if it turns out that it
does not, then you can use the processx package, e.g.:
> p <- processx::process$new("sed", c("-l",
"s/a/x/g"), stdin = "|", stdout = "|")
> p$write_input("foobar\n")
> p$read_output()
[1] "foobxr\n"

The `-l` sed flag is to make sed line-buffered, otherwise it is will
not produce output until there is enough.

`$write_input()` and `$read_output()` are not easy to program, in particular:
* `$write_input()` returns the chunk of data that it hasn't managed to
write into the pipe. You need to call `$write_input() again, with this
data next, usually.
* `$read_output()` returns an empty string if there is no data to
read, so typically you want to call `p$poll()` first, to make sure
that there is something to read.
* `$read_output()` might not read whole lines, so maybe
`$read_output_lines()` is better for you.
* Close the stdin of the process if you want to quit cleanly:
`close(p$get_input_connection())`.
* There is currently no way to poll the input side of the pipe. :(

HTH, Gabor

On Mon, Mar 16, 2020 at 11:31 AM Greg Minshall <minshall at umich.edu>
wrote:
>
> hi.  i'd like to instantiate sed(1), send it some input, and retrieve
> its output, all via pipes (rather than an intermediate file).
>
> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing such a
> facility?
>
> cheers, Greg
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

On Fri, 13 Mar 2020 20:26:43 +0300
Greg Minshall <minshall at umich.edu> wrote:
> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing
> such a facility?
Pipes (including those created by popen(3), which R pipe() uses
internally) are uni-directional data channels. While it could be
possible to open two pipes for both stdin and stdout of the child
process, doing so correctly is complicated because of differences in
buffering provided by the runtime: when stdin/stdout is not a terminal,
buffering mode may be set to block-oriented instead of line-oriented,
resulting in both parent and child being dead-locked, waiting to fill
the buffer instead of returning from the blocking call after the first
newline. (Hence the -l flag to sed mentioned by G?bor Cs?rdi, which
avoids this problem for sed).

Programs designed to first read stdin until end-of-file, then process
the input and print results on the stdout are usually safe to use in
this way, but others may be not. Software specifically designed to
control other software (e.g. Expect [*]) gets around this limitation by
running the child processes inside pseudo-terminals and/or running in
event-driven manner, being ready to service the child process whether
it wants to read its stdin or write to stdout.

Since sed has its -l flag, it should be possible to safely drive it
line-by-line with the help of processx, but not via pipe().

-- 
Best regards,
Ivan

[*] https://core.tcl-lang.org/expect/index

On 13 March 2020 at 20:26, Greg Minshall wrote:
| hi.  i'd like to instantiate sed(1), send it some input, and retrieve
| its output, all via pipes (rather than an intermediate file).
| 
| my sense from pipe and looking at the sources (sys-unix.c) is that is
| not possible.  is that true?  are there any thoughts of providing such a
| facility?

Octave had this already in the 1990s, see documentation for 'popen2'
here:

  https://octave.org/doc/v4.2.1/Controlling-Subprocesses.html

As it says 'Start a subprocess with two-way communication'.

Dirk


-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org

Dirk,
> Octave had this already in the 1990s, see documentation for
'popen2' here:
thanks.  unix that had since the 1970s... :)

cheers, Greg

Gabor, thanks.  yes, managing the two-way communication is always a bit
error-prone, as it depends on the input/output characteristics of the
two ends -- they either match, or deadlock.  it's too bad if polling is
always *required* -- i'd think sometimes a programmer would be happy
blocking, though other times one wants better control over when to
block.  cheers, Greg

FWIW if you're on unix, you can use named pipes (fifos) for that:
> system("mkfifo my.output")
> p = pipe("sed -l s:hello:oops: > my.output", "w")
> i = file("my.output", "r", blocking=FALSE, raw=TRUE)
> writeLines("hello!\n", p)
> flush(p)
> readLines(i, 1)
[1] "oops!"

Cheers,
Simon


> On 14/03/2020, at 6:26 AM, Greg Minshall <minshall at umich.edu>
wrote:
> 
> hi.  i'd like to instantiate sed(1), send it some input, and retrieve
> its output, all via pipes (rather than an intermediate file).
> 
> my sense from pipe and looking at the sources (sys-unix.c) is that is
> not possible.  is that true?  are there any thoughts of providing such a
> facility?
> 
> cheers, Greg
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

Simon,
> FWIW if you're on unix, you can use named pipes (fifos) for that:
i've always wondered what named pipes actually were.  thanks!

cheers, Greg