Duncan Murdoch
2021-Jan-27 21:25 UTC
[R] Error when calling (R 4.0.x on Windows) from Python
On 27/01/2021 3:40 p.m., Bill Dunlap wrote:> I believe the problem is from svn 77925 in gnuwin/front-ends/rcmdfn.c, > which was committed a few days after 3.6.3 was released. Rterm used > to put double quotes around a command line argument only if it > contained a space, now it double quotes all arguments. It sees shell > constructs like "1>" and the following file name as arguments and > double quoting them hides them from the shell, leading to this > problem. I think we may have to rely on the user supplying quotes as > needed instead of blindly adding them.Okay, now I see what you mean. If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe, so it is possible that redirection would be handled. If you invoke R directly using Rterm.exe, then my description down below would be correct. Duncan Murdoch> > -Bill > > On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch > <murdoch.duncan at gmail.com> wrote: >> >> On 27/01/2021 3:17 p.m., Duncan Murdoch wrote: >>> On 27/01/2021 3:38 a.m., Martin Maechler wrote: >>>>>>>>> Martin Maechler >>>>>>>>> on Tue, 26 Jan 2021 12:37:58 +0100 writes: >>>> >>>>>>>>> Marcel Baumgartner >>>>>>>>> on Tue, 26 Jan 2021 08:55:48 +0100 writes: >>>> >>>> >> Dear all, my colleague posted our issue on stackoverflow: >>>> >>>> >> Calling R script from Python does not save log file in >>>> >> version 4 - Stack Overflow >>>> >> [https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4] >>>> >>>> >> It is about this kind of call to R: >>>> >>>> >> R.exe -f code.R --args "~/file.txt" 1> "~/log.txt" 2>&1". >>>> >>>> >> The issue is that the log.txt file is not created when >>>> >> running R 4.x.x. The same code works perfectly fine with >>>> >> R 3.6.x. >>>> >>>> >> Any idea what's going wrong as of version 4? Regards >>>> >> Marcel >>>> >>>> > Dear Marcel, I think the solution is embarrassingly >>>> > simple: >>>> >>>> >> From the SO post, where she showed a bit more detail than you >>>> > show here, it's clear you have confused 'R.exe' and >>>> > 'Rscript.exe' and what you say above is not true: >>>> >>>> > 'R.exe' was used for R 3.6.0 but for R 4.0.3, you/she used >>>> > 'Rscript.exe' instead. >>>> >>>> >>>> > ... as you've noticed now, they do behave differently, >>>> > indeed! >>>> >>>> Well, this was not the solution to their -- Windows-only -- problem. >>>> The problem *is* indeed visible if they only use R.exe (also >>>> for R 4.0.3). >>>> >>>> I've commented more on the SO issue (see above), >>>> notably asking for a *minimal* repr.ex. (reproducible example), >>>> and one *not* using "<YOUR PATH>" and setwd() .. >>>> >>> >>> Isn't this purely a Python or user problem? R shouldn't process >>> redirection directives like >>> >>> 1> "~/log.txt" 2>&1 >>> >>> because it's the shell's job to process those. If Python is acting as >>> the shell, it needs to handle those things. If R was handling the >>> command via >> >> Oops, sent before finishing: >> >> If R was handling the command via system() or system2(), it would handle >> redirection itself. If it was using the Windows-only shell(), it would >> call cmd.exe (by default) to handle redirection. (This is a difference >> between R on Windows and R in Unix: in Unix a shell is always used.) >> >> Duncan Murdoch >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code.
I tried the following change, that adds quotes if the argument does
not include ">".
Index: front-ends/rcmdfn.c
==================================================================---
front-ends/rcmdfn.c (revision 79883)
+++ front-ends/rcmdfn.c (working copy)
@@ -173,9 +173,13 @@
fprintf(stderr, "command line too long\n");
return(27);
}
- strcat(cmd, "\"");
+ if (!strchr(argv[i], '>')) {
+ strcat(cmd, "\"");
+ }
strcat(cmd, argv[i]);
- strcat(cmd, "\"");
+ if (!strchr(argv[i], '>')) {
+ strcat(cmd, "\"");
+ }
}
/* the outermost double quotes are needed for cmd.exe */
strcat(cmd, "\"");
It lets the python example work. I am not sure that quoting all the
arguments buys you much, as shQuote() is still needed for arguments
that include spaces. E.g., with 3.6.3, 4.0.3, and my development
build with the above patch we get
> stopifnot(dir.create(dirname <- file.path(tempfile(), "A
SPACE"), recursive=TRUE))
> logname <- file.path(dirname, "log.txt")
> unlink(logname)
> system(paste( "C:\\R\\R-3.6.3\\bin\\R.exe --quiet --vanilla -e
\"commandArgs()\" 1>", logname))
ARGUMENT 'SPACE/log.txt' __ignored__
[1] 0> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory>
> system(paste( "C:\\R\\R-4.0.3\\bin\\R.exe --quiet --vanilla -e
\"commandArgs()\" 1>", logname))
> commandArgs()
[1] "C:\\R\\R-40~1.3/bin/x64/Rterm.exe"
[2] "--quiet"
[3] "--vanilla"
[4] "-e"
[5] "commandArgs()"
[6] "1>"
[7]
"C:\\Users\\willi\\AppData\\Local\\Temp\\RtmpM5tsC7\\file1a1068734a49/A"
[8] "SPACE/log.txt">
>
[1] 0> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory>
> unlink(logname)
> system(paste( "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe
--quiet --vanilla -e \"commandArgs()\" 1>", logname))
[1] 0> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "cannot open the connection"
Warning message:
In file(con, "r") :
cannot open file
'C:\Users\willi\AppData\Local\Temp\RtmpM5tsC7\file1a1068734a49/A
SPACE/log.txt': No such file or directory> tryCatch(readLines(sub(" .*$", "", logname)),
error=function(e)conditionMessage(e))
[1] "> commandArgs()"
"[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
[3] "[2] \"--quiet\"
" "[3] \"--vanilla\"
"
[5] "[4] \"-e\"
" "[5] \"commandArgs()\"
"
[7] "[6] \"SPACE/log.txt\"
" "> "
[9] "> ">
> unlink(logname)
> system(paste( "C:\\msys64\\home\\willi\\ucrt3\\r\\trunk\\bin\\R.exe
--quiet --vanilla -e \"commandArgs()\" 1>", shQuote(logname)))
[1] 0> tryCatch(readLines(logname), error=function(e)conditionMessage(e))
[1] "> commandArgs()"
"[1]
\"C:\\\\msys64\\\\home\\\\willi\\\\ucrt3\\\\r\\\\trunk/bin/x64/Rterm.exe\""
[3] "[2] \"--quiet\"
" "[3] \"--vanilla\"
"
[5] "[4] \"-e\"
" "[5] \"commandArgs()\"
"
[7] "> "
"> "
-Bill
On Wed, Jan 27, 2021 at 1:25 PM Duncan Murdoch <murdoch.duncan at
gmail.com> wrote:>
> On 27/01/2021 3:40 p.m., Bill Dunlap wrote:
> > I believe the problem is from svn 77925 in gnuwin/front-ends/rcmdfn.c,
> > which was committed a few days after 3.6.3 was released. Rterm used
> > to put double quotes around a command line argument only if it
> > contained a space, now it double quotes all arguments. It sees shell
> > constructs like "1>" and the following file name as
arguments and
> > double quoting them hides them from the shell, leading to this
> > problem. I think we may have to rely on the user supplying quotes as
> > needed instead of blindly adding them.
>
> Okay, now I see what you mean.
>
> If you invoke R using R.exe, it asks cmd.exe to run Rterm.exe, so it is
> possible that redirection would be handled.
>
> If you invoke R directly using Rterm.exe, then my description down below
> would be correct.
>
> Duncan Murdoch
>
>
> >
> > -Bill
> >
> > On Wed, Jan 27, 2021 at 12:28 PM Duncan Murdoch
> > <murdoch.duncan at gmail.com> wrote:
> >>
> >> On 27/01/2021 3:17 p.m., Duncan Murdoch wrote:
> >>> On 27/01/2021 3:38 a.m., Martin Maechler wrote:
> >>>>>>>>> Martin Maechler
> >>>>>>>>> on Tue, 26 Jan 2021 12:37:58
+0100 writes:
> >>>>
> >>>>>>>>> Marcel Baumgartner
> >>>>>>>>> on Tue, 26 Jan 2021 08:55:48
+0100 writes:
> >>>>
> >>>> >> Dear all, my colleague posted our issue on
stackoverflow:
> >>>>
> >>>> >> Calling R script from Python does not save
log file in
> >>>> >> version 4 - Stack Overflow
> >>>> >>
[https://stackoverflow.com/questions/65887485/calling-r-script-from-python-does-not-save-log-file-in-version-4]
> >>>>
> >>>> >> It is about this kind of call to R:
> >>>>
> >>>> >> R.exe -f code.R --args
"~/file.txt" 1> "~/log.txt" 2>&1".
> >>>>
> >>>> >> The issue is that the log.txt file is not
created when
> >>>> >> running R 4.x.x. The same code works
perfectly fine with
> >>>> >> R 3.6.x.
> >>>>
> >>>> >> Any idea what's going wrong as of
version 4? Regards
> >>>> >> Marcel
> >>>>
> >>>> > Dear Marcel, I think the solution is
embarrassingly
> >>>> > simple:
> >>>>
> >>>> >> From the SO post, where she showed a bit
more detail than you
> >>>> > show here, it's clear you have confused
'R.exe' and
> >>>> > 'Rscript.exe' and what you say above
is not true:
> >>>>
> >>>> > 'R.exe' was used for R 3.6.0 but for R
4.0.3, you/she used
> >>>> > 'Rscript.exe' instead.
> >>>>
> >>>>
> >>>> > ... as you've noticed now, they do behave
differently,
> >>>> > indeed!
> >>>>
> >>>> Well, this was not the solution to their -- Windows-only
-- problem.
> >>>> The problem *is* indeed visible if they only use R.exe
(also
> >>>> for R 4.0.3).
> >>>>
> >>>> I've commented more on the SO issue (see above),
> >>>> notably asking for a *minimal* repr.ex. (reproducible
example),
> >>>> and one *not* using "<YOUR PATH>" and
setwd() ..
> >>>>
> >>>
> >>> Isn't this purely a Python or user problem? R
shouldn't process
> >>> redirection directives like
> >>>
> >>> 1> "~/log.txt" 2>&1
> >>>
> >>> because it's the shell's job to process those. If
Python is acting as
> >>> the shell, it needs to handle those things. If R was handling
the
> >>> command via
> >>
> >> Oops, sent before finishing:
> >>
> >> If R was handling the command via system() or system2(), it would
handle
> >> redirection itself. If it was using the Windows-only shell(), it
would
> >> call cmd.exe (by default) to handle redirection. (This is a
difference
> >> between R on Windows and R in Unix: in Unix a shell is always
used.)
> >>
> >> Duncan Murdoch
> >>
> >> ______________________________________________
> >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more,
see
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
>