R help - Oct 2018 - Cannot use R on Windows when installed to an external drive with a space in the path

Hi all,

I'm considering filing this as a bug report but wanted to make sure first
that I'm not doing something wrong.

A piece of software which I develop installs R as part of its dependencies.
This normally works perfectly fine. However we noticed a bug when
installing to external drives on a path that includes a space, RScript.exe
and R.exe will not work.

E:\Program Files (x86)\R-3-3-3\bin>Rscript.exe
'E:\Program' is not recognized as an internal or external command,
operable program or batch file.

As you can see I'm in the directory itself, and simply calling the
executable without entering a path.

Similarly, if I do enter the full path with quote marks, I get the same
issue.

E:\>"E:\Program Files (x86)\R-3-3-3\bin\Rscript.exe"
'E:\Program' is not recognized as an internal or external command,
operable program or batch file.

At this stage I'm thinking it's either an R bug or a Windows bug, but I
can
run the following just fine.

E:\Program Files (x86)\HelloWorldProj\bin>HelloWorldProj.exe
Hello, world!

I've tested this with two variables.

 A - Using the R installer OR Simply copying the binaries
 B - Writing to E:\R-3-3-3 OR Writing to E:\Program Files (x86)\R-3-3-3

A does not affect the outcome in any way, however R only works when B uses
the path without spaces.

I searched elsewhere on the internet for the same and interestingly it was
another R install with this issue. I believe the commenters here were
erroneously attributing it to the user.

https://stackoverflow.com/questions/43510438/how-to-run-an-executable-that-contains-a-space-in-path-from-command-line-on-wind

Any thoughts? I'm open to a sensible workaround, and if it is a bug I'm
happy to raise it.

Thanks,
Marc

	[[alternative HTML version deleted]]

Hi all,

This is my second time posting this, I accidentally sent the first email
without the proper subject header. Hopefully this won't create any noise
and a moderator can dismiss the first one.

I'm considering filing this as a bug report but wanted to make sure first
that I'm not doing something wrong.

A piece of software which I develop installs R as part of its dependencies.
This normally works perfectly fine. However we noticed a bug when
installing to external drives on a path that includes a space, RScript.exe
and R.exe will not work.

E:\Program Files (x86)\R-3-3-3\bin>Rscript.exe
'E:\Program' is not recognized as an internal or external command,
operable program or batch file.

As you can see I'm in the directory itself, and simply calling the
executable without entering a path.

Similarly, if I do enter the full path with quote marks, I get the same
issue.

E:\>"E:\Program Files (x86)\R-3-3-3\bin\Rscript.exe"
'E:\Program' is not recognized as an internal or external command,
operable program or batch file.

At this stage I'm thinking it's either an R bug or a Windows bug, but I
can
run the following just fine.

E:\Program Files (x86)\HelloWorldProj\bin>HelloWorldProj.exe
Hello, world!

I've tested this with two variables.

 A - Using the R installer OR Simply copying the binaries
 B - Writing to E:\R-3-3-3 OR Writing to E:\Program Files (x86)\R-3-3-3

A does not affect the outcome in any way, however R only works when B uses
the path without spaces.

I searched elsewhere on the internet for the same and interestingly it was
another R install with this issue. I believe the commenters here were
erroneously attributing it to the user.

https://stackoverflow.com/questions/43510438/how-to-run-an-executable-that-contains-a-space-in-path-from-command-line-on-wind

Any thoughts? I'm open to a sensible workaround, and if it is a bug I'm
happy to raise it.

Thanks,
Marc

	[[alternative HTML version deleted]]

Dear Marc,

this is probably because short file names are disabled on that external 
drive. R needs short file names to be enabled on the drive where it is 
installed - please check/follow according to Microsoft documentation on 
turning on Short File Names (SFN, aka 8.3 names, 8dot3 names).

Best,
Tomas

On 10/22/2018 03:34 PM, Marc Capavanni via R-help wrote:
> Hi all,
>
> This is my second time posting this, I accidentally sent the first email
> without the proper subject header. Hopefully this won't create any
noise
> and a moderator can dismiss the first one.
>
> I'm considering filing this as a bug report but wanted to make sure
first
> that I'm not doing something wrong.
>
> A piece of software which I develop installs R as part of its dependencies.
> This normally works perfectly fine. However we noticed a bug when
> installing to external drives on a path that includes a space, RScript.exe
> and R.exe will not work.
>
> E:\Program Files (x86)\R-3-3-3\bin>Rscript.exe
> 'E:\Program' is not recognized as an internal or external command,
> operable program or batch file.
>
> As you can see I'm in the directory itself, and simply calling the
> executable without entering a path.
>
> Similarly, if I do enter the full path with quote marks, I get the same
> issue.
>
> E:\>"E:\Program Files (x86)\R-3-3-3\bin\Rscript.exe"
> 'E:\Program' is not recognized as an internal or external command,
> operable program or batch file.
>
> At this stage I'm thinking it's either an R bug or a Windows bug,
but I can
> run the following just fine.
>
> E:\Program Files (x86)\HelloWorldProj\bin>HelloWorldProj.exe
> Hello, world!
>
> I've tested this with two variables.
>
>   A - Using the R installer OR Simply copying the binaries
>   B - Writing to E:\R-3-3-3 OR Writing to E:\Program Files (x86)\R-3-3-3
>
> A does not affect the outcome in any way, however R only works when B uses
> the path without spaces.
>
> I searched elsewhere on the internet for the same and interestingly it was
> another R install with this issue. I believe the commenters here were
> erroneously attributing it to the user.
>
>
https://stackoverflow.com/questions/43510438/how-to-run-an-executable-that-contains-a-space-in-path-from-command-line-on-wind
>
> Any thoughts? I'm open to a sensible workaround, and if it is a bug
I'm
> happy to raise it.
>
> Thanks,
> Marc
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Hi Tomas,

You're likely right, I didn't restart, unplug/replug the hard drive, or
even close CMD, so I can give that a go. Thanks for your insight.

In any case I think I'll go with the solution of using 3.5.1 as it seems to
solve the issue, and would mean I wouldn't have to modify hard drive or
registry settings on my user's PCs.

Many thanks,
Marc

P.S. I accidentally didn't include the r-help address, I'll bring it
back
in in case others might find this useful.

On Mon, 22 Oct 2018 at 15:47, Tomas Kalibera <tomas.kalibera at gmail.com>
wrote:
> Hi Marc,
>
> this seems to me that the directory where R is installed still does not
> have a short name. I don't remember the details, but maybe you have to
> reboot or something for them to be generated, windows docs should say. It
> would be good to ensure that the short name exists, e.g. using "dir
/x" or
> just from inside R, checking R home. R internally uses the short name when
> possible, because it is not completely robust against space in file name.
>
> I've fixed some most obvious places in R recently (already in R 3.5),
but
> maybe not all and certainly packages may break with spaces in R home
> directory name and some will definitely break. So it is unsafe to use R
> without SFN, even though it may start and many packages may work fine in
> 3.5.
>
> Best
> Tomas
>
>
> On 10/22/2018 04:40 PM, Marc Capavanni wrote:
>
> Hi Tomas,
>
> Thanks for your input. I tried to enabled those settings but the issue
> persists.
>
>     E:\Program Files (x86)\R-3.3.3\bin>fsutil 8dot3name query E:
>     The volume state is: 0 (8dot3 name creation is enabled).
>     The registry state is: 2 (Per volume setting - the default).
>
>     Based on the above settings, 8dot3 name creation is enabled on E:
>
>     E:\Program Files (x86)\R-3.3.3\bin>Rscript.exe
>     'E:\Program' is not recognized as an internal or external
command,
>     operable program or batch file.
>
> However I did try the latest version of R (3.5.1 as of writing) and it
> works following the same steps in which 3.3.3 does not. I've verified
this
> on a few machines and different external devices, so it seems like this has
> been remedied in the past few months.
>
> Thanks,
> Marc
>
> On Mon, 22 Oct 2018 at 14:47, Tomas Kalibera <tomas.kalibera at
gmail.com>
> wrote:
>
>> Dear Marc,
>>
>> this is probably because short file names are disabled on that external
>> drive. R needs short file names to be enabled on the drive where it is
>> installed - please check/follow according to Microsoft documentation on
>> turning on Short File Names (SFN, aka 8.3 names, 8dot3 names).
>>
>> Best,
>> Tomas
>>
>> On 10/22/2018 03:34 PM, Marc Capavanni via R-help wrote:
>> > Hi all,
>> >
>> > This is my second time posting this, I accidentally sent the first
email
>> > without the proper subject header. Hopefully this won't create
any noise
>> > and a moderator can dismiss the first one.
>> >
>> > I'm considering filing this as a bug report but wanted to make
sure
>> first
>> > that I'm not doing something wrong.
>> >
>> > A piece of software which I develop installs R as part of its
>> dependencies.
>> > This normally works perfectly fine. However we noticed a bug when
>> > installing to external drives on a path that includes a space,
>> RScript.exe
>> > and R.exe will not work.
>> >
>> > E:\Program Files (x86)\R-3-3-3\bin>Rscript.exe
>> > 'E:\Program' is not recognized as an internal or external
command,
>> > operable program or batch file.
>> >
>> > As you can see I'm in the directory itself, and simply calling
the
>> > executable without entering a path.
>> >
>> > Similarly, if I do enter the full path with quote marks, I get the
same
>> > issue.
>> >
>> > E:\>"E:\Program Files (x86)\R-3-3-3\bin\Rscript.exe"
>> > 'E:\Program' is not recognized as an internal or external
command,
>> > operable program or batch file.
>> >
>> > At this stage I'm thinking it's either an R bug or a
Windows bug, but I
>> can
>> > run the following just fine.
>> >
>> > E:\Program Files (x86)\HelloWorldProj\bin>HelloWorldProj.exe
>> > Hello, world!
>> >
>> > I've tested this with two variables.
>> >
>> >   A - Using the R installer OR Simply copying the binaries
>> >   B - Writing to E:\R-3-3-3 OR Writing to E:\Program Files
(x86)\R-3-3-3
>> >
>> > A does not affect the outcome in any way, however R only works
when B
>> uses
>> > the path without spaces.
>> >
>> > I searched elsewhere on the internet for the same and
interestingly it
>> was
>> > another R install with this issue. I believe the commenters here
were
>> > erroneously attributing it to the user.
>> >
>> >
>>
https://stackoverflow.com/questions/43510438/how-to-run-an-executable-that-contains-a-space-in-path-from-command-line-on-wind
>> >
>> > Any thoughts? I'm open to a sensible workaround, and if it is
a bug I'm
>> > happy to raise it.
>> >
>> > Thanks,
>> > Marc
>> >
>> >       [[alternative HTML version deleted]]
>> >
>> > ______________________________________________
>> > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more,
see
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>>
>>
>>
>
	[[alternative HTML version deleted]]

This is a bin\R.exe + bin\Rscript.exe bug.

On Windows, where the system call to create a process only takes one
command line string instead of an array of command line parameters[0]
(and the C runtimes usually call CommandLineToArgvW[1] behind the
scenes to get an array back), execv() does nothing to prepare a
properly-quoted command line and just concatenates its arguments[2].
Moreover, Windows version of execv() is effectively undocumented[3],
nor there is any information about the suggested replacement[4].
(You can call it a Windows bug instead.)

Short file names are not a guaranteed solution, because they may
not be available at all on some filesystems[5] (which is how this
error has been encountered). In the general case, "argument quoting
hell"[6] is unavoidable on Windows, but the code from [2] implements
the proper conversion from an array of strings to a single quoted
command line, ready to be parsed by the launched process.

char* getRHOME(int m) in src/gnuwin32/rhome.c tries to return a short
path (assuming it won't contain spaces). This is used by rcmdfn() from
src/gnuwin32/front-ends/rcmdfn.c to system() a command line built with
no quoting, causing the error. rcmdfn() is called from
src/gnuwin32/front-ends/R.c, which seems to contain the entry point for
R\R-3.3.3\bin\R.exe.

Fortunately, in src/unix/Rscript.c the whole mess seems to be avoided by
having it directly include rterm.c and then call AppMain() on Windows,
passing the command line parameter array. Same goes for
src/gnuwin32/front-ends/graphappmain.c. The corresponding executables
reside in the architecture-specific subdirectory of ...\bin\.

So if you run Rgui.exe, Rterm.exe or Rscript.exe from
R-3.3.3\bin\i386 instead of R-3.3.3\bin, no extra cmd.exe processes
should be spawned and the badly-formed system() should be avoided.

-- 
Best regards,
Ivan

[0]
https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-createprocessa

That's probably for historical reasons, because Win32 API wanted to
stay mostly source-compatible with Win16 API, and Win16 API may have
evolved from DOS system calls, where the command line was, indeed, a
single string pointer.

[1]
https://docs.microsoft.com/en-us/windows/desktop/api/shellapi/nf-shellapi-commandlinetoargvw

[2]
https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/

[3]
https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2012/ms235416(v=vs.110)

[4]
https://docs.microsoft.com/en-us/cpp/c-runtime-library/exec-wexec-functions?view=vs-2017

[5] https://blogs.msdn.microsoft.com/oldnewthing/20181004-00/?p=99895

[6] quoted from Rscript.c

On 10/22/2018 07:15 PM, Ivan Krylov wrote:
> This is a bin\R.exe + bin\Rscript.exe bug.
Please note this has been fixed in later versions of R, quoting has been 
added. If you can find some corner case in which the current quoting 
(R-devel) is incorrect on Windows, in the front-ends or anywhere else, 
please send a bug report with a reproducible example.

But please keep in mind my comment that some packages will not be 
working with space in the name of R home directory. Some packages will 
not build from source. CRAN has been checking for the most obvious cause 
of this (certain use of include directives in make files), but it is 
still not safe to use R on file systems without SFN.

Best,
Tomas
>
> On Windows, where the system call to create a process only takes one
> command line string instead of an array of command line parameters[0]
> (and the C runtimes usually call CommandLineToArgvW[1] behind the
> scenes to get an array back), execv() does nothing to prepare a
> properly-quoted command line and just concatenates its arguments[2].
> Moreover, Windows version of execv() is effectively undocumented[3],
> nor there is any information about the suggested replacement[4].
> (You can call it a Windows bug instead.)
>
> Short file names are not a guaranteed solution, because they may
> not be available at all on some filesystems[5] (which is how this
> error has been encountered). In the general case, "argument quoting
> hell"[6] is unavoidable on Windows, but the code from [2] implements
> the proper conversion from an array of strings to a single quoted
> command line, ready to be parsed by the launched process.
>
> char* getRHOME(int m) in src/gnuwin32/rhome.c tries to return a short
> path (assuming it won't contain spaces). This is used by rcmdfn() from
> src/gnuwin32/front-ends/rcmdfn.c to system() a command line built with
> no quoting, causing the error. rcmdfn() is called from
> src/gnuwin32/front-ends/R.c, which seems to contain the entry point for
> R\R-3.3.3\bin\R.exe.
>
> Fortunately, in src/unix/Rscript.c the whole mess seems to be avoided by
> having it directly include rterm.c and then call AppMain() on Windows,
> passing the command line parameter array. Same goes for
> src/gnuwin32/front-ends/graphappmain.c. The corresponding executables
> reside in the architecture-specific subdirectory of ...\bin\.
>
> So if you run Rgui.exe, Rterm.exe or Rscript.exe from
> R-3.3.3\bin\i386 instead of R-3.3.3\bin, no extra cmd.exe processes
> should be spawned and the badly-formed system() should be avoided.
>