Ivan - thanks for looking into this. My answers to your comments are below.
Cheers
David
On 12/13/2019 1:46 AM, Ivan Krylov wrote:
On Thu, 12 Dec 2019 15:48:13 +0000
David Stevens <david.stevens at usu.edu><mailto:david.stevens at
usu.edu> wrote:
Certain R packages will not install properly on my Windows 10
computer.
Certain, but not all? Which packages are you able to install on that
computer?
Too many to name - one is nlstools. Another is tinytex. I updated many package
after bookdown wouldn't install for the same reasons. Out of a list of ~20
packages to install, ~5 failed with a similar error message. That sent me down
the rabbit hole that brought me here.
Warning: invalid package 'C:\Users\David'
Warning: invalid package
'Stevens\AppData\Local\Temp\Rtmpk5NqrI/downloaded_packages/callr_3.4.0.tar.gz'
This looks like a command line argument quoting issue.
I wonder why doesn't install.packages use shQuote to quote the file
name when calling system2(c("R", "CMD", "INSTALL",
file)) to install a
downloaded source package:
https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/library/utils/R/packages2.R#L833
Maybe there should be shQuote(fil) instead, especially since R does
quote the file name when installing from local source tarballs:
https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/library/utils/R/packages2.R#L676
This happens on only a subset of packages I install or update.
I have a hypothesis: perhaps this only happens for packages with no
pre-built Windows binary available yet, since "win.binary" packages
may
be installed by unpacking a zip file, without calling a command line
with potential space quoting issues:
I read a thread elsewhere that said a work around is to run
options(pkgType='binary') before installation and the problem went away.
Does this help.
https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/library/utils/R/packages2.R#L491
->
https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/library/utils/R/windows/install.packages.R#L263
Calling install.packages(..., verbose = TRUE) for both a succeeding and
a failing package might help to verify whether this is the case.
This is a relatively recent issue (i.e. I never saw it before
November - I've used R for ~15 years).
Just to confirm it: you had no problems on the same Windows 10 computer
with the same user name and %USERPROFILE% path as before?
Yes, this is the case. I do the regular Windows 10 updates and update R and
RStudio as soon as I am aware there's a new version out. I haven't
explicitly change %USERPROFILE%.
Apparently,
tempdir() used to return a 8.3 directory path on your computer, but now
doesn't - but that should not happen, since R_reInitTempDir()
explicitly asks for a 8.3 path:
tempdir() gives
tempdir()
[1] "C:\\Users\\David Stevens\\AppData\\Local\\Temp\\RtmpQpqh0t"
>
https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/main/sysutils.c#L1810
Microsoft Docs page for GetShortPathName() says:
If you call GetShortPathName on a path that doesn't have any short
names on-disk, the call will succeed, but will return the long-name
path instead. This outcome is also possible with NTFS volumes
because there's no guarantee that a short name will exist for a
given long name.
Some newer Windows 10 installations may have
NtfsDisable8dot3NameCreation enabled, thus preventing R from getting a
8.3 path to the temp directory.
I am taking the liberty of Cc-ing R-devel because this might warrant
adding a shQuote() call to install.packages().
Thanks for looking into this.
--
David K Stevens, PhD, PE
Environmental Engineering Division
Civil and Environmental Engineering
Utah State University
8200 Old Main Hill
Logan, UT 83200-8200
(435) 797-3229
david.stevens at usu.edu<mailto:david.stevens at usu.edu>
[[alternative HTML version deleted]]
On Fri, 13 Dec 2019 13:19:54 +0000 David Stevens <david.stevens at usu.edu> wrote:> There are binary versions available but the source versions are > later:Okay, that would be the reason why would R on Windows try to install a source package instead of a binary package. One can also see that callr has just been successfully installed from a binary package (that took time to be built from a freshly updated source package and was unavailable yesterday) -- but now there are other updated packages that cannot be installed. On Fri, 13 Dec 2019 13:14:28 +0000 David Stevens <david.stevens at usu.edu> wrote:> I read a thread elsewhere that said a work around is to run > options(pkgType='binary') before installation and the problem went > away. Does this help.Yes, this again points us at the differences between installing source packages and "win.binary" packages.> Yes, this is the case. I do the regular Windows 10 updates and update > R and RStudio as soon as I am aware there's a new version out. I > haven't explicitly change %USERPROFILE%.Since new packages are published all the time, it is likely that your R installation was able to install source packages successfully, until recently.> tempdir() gives > > tempdir() > [1] "C:\\Users\\David Stevens\\AppData\\Local\\Temp\\RtmpQpqh0t"Yes, this is a problem. file.path(tempdir(), "downloaded_packages") is passed to download.packages() and used to store the downloaded files. Eventually, download.packages() returns destination file paths (which now contain spaces), which are then passed to what amounts to: system2( command = file.path(R.home("bin"), "R"), args = c("CMD", "INSTALL", path), ... ) system2() uses paste(c(env, shQuote(command), args), collapse = " ") to form a command line and eventually passes that command line to CreateProcess(). The path value is left unquoted, causing the error observed above. I believe that this is a bug in install.packages() and that the `fil` argument in [*] should be quoted using shQuote() just like it is quoted in all other invocations of R CMD INSTALL in the same file. A workaround that should have worked but didn't was to pass a writeable path without spaces as a destdir = ... argument to install.packages(). I am not sure why did install.packages() decide to use C:/myRLib as the library instead of a temp directory to download files in (we just need a temp directory, not a separate library). Try the following again in a clean session? install.packages( c('lmerTest', 'quantreg', 'rmarkdown', 'SparseM'), lib = .libPaths()[1L], destdir = 'c:/myRLib' ) -- Best regards, Ivan [*] https://github.com/wch/r-source/blob/e554f7f12b22868bdae51aadaeea4d56c9f87a32/src/library/utils/R/packages2.R#L839
I started a new session and entered> rm(list=ls()) > install.packages(+ c('lmerTest', 'quantreg', 'rmarkdown', 'SparseM'), + lib = .libPaths()[1L], + destdir = 'c:/myRLib' + ) And this is the result: There are binary versions available but the source versions are later: binary source needs_compilation lmerTest 3.1-0 3.1-1 FALSE quantreg 5.52 5.54 TRUE rmarkdown 1.18 2.0 FALSE SparseM 1.77 1.78 TRUE installing the source packages ?lmerTest?, ?quantreg?, ?rmarkdown?, ?SparseM? trying URL 'https://cloud.r-project.org/src/contrib/lmerTest_3.1-1.tar.gz' Content type 'application/x-gzip' length 194447 bytes (189 KB) downloaded 189 KB trying URL 'https://cloud.r-project.org/src/contrib/quantreg_5.54.tar.gz' Content type 'application/x-gzip' length 995368 bytes (972 KB) downloaded 972 KB trying URL 'https://cloud.r-project.org/src/contrib/rmarkdown_2.0.tar.gz' Content type 'application/x-gzip' length 3186667 bytes (3.0 MB) downloaded 3.0 MB trying URL 'https://cloud.r-project.org/src/contrib/SparseM_1.78.tar.gz' Content type 'application/x-gzip' length 735024 bytes (717 KB) downloaded 717 KB * installing *source* package 'lmerTest' ... ** package 'lmerTest' successfully unpacked and MD5 sums checked ** using staged installation ** R ** data *** moving datasets to lazyload DB ** inst ** byte-compile and prepare package for lazy loading Fatal error: cannot open file 'C:\Users\David': No such file or directory ERROR: lazy loading failed for package 'lmerTest' * removing 'C:/myRLib/lmerTest' Warning in install.packages : installation of package ?lmerTest? had non-zero exit status * installing *source* package 'rmarkdown' ... ** package 'rmarkdown' successfully unpacked and MD5 sums checked ** using staged installation ** R ** inst ** byte-compile and prepare package for lazy loading Fatal error: cannot open file 'C:\Users\David': No such file or directory ERROR: lazy loading failed for package 'rmarkdown' * removing 'C:/myRLib/rmarkdown' Warning in install.packages : installation of package ?rmarkdown? had non-zero exit status * installing *source* package 'SparseM' ... ** package 'SparseM' successfully unpacked and MD5 sums checked ** using staged installation ** libs *** arch - i386 C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c bckslv.f -o bckslv.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c chol.f -o chol.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c chol2csr.f -o chol2csr.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c cholesky.f -o cholesky.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c csr.f -o csr.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c extract.f -o extract.o C:/Rtools/mingw_32/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O3 -Wall -std=gnu99 -mtune=generic -c init.c -o init.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c sparskit.f -o sparskit.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c subscr.f -o subscr.o C:/Rtools/mingw_32/bin/gcc -shared -s -static-libgcc -o SparseM.dll tmp.def bckslv.o chol.o chol2csr.o cholesky.o csr.o extract.o init.o sparskit.o subscr.o -lgfortran -lm -lquadmath -LC:/PROGRA~1/R/R-3.6.2/bin/i386 -lR installing to C:/myRLib/00LOCK-SparseM/00new/SparseM/libs/i386 *** arch - x64 C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c bckslv.f -o bckslv.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c chol.f -o chol.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c chol2csr.f -o chol2csr.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c cholesky.f -o cholesky.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c csr.f -o csr.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c extract.f -o extract.o C:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O2 -Wall -std=gnu99 -mtune=generic -c init.c -o init.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c sparskit.f -o sparskit.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c subscr.f -o subscr.o C:/Rtools/mingw_64/bin/gcc -shared -s -static-libgcc -o SparseM.dll tmp.def bckslv.o chol.o chol2csr.o cholesky.o csr.o extract.o init.o sparskit.o subscr.o -lgfortran -lm -lquadmath -LC:/PROGRA~1/R/R-3.6.2/bin/x64 -lR installing to C:/myRLib/00LOCK-SparseM/00new/SparseM/libs/x64 ** R ** data ** demo ** inst ** byte-compile and prepare package for lazy loading Fatal error: cannot open file 'C:\Users\David': No such file or directory ERROR: lazy loading failed for package 'SparseM' * removing 'C:/myRLib/SparseM' Warning in install.packages : installation of package ?SparseM? had non-zero exit status * installing *source* package 'quantreg' ... ** package 'quantreg' successfully unpacked and MD5 sums checked ** using staged installation ** libs *** arch - i386 C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c boot.f -o boot.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c bound.f -o bound.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c boundc.f -o boundc.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c brute.f -o brute.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c chlfct.f -o chlfct.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c cholesky.f -o cholesky.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c combos.f -o combos.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c crqf.f -o crqf.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c crqfnb.f -o crqfnb.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c dsel05.f -o dsel05.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c etime.f -o etime.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c extract.f -o extract.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c idmin.f -o idmin.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c iswap.f -o iswap.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c kuantiles.f -o kuantiles.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c linpack.f -o linpack.o C:/Rtools/mingw_32/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O3 -Wall -std=gnu99 -mtune=generic -c mcmb.c -o mcmb.o mcmb.c: In function 'func': mcmb.c:176:32: warning: variable 'large' set but not used [-Wunused-but-set-variable] double taustar, pwtsum, ans, large; ^ mcmb.c:175:16: warning: variable 'mm' set but not used [-Wunused-but-set-variable] unsigned int mm; ^ C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c penalty.f -o penalty.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c powell.f -o powell.o C:/Rtools/mingw_32/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O3 -Wall -std=gnu99 -mtune=generic -c quantreg_init.c -o quantreg_init.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rls.f -o rls.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rq0.f -o rq0.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rq1.f -o rq1.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rqbr.f -o rqbr.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rqfn.f -o rqfn.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rqfnb.f -o rqfnb.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rqfnc.f -o rqfnc.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c rqs.f -o rqs.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c sakj.f -o sakj.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c sparskit2.f -o sparskit2.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c srqfn.f -o srqfn.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c srqfnc.f -o srqfnc.o C:/Rtools/mingw_32/bin/gfortran -O3 -mtune=generic -c srtpai.f -o srtpai.o C:/Rtools/mingw_32/bin/gcc -shared -s -static-libgcc -o quantreg.dll tmp.def boot.o bound.o boundc.o brute.o chlfct.o cholesky.o combos.o crqf.o crqfnb.o dsel05.o etime.o extract.o idmin.o iswap.o kuantiles.o linpack.o mcmb.o penalty.o powell.o quantreg_init.o rls.o rq0.o rq1.o rqbr.o rqfn.o rqfnb.o rqfnc.o rqs.o sakj.o sparskit2.o srqfn.o srqfnc.o srtpai.o -LC:/PROGRA~1/R/R-3.6.2/bin/i386 -lRlapack -LC:/PROGRA~1/R/R-3.6.2/bin/i386 -lRblas -lgfortran -lm -lquadmath -lgfortran -lm -lquadmath -LC:/PROGRA~1/R/R-3.6.2/bin/i386 -lR installing to C:/myRLib/00LOCK-quantreg/00new/quantreg/libs/i386 *** arch - x64 C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c boot.f -o boot.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c bound.f -o bound.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c boundc.f -o boundc.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c brute.f -o brute.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c chlfct.f -o chlfct.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c cholesky.f -o cholesky.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c combos.f -o combos.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c crqf.f -o crqf.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c crqfnb.f -o crqfnb.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c dsel05.f -o dsel05.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c etime.f -o etime.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c extract.f -o extract.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c idmin.f -o idmin.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c iswap.f -o iswap.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c kuantiles.f -o kuantiles.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c linpack.f -o linpack.o C:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O2 -Wall -std=gnu99 -mtune=generic -c mcmb.c -o mcmb.o mcmb.c: In function 'func': mcmb.c:176:32: warning: variable 'large' set but not used [-Wunused-but-set-variable] double taustar, pwtsum, ans, large; ^ mcmb.c:175:16: warning: variable 'mm' set but not used [-Wunused-but-set-variable] unsigned int mm; ^ C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c penalty.f -o penalty.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c powell.f -o powell.o C:/Rtools/mingw_64/bin/gcc -I"C:/PROGRA~1/R/R-3.6.2/include" -DNDEBUG -O2 -Wall -std=gnu99 -mtune=generic -c quantreg_init.c -o quantreg_init.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rls.f -o rls.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rq0.f -o rq0.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rq1.f -o rq1.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rqbr.f -o rqbr.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rqfn.f -o rqfn.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rqfnb.f -o rqfnb.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rqfnc.f -o rqfnc.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c rqs.f -o rqs.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c sakj.f -o sakj.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c sparskit2.f -o sparskit2.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c srqfn.f -o srqfn.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c srqfnc.f -o srqfnc.o C:/Rtools/mingw_64/bin/gfortran -O2 -mtune=generic -c srtpai.f -o srtpai.o C:/Rtools/mingw_64/bin/gcc -shared -s -static-libgcc -o quantreg.dll tmp.def boot.o bound.o boundc.o brute.o chlfct.o cholesky.o combos.o crqf.o crqfnb.o dsel05.o etime.o extract.o idmin.o iswap.o kuantiles.o linpack.o mcmb.o penalty.o powell.o quantreg_init.o rls.o rq0.o rq1.o rqbr.o rqfn.o rqfnb.o rqfnc.o rqs.o sakj.o sparskit2.o srqfn.o srqfnc.o srtpai.o -LC:/PROGRA~1/R/R-3.6.2/bin/x64 -lRlapack -LC:/PROGRA~1/R/R-3.6.2/bin/x64 -lRblas -lgfortran -lm -lquadmath -lgfortran -lm -lquadmath -LC:/PROGRA~1/R/R-3.6.2/bin/x64 -lR installing to C:/myRLib/00LOCK-quantreg/00new/quantreg/libs/x64 ** R ** data ** demo ** inst ** byte-compile and prepare package for lazy loading Fatal error: cannot open file 'C:\Users\David': No such file or directory ERROR: lazy loading failed for package 'quantreg' * removing 'C:/myRLib/quantreg' Warning in install.packages : installation of package ?quantreg? had non-zero exit status On 12/13/2019 7:12 AM, Ivan Krylov wrote: install.packages( c('lmerTest', 'quantreg', 'rmarkdown', 'SparseM'), lib = .libPaths()[1L], destdir = 'c:/myRLib' ) -- David K Stevens, PhD, PE Environmental Engineering Division Civil and Environmental Engineering Utah State University 8200 Old Main Hill Logan, UT 83200-8200 (435) 797-3229 david.stevens at usu.edu<mailto:david.stevens at usu.edu> [[alternative HTML version deleted]]