R help - Jun 2017 - Error in readRDS(dest) SOLVED

On Tue, 23-May-2017 at 12:20PM +0200, Martin Maechler wrote:

[...]

|> 
|> Given the above stack trace. 
|> It may be easier to just do
|> 
|>     debugonce(available.packages)
|>     install.packages("withr")
|> 
|> and then inside available.packages, (using 'n') step to the
|> point _before_ the tryCatch(...) call happens; there, e.g. use
|> 
|>       ls.str()
|> 
|> which gives an str() of all your local objects, notably 'dest'
|> and 'method'.
|> but you can also try other things once inside
|> available.packages().

I couldn't see any differences between R-3.3.3 (which works) and
R-3.4.0 (which doesn't) until I got to here, a few lines before the
download.file line:

Browse[2]> 
debug: dest <- file.path(tempdir(), paste0("repos_",
URLencode(repos,
    TRUE), ".rds"))
Browse[2]> 

When I check out those directories in a terminal, there's a big diffrence:

With R-3.4.0
~ > ll /tmp/RtmpFUhtpY
total 4
drwxr-xr-x 2 hrapgc hrapgc 4096 May 31 10:45 downloaded_packages/
-rw-r--r-- 1 hrapgc hrapgc    0 May 31 10:56
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds


With R-3.3.3
~ > ll /tmp/RtmpkPgL3A
total 380
drwxr-xr-x 2 hrapgc hrapgc   4096 May 31 11:01 downloaded_packages/
-rw-r--r-- 1 hrapgc hrapgc   8214 May 31 11:01 libloc_185_3165c7f52d5fdf96.rds
-rw-r--r-- 1 hrapgc hrapgc 372263 May 31 11:01
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds

So, if I could figure out what makes *that* difference I could get
somewhere.  I see there's considerably extra code in the newer of the
two versions of available.packages() but being a bear with a small
brain, I can't figure out what differences should be expected.  I have
no idea what populates those 'dest' directories.

TIA
-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}                   Great minds discuss ideas    
 _( Y )_  	         Average minds discuss events 
(:_~*~_:)                  Small minds discuss people  
 (_)-(_)  	                      ..... Eleanor Roosevelt
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.

On 05/31/2017 04:38 AM, Patrick Connolly wrote:
> On Tue, 23-May-2017 at 12:20PM +0200, Martin Maechler wrote:
> 
> [...]
> 
> |>
> |> Given the above stack trace.
> |> It may be easier to just do
> |>
> |>     debugonce(available.packages)
> |>     install.packages("withr")
> |>
> |> and then inside available.packages, (using 'n') step to the
> |> point _before_ the tryCatch(...) call happens; there, e.g. use
> |>
> |>       ls.str()
> |>
> |> which gives an str() of all your local objects, notably
'dest'
> |> and 'method'.
> |> but you can also try other things once inside
> |> available.packages().
> 
> I couldn't see any differences between R-3.3.3 (which works) and
> R-3.4.0 (which doesn't) until I got to here, a few lines before the
> download.file line:
> 
> Browse[2]>
> debug: dest <- file.path(tempdir(), paste0("repos_",
URLencode(repos,
>      TRUE), ".rds"))
> Browse[2]>
> 
> When I check out those directories in a terminal, there's a big
diffrence:
> 
> With R-3.4.0
> ~ > ll /tmp/RtmpFUhtpY
> total 4
> drwxr-xr-x 2 hrapgc hrapgc 4096 May 31 10:45 downloaded_packages/
> -rw-r--r-- 1 hrapgc hrapgc    0 May 31 10:56
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds
> 
> 
The file repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds 
was likely created earlier in your R session. Likely the download a few 
lines down

                     download.file(url = paste0(repos,
"/PACKAGES.rds"),
                                   destfile = dest, method = method,
                                   cacheOK = FALSE, quiet = TRUE, mode = 
"wb")

'succeeded' but created a zero-length file.

You could try to troubleshoot this with something like the following, 
downloading to a temporary location

   dest = tempfile()
   url = "http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds"
   download.file(url, dest)
   file.size(dest)

If this succeeds (it should download a file of several hundred KB), then 
try adding the options method, cacheOK, quiet, mode to the 
download.file() call. 'method' can be determined when you are in 
available.packages while debugging; if R says that it is missing, then 
it will be assigned, in download.file, to either 
getOption("download.file.method") or (if the option is NULL or
"auto")
"libcurl".

If the download 'succeeds' but the temporary file created is 0 bytes, 
then it would be good to share the problematic command with us.

Martin Morgan
> With R-3.3.3
> ~ > ll /tmp/RtmpkPgL3A
> total 380
> drwxr-xr-x 2 hrapgc hrapgc   4096 May 31 11:01 downloaded_packages/
> -rw-r--r-- 1 hrapgc hrapgc   8214 May 31 11:01
libloc_185_3165c7f52d5fdf96.rds
> -rw-r--r-- 1 hrapgc hrapgc 372263 May 31 11:01
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds
> 
> So, if I could figure out what makes *that* difference I could get
> somewhere.  I see there's considerably extra code in the newer of the
> two versions of available.packages() but being a bear with a small
> brain, I can't figure out what differences should be expected.  I have
> no idea what populates those 'dest' directories.
> 
> TIA
> 

This email message may contain legally privileged and/or...{{dropped:2}}

On Wed, 31-May-2017 at 10:05AM -0400, Martin Morgan wrote:

|> On 05/31/2017 04:38 AM, Patrick Connolly wrote:

|> >When I check out those directories in a terminal, there's a big
diffrence:
|> >
|> >With R-3.4.0
|> >~ > ll /tmp/RtmpFUhtpY
|> >total 4
|> >drwxr-xr-x 2 hrapgc hrapgc 4096 May 31 10:45 downloaded_packages/
|> >-rw-r--r-- 1 hrapgc hrapgc    0 May 31 10:56
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds
|> >
|> >
|> 
|> The file
|> repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds was
|> likely created earlier in your R session. Likely the download a few
|> lines down
|> 
|>                     download.file(url = paste0(repos,
"/PACKAGES.rds"),
|>                                   destfile = dest, method = method,
|>                                   cacheOK = FALSE, quiet = TRUE,
|> mode = "wb")
|> 
|> 'succeeded' but created a zero-length file.
|> 
|> You could try to troubleshoot this with something like the
|> following, downloading to a temporary location
|> 
|>   dest = tempfile()
|>   url =
"http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds"
|>   download.file(url, dest)
|>   file.size(dest)

That works fine using R-3.3.3 but with R-3.4.0, this happens:
>  download.file(url, dest)
trying URL 'http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds'
Error in download.file(url, dest) : 
  cannot open URL
'http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds'
In addition: Warning message:
In download.file(url, dest) :
  URL 'http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds': status
was 'Couldn't connect to server'
> 
Which seems to indicate that something else is preventing connexion to
the server.  Is there something that's changed with the way proxy
servers are used? was the question that led me to check out the
changes in R 3.4.0 which mentions visible changes to download.file().
So I tried method = "internal" which worked!

So making the same setting to the call to install.packages() fixed my
problem.


Thanks for all the suggestions which led me to the solution.  I just
wonder if there could be a more informative error message.

-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}                   Great minds discuss ideas    
 _( Y )_  	         Average minds discuss events 
(:_~*~_:)                  Small minds discuss people  
 (_)-(_)  	                      ..... Eleanor Roosevelt
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.