R help - May 2017 - Error in readRDS(dest) (was Re: Error with installed.packages with R 3.4.0 on Windows)

On Mon, 22-May-2017 at 05:43AM -0400, Martin Morgan wrote:

|> On 05/22/2017 05:10 AM, Patrick Connolly wrote:

|> >Apparently it isn't harmless.
|> >
|> >>install.packages("withr")
|> >Error in readRDS(dest) : error reading from connection
|> 
|> that seems like a plain-old network connectivity issue, or perhaps
|> an issue with the CRAN mirror you're using. Can you debug on your
|> end, e.g,.
|> 
|>   options(error=recover)
|>   install.packages("withr")
|>   ...
|> 
|> then select the 'frame' where the error occurs, look around
|> 
|>   ls()
|> 
|> find the value of 'dest', and e.g., try to open dest in your 
browser.

This is what I get
>   options(error=recover)
>   install.packages("withr")
^C

Enter a frame number, or 0 to exit   

1: install.packages("withr")
2: available.packages(contriburl = contriburl, method = method)
3: tryCatch({
    download.file(url = paste0(repos, "/PACKAGES.rds"), destfile
4: tryCatchList(expr, classes, parentenv, handlers)
5: tryCatchOne(expr, names, parentenv, handlers[[1]])
6: doTryCatch(return(expr), name, parentenv, handler)
7: download.file(url = paste0(repos, "/PACKAGES.rds"), destfile =
dest, method

Selection: 7
Called from: eval(substitute(browser(skipCalls = skip), list(skip = 7 - which)),
    envir = sys.frame(which))
Browse[1]> dest
Error during wrapup: object 'dest' not found

That indicates to me that the problem is further back but I have no
idea how make use of that information.


Browse[1]> ls()
[1] "cacheOK"  "destfile" "extra"   
"method"   "mode"     "quiet"    "url"
Browse[1]> url
[1] "http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds"
Browse[1]> destfile
[1]
"/tmp/RtmpplJSrB/repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds"
Browse[1]> 

The destfile above is zero-length and I suppose is where dest is
intended to end up.

Where else should I be looking?  Earlier installations never had this
issue so I don't have anything to compare.

TIA
-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}                   Great minds discuss ideas    
 _( Y )_  	         Average minds discuss events 
(:_~*~_:)                  Small minds discuss people  
 (_)-(_)  	                      ..... Eleanor Roosevelt
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.

>>>>> Patrick Connolly <p_connolly at slingshot.co.nz>
>>>>>     on Tue, 23 May 2017 20:47:22 +1200 writes:
    > On Mon, 22-May-2017 at 05:43AM -0400, Martin Morgan wrote:
    > |> On 05/22/2017 05:10 AM, Patrick Connolly wrote:

    > |> >Apparently it isn't harmless.
    > |> >
    > |> >>install.packages("withr")
    > |> >Error in readRDS(dest) : error reading from connection
    > |> 
    > |> that seems like a plain-old network connectivity issue, or
perhaps
    > |> an issue with the CRAN mirror you're using. Can you debug on
your
    > |> end, e.g,.
    > |> 
    > |>   options(error=recover)
    > |>   install.packages("withr")
    > |>   ...
    > |> 
    > |> then select the 'frame' where the error occurs, look
around
    > |> 
    > |>   ls()
    > |> 
    > |> find the value of 'dest', and e.g., try to open dest in
your  browser.

    > This is what I get

    >> options(error=recover)
    >> install.packages("withr")
    > ^C

    > Enter a frame number, or 0 to exit   

    > 1: install.packages("withr")
    > 2: available.packages(contriburl = contriburl, method = method)
    > 3: tryCatch({
    > download.file(url = paste0(repos, "/PACKAGES.rds"), destfile
    > 4: tryCatchList(expr, classes, parentenv, handlers)
    > 5: tryCatchOne(expr, names, parentenv, handlers[[1]])
    > 6: doTryCatch(return(expr), name, parentenv, handler)
    > 7: download.file(url = paste0(repos, "/PACKAGES.rds"),
destfile = dest, method

    > Selection: 7

'7'  was the wrong choice:  'dest' exists in the frame that
     *calls* download.file, in this case, it is frame 2, i.e.,
     inside available.packages(.)  where the  tryCatch() call to
     download.file() happens.

Given the above stack trace. 
It may be easier to just do

    debugonce(available.packages)
    install.packages("withr")

and then inside available.packages, (using 'n') step to the
point _before_ the tryCatch(...) call happens; there, e.g. use

      ls.str()

which gives an str() of all your local objects, notably 'dest'
and 'method'.
but you can also try other things once inside
available.packages().

Martin


    > Called from: eval(substitute(browser(skipCalls = skip), list(skip = 7 -
which)),
    > envir = sys.frame(which))
    > Browse[1]> dest
    > Error during wrapup: object 'dest' not found

    > That indicates to me that the problem is further back but I have no
    > idea how make use of that information.


    > Browse[1]> ls()
    > [1] "cacheOK"  "destfile" "extra"   
"method"   "mode"     "quiet"    "url"
    > Browse[1]> url
    > [1]
"http://cran.stat.auckland.ac.nz/src/contrib/PACKAGES.rds"
    > Browse[1]> destfile
    > [1]
"/tmp/RtmpplJSrB/repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds"
    > Browse[1]> 

    > The destfile above is zero-length and I suppose is where dest is
    > intended to end up.

    > Where else should I be looking?  Earlier installations never had this
    > issue so I don't have anything to compare.

    > TIA
    > --

On Tue, 23-May-2017 at 12:20PM +0200, Martin Maechler wrote:

[...]

|> 
|> Given the above stack trace. 
|> It may be easier to just do
|> 
|>     debugonce(available.packages)
|>     install.packages("withr")
|> 
|> and then inside available.packages, (using 'n') step to the
|> point _before_ the tryCatch(...) call happens; there, e.g. use
|> 
|>       ls.str()

I got:

contriburl :  chr [1:2] "http://cran.stat.auckland.ac.nz/src/contrib"
...
dest :  chr
"/tmp/Rtmp6D0KNY/repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds"
fields :  chr [1:16] "Package" "Version"
"Priority" "Depends" "Imports"
"LinkingTo" ...
filters :  NULL
localcran :  logi FALSE
method : <missing>
repos :  chr "http://cran.stat.auckland.ac.nz/src/contrib"
requiredFields :  chr [1:16] "Package" "Version"
"Priority" "Depends" "Imports"
"LinkingTo" ...
res :  chr[0 , 1:17] 
type :  chr "source"


So I thought the missing method was the problem.  However, it's
exactly the same with R-3.3.3.  So still no answwer.

Then I tried:

debugonce(readRDS)
install.packages("withr")

Browse[2]> con
A connection with    
description
"/tmp/RtmpqzKzzK/repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds"
class       "gzfile"               
mode        "rb"                   
text        "binary"               
opened      "opened"               
can read    "yes"                  
can write   "no"

## With R-3.3.3

Browse[2]> con       
                                                description 
"/home/hrapgc/local/R-3.3.3/library/abind/Meta/package.rds" 
                                                      class 
                                                   "gzfile" 
                                                       mode 
                                                       "rb" 
                                                       text 
                                                   "binary" 
                                                     opened 
                                                   "opened" 
                                                   can read 
                                                      "yes" 
                                                  can write 
                                                       "no" 

So, in 3.4.0, con refers to a repo and is zero length whereas 3.3.3
refers to a library.  To me that would appear to be a major
difference.  That must have something to do with it.  But what?

Since the latter works and the former doesn't, it seems as though the
problem is with 3.4.0's readRDS().  But others seem not to have the
same problem.  I'm no closer to understanding what's happenning.


-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}                   Great minds discuss ideas    
 _( Y )_  	         Average minds discuss events 
(:_~*~_:)                  Small minds discuss people  
 (_)-(_)  	                      ..... Eleanor Roosevelt
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.

On Tue, 23-May-2017 at 12:20PM +0200, Martin Maechler wrote:

[...]

|> 
|> Given the above stack trace. 
|> It may be easier to just do
|> 
|>     debugonce(available.packages)
|>     install.packages("withr")
|> 
|> and then inside available.packages, (using 'n') step to the
|> point _before_ the tryCatch(...) call happens; there, e.g. use
|> 
|>       ls.str()
|> 
|> which gives an str() of all your local objects, notably 'dest'
|> and 'method'.
|> but you can also try other things once inside
|> available.packages().

I couldn't see any differences between R-3.3.3 (which works) and
R-3.4.0 (which doesn't) until I got to here, a few lines before the
download.file line:

Browse[2]> 
debug: dest <- file.path(tempdir(), paste0("repos_",
URLencode(repos,
    TRUE), ".rds"))
Browse[2]> 

When I check out those directories in a terminal, there's a big diffrence:

With R-3.4.0
~ > ll /tmp/RtmpFUhtpY
total 4
drwxr-xr-x 2 hrapgc hrapgc 4096 May 31 10:45 downloaded_packages/
-rw-r--r-- 1 hrapgc hrapgc    0 May 31 10:56
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds


With R-3.3.3
~ > ll /tmp/RtmpkPgL3A
total 380
drwxr-xr-x 2 hrapgc hrapgc   4096 May 31 11:01 downloaded_packages/
-rw-r--r-- 1 hrapgc hrapgc   8214 May 31 11:01 libloc_185_3165c7f52d5fdf96.rds
-rw-r--r-- 1 hrapgc hrapgc 372263 May 31 11:01
repos_http%3A%2F%2Fcran.stat.auckland.ac.nz%2Fsrc%2Fcontrib.rds

So, if I could figure out what makes *that* difference I could get
somewhere.  I see there's considerably extra code in the newer of the
two versions of available.packages() but being a bear with a small
brain, I can't figure out what differences should be expected.  I have
no idea what populates those 'dest' directories.

TIA
-- 
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.   
   ___    Patrick Connolly   
 {~._.~}                   Great minds discuss ideas    
 _( Y )_  	         Average minds discuss events 
(:_~*~_:)                  Small minds discuss people  
 (_)-(_)  	                      ..... Eleanor Roosevelt
	  
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.