thr3ads.net - R devel - [Rd] Issues with libcurl + HTTP status codes (eg. 403, 404) [Aug 2015]

If this information is useful, please help other people find it:
Share via:

Kevin Ushey

2015-Aug-25 19:54 UTC

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

Hi all,

The following fails for me (on OS X, although I imagine it's the same
on other platforms using libcurl):

    options(download.file.method = "libcurl")
    options(repos = c(CRAN = "https://cran.rstudio.com/", CRANextra
"http://www.stats.ox.ac.uk/pub/RWin"))
    install.packages("lattice") ## could be any package

gives me:

    > options(download.file.method = "libcurl")
    > options(repos = c(CRAN = "https://cran.rstudio.com/",
CRANextra
= "http://www.stats.ox.ac.uk/pub/RWin"))
    > install.packages("lattice") ## coudl be any package
    Installing package into ?/Users/kevinushey/Library/R/3.2/library?
    (as ?lib? is unspecified)
    Error: Line starting '<!DOCTYPE HTML PUBLI ...' is malformed!

This seems to come from a call to `available.packages()` to a URL that
doesn't exist on the server (likely when querying PACKAGES on the
CRANextra repo)

Eg.

    > URL <- "http://www.stats.ox.ac.uk/pub/RWin"
    > available.packages(URL, method = "internal")
    Warning: unable to access index for repository
http://www.stats.ox.ac.uk/pub/RWin
         Package Version Priority Depends Imports LinkingTo Suggests
Enhances License License_is_FOSS
        License_restricts_use OS_type Archs MD5sum NeedsCompilation
File Repository
    > available.packages(URL, method = "libcurl")
    Error: Line starting '<!DOCTYPE HTML PUBLI ...' is malformed!

It looks like libcurl downloads and retrieves the 403 page itself,
rather than reporting that it was actually forbidden, e.g.:

    >
download.file("http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz",
tempfile(), method = "libcurl")
    trying URL
'http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz'
    Content type 'text/html; charset=iso-8859-1' length 339 bytes
    =================================================    downloaded 339 bytes

Using `method = "internal"` gives an error related to the inability to
access that URL due to the HTTP status 403.

The overarching issue here is that package installation shouldn't fail
even if libcurl fails to access one of the repositories set.
> sessionInfo()R version 3.2.2 (2015-08-14)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.4 (Yosemite)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] testthat_0.8.1.0.99  knitr_1.11           devtools_1.5.0.9001
[4] BiocInstaller_1.15.5

loaded via a namespace (and not attached):
 [1] httr_1.0.0     R6_2.0.0.9000  tools_3.2.2    parallel_3.2.2 whisker_0.3-2
 [6] RCurl_1.95-4.1 memoise_0.2.1  stringr_0.6.2  digest_0.6.4   evaluate_0.7.2

Thanks,
Kevin

Martin Morgan

2015-Aug-25 20:11 UTC

head link

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

On 08/25/2015 12:54 PM, Kevin Ushey wrote:> Hi all,
>
> The following fails for me (on OS X, although I imagine it's the same
> on other platforms using libcurl):
>
>      options(download.file.method = "libcurl")
>      options(repos = c(CRAN = "https://cran.rstudio.com/",
CRANextra > "http://www.stats.ox.ac.uk/pub/RWin"))
>      install.packages("lattice") ## could be any package
>
> gives me:
>
>      > options(download.file.method = "libcurl")
>      > options(repos = c(CRAN = "https://cran.rstudio.com/",
CRANextra
> = "http://www.stats.ox.ac.uk/pub/RWin"))
>      > install.packages("lattice") ## coudl be any package
>      Installing package into ?/Users/kevinushey/Library/R/3.2/library?
>      (as ?lib? is unspecified)
>      Error: Line starting '<!DOCTYPE HTML PUBLI ...' is
malformed!
>
> This seems to come from a call to `available.packages()` to a URL that
> doesn't exist on the server (likely when querying PACKAGES on the
> CRANextra repo)
>
> Eg.
>
>      > URL <- "http://www.stats.ox.ac.uk/pub/RWin"
>      > available.packages(URL, method = "internal")
>      Warning: unable to access index for repository
> http://www.stats.ox.ac.uk/pub/RWin
>           Package Version Priority Depends Imports LinkingTo Suggests
> Enhances License License_is_FOSS
>          License_restricts_use OS_type Archs MD5sum NeedsCompilation
> File Repository
>      > available.packages(URL, method = "libcurl")
>      Error: Line starting '<!DOCTYPE HTML PUBLI ...' is
malformed!
>
> It looks like libcurl downloads and retrieves the 403 page itself,
> rather than reporting that it was actually forbidden, e.g.:
>
>      >
download.file("http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz",
> tempfile(), method = "libcurl")
>      trying URL
'http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz'
>      Content type 'text/html; charset=iso-8859-1' length 339 bytes
>      =================================================>      downloaded
339 bytes
>
> Using `method = "internal"` gives an error related to the
inability to
> access that URL due to the HTTP status 403.
>
> The overarching issue here is that package installation shouldn't fail
> even if libcurl fails to access one of the repositories set.
>
With

 > R.version.string
[1] "R version 3.2.2 Patched (2015-08-25 r69179)"

the behavior is to warn with an indication of the repository for which the 
problem occurs

 > URL <- "http://www.stats.ox.ac.uk/pub/RWin"
 > available.packages(URL, method="libcurl")
Warning: unable to access index for repository
http://www.stats.ox.ac.uk/pub/RWin:
   Line starting '<!DOCTYPE HTML PUBLI ...' is malformed!
      Package Version Priority Depends Imports LinkingTo Suggests Enhances
      License License_is_FOSS License_restricts_use OS_type Archs MD5sum
      NeedsCompilation File Repository
 > available.packages(URL, method="internal")
Warning: unable to access index for repository
http://www.stats.ox.ac.uk/pub/RWin:
   cannot open URL 'http://www.stats.ox.ac.uk/pub/RWin/PACKAGES'
      Package Version Priority Depends Imports LinkingTo Suggests Enhances
      License License_is_FOSS License_restricts_use OS_type Archs MD5sum
      NeedsCompilation File Repository

Does that work for you / address the problem?

Martin
>> sessionInfo()
> R version 3.2.2 (2015-08-14)
> Platform: x86_64-apple-darwin13.4.0 (64-bit)
> Running under: OS X 10.10.4 (Yosemite)
>
> locale:
> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
>
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
>
> other attached packages:
> [1] testthat_0.8.1.0.99  knitr_1.11           devtools_1.5.0.9001
> [4] BiocInstaller_1.15.5
>
> loaded via a namespace (and not attached):
>   [1] httr_1.0.0     R6_2.0.0.9000  tools_3.2.2    parallel_3.2.2
whisker_0.3-2
>   [6] RCurl_1.95-4.1 memoise_0.2.1  stringr_0.6.2  digest_0.6.4  
evaluate_0.7.2
>
> Thanks,
> Kevin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

Kevin Ushey

2015-Aug-25 20:30 UTC

head link

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

Hi Martin,

Indeed it does (and I should have confirmed myself with R-patched and
R-devel before posting...)

Thanks, and sorry for the noise.
Kevin

On Tue, Aug 25, 2015, 13:11 Martin Morgan <mtmorgan at fredhutch.org>
wrote:
> On 08/25/2015 12:54 PM, Kevin Ushey wrote:
> > Hi all,
> >
> > The following fails for me (on OS X, although I imagine it's the
same
> > on other platforms using libcurl):
> >
> >      options(download.file.method = "libcurl")
> >      options(repos = c(CRAN = "https://cran.rstudio.com/",
CRANextra > > "http://www.stats.ox.ac.uk/pub/RWin"))
> >      install.packages("lattice") ## could be any package
> >
> > gives me:
> >
> >      > options(download.file.method = "libcurl")
> >      > options(repos = c(CRAN =
"https://cran.rstudio.com/", CRANextra
> > = "http://www.stats.ox.ac.uk/pub/RWin"))
> >      > install.packages("lattice") ## coudl be any
package
> >      Installing package into ?/Users/kevinushey/Library/R/3.2/library?
> >      (as ?lib? is unspecified)
> >      Error: Line starting '<!DOCTYPE HTML PUBLI ...' is
malformed!
> >
> > This seems to come from a call to `available.packages()` to a URL that
> > doesn't exist on the server (likely when querying PACKAGES on the
> > CRANextra repo)
> >
> > Eg.
> >
> >      > URL <- "http://www.stats.ox.ac.uk/pub/RWin"
> >      > available.packages(URL, method = "internal")
> >      Warning: unable to access index for repository
> > http://www.stats.ox.ac.uk/pub/RWin
> >           Package Version Priority Depends Imports LinkingTo Suggests
> > Enhances License License_is_FOSS
> >          License_restricts_use OS_type Archs MD5sum NeedsCompilation
> > File Repository
> >      > available.packages(URL, method = "libcurl")
> >      Error: Line starting '<!DOCTYPE HTML PUBLI ...' is
malformed!
> >
> > It looks like libcurl downloads and retrieves the 403 page itself,
> > rather than reporting that it was actually forbidden, e.g.:
> >
> >      > download.file("
>
http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz
> ",
> > tempfile(), method = "libcurl")
> >      trying URL '
>
http://www.stats.ox.ac.uk/pub/RWin/bin/macosx/mavericks/contrib/3.2/PACKAGES.gz
> '
> >      Content type 'text/html; charset=iso-8859-1' length 339
bytes
> >      =================================================> >     
downloaded 339 bytes
> >
> > Using `method = "internal"` gives an error related to the
inability to
> > access that URL due to the HTTP status 403.
> >
> > The overarching issue here is that package installation shouldn't
fail
> > even if libcurl fails to access one of the repositories set.
> >
>
> With
>
>  > R.version.string
> [1] "R version 3.2.2 Patched (2015-08-25 r69179)"
>
> the behavior is to warn with an indication of the repository for which the
> problem occurs
>
>  > URL <- "http://www.stats.ox.ac.uk/pub/RWin"
>  > available.packages(URL, method="libcurl")
> Warning: unable to access index for repository
> http://www.stats.ox.ac.uk/pub/RWin:
>    Line starting '<!DOCTYPE HTML PUBLI ...' is malformed!
>       Package Version Priority Depends Imports LinkingTo Suggests Enhances
>       License License_is_FOSS License_restricts_use OS_type Archs MD5sum
>       NeedsCompilation File Repository
>  > available.packages(URL, method="internal")
> Warning: unable to access index for repository
> http://www.stats.ox.ac.uk/pub/RWin:
>    cannot open URL 'http://www.stats.ox.ac.uk/pub/RWin/PACKAGES'
>       Package Version Priority Depends Imports LinkingTo Suggests Enhances
>       License License_is_FOSS License_restricts_use OS_type Archs MD5sum
>       NeedsCompilation File Repository
>
> Does that work for you / address the problem?
>
> Martin
>
> >> sessionInfo()
> > R version 3.2.2 (2015-08-14)
> > Platform: x86_64-apple-darwin13.4.0 (64-bit)
> > Running under: OS X 10.10.4 (Yosemite)
> >
> > locale:
> > [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
> >
> > attached base packages:
> > [1] stats     graphics  grDevices utils     datasets  methods   base
> >
> > other attached packages:
> > [1] testthat_0.8.1.0.99  knitr_1.11           devtools_1.5.0.9001
> > [4] BiocInstaller_1.15.5
> >
> > loaded via a namespace (and not attached):
> >   [1] httr_1.0.0     R6_2.0.0.9000  tools_3.2.2    parallel_3.2.2
> whisker_0.3-2
> >   [6] RCurl_1.95-4.1 memoise_0.2.1  stringr_0.6.2  digest_0.6.4
>  evaluate_0.7.2
> >
> > Thanks,
> > Kevin
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
>
>
> --
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M1 B861
> Phone: (206) 667-2793
>
	[[alternative HTML version deleted]]

Possibly Parallel Threads

Search for more possibly parallel threads

R devel - Aug 2015 - Issues with libcurl + HTTP status codes (eg. 403, 404)

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

[Rd] Issues with libcurl + HTTP status codes (eg. 403, 404)

Possibly Parallel Threads