[R] [External] Re: read.csv fails in R console in Ubuntu terminal but works in RStudio after R 3.6.3 upgrade to R 4.0.2?

Rasmus Liland
2020-07-17T13:23:43Z

On 2020-07-17 07:54 -0400, Sam H wrote:
| On 2020-07-17 09:30 +0100, ruipbarradas wrote:
| | On 2020-07-16 20:59 -0500, luke-tierney at uiowa.edu wrote:
| | | ?s 08:45 de 15/07/20, Sam H escreveu:
| | | | Hi,
| | | |
| | | | I am trying to download some
| | | | data using read.csv and it works
| | | | perfectly in RStudio and fails
| | | | in the R console in the terminal
| | | | in Ubuntu 18.04 after upgrading
| | | | from R 3.6.3 to 4.0.2.
| | |
| | | On my Ubuntu system the download
| | | with read.csv succeeds in an R
| | | console if I set the HTTPUserAgent
| | | and download.file.method options to
| | | match the ones used by RStudio.
| | |
| | | Given how picky the server is being
| | | I would worry about whether this use
| | | is in line with the site's terms of
| | | service.
| |
| | Yes, I thought it's a site policy
| | issue too. But the file can be
| | accessed and read/downloaded from
| | RStudio and Firefox so apparently
| | there's no reason why R console
| | shouldn't .
|
| Hello,
|
| Thank you very much to you all to look into this.
|
| I came across this problem when I was using TTR::stockSymbols() (
|
https://github.com/joshuaulrich/TTR/blob/e6609b9f7621f3a4b1a204c159af61aebc89997e/R/WebData.R)
| .
|
| As a workaround I added this function
| to my private R package and instead of
| read.csv I am now using
| data.table::fread() which properly
| (without failing) downloads the file
| and reads it.

Dear Sam,

Good thing you solved this.

Like Luke said, to use read.csv you need
to set the HTTPUserAgent option:

	options("HTTPUserAgent"="User-Agent: RStudio Desktop
(1.3.959)")

... or with cURL directly:

	rasmus at twentyfive ~ % curl -H 'User-Agent: RStudio Desktop
(1.3.959)'
'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download'

?s 08:45 de 15/07/20, Sam H escreveu:
| Before upgrading this worked in the R
| console in the terminal also without
| any issues.

In version 3.6.3, I was not able to
run the lines

	> R.Version()$version.string
	[1] "R version 3.6.3 (2020-02-29)"
	> options()[c("download.file.method", "HTTPUserAgent")]
	$<NA>
	NULL

	$HTTPUserAgent
	[1] "R (3.6.3 x86_64-pc-linux-gnu x86_64 linux-gnu)"

	>
x<-"https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download"
	> read.csv(x, as.is=TRUE, na="n/a")
	Error in file(file, "rt") :
	  cannot open the connection to
'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download'
	In addition: Warning message:
	In file(file, "rt") :
	  cannot open URL
'https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download':
HTTP status was '403 Forbidden'
	>

Running data.table::fread in 4.0.2:

	> options()[c("download.file.method", "HTTPUserAgent")]
	$<NA>
	NULL

	$HTTPUserAgent
	[1] "R (4.0.2 x86_64-pc-linux-gnu x86_64 linux-gnu)"
	> x <-
"https://old.nasdaq.com/screening/companies-by-name.aspx?letter=0&exchange=1&render=download"
	> data.table::fread(x, header=TRUE)[1:2,]
	   Symbol               Name LastSale
	1:    TXG 10x Genomics, Inc.    89.19
	2:     YI          111, Inc.     6.53
	   MarketCap IPOyear        Sector
	1:    $8.77B    2019 Capital Goods
	2:  $537.81M    2018   Health Care
	                                           industry
	1: Biotechnology: Laboratory Analytical Instruments
	2:                         Medical/Nursing Services
	                       Summary Quote V9
	1: https://old.nasdaq.com/symbol/txg NA
	2:  https://old.nasdaq.com/symbol/yi NA

Does anyone know what data.table::fread
does different to read.csv here (so
setting HTTPUserAgent is not needed)?

Without HTTPUserAgent, I think
data.table::fread just reports something
like "libcurl/7.71.1", like read.csv
would have done ...

Best,
Rasmus