Hello I am trying read nc rainfall files directly from the UK centre for
hydrology and ecology website -
hourly:about 300Mb
https://catalogue.ceh.ac.uk/datastore/eidchub/fc9423d6-3d54-467f-bb2b-fc7357a3941f/
and daily about 90Mb
https://catalogue.ceh.ac.uk/datastore/eidchub/2ab15bf0-ad08-415c-ba64-831168be7293/precip/
I can download them "by hand" no problem using the ncdf4 package but
there's lots and I wanted to streamline the process
code I've used (which works for "by hand" downloading) is (for
example)
nc_data<-nc_open("
https://catalogue.ceh.ac.uk/datastore/eidchub/2ab15bf0-ad08-415c-ba64-831168be7293/precip/chess-met_precip_gb_1km_daily_19610301-19610331.nc
")
However, when I run this I don't get an error message but R just sits there
with the little red circle (at least 30 minutes for the 90Mb files)
What I'm wondering is two things - a)is the data in fact downloading but
it's just taking ages and I need to let it run and go and have a coffee
b)is there (I can't find
anything) within R which allows me to monitor the progress of the download,
if in fact it is taking place?
Thanks Nick Wray
[[alternative HTML version deleted]]
? Mon, 14 Nov 2022 13:04:31 +0000 Nick Wray <nickmwray at gmail.com> ?????:> nc_data<-nc_open(" > https://catalogue.ceh.ac.uk/datastore/eidchub/2ab15bf0-ad08-415c-ba64-831168be7293/precip/chess-met_precip_gb_1km_daily_19610301-19610331.nc > ") > > However, when I run this I don't get an error message but R just sits > there with the little red circle (at least 30 minutes for the 90Mb > files)Not sure why it just sits there, but looking at the package source code, I can see that ncdf4::nc_open eventually forwards its arguments to the nc_open function from the netcdf library: https://github.com/cran/ncdf4/blob/1.19/R/ncdf4.R#L568 https://github.com/cran/ncdf4/blob/1.19/src/ncdf.c#L862 https://docs.unidata.ucar.edu/netcdf-c/current/group__datasets.html#gaccbdb128a1640e204831256dbbc24d3e netcdf does support some form of remote file access, but only OPeNDAP-compatible URLs are supposed to work. Your links ask me to log in, which may mean that they won't work with netcdf. It might be easier to use one of the R packages that let you work with libcurl and give them the cookies from your browser in order to download the files first, then access them. See also: https://stackoverflow.com/a/70368135 What does ncdump say about these URLs? -- Best regards, Ivan [1]
On Mon, 14 Nov 2022, Nick Wray writes:> Hello I am trying read nc rainfall files directly from the UK centre for > hydrology and ecology website - > hourly:about 300Mb > https://catalogue.ceh.ac.uk/datastore/eidchub/fc9423d6-3d54-467f-bb2b-fc7357a3941f/ > and daily about 90Mb > https://catalogue.ceh.ac.uk/datastore/eidchub/2ab15bf0-ad08-415c-ba64-831168be7293/precip/ > > I can download them "by hand" no problem using the ncdf4 package but > there's lots and I wanted to streamline the process > > code I've used (which works for "by hand" downloading) is (for example) > > nc_data<-nc_open(" > https://catalogue.ceh.ac.uk/datastore/eidchub/2ab15bf0-ad08-415c-ba64-831168be7293/precip/chess-met_precip_gb_1km_daily_19610301-19610331.nc > ") > > However, when I run this I don't get an error message but R just sits there > with the little red circle (at least 30 minutes for the 90Mb files) > > What I'm wondering is two things - a)is the data in fact downloading but > it's just taking ages and I need to let it run and go and have a coffee > b)is there (I can't find > anything) within R which allows me to monitor the progress of the download, > if in fact it is taking place? > Thanks Nick Wray >Have you tried 'download.file'? It should provide a progress bar if the file size is known. I'd first download the file, and then read it. -- Enrico Schumann Lucerne, Switzerland http://enricoschumann.net