R help - Apr 2016 - Unexpected values obtained when reading in data using ncdf and ncdf4

Dear R Users,

I am encountering a problem when reading nc files into R using the ncdf and
ncdf4 libraries. The nc files are too large to attach an example (but if someone
is interested in helping out I could send a file privately via an online drive),
but the code is basic:

for(i in 1:length(thesenames[,1])){
   data <- nc_open(paste(INDIR, thesenames[i,c("wholename")],
sep=""), write=F)
   d.vars <- names(data$var)
   d.size <- (data$var[[length(d.vars)]])$size

   # Obtaining longitude and latitude values
   d.lon <- as.vector(ncvar_get(data, varid="lon", start=c(1,1),
count=c(d.size[1],d.size[2])))
   d.lat <- as.vector(ncvar_get(data, varid="lat", start=c(1,1),
count=c(d.size[1],d.size[2])))

   # Obtaining climate data values
   df.clim <- data.frame(rn=seq(1:length(d.lon)))
   for(y in 1:d.size[3]){
     df.clim[,1+y] <- as.vector(ncvar_get(data, varid=d.vars[length(d.vars)],
start=c(1,1,y), count=c(d.size[1],d.size[2],1)))
      names(df.clim)[1+y] <- paste("y",y,sep="")  }
   tosummarise[,,i] <- as.matrix(df.clim[,-1])
}

The data are temperature or precipitation, across space and time.

For most of the >250 files I have, there are no problems, but for around 8 of
these files, I get strange values. The data should be within a relatively narrow
range, yet I get values such as -8.246508e+07  or  7.659506e+11. The
particularly strange part is that these kind of values occur at regularly spaced
intervals across the data, usually within a single time step.

I have the same problem (including the exact same strange values) when using
ArcMap, yet the data provider assures me that the data look normal when using
CDO (climate data operators) to view them, and that there are no missing values.

I realise this is very difficult to diagnose without the nc files themselves, so
my questions are (1) Has anyone encountered something like this before?, (2) Is
there something I am failing to specify in the code when reading in?, and (3) Is
anyone interested in digging into this and willing to play around with the nc
files if I make them available privately?

Thanks very much in advance!
Louise





	[[alternative HTML version deleted]]

On Fri, Apr 22, 2016 at 1:32 AM, Louise Mair <louise.mair at slu.se>
wrote:
> Dear R Users,
>
> I am encountering a problem when reading nc files into R using the ncdf
> and ncdf4 libraries. The nc files are too large to attach an example (but
> if someone is interested in helping out I could send a file privately via
> an online drive), but the code is basic:
>
?[...]?


?Hi Louise,

I'm the author of the ncdf and ncdf4 libraries. What are the details --
what operating system are you running on, what version of R and the netcdf
library are you using?

If you make the files available to me I can take a look.

Regards,

--Dave Pierce
?




> for(i in 1:length(thesenames[,1])){
>    data <- nc_open(paste(INDIR, thesenames[i,c("wholename")],
sep=""),
> write=F)
>    d.vars <- names(data$var)
>    d.size <- (data$var[[length(d.vars)]])$size
>
>    # Obtaining longitude and latitude values
>    d.lon <- as.vector(ncvar_get(data, varid="lon",
start=c(1,1),
> count=c(d.size[1],d.size[2])))
>    d.lat <- as.vector(ncvar_get(data, varid="lat",
start=c(1,1),
> count=c(d.size[1],d.size[2])))
>
>    # Obtaining climate data values
>    df.clim <- data.frame(rn=seq(1:length(d.lon)))
>    for(y in 1:d.size[3]){
>      df.clim[,1+y] <- as.vector(ncvar_get(data,
> varid=d.vars[length(d.vars)], start=c(1,1,y),
> count=c(d.size[1],d.size[2],1)))
>       names(df.clim)[1+y] <- paste("y",y,sep="")  }
>    tosummarise[,,i] <- as.matrix(df.clim[,-1])
> }
>
> The data are temperature or precipitation, across space and time.
>
> For most of the >250 files I have, there are no problems, but for around
8
> of these files, I get strange values. The data should be within a
> relatively narrow range, yet I get values such as -8.246508e+07  or
> 7.659506e+11. The particularly strange part is that these kind of values
> occur at regularly spaced intervals across the data, usually within a
> single time step.
>
> I have the same problem (including the exact same strange values) when
> using ArcMap, yet the data provider assures me that the data look normal
> when using CDO (climate data operators) to view them, and that there are no
> missing values.
>
> I realise this is very difficult to diagnose without the nc files
> themselves, so my questions are (1) Has anyone encountered something like
> this before?, (2) Is there something I am failing to specify in the code
> when reading in?, and (3) Is anyone interested in digging into this and
> willing to play around with the nc files if I make them available
privately?
>
> Thanks very much in advance!
> Louise
>
>
>
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, La Jolla, California, USA
(858) 534-8276 (voice)  /  (858) 534-8561 (fax)    dpierce at ucsd.edu

	[[alternative HTML version deleted]]

Hi Louise:

If Dave can?t figure it out, I can give a look also.  A couple of things I would
suggest:

1.  Don?t use the name ?data? in the nc_open command, that is a reserved command
in R and you never know what problems that can cause.

2. You are doing calculations to get set the start and count values in the
ncvar_get commands, print those values out before you make the calls to make
certain they are valid.

HTH,

-Roy
> On Apr 22, 2016, at 8:08 AM, David W. Pierce <dpierce at ucsd.edu>
wrote:
> 
> On Fri, Apr 22, 2016 at 1:32 AM, Louise Mair <louise.mair at slu.se>
wrote:
> 
>> Dear R Users,
>> 
>> I am encountering a problem when reading nc files into R using the ncdf
>> and ncdf4 libraries. The nc files are too large to attach an example
(but
>> if someone is interested in helping out I could send a file privately
via
>> an online drive), but the code is basic:
>> 
> ?[...]?
> 
> 
> ?Hi Louise,
> 
> I'm the author of the ncdf and ncdf4 libraries. What are the details --
> what operating system are you running on, what version of R and the netcdf
> library are you using?
> 
> If you make the files available to me I can take a look.
> 
> Regards,
> 
> --Dave Pierce
> ?
> 
> 
> 
> 
> 
>> for(i in 1:length(thesenames[,1])){
>>   data <- nc_open(paste(INDIR,
thesenames[i,c("wholename")], sep=""),
>> write=F)
>>   d.vars <- names(data$var)
>>   d.size <- (data$var[[length(d.vars)]])$size
>> 
>>   # Obtaining longitude and latitude values
>>   d.lon <- as.vector(ncvar_get(data, varid="lon",
start=c(1,1),
>> count=c(d.size[1],d.size[2])))
>>   d.lat <- as.vector(ncvar_get(data, varid="lat",
start=c(1,1),
>> count=c(d.size[1],d.size[2])))
>> 
>>   # Obtaining climate data values
>>   df.clim <- data.frame(rn=seq(1:length(d.lon)))
>>   for(y in 1:d.size[3]){
>>     df.clim[,1+y] <- as.vector(ncvar_get(data,
>> varid=d.vars[length(d.vars)], start=c(1,1,y),
>> count=c(d.size[1],d.size[2],1)))
>>      names(df.clim)[1+y] <- paste("y",y,sep="") 
}
>>   tosummarise[,,i] <- as.matrix(df.clim[,-1])
>> }
>> 
>> The data are temperature or precipitation, across space and time.
>> 
>> For most of the >250 files I have, there are no problems, but for
around 8
>> of these files, I get strange values. The data should be within a
>> relatively narrow range, yet I get values such as -8.246508e+07  or
>> 7.659506e+11. The particularly strange part is that these kind of
values
>> occur at regularly spaced intervals across the data, usually within a
>> single time step.
>> 
>> I have the same problem (including the exact same strange values) when
>> using ArcMap, yet the data provider assures me that the data look
normal
>> when using CDO (climate data operators) to view them, and that there
are no
>> missing values.
>> 
>> I realise this is very difficult to diagnose without the nc files
>> themselves, so my questions are (1) Has anyone encountered something
like
>> this before?, (2) Is there something I am failing to specify in the
code
>> when reading in?, and (3) Is anyone interested in digging into this and
>> willing to play around with the nc files if I make them available
privately?
>> 
>> Thanks very much in advance!
>> Louise
>> 
>> 
>> 
>> 
>> 
>>        [[alternative HTML version deleted]]
>> 
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>> 
> 
> 
> 
> -- 
> David W. Pierce
> Division of Climate, Atmospheric Science, and Physical Oceanography
> Scripps Institution of Oceanography, La Jolla, California, USA
> (858) 534-8276 (voice)  /  (858) 534-8561 (fax)    dpierce at ucsd.edu
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn at noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 
"the arc of the moral universe is long, but it bends toward justice"
-MLK Jr.