R help - Mar 2020 - Request for help about running a loop and reading .NC files

Hello Everyone,

I have written a loop which reads hundreds of .nc files
and extract information from each .nc file and
exports that corresponding information as a csv file.

The loop works fine until it encounters a .nc file which it cannot read
and the loop stops.

 I would appreciate if anyone can suggest how can I modify
the loop, whereby the loop will continue to run by bypassing those
particular
files which it cannot read or if any particular file has an error.

In the end, I was also hoping to modify the loop such that it
will generate a report which will inform me which
files were not read by the loop.  The codes are given below

Thanks for your help,

Regards,
Bhaskar Mitra


#_------------------------------------------------------------------


library(ncdf4)
library(reshape2)
library(dplyr)
library(stringr)


setwd("directory path")

Output <- "directory path"

flist <- list.files(path ="NCFiles/", pattern =
"^.*\\.(nc|NC|Nc|Nc)$")

for (i in 1: length(flist))
{

  nc <- nc_open(paste0("NCFiles/",flist[i]))
  mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
  nc_close(nc)

  mean_chl_df <- melt(mean1)
  trial.table.df <-as.data.frame(mean_chl_df)

write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))

}

	[[alternative HTML version deleted]]

On Mon, 23 Mar 2020 16:16:42 -0700
Bhaskar Mitra <bhaskar.kolkata at gmail.com> wrote:
> I would appreciate if anyone can suggest how can I modify
>the loop, whereby the loop will continue to run by bypassing those
>particular
>files which it cannot read or if any particular file has an error.
See ?try and ?tryCatch. With `try`, you can use inherits(x,
'try-error') to check whether execution failed. With `tryCatch` you can
choose your own value to return instead of the error object.

-- 
Best regards,
Ivan

?tryCatch

See also:
https://www.r-bloggers.com/error-handling-in-r/


Bert Gunter

"The trouble with having an open mind is that people keep coming along
and sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )

On Tue, Mar 24, 2020 at 5:13 AM Bhaskar Mitra <bhaskar.kolkata at
gmail.com> wrote:
>
> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops.
>
>  I would appreciate if anyone can suggest how can I modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read or if any particular file has an error.
>
> In the end, I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below
>
> Thanks for your help,
>
> Regards,
> Bhaskar Mitra
>
>
> #_------------------------------------------------------------------
>
>
> library(ncdf4)
> library(reshape2)
> library(dplyr)
> library(stringr)
>
>
> setwd("directory path")
>
> Output <- "directory path"
>
> flist <- list.files(path ="NCFiles/", pattern =
"^.*\\.(nc|NC|Nc|Nc)$")
>
> for (i in 1: length(flist))
> {
>
>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>   nc_close(nc)
>
>   mean_chl_df <- melt(mean1)
>   trial.table.df <-as.data.frame(mean_chl_df)
>
>
write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>
> }
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Hey Bhaskar,

Dave Pierce here, the author and maintainer of the ncdf4 package. What
platform are you working on? If you work on Linux, I'd be happy to send you
a new source tar version of the ncdf4 package I've made that handles this
case. It now has a settable parameter in nc_open called
"return_on_error",
and if you set that to TRUE, then nc_open always returns. If no error was
encountered, then the return list element $error is FALSE, and if an error
was encountered, $error is set to TRUE. So you could do this:

for( ifile in 1:nfiles ) {
    nc = nc_open( filename[ifile], return_on_error=TRUE )
    if( nc$error == FALSE ) {
        ... do your processing here ...
        }
  else
        print(paste("Error encountered, skipping file",
filename[ifile]))

Regards,

--Dave

On Tue, Mar 24, 2020 at 5:13 AM Bhaskar Mitra <bhaskar.kolkata at
gmail.com>
wrote:
> Hello Everyone,
>
> I have written a loop which reads hundreds of .nc files
> and extract information from each .nc file and
> exports that corresponding information as a csv file.
>
> The loop works fine until it encounters a .nc file which it cannot read
> and the loop stops.
>
>  I would appreciate if anyone can suggest how can I modify
> the loop, whereby the loop will continue to run by bypassing those
> particular
> files which it cannot read or if any particular file has an error.
>
> In the end, I was also hoping to modify the loop such that it
> will generate a report which will inform me which
> files were not read by the loop.  The codes are given below
>
> Thanks for your help,
>
> Regards,
> Bhaskar Mitra
>
>
> #_------------------------------------------------------------------
>
>
> library(ncdf4)
> library(reshape2)
> library(dplyr)
> library(stringr)
>
>
> setwd("directory path")
>
> Output <- "directory path"
>
> flist <- list.files(path ="NCFiles/", pattern =
"^.*\\.(nc|NC|Nc|Nc)$")
>
> for (i in 1: length(flist))
> {
>
>   nc <- nc_open(paste0("NCFiles/",flist[i]))
>   mean1 <- ncvar_get(nc,attributes(nc$dim)$names[3])
>   nc_close(nc)
>
>   mean_chl_df <- melt(mean1)
>   trial.table.df <-as.data.frame(mean_chl_df)
>
>
>
write.csv(trial.table.df,paste0(Output,"/",tools::file_path_sans_ext(flist[i]),".csv"))
>
> }
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

-- 

-------------------------------------------------------------------
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography
(858) 534-8276 (voice)  /  (858) 534-8561 (fax)    dpierce at ucsd.edu
-------------------------------------------------------------------

	[[alternative HTML version deleted]]