R help - Jul 2009 - Problem With Repeated Use Of Load/Save/Close Commands

Hello,

I'm having a problem in R with repeated use of the "load",
"save", and
"close" commands. I'm getting an error message that reads,
"Too many
open files". I'm opening files and closing them (and unlinking them), 
but when I go through that process 509 times, the program halts and I 
get this error message: "cannot open the connection" with warning 
messages: "Too many open files".  I've been working on this
problem for
a couple of weeks and have gleaned a bit of info from different internet 
threads, but no solutions yet.

I'm using Windows XP, SP3 and R 2.9.1.
Here is my session info:

R version 2.9.1 (2009-06-26)
i286-pc-mingw32

locale:
LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
States. 1252;LC_MONETARY=English_United
States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

I'm using the vanilla load of R, with nothing added after booting up.
The problem also occurs on my Vista machine as well.

The program below will induce the problem.
------------------------------------------------------------
   junk       = 1
   PathA    = tempdir()
   conX     = paste(PathA,"junk",sep="\\")
   outJunk  = file(conX, open="wb")
   save(junk, file=outJunk)
   close(outJunk)
   for(i in 1:4000){
       outMIA = file(conX, open="rb")
       load(file=outMIA)
       close(outMIA)
       closeAllConnections()
       unlink(conX)
       rm(outMIA)
       rm(junk)
       cat(" i = ",i,sep=" ")
       gc()
       zzz = showConnections(all=FALSE)
       cat(" zzz = ",zzz,"\n",sep=" ")
   }
--------------------------------------------------------------------
There is some talk on the internet that some windows systems have a 
limit of 512 files that can be open at one time. Even though I'm closing 
my files each time, something is keeping track of how many times I've 
opened and closed a file in a session. I've talked to Microsoft and run 
a test program in Visual Studio C#, and, at the moment, it looks like 
the problem does not lie in the Microsoft arena. The C# program 
performed a similar task 10,000 times without a problem. I'm not totally 
convinced, but the current evidence says to look elsewhere.

I've attached a script that will induce the problem. However, be warned 
that, if you use it, you will have to get out of R after you run it. R 
will no longer be able to access files such as help or sessionInfo(). 
This can be solved by getting out of the R GUI and back in.

R E-mails from as far back as 2006 ask for help on the issue. There have 
been several suggestions, but no confirmed solution that I can find. You 
will see my attempt at these suggestions in the script [ rm(outMIA); 
rm(junk); closeAllConnections(); and gc(); after close(outMIA);  and 
unlink(conX);]. For me, this becomes important because it limits the 
total number if iterations in nested do-loops. This is where I ran into 
the problem. The program above will allow you to reproduce the problem.

Any suggestion would be greatly appreciated.

Thanks for your attention,
Rich Short

On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote:
> Hello,
> 
> I'm having a problem in R with repeated use of the "load",
"save", and
> "close" commands. I'm getting an error message that reads,
"Too many
> open files". I'm opening files and closing them (and unlinking
them),
> but when I go through that process 509 times, the program halts and I 
> get this error message: "cannot open the connection" with warning
> messages: "Too many open files".  I've been working on this
problem for
> a couple of weeks and have gleaned a bit of info from different internet 
> threads, but no solutions yet.
> 
> I'm using Windows XP, SP3 and R 2.9.1.
> Here is my session info:
> 
> R version 2.9.1 (2009-06-26)
> i286-pc-mingw32
> 
> locale:
> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United
> States. 1252;LC_MONETARY=English_United
> States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252
> 
> attached base packages:
> [1] stats     graphics  grDevices utils     datasets  methods   base
> 
> I'm using the vanilla load of R, with nothing added after booting up.
> The problem also occurs on my Vista machine as well.
> 
> The program below will induce the problem.
> ------------------------------------------------------------
>    junk       = 1
>    PathA    = tempdir()
>    conX     = paste(PathA,"junk",sep="\\")
>    outJunk  = file(conX, open="wb")
>    save(junk, file=outJunk)
>    close(outJunk)
>    for(i in 1:4000){
>        outMIA = file(conX, open="rb")
>        load(file=outMIA)
>        close(outMIA)
>        closeAllConnections()
>        unlink(conX)
>        rm(outMIA)
>        rm(junk)
>        cat(" i = ",i,sep=" ")
>        gc()
>        zzz = showConnections(all=FALSE)
>        cat(" zzz = ",zzz,"\n",sep=" ")
>    }
> --------------------------------------------------------------------
> There is some talk on the internet that some windows systems have a 
> limit of 512 files that can be open at one time. Even though I'm
closing
> my files each time, something is keeping track of how many times I've 
> opened and closed a file in a session. I've talked to Microsoft and run
> a test program in Visual Studio C#, and, at the moment, it looks like 
> the problem does not lie in the Microsoft arena. The C# program 
> performed a similar task 10,000 times without a problem. I'm not
totally
> convinced, but the current evidence says to look elsewhere.
> 
> I've attached a script that will induce the problem. However, be warned
> that, if you use it, you will have to get out of R after you run it. R 
> will no longer be able to access files such as help or sessionInfo(). 
> This can be solved by getting out of the R GUI and back in.
> 
> R E-mails from as far back as 2006 ask for help on the issue. There have 
> been several suggestions, but no confirmed solution that I can find. You 
> will see my attempt at these suggestions in the script [ rm(outMIA); 
> rm(junk); closeAllConnections(); and gc(); after close(outMIA);  and 
> unlink(conX);]. For me, this becomes important because it limits the 
> total number if iterations in nested do-loops. This is where I ran into 
> the problem. The program above will allow you to reproduce the problem.
> 
> Any suggestion would be greatly appreciated.
I've somewhat localized the problem.  load(file=outMIA) passes the 
outMIA connection to gzcon(), which handles decompression of the data. 
gzcon() re-opens the file, and it looks as though the original file 
handle is lost, because closing either the result of gzcon() or the 
original connection results in only the second file handle being closed.

So a second workaround besides the one I sent earlier is just not to 
open or close outMIA.  That is, rewrite it as

    junk       = 1
    PathA    = tempdir()
    conX     = paste(PathA,"junk",sep="\\")
    outJunk  = file(conX, open="wb")
    save(junk, file=outJunk)
    close(outJunk)
    for(i in 1:4000){
        outMIA = file(conX)
        load(file=outMIA)
        rm(outMIA)
        rm(junk)
        cat(" i = ",i,sep=" ")
        gc()
        zzz = showConnections(all=FALSE)
        cat(" zzz = ",zzz,"\n",sep=" ")
    }

so that load() does the open and close, and things are fine.

Duncan Murdoch