murdoch at stats.uwo.ca
2009-Jul-18 12:15 UTC
[Rd] [R] Problem With Repeated Use Of Load/Save/Close Commands (PR#13841)
On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote:> Hello, > > I'm having a problem in R with repeated use of the "load", "save", and > "close" commands. I'm getting an error message that reads, "Too many > open files". I'm opening files and closing them (and unlinking them), > but when I go through that process 509 times, the program halts and I > get this error message: "cannot open the connection" with warning > messages: "Too many open files". I've been working on this problem for > a couple of weeks and have gleaned a bit of info from different internet > threads, but no solutions yet. > > I'm using Windows XP, SP3 and R 2.9.1. > Here is my session info: > > R version 2.9.1 (2009-06-26) > i286-pc-mingw32 > > locale: > LC_COLLATE=English_United States.1252;LC_CTYPE=English_United > States. 1252;LC_MONETARY=English_United > States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > I'm using the vanilla load of R, with nothing added after booting up. > The problem also occurs on my Vista machine as well. > > The program below will induce the problem. > ------------------------------------------------------------ > junk = 1 > PathA = tempdir() > conX = paste(PathA,"junk",sep="\\") > outJunk = file(conX, open="wb") > save(junk, file=outJunk) > close(outJunk) > for(i in 1:4000){ > outMIA = file(conX, open="rb") > load(file=outMIA) > close(outMIA) > closeAllConnections() > unlink(conX) > rm(outMIA) > rm(junk) > cat(" i = ",i,sep=" ") > gc() > zzz = showConnections(all=FALSE) > cat(" zzz = ",zzz,"\n",sep=" ") > } > -------------------------------------------------------------------- > There is some talk on the internet that some windows systems have a > limit of 512 files that can be open at one time. Even though I'm closing > my files each time, something is keeping track of how many times I've > opened and closed a file in a session. I've talked to Microsoft and run > a test program in Visual Studio C#, and, at the moment, it looks like > the problem does not lie in the Microsoft arena. The C# program > performed a similar task 10,000 times without a problem. I'm not totally > convinced, but the current evidence says to look elsewhere. > > I've attached a script that will induce the problem. However, be warned > that, if you use it, you will have to get out of R after you run it. R > will no longer be able to access files such as help or sessionInfo(). > This can be solved by getting out of the R GUI and back in. > > R E-mails from as far back as 2006 ask for help on the issue. There have > been several suggestions, but no confirmed solution that I can find. You > will see my attempt at these suggestions in the script [ rm(outMIA); > rm(junk); closeAllConnections(); and gc(); after close(outMIA); and > unlink(conX);]. For me, this becomes important because it limits the > total number if iterations in nested do-loops. This is where I ran into > the problem. The program above will allow you to reproduce the problem. > > Any suggestion would be greatly appreciated.This looks like a bug to me, and so I've posted it to the bugs list, but your script doesn't make sense for normal use. Before the loop you create a file named according to conX containing the junk variable, and then within the loop you load that file, *then delete it*, using unlink(). However, if you examine the value of unlink(), you'll see that it fails every time. Moreover, the use of connections is unnecessary: load() can work with a filename. So you can skip the file() and close() calls. If I simplify your script by removing the unnecessary stuff, then it works. That is, this works: junk = 1 PathA = tempdir() conX = paste(PathA,"junk",sep="\\") save(junk, file=conX) for(i in 1:4000){ load(file=conX) rm(junk) cat(" i = ",i,sep=" ") gc() zzz = showConnections(all=FALSE) cat(" zzz = ",zzz,"\n",sep=" ") } So a suggestion for a workaround is simply to follow the pattern above, rather than that of your demonstration code. I'll spend a bit more time and see if I can find what is causing the open file leak. Duncan Murdoch
Duncan Murdoch
2009-Jul-18 16:19 UTC
[Rd] [R] Problem With Repeated Use Of Load/Save/Close Commands (PR#13841)
On 18/07/2009 8:15 AM, murdoch at stats.uwo.ca wrote:> On 17/07/2009 7:57 PM, Marilyn & Rich Short wrote: >> Hello, >> >> I'm having a problem in R with repeated use of the "load", "save", and >> "close" commands. I'm getting an error message that reads, "Too many >> open files". I'm opening files and closing them (and unlinking them), >> but when I go through that process 509 times, the program halts and I >> get this error message: "cannot open the connection" with warning >> messages: "Too many open files". I've been working on this problem for >> a couple of weeks and have gleaned a bit of info from different internet >> threads, but no solutions yet. >> >> I'm using Windows XP, SP3 and R 2.9.1. >> Here is my session info: >> >> R version 2.9.1 (2009-06-26) >> i286-pc-mingw32 >> >> locale: >> LC_COLLATE=English_United States.1252;LC_CTYPE=English_United >> States. 1252;LC_MONETARY=English_United >> States. 1252;LC_NUMERIC=C;LC_TIME=English_United States.1252 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> I'm using the vanilla load of R, with nothing added after booting up. >> The problem also occurs on my Vista machine as well. >> >> The program below will induce the problem. >> ------------------------------------------------------------ >> junk = 1 >> PathA = tempdir() >> conX = paste(PathA,"junk",sep="\\") >> outJunk = file(conX, open="wb") >> save(junk, file=outJunk) >> close(outJunk) >> for(i in 1:4000){ >> outMIA = file(conX, open="rb") >> load(file=outMIA) >> close(outMIA) >> closeAllConnections() >> unlink(conX) >> rm(outMIA) >> rm(junk) >> cat(" i = ",i,sep=" ") >> gc() >> zzz = showConnections(all=FALSE) >> cat(" zzz = ",zzz,"\n",sep=" ") >> } >> -------------------------------------------------------------------- >> There is some talk on the internet that some windows systems have a >> limit of 512 files that can be open at one time. Even though I'm closing >> my files each time, something is keeping track of how many times I've >> opened and closed a file in a session. I've talked to Microsoft and run >> a test program in Visual Studio C#, and, at the moment, it looks like >> the problem does not lie in the Microsoft arena. The C# program >> performed a similar task 10,000 times without a problem. I'm not totally >> convinced, but the current evidence says to look elsewhere. >> >> I've attached a script that will induce the problem. However, be warned >> that, if you use it, you will have to get out of R after you run it. R >> will no longer be able to access files such as help or sessionInfo(). >> This can be solved by getting out of the R GUI and back in. >> >> R E-mails from as far back as 2006 ask for help on the issue. There have >> been several suggestions, but no confirmed solution that I can find. You >> will see my attempt at these suggestions in the script [ rm(outMIA); >> rm(junk); closeAllConnections(); and gc(); after close(outMIA); and >> unlink(conX);]. For me, this becomes important because it limits the >> total number if iterations in nested do-loops. This is where I ran into >> the problem. The program above will allow you to reproduce the problem. >> >> Any suggestion would be greatly appreciated. > > This looks like a bug to me, and so I've posted it to the bugs list, but > your script doesn't make sense for normal use. Before the loop you > create a file named according to conX containing the junk variable, and > then within the loop you load that file, *then delete it*, using > unlink(). However, if you examine the value of unlink(), you'll see > that it fails every time. > > Moreover, the use of connections is unnecessary: load() can work with a > filename. So you can skip the file() and close() calls. > > If I simplify your script by removing the unnecessary stuff, then it > works. That is, this works: > > junk = 1 > PathA = tempdir() > conX = paste(PathA,"junk",sep="\\") > save(junk, file=conX) > for(i in 1:4000){ > load(file=conX) > rm(junk) > cat(" i = ",i,sep=" ") > gc() > zzz = showConnections(all=FALSE) > cat(" zzz = ",zzz,"\n",sep=" ") > } > > So a suggestion for a workaround is simply to follow the pattern above, > rather than that of your demonstration code. > > I'll spend a bit more time and see if I can find what is causing the > open file leak.Now fixed in R-devel, R-patched coming soon. Duncan Murdoch