R help - Oct 2010 - what does this err mean and how to solve it? Error in file(file, ifelse(append, "a", "w"))

Dear List
I am running a loop downloading ?web pages and save the html to a
temporary file (use download.file() )
?then read (using readLines) ?it in for processing;
finally write useful info from each processed page to a unique file

the problem is once the loop runs up to somewhere near ?5000, it will
throw out an err like below and won't go further.

----------------------------------------------------------------
Error in file(file, ifelse(append, "a", "w")) :
?cannot open the connection
-----------------------------------------------------------------

In the meantime, a request for new connection won't be successful, for
example, a request for the help page of "file" will trigger err below

-----------------------------------------------------------------------
??file
Error in gzfile(file, "rb") : cannot open the connection
In addition: Warning message:
In gzfile(file, "rb") :
?cannot open compressed file
'C:/PROGRA~1/R/R-211~1.1/library/stats/help/aliases.rds', probable
reason 'Too many open files'
-----------------------------------------------------------------------

I am not sure if the problem is too many connections not closed. since
I close the file connection after each readLines.
checking with showConnections(all=T) does not show excessive
connections and closeAllConnections() does not help.

Can any one help me on this?


Any answer highly appreciated.

yong

you should close files that you do not use anymore. the maximum number
of open files is likely 4000 or so. Use close(file) before you open
the next one.

Peter

On Wed, Oct 6, 2010 at 4:55 PM, Yong Wang <wangyong1 at gmail.com>
wrote:
> Dear List
> I am running a loop downloading ?web pages and save the html to a
> temporary file (use download.file() )
> ?then read (using readLines) ?it in for processing;
> finally write useful info from each processed page to a unique file
>
> the problem is once the loop runs up to somewhere near ?5000, it will
> throw out an err like below and won't go further.
>
> ----------------------------------------------------------------
> Error in file(file, ifelse(append, "a", "w")) :
> ?cannot open the connection
> -----------------------------------------------------------------
>
> In the meantime, a request for new connection won't be successful, for
> example, a request for the help page of "file" will trigger err
below
>
> -----------------------------------------------------------------------
> ??file
> Error in gzfile(file, "rb") : cannot open the connection
> In addition: Warning message:
> In gzfile(file, "rb") :
> ?cannot open compressed file
> 'C:/PROGRA~1/R/R-211~1.1/library/stats/help/aliases.rds', probable
> reason 'Too many open files'
> -----------------------------------------------------------------------
>
> I am not sure if the problem is too many connections not closed. since
> I close the file connection after each readLines.
> checking with showConnections(all=T) does not show excessive
> connections and closeAllConnections() does not help.
>
> Can any one help me on this?
>
>
> Any answer highly appreciated.
>
> yong
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>