I have a loop that processes all the raw data files in agiven directory. It should allocate only the memory to accomodate the filenames list. Each file is loaded into RAM one at a time and saved to another directory after processing. The R script runs fine but it had some memory problems so it was terminated before getting to the end of the list. This is a Windows XP desktop with ~ 1GB RAM In the following I am pasting R messages. I would appreciate some suggestion about memory handling: Error: cannot allocate vector of size 32.0 Mb In addition: Warning messages: 1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y -- Maura E.M
On Tue, 9 Oct 2007, Maura E Monville wrote:> I have a loop that processes all the raw data files in agiven directory. > It should allocate only the memory to accomodate the filenames list.Without seeeing the script (see the footer of this and every R-help message, including the second copy you sent), all we can say is that was not what your script does. But that did seem implausible, as R needs memory to 'process' each file.> Each file is loaded into RAM one at a time and saved to another > directory after processing. The R script runs fine but it had some > memory problems so it was terminated before getting to the end of the > list. > This is a Windows XP desktop with ~ 1GB RAM > In the following I am pasting R messages. I would appreciate some > suggestion about memory handling: > > Error: cannot allocate vector of size 32.0 Mb > In addition: Warning messages: > 1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > 2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > 3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > 4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
I forgot to attach my script. I work under pressure and make a lot of these mistakes. Sorry. I have attached it right now. I'm lso pasting the messages appearing on R GUI in the following. I see the average becomes NA which makes me think that for some reason R interpreter is reading out of the loaded file ... I will try to locate the file that causes this problem. Thanks to all of you. Maura start 4563 end 161 avg -0.04171546 start 4724 end 4871 avg -2.880717 gc(): 242445 107937 6.5 0.9 467875 786432 12.5 6 350000 786432 9.4 6 Read 1 item i from which: 7 start 1 end 135 avg -2.993604 start 136 end 135 avg -1.807748 start 271 end 176 avg -3.047341 start 447 end 154 avg -2.036649 start 601 end 141 avg -1.0865 start 742 end 152 avg -0.6664973 start 894 end 121 avg -0.6277448 start 1015 end 160 avg -0.4067699 start 1175 end 161 avg -0.4687507 start 1336 end 185 avg -0.478178 start 1521 end 211 avg -0.5239797 start 1732 end 600 avg -1.581398 start 2332 end 288 avg -0.5022956 start 2620 end 144 avg -0.07848339 start 2764 end 215 avg -0.2710712 start 2979 end 178 avg -0.1480546 start 3157 end 205 avg -0.2299476 start 3362 end 180 avg -0.1381425 start 3542 end 141 avg -0.1376570 start 3683 end 151 avg -0.111202 start 3834 end 134 avg -0.1246371 start 3968 end 142 avg -0.09887487 start 4110 end 126 avg -0.1023989 start 4236 end 118 avg -0.08591205 start 4354 end 154 avg -0.08125074 start 4508 end 187 avg -0.1366167 start 4695 end 178 avg -0.1046542 start 4873 end 178 avg -0.1076104 start 5051 end 2 avg -0.08889775 start 5053 end 4 avg NA start 5057 end 8 avg NA start 5065 end 16 avg NA start 5081 end 32 avg NA start 5113 end 64 avg NA start 5177 end 128 avg NA start 5305 end 256 avg NA start 5561 end 512 avg NA start 6073 end 1024 avg NA start 7097 end 2048 avg NA start 9145 end 4096 avg NA start 13241 end 8192 avg NA start 21433 end 16384 avg NA start 37817 end 32768 avg NA start 70585 end 65536 avg NA start 136121 end 131072 avg NA start 267193 end 262144 avg NA start 529337 end 524288 avg NA start 1053625 end 1048576 avg NA start 2102201 end 2097152 avg NA start 4199353 end 4194304 avg NA start 8393657 end 8388608 avg NA start 16782265 end 16777216 avg NA Error: cannot allocate vector of size 32.0 Mb In addition: Warning messages: 1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y 4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y>On 10/9/07, jim holtman <jholtman at gmail.com> wrote:> Have you tried using 'gc()' at the end of your loop each time you free > up the objects that have been use 'rm(.....)'? You might also look at > the output from gc() to see if memory is being used up. You may be > getting fragmentation in memory. > > If you are processing a list of files and this continues, then one way > is to process a subset, then exit and restart R and process the next > subset. It would be useful to know the sizes of the objects that you > are processing and how you are writing them out. Are you using 'save' > or 'write.table' ('save' would be a lot more efficient). Since we > have no idea of what you loop looks like it is hard to provide more > suggestions. > > On 10/9/07, Maura E Monville <maura.monville at gmail.com> wrote: > > I have a loop that processes all the raw data files in agiven directory. > > It should allocate only the memory to accomodate the filenames list. > > Each file is loaded into RAM one at a time and saved to another > > directory after processing. The R script runs fine but it had some > > memory problems so it was terminated before getting to the end of the > > list. > > This is a Windows XP desktop with ~ 1GB RAM > > In the following I am pasting R messages. I would appreciate some > > suggestion about memory handling: > > > > Error: cannot allocate vector of size 32.0 Mb > > In addition: Warning messages: > > 1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > > 2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > > 3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > > 4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y > > > > > > -- > > Maura E.M > > > > ______________________________________________ > > R-help at r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-help > > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > > and provide commented, minimal, self-contained, reproducible code. > > > > > -- > Jim Holtman > Cincinnati, OH > +1 513 646 9390 > > What is the problem you are trying to solve? >-- Maura E.M