Christopher Lloyd
2021-Jan-19 20:22 UTC
[R] Error: cons memory exhausted (limit reached?): Memory Management?
Hi all, I think this is only the second time that I have posted so I apologise if my etiquette isn't quite correct. I'm loading a large (~30GB) geojson file into R using readOGR on a HPC. I am also loading a small shapefile, and then trying to undertake some processing on the large geojson using gBuffer from the rgeos package. I believe that the HPC is running?Red Hat Enterprise Linux 7.4, and it certainly has?around 750 GB free for user jobs. I have allocated the full amount of ram to the job. I previously used the following modules to undertake this task and it ran successfully, although only after tweaking the settings that I detail below - otherwise I had the same error: module load proj/5.0.0 module load gdal/2.3.1 module load geos/3.6.2 module load gcc/6.4.0 module load R/3.5.2 module load python? # python 3 by default module load numpy/1.14.0 # requires module load python Settings at linux command line that previously allowed a successful run: R_MAX_VSIZE=720G R_GC_MEM_GROW=0 --min-nsize=50000k --min-vsize=12M --max-ppsize=500000 (when executing the R script from command line) However, the modules have now been updated on the HPC, and so I am now using: module load proj/6.1.1 module load R/3.6.2 (other modules remain the same) I get the following error whilst processing (loading the file into R is ok), with gcinfo() turned on: Garbage collection 144 = 86+22+36 (level 0) ...? 288541.3 Mbytes of cons cells used (66%) 55320.1 Mbytes of vectors used (98%) Garbage collection 145 = 86+23+36 (level 1) ...? 66679.3 Mbytes of cons cells used (15%) 56447.7 Mbytes of vectors used (100%) Garbage collection 146 = 86+23+37 (level 2) ...? 39852.3 Mbytes of cons cells used (11%) 49032.4 Mbytes of vectors used (72%) Garbage collection 147 = 87+23+37 (level 0) ...? 124935.0 Mbytes of cons cells used (36%) 64961.0 Mbytes of vectors used (95%) Garbage collection 148 = 87+24+37 (level 1) ...? 985162418403226.2 Mbytes of cons cells used (-2147483648%) 35274.8 Mbytes of vectors used (52%) Error: cons memory exhausted (limit reached?) In addition: Warning message: Garbage collection 149 = 88+24+37 (level 0) ...? 985162418403226.2 Mbytes of cons cells used (-2147483648%) 35274.8 Mbytes of vectors used (52%) Lost warning messages Execution halted Garbage collection 150 = 89+24+37 (level 0) ...? 985162418403226.2 Mbytes of cons cells used (-2147483648%) 35274.8 Mbytes of vectors used (52%) Error: cons memory exhausted (limit reached?) The job halts with Memory Utilized: 411.29 GB I cannot understand why the job worked previously (just) but now does not when seemingly the only change is an updated proj and R version (3.5.2 to 3.6.2). Might anyone have any suggestions as to why this is the case? And/or how to alter the memory management so that memory is not exhausted so easily? Many thanks, Chris [[alternative HTML version deleted]]
Bert Gunter
2021-Jan-20 00:35 UTC
[R] Error: cons memory exhausted (limit reached?): Memory Management?
You might do better posting this on r-sig-geo -- I believe you will more likely find the expertise you seek there. Could be wrong of course. Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Tue, Jan 19, 2021 at 3:01 PM Christopher Lloyd via R-help < r-help at r-project.org> wrote:> Hi all, I think this is only the second time that I have posted so I > apologise if my etiquette isn't quite correct. > I'm loading a large (~30GB) geojson file into R using readOGR on a HPC. I > am also loading a small shapefile, and then trying to undertake some > processing on the large geojson using gBuffer from the rgeos package. > > I believe that the HPC is running Red Hat Enterprise Linux 7.4, and it > certainly has around 750 GB free for user jobs. I have allocated the full > amount of ram to the job. > I previously used the following modules to undertake this task and it ran > successfully, although only after tweaking the settings that I detail below > - otherwise I had the same error: > > module load proj/5.0.0 > module load gdal/2.3.1 > module load geos/3.6.2 > module load gcc/6.4.0 > module load R/3.5.2 > module load python # python 3 by default > module load numpy/1.14.0 # requires module load python > > Settings at linux command line that previously allowed a successful run: > R_MAX_VSIZE=720G > R_GC_MEM_GROW=0 > > --min-nsize=50000k --min-vsize=12M --max-ppsize=500000 (when executing the > R script from command line) > > However, the modules have now been updated on the HPC, and so I am now > using: > module load proj/6.1.1 > module load R/3.6.2 > (other modules remain the same) > > I get the following error whilst processing (loading the file into R is > ok), with gcinfo() turned on: > > Garbage collection 144 = 86+22+36 (level 0) ... > 288541.3 Mbytes of cons cells used (66%) > 55320.1 Mbytes of vectors used (98%) > Garbage collection 145 = 86+23+36 (level 1) ... > 66679.3 Mbytes of cons cells used (15%) > 56447.7 Mbytes of vectors used (100%) > Garbage collection 146 = 86+23+37 (level 2) ... > 39852.3 Mbytes of cons cells used (11%) > 49032.4 Mbytes of vectors used (72%) > Garbage collection 147 = 87+23+37 (level 0) ... > 124935.0 Mbytes of cons cells used (36%) > 64961.0 Mbytes of vectors used (95%) > Garbage collection 148 = 87+24+37 (level 1) ... > 985162418403226.2 Mbytes of cons cells used (-2147483648%) > 35274.8 Mbytes of vectors used (52%) > > Error: cons memory exhausted (limit reached?) > In addition: Warning message: > Garbage collection 149 = 88+24+37 (level 0) ... > 985162418403226.2 Mbytes of cons cells used (-2147483648%) > 35274.8 Mbytes of vectors used (52%) > Lost warning messages > Execution halted > Garbage collection 150 = 89+24+37 (level 0) ... > 985162418403226.2 Mbytes of cons cells used (-2147483648%) > 35274.8 Mbytes of vectors used (52%) > > Error: cons memory exhausted (limit reached?) > > The job halts with Memory Utilized: 411.29 GB > > I cannot understand why the job worked previously (just) but now does not > when seemingly the only change is an updated proj and R version (3.5.2 to > 3.6.2). > > Might anyone have any suggestions as to why this is the case? And/or how > to alter the memory management so that memory is not exhausted so easily? > > Many thanks, Chris > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >[[alternative HTML version deleted]]