Samuel Granjeaud IR/Inserm
2020-Apr-07 13:57 UTC
[Rd] Hard memory limit of 16GB under Windows?
Hi Tomas, Many thanks for your answer. Here is a copy of a fresh session under RStudio, and after a copy under Rgui. Strangely enough the result of memory.limit() is not the same. Without your question I would not have looked to RGui, being used to work with RStudio. The value under RGui sounds to correspond to the total RAM of the computer. It makes me noticing that the value is in MB. The value under Rstudio was so huge (1.759219e+13) that I just interpreted it as GB. But I was totally wrong. So in fact I don't know what it refers to. The documentation says "For a 64-bit versions of R under 64-bit Windows the limit is currently 8Tb.", but it looks like being 16TB, which my computer don't have of course. I still have to understand why my colleague has a problem of memory allocation (cannot allocate vector of size 12.6 Gb). Sorry for the wrong interpretation, but thanks for the help, Samuel --- RStudio R version 3.6.3 (2020-02-29) -- "Holding the Windsock" Copyright (C) 2020 The R Foundation for Statistical Computing Platform: x86_64-w64-mingw32/x64 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > memory.limit() [1] 1.759219e+13 > sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18363) Matrix products: default locale: [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 LC_MONETARY=French_France.1252 LC_NUMERIC=C [5] LC_TIME=French_France.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_3.6.3 tools_3.6.3 > --- RGui R version 3.6.3 (2020-02-29) -- "Holding the Windsock" Copyright (C) 2020 The R Foundation for Statistical Computing Platform: x86_64-w64-mingw32/x64 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > ls() character(0) > memory.limit() [1] 32627 > sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18363) Matrix products: default locale: [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 LC_MONETARY=French_France.1252 [4] LC_NUMERIC=C LC_TIME=French_France.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] compiler_3.6.3 >
Hi Samuel, please also have a look at ?memory.limit. You can set this limit at R startup. It is in megabytes. Maybe R Studio sets it at runtime. Best Tomas On 4/7/20 3:57 PM, Samuel Granjeaud IR/Inserm wrote:> Hi Tomas, > > Many thanks for your answer. > > Here is a copy of a fresh session under RStudio, and after a copy > under Rgui. > Strangely enough the result of memory.limit() is not the same. Without > your question I would not have looked to RGui, being used to work with > RStudio. > > The value under RGui sounds to correspond to the total RAM of the > computer. It makes me noticing that the value is in MB. > > The value under Rstudio was so huge (1.759219e+13) that I just > interpreted it as GB. But I was totally wrong. So in fact I don't know > what it refers to. The documentation says "For a 64-bit versions of R > under 64-bit Windows the limit is currently 8Tb.", but it looks like > being 16TB, which my computer don't have of course. > > I still have to understand why my colleague has a problem of memory > allocation (cannot allocate vector of size 12.6 Gb). > > Sorry for the wrong interpretation, but thanks for the help, > Samuel > > --- RStudio > > R version 3.6.3 (2020-02-29) -- "Holding the Windsock" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-w64-mingw32/x64 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > memory.limit() > [1] 1.759219e+13 > > sessionInfo() > R version 3.6.3 (2020-02-29) > Platform: x86_64-w64-mingw32/x64 (64-bit) > Running under: Windows 10 x64 (build 18363) > > Matrix products: default > > locale: > [1] LC_COLLATE=French_France.1252? LC_CTYPE=French_France.1252 > LC_MONETARY=French_France.1252 LC_NUMERIC=C > [5] LC_TIME=French_France.1252 > > attached base packages: > [1] stats???? graphics? grDevices utils???? datasets? methods base > > loaded via a namespace (and not attached): > [1] compiler_3.6.3 tools_3.6.3 > > > > --- RGui > > R version 3.6.3 (2020-02-29) -- "Holding the Windsock" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-w64-mingw32/x64 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > ls() > character(0) > > memory.limit() > [1] 32627 > > sessionInfo() > R version 3.6.3 (2020-02-29) > Platform: x86_64-w64-mingw32/x64 (64-bit) > Running under: Windows 10 x64 (build 18363) > > Matrix products: default > > locale: > [1] LC_COLLATE=French_France.1252? LC_CTYPE=French_France.1252 > LC_MONETARY=French_France.1252 > [4] LC_NUMERIC=C?????????????????? LC_TIME=French_France.1252 > > attached base packages: > [1] stats???? graphics? grDevices utils???? datasets? methods base > > loaded via a namespace (and not attached): > [1] compiler_3.6.3 > > >
Hi Samuel, You may already be aware, but if not, RStudio has their own support mechanisms here: https://support.rstudio.com/hc/en-us If this does turn out to be RStudio specific, you may wish to check there for additional insights. Regards, Marc Schwartz> On Apr 7, 2020, at 10:24 AM, Tomas Kalibera <tomas.kalibera at gmail.com> wrote: > > Hi Samuel, > > please also have a look at ?memory.limit. You can set this limit at R startup. It is in megabytes. Maybe R Studio sets it at runtime. > > Best > Tomas > > On 4/7/20 3:57 PM, Samuel Granjeaud IR/Inserm wrote: >> Hi Tomas, >> >> Many thanks for your answer. >> >> Here is a copy of a fresh session under RStudio, and after a copy under Rgui. >> Strangely enough the result of memory.limit() is not the same. Without your question I would not have looked to RGui, being used to work with RStudio. >> >> The value under RGui sounds to correspond to the total RAM of the computer. It makes me noticing that the value is in MB. >> >> The value under Rstudio was so huge (1.759219e+13) that I just interpreted it as GB. But I was totally wrong. So in fact I don't know what it refers to. The documentation says "For a 64-bit versions of R under 64-bit Windows the limit is currently 8Tb.", but it looks like being 16TB, which my computer don't have of course. >> >> I still have to understand why my colleague has a problem of memory allocation (cannot allocate vector of size 12.6 Gb). >> >> Sorry for the wrong interpretation, but thanks for the help, >> Samuel >> >> --- RStudio >> >> R version 3.6.3 (2020-02-29) -- "Holding the Windsock" >> Copyright (C) 2020 The R Foundation for Statistical Computing >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> >> R is free software and comes with ABSOLUTELY NO WARRANTY. >> You are welcome to redistribute it under certain conditions. >> Type 'license()' or 'licence()' for distribution details. >> >> R is a collaborative project with many contributors. >> Type 'contributors()' for more information and >> 'citation()' on how to cite R or R packages in publications. >> >> Type 'demo()' for some demos, 'help()' for on-line help, or >> 'help.start()' for an HTML browser interface to help. >> Type 'q()' to quit R. >> >> > memory.limit() >> [1] 1.759219e+13 >> > sessionInfo() >> R version 3.6.3 (2020-02-29) >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> Running under: Windows 10 x64 (build 18363) >> >> Matrix products: default >> >> locale: >> [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 LC_MONETARY=French_France.1252 LC_NUMERIC=C >> [5] LC_TIME=French_France.1252 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.6.3 tools_3.6.3 >> > >> >> --- RGui >> >> R version 3.6.3 (2020-02-29) -- "Holding the Windsock" >> Copyright (C) 2020 The R Foundation for Statistical Computing >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> >> R is free software and comes with ABSOLUTELY NO WARRANTY. >> You are welcome to redistribute it under certain conditions. >> Type 'license()' or 'licence()' for distribution details. >> >> R is a collaborative project with many contributors. >> Type 'contributors()' for more information and >> 'citation()' on how to cite R or R packages in publications. >> >> Type 'demo()' for some demos, 'help()' for on-line help, or >> 'help.start()' for an HTML browser interface to help. >> Type 'q()' to quit R. >> >> > ls() >> character(0) >> > memory.limit() >> [1] 32627 >> > sessionInfo() >> R version 3.6.3 (2020-02-29) >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> Running under: Windows 10 x64 (build 18363) >> >> Matrix products: default >> >> locale: >> [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 LC_MONETARY=French_France.1252 >> [4] LC_NUMERIC=C LC_TIME=French_France.1252 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.6.3 >> > >> > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
Dear Samuel, The most important information from your mail was the actual error message: "cannot allocate vector of size 12.6 Gb". You'll need to know what code generated this error message. And then figure out if the code does something sensible. Often that is not the case when you get a similar error message. Best regards, ir. Thierry Onkelinx Statisticus / Statistician Vlaamse Overheid / Government of Flanders INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND FOREST Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance thierry.onkelinx at inbo.be Havenlaan 88 bus 73, 1000 Brussel www.inbo.be /////////////////////////////////////////////////////////////////////////////////////////// To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey /////////////////////////////////////////////////////////////////////////////////////////// <https://www.inbo.be> Op di 7 apr. 2020 om 16:16 schreef Samuel Granjeaud IR/Inserm < samuel.granjeaud at inserm.fr>:> Hi Tomas, > > Many thanks for your answer. > > Here is a copy of a fresh session under RStudio, and after a copy under > Rgui. > Strangely enough the result of memory.limit() is not the same. Without > your question I would not have looked to RGui, being used to work with > RStudio. > > The value under RGui sounds to correspond to the total RAM of the > computer. It makes me noticing that the value is in MB. > > The value under Rstudio was so huge (1.759219e+13) that I just > interpreted it as GB. But I was totally wrong. So in fact I don't know > what it refers to. The documentation says "For a 64-bit versions of R > under 64-bit Windows the limit is currently 8Tb.", but it looks like > being 16TB, which my computer don't have of course. > > I still have to understand why my colleague has a problem of memory > allocation (cannot allocate vector of size 12.6 Gb). > > Sorry for the wrong interpretation, but thanks for the help, > Samuel > > --- RStudio > > R version 3.6.3 (2020-02-29) -- "Holding the Windsock" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-w64-mingw32/x64 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > memory.limit() > [1] 1.759219e+13 > > sessionInfo() > R version 3.6.3 (2020-02-29) > Platform: x86_64-w64-mingw32/x64 (64-bit) > Running under: Windows 10 x64 (build 18363) > > Matrix products: default > > locale: > [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 > LC_MONETARY=French_France.1252 LC_NUMERIC=C > [5] LC_TIME=French_France.1252 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] compiler_3.6.3 tools_3.6.3 > > > > --- RGui > > R version 3.6.3 (2020-02-29) -- "Holding the Windsock" > Copyright (C) 2020 The R Foundation for Statistical Computing > Platform: x86_64-w64-mingw32/x64 (64-bit) > > R is free software and comes with ABSOLUTELY NO WARRANTY. > You are welcome to redistribute it under certain conditions. > Type 'license()' or 'licence()' for distribution details. > > R is a collaborative project with many contributors. > Type 'contributors()' for more information and > 'citation()' on how to cite R or R packages in publications. > > Type 'demo()' for some demos, 'help()' for on-line help, or > 'help.start()' for an HTML browser interface to help. > Type 'q()' to quit R. > > > ls() > character(0) > > memory.limit() > [1] 32627 > > sessionInfo() > R version 3.6.3 (2020-02-29) > Platform: x86_64-w64-mingw32/x64 (64-bit) > Running under: Windows 10 x64 (build 18363) > > Matrix products: default > > locale: > [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 > LC_MONETARY=French_France.1252 > [4] LC_NUMERIC=C LC_TIME=French_France.1252 > > attached base packages: > [1] stats graphics grDevices utils datasets methods base > > loaded via a namespace (and not attached): > [1] compiler_3.6.3 > > > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >[[alternative HTML version deleted]]
Agreed. I don't see any evidence of a "hard limit" here. It just looks like the code you're running has exhausted the memory of your machine. It doesn't require creating a lot of 12.6 Gb vectors to do that, even on a machine with a lot of memory ;-) H. On 4/7/20 08:05, Thierry Onkelinx via R-devel wrote:> Dear Samuel, > > The most important information from your mail was the actual error message: > "cannot allocate vector of size 12.6 Gb". > > You'll need to know what code generated this error message. And then figure > out if the code does something sensible. Often that is not the case when > you get a similar error message. > > Best regards, > > ir. Thierry Onkelinx > Statisticus / Statistician > > Vlaamse Overheid / Government of Flanders > INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND > FOREST > Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance > thierry.onkelinx at inbo.be > Havenlaan 88 bus 73, 1000 Brussel > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.inbo.be&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=r6WLJ5dXWo2qb7mQwONaCxYeeWgKwycd3y89JoqY-oY&s=VNmTfuiByvneqd4lcQS0o961TZa3ygo2Rdp7AtiGnk0&e> > /////////////////////////////////////////////////////////////////////////////////////////// > To call in the statistician after the experiment is done may be no more > than asking him to perform a post-mortem examination: he may be able to say > what the experiment died of. ~ Sir Ronald Aylmer Fisher > The plural of anecdote is not data. ~ Roger Brinner > The combination of some data and an aching desire for an answer does not > ensure that a reasonable answer can be extracted from a given body of data. > ~ John Tukey > /////////////////////////////////////////////////////////////////////////////////////////// > > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.inbo.be&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=r6WLJ5dXWo2qb7mQwONaCxYeeWgKwycd3y89JoqY-oY&s=qqsNew3vl9ZvyUMRz_GQy_fAfoj9axzmbcsUUayPVAs&e= > > > > Op di 7 apr. 2020 om 16:16 schreef Samuel Granjeaud IR/Inserm < > samuel.granjeaud at inserm.fr>: > >> Hi Tomas, >> >> Many thanks for your answer. >> >> Here is a copy of a fresh session under RStudio, and after a copy under >> Rgui. >> Strangely enough the result of memory.limit() is not the same. Without >> your question I would not have looked to RGui, being used to work with >> RStudio. >> >> The value under RGui sounds to correspond to the total RAM of the >> computer. It makes me noticing that the value is in MB. >> >> The value under Rstudio was so huge (1.759219e+13) that I just >> interpreted it as GB. But I was totally wrong. So in fact I don't know >> what it refers to. The documentation says "For a 64-bit versions of R >> under 64-bit Windows the limit is currently 8Tb.", but it looks like >> being 16TB, which my computer don't have of course. >> >> I still have to understand why my colleague has a problem of memory >> allocation (cannot allocate vector of size 12.6 Gb). >> >> Sorry for the wrong interpretation, but thanks for the help, >> Samuel >> >> --- RStudio >> >> R version 3.6.3 (2020-02-29) -- "Holding the Windsock" >> Copyright (C) 2020 The R Foundation for Statistical Computing >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> >> R is free software and comes with ABSOLUTELY NO WARRANTY. >> You are welcome to redistribute it under certain conditions. >> Type 'license()' or 'licence()' for distribution details. >> >> R is a collaborative project with many contributors. >> Type 'contributors()' for more information and >> 'citation()' on how to cite R or R packages in publications. >> >> Type 'demo()' for some demos, 'help()' for on-line help, or >> 'help.start()' for an HTML browser interface to help. >> Type 'q()' to quit R. >> >> > memory.limit() >> [1] 1.759219e+13 >> > sessionInfo() >> R version 3.6.3 (2020-02-29) >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> Running under: Windows 10 x64 (build 18363) >> >> Matrix products: default >> >> locale: >> [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 >> LC_MONETARY=French_France.1252 LC_NUMERIC=C >> [5] LC_TIME=French_France.1252 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.6.3 tools_3.6.3 >> > >> >> --- RGui >> >> R version 3.6.3 (2020-02-29) -- "Holding the Windsock" >> Copyright (C) 2020 The R Foundation for Statistical Computing >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> >> R is free software and comes with ABSOLUTELY NO WARRANTY. >> You are welcome to redistribute it under certain conditions. >> Type 'license()' or 'licence()' for distribution details. >> >> R is a collaborative project with many contributors. >> Type 'contributors()' for more information and >> 'citation()' on how to cite R or R packages in publications. >> >> Type 'demo()' for some demos, 'help()' for on-line help, or >> 'help.start()' for an HTML browser interface to help. >> Type 'q()' to quit R. >> >> > ls() >> character(0) >> > memory.limit() >> [1] 32627 >> > sessionInfo() >> R version 3.6.3 (2020-02-29) >> Platform: x86_64-w64-mingw32/x64 (64-bit) >> Running under: Windows 10 x64 (build 18363) >> >> Matrix products: default >> >> locale: >> [1] LC_COLLATE=French_France.1252 LC_CTYPE=French_France.1252 >> LC_MONETARY=French_France.1252 >> [4] LC_NUMERIC=C LC_TIME=French_France.1252 >> >> attached base packages: >> [1] stats graphics grDevices utils datasets methods base >> >> loaded via a namespace (and not attached): >> [1] compiler_3.6.3 >> > >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=r6WLJ5dXWo2qb7mQwONaCxYeeWgKwycd3y89JoqY-oY&s=ABvG3sGKR5ln27FVCM8dlmZ82X93ZCTigbMxHeBEb6E&e>> > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_r-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=r6WLJ5dXWo2qb7mQwONaCxYeeWgKwycd3y89JoqY-oY&s=ABvG3sGKR5ln27FVCM8dlmZ82X93ZCTigbMxHeBEb6E&e>-- Herv? Pag?s Program in Computational Biology Division of Public Health Sciences Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA 98109-1024 E-mail: hpages at fredhutch.org Phone: (206) 667-5791 Fax: (206) 667-1319