R devel - Aug 2008 - Allocated Memory Warnings in Vista 32 bit with 4 GB (PR#12557)

Hello

I am running Windows Vista 32 with 4 GB (installed, though Windows of course
only recognizes 3326 MB, as reported by Windows "My Computer")

I am running R 2.7.1

I was trying to read in a comma delimited single column CSV file, assign that
file to a variable ("data") and then extract a sample (assigned to
"partial").  I was getting memory allocation errors, and from the log
below, I started to see a pattern which indicates that the "error" is
related to how R views memory allocation, and perhaps how R is using or reading
memory on my specific system.  The error, I would guess, is probably related to
how R is reporting the memory available.

My source files have the same number of rows (or cases) as the name of the file.
Thus, data10000 has 10,000 observations, and data300000 has 300,000
observations.

Here is my history which shows the problem, and inconsistency in memory
allocation reporting:

  > data=3Dread.csv("data10000.csv",header=3DFALSE)
  > partial=3Dsample(data,5000,T)
  > data=3Dread.csv("data100000.csv",header=3DFALSE)
  Error in file(file, "r") : cannot open the connection
  In addition: Warning message:
  In file(file, "r") :
    cannot open file 'data100000.csv': No such file or directory
  > data=3Dread.csv("data300000.csv",header=3DFALSE)
  > partial=3Dsample(data,5000,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial=3Dsample(data,5,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial=3Dsample(data,1,T)
  Warning messages:
  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial=3Dsample(data,1,T)
  > partial=3Dsample(data,5000,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial[1:5,]
  [1]  0.3204279  1.6583593 -0.3456585  1.2951363 -1.1096974
  > rm(partial)
  > partial=3Dsample(data,5000,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  5: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  6: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial[1:5,]
  Error: object "partial" not found
  > partial=3Dsample(data,1,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial=3Dsample(data,1,T)
  Warning messages:
  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
    Reached total allocation of 1535Mb: see help(memory.size)
  > partial=3Dsample(data,1,T)
  > partial=3Dsample(data,5000,T)
  Error: cannot allocate vector of size 2.3 Mb
  In addition: Warning messages:
  1: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  2: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  3: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  4: In names(y) <- make.unique(cols) :
    Reached total allocation of 1535Mb: see help(memory.size)
  >=20


I have been able to reproduce the situation several times.  For example, what is
curious is that a command like partial=3Dsample(data,1,T) may or may not produce
a warning message.  In the history, obviously 2.3MB is well below the allocated
1535MB.  I had been getting this error on my original larger source data
(1,000,000 observations), but since I am a new R user, it was not clear to me
what these errors mean.

If you need the original source files, or have additional questions, please let
me know.  I am a certified .NET software developer, and although I do not know
everything about Windows development, I believe my abilities are above average.



Mark Tabladillo=20
Atlanta, GA
marktab at email.phoenix.edu
Alternate:  x2008 at marktab.com


	[[alternative HTML version deleted]]

I see no bug here.  R is telling you that you have insufficient memory 
available: see also ?"Memory-limits".

See also rw-FAQ Q2.9.

Note that we don't have a reproducible example and in particular have no 
idea of how many columns this data frame has.

The 'R Data Import/Export' manual gives you many hints on how to do this
more efficiently, and it is referenced from the help page.  I see no sign 
that you have consulted it, so doing your homework is your next (belated) 
step.


On Sun, 17 Aug 2008, drmark at marktab.com wrote:
> Hello
>
> I am running Windows Vista 32 with 4 GB (installed, though Windows of
cours> e only recognizes 3326 MB, as reported by Windows "My
Computer")
And only 2Gb of that is available to a user process.
> I am running R 2.7.1
>
> I was trying to read in a comma delimited single column CSV file, assign
th> at file to a variable ("data") and then extract a sample
(assigned to "part> ial").  I was getting memory allocation errors,
and from the log below, I s> tarted to see a pattern which indicates that the
"error" is related to how > R views memory allocation, and perhaps
how R is using or reading memory on > my specific system.  The error, I would
guess, is probably related to how R> is reporting the memory available.
>
> My source files have the same number of rows (or cases) as the name of the
> file.  Thus, data10000 has 10,000 observations, and data300000 has 300,000
> observations.
>
> Here is my history which shows the problem, and inconsistency in memory
all> ocation reporting:
>
>  > data=3Dread.csv("data10000.csv",header=3DFALSE)
>  > partial=3Dsample(data,5000,T)
>  > data=3Dread.csv("data100000.csv",header=3DFALSE)
>  Error in file(file, "r") : cannot open the connection
>  In addition: Warning message:
>  In file(file, "r") :
>    cannot open file 'data100000.csv': No such file or directory
>  > data=3Dread.csv("data300000.csv",header=3DFALSE)
>  > partial=3Dsample(data,5000,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial=3Dsample(data,5,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial=3Dsample(data,1,T)
>  Warning messages:
>  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial=3Dsample(data,1,T)
>  > partial=3Dsample(data,5000,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial[1:5,]
>  [1]  0.3204279  1.6583593 -0.3456585  1.2951363 -1.1096974
>  > rm(partial)
>  > partial=3Dsample(data,5000,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  5: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  6: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial[1:5,]
>  Error: object "partial" not found
>  > partial=3Dsample(data,1,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial=3Dsample(data,1,T)
>  Warning messages:
>  1: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In attributes(.Data) <- c(attributes(.Data), attrib) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  > partial=3Dsample(data,1,T)
>  > partial=3Dsample(data,5000,T)
>  Error: cannot allocate vector of size 2.3 Mb
>  In addition: Warning messages:
>  1: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  2: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  3: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  4: In names(y) <- make.unique(cols) :
>    Reached total allocation of 1535Mb: see help(memory.size)
>  >=20
>
>
> I have been able to reproduce the situation several times.  For example,
wh> at is curious is that a command like partial=3Dsample(data,1,T) may or
may > not produce a warning message.  In the history, obviously 2.3MB is well
bel> ow the allocated 1535MB.  I had been getting this error on my original
larg> er source data (1,000,000 observations), but since I am a new R user,
it wa> s not clear to me what these errors mean.
>
> If you need the original source files, or have additional questions,
please> let me know.  I am a certified .NET software developer, and although
I do > not know everything about Windows development, I believe my abilities
are a> bove average.
>
>
>
> Mark Tabladillo=20
> Atlanta, GA
> marktab at email.phoenix.edu
> Alternate:  x2008 at marktab.com
>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595