R help - Aug 2007 - Mixture of Normals with Large Data

All:

I am trying to fit a mixture of 2 normals with > 110 million observations. I
am running R 2.5.1 on a box with 1gb RAM running 32-bit windows and I
continue to run out of memory. Does anyone have any suggestions.

Thanks so much,

Tim

	[[alternative HTML version deleted]]

On Sat, 4 Aug 2007, Tim Victor wrote:
> All:
>
> I am trying to fit a mixture of 2 normals with > 110 million
observations. I
> am running R 2.5.1 on a box with 1gb RAM running 32-bit windows and I
> continue to run out of memory. Does anyone have any suggestions.

If the first few million observations can be regarded as a SRS of the 
rest, then just use them. Or read in blocks of a convenient size and 
sample some observations from each block. You can repeat this process a 
few times to see if the results are sufficiently accurate.

Otherwise, read in blocks of a convenient size (perhaps 1 million 
observations at a time), quantize the data to a manageable number of 
intervals - maybe a few thousand - and tabulate it. Add the counts over 
all the blocks.

Then use mle() to fit a multinomial likelihood whose probabilities are the 
masses associated with each bin under a mixture of normals law.

Chuck
>
> Thanks so much,
>
> Tim
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
Charles C. Berry                            (858) 534-2098
                                             Dept of Family/Preventive Medicine
E mailto:cberry at tajo.ucsd.edu	            UC San Diego
http://famprevmed.ucsd.edu/faculty/cberry/  La Jolla, San Diego 92093-0901

Another possibility is to use "data squashing" methods.  Relevant
papers are: (1) DuMouchel et al. (1999), (2) Madigan et al. (2002), and (3) Owen
(1999).

Ravi.
____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. (410) 502-2619
email: rvaradhan at jhmi.edu


----- Original Message -----
From: "Charles C. Berry" <cberry at tajo.ucsd.edu>
Date: Saturday, August 4, 2007 8:01 pm
Subject: Re: [R] Mixture of Normals with Large Data
To: tvictor at dolphin.upenn.edu
Cc: r-help at stat.math.ethz.ch

> On Sat, 4 Aug 2007, Tim Victor wrote:
>  
>  > All:
>  >
>  > I am trying to fit a mixture of 2 normals with > 110 million 
> observations. I
>  > am running R 2.5.1 on a box with 1gb RAM running 32-bit windows and 
> I
>  > continue to run out of memory. Does anyone have any suggestions.
>  
>  
>  If the first few million observations can be regarded as a SRS of the 
> 
>  rest, then just use them. Or read in blocks of a convenient size and 
> 
>  sample some observations from each block. You can repeat this process 
> a 
>  few times to see if the results are sufficiently accurate.
>  
>  Otherwise, read in blocks of a convenient size (perhaps 1 million 
>  observations at a time), quantize the data to a manageable number of 
> 
>  intervals - maybe a few thousand - and tabulate it. Add the counts 
> over 
>  all the blocks.
>  
>  Then use mle() to fit a multinomial likelihood whose probabilities 
> are the 
>  masses associated with each bin under a mixture of normals law.
>  
>  Chuck
>  
>  >
>  > Thanks so much,
>  >
>  > Tim
>  >
>  > 	[[alternative HTML version deleted]]
>  >
>  > ______________________________________________
>  > R-help at stat.math.ethz.ch mailing list
>  > 
>  > PLEASE do read the posting guide 
>  > and provide commented, minimal, self-contained, reproducible code.
>  >
>  
>  Charles C. Berry                            (858) 534-2098
>                                               Dept of 
> Family/Preventive Medicine
>  E                     UC San Diego
>    La Jolla, San Diego 92093-0901
>  
>  ______________________________________________
>  R-help at stat.math.ethz.ch mailing list
>  
>  PLEASE do read the posting guide 
>  and provide commented, minimal, self-contained, reproducible code.