R help - Aug 2011 - Exception while using NeweyWest function with doMC

Dear R users,

I am using R right now for a simulation of a model that needs a lot of
memory. Therefore I use the *bigmemory* package and - to make it faster -
the *doMC* package. See my code posted on http://pastebin.com/dFRGdNrG

Now, if I use the foreach loop with the addon %do% (for sequential run) I
have no problems at all - only here and there some singularities in
regressor matrices which should be ok.
BUT if I run the loop on multiple cores I get very often a bad exception. I
have posted the exception on http://pastebin.com/eMWF4cu0 The exception
comes from the NeweyWest function loaded within the sandwich library.

I have no clue, what it want to say me and why it is so weirdly printed to
the terminal. I am used to receive here and there errors....but the messages
never look like this.

Does anyone have a useful answer for me, where to look for the cause of this
weird error?

Here some additional information:

Hardware: MacBook Pro 2.66 GHz Intel Core Duo, 4 GB Memory 1067 MHz DDR3
Software System: Mac Os X Lion 10.7.1 (11B26)
Software App: R64 version 2.11.1 run via Mac terminal

I hope someone has a good suggestion!

Thank u all!

Simon

	[[alternative HTML version deleted]]

Simon,

Though we're please to see another use of bigmemory, it really isn't
clear that it is gaining you
anything in your example; anything like as.big.matrix(matrix(...))
still consumes full RAM for both
the inner matrix() and the new big.matrix -- is the filebacking really
necessary.  It also doesn't
appear that you are making use of shared memory, so I'm unsure what
the gains are.  However,
I don't have any particular insight as to the subsequent problem with
NeweyWest (which doesn't
seem to be using the big.matrix objects).

Jay

----------------------
Message: 32
Date: Sat, 27 Aug 2011 21:37:55 +0200
From: Simon Zehnder <simon.zehnder at googlemail.com>
To: r-help at r-project.org
Subject: [R] Exception while using NeweyWest function with doMC
Message-ID:
       <CAGQVrP_Gk+T0oWBv1ste-y0ZAFmi9s_ZWqRxyXuGsUi18mScQA at
mail.gmail.com>
Content-Type: text/plain

Dear R users,

I am using R right now for a simulation of a model that needs a lot of
memory. Therefore I use the *bigmemory* package and - to make it faster -
the *doMC* package. See my code posted on http://pastebin.com/dFRGdNrG

< snip >
-------------------------

-- 
John W. Emerson (Jay)
Associate Professor of Statistics
Department of Statistics
Yale University
http://www.stat.yale.edu/~jay

On Aug 27, 2011, at 3:37 PM, Simon Zehnder wrote:
> Dear R users,
>
> I am using R right now for a simulation of a model that needs a lot of
> memory. Therefore I use the *bigmemory* package and - to make it  
> faster -
> the *doMC* package. See my code posted on http://pastebin.com/dFRGdNrG
>
> Now, if I use the foreach loop with the addon %do% (for sequential  
> run) I
> have no problems at all - only here and there some singularities in
> regressor matrices which should be ok.
> BUT if I run the loop on multiple cores I get very often a bad  
> exception. I
> have posted the exception on http://pastebin.com/eMWF4cu0 The  
> exception
> comes from the NeweyWest function loaded within the sandwich library.
>
> I have no clue, what it want to say me and why it is so weirdly  
> printed to
> the terminal. I am used to receive here and there errors....but the  
> messages
> never look like this.
>
> Does anyone have a useful answer for me, where to look for the cause  
> of this
> weird error?
>
> Here some additional information:
>
> Hardware: MacBook Pro 2.66 GHz Intel Core Duo, 4 GB Memory 1067 MHz  
> DDR3
> Software System: Mac Os X Lion 10.7.1 (11B26)
> Software App: R64 version 2.11.1 run via Mac terminal
Using the R64 version in a 4GB environment will reduce the effective  
memory capacity since the larger pointers take up more space, and  
using parallel methods is unlikely to improve performance very much  
with only two cores. It also seems likely that there have been several  
bug fixes in the last couple of years since that version of R was  
released, so the package authors are unlikely to be very interested in  
segfault errors thrown by outdated software.
> I hope someone has a good suggestion!
Update R. Don't use features that only reduce performance and make  
unstable a machine that has limited resources.

-- 

David Winsemius, MD
West Hartford, CT

On Aug 30, 2011, at 11:29 AM, Simon Zehnder wrote:
> Hi David,
>
> thank you very much for your advice! I updated R and all my  
> packages. Regrettably it doesn't work yet. But, I think, that the  
> parallel processing (using 32bit) does improve time, especially when  
> it comes to higher dimensions:
>
> system.time(simuFunctionSeq(0.03, 0.015, 1, 5, 1000, 100,"/Users/ 
> simon/Documents/R/BigMTest"))
>   system.time(simuFunctionPar(0.03, 0.015, 1, 5, 1000, 100,"/Users/ 
> simon/Documents/R/BigMTest"))
> [1] "Sequential Processing with N =  1000  and K =  100"
>   user  system elapsed
>  5.157   0.086   5.587
> [1] "Parallel Processing with N =  1000  and K =  100"
>   user  system elapsed
>  6.069   0.220   3.895
>
> :> system.time(simuFunctionSeq(0.03, 0.015, 1, 5, 10000,
100,"/Users/
> simon/Documents/R/BigMTest"))
>   system.time(simuFunctionPar(0.03, 0.015, 1, 5, 10000, 100,"/Users/ 
> simon/Documents/R/BigMTest"))
> [1] "Sequential Processing with N =  10000  and K =  100"
>   user  system elapsed
>  8.129   0.689  12.747
> [1] "Parallel Processing with N =  10000  and K =  100"
>   user  system elapsed
>  8.387   0.772  12.005
>
> :> system.time(simuFunctionSeq(0.03, 0.015, 1, 5, 10000, 1000,"/ 
> Users/simon/Documents/R/BigMTest"))
>   system.time(simuFunctionPar(0.03, 0.015, 1, 5, 10000, 1000,"/Users/ 
> simon/Documents/R/BigMTest"))
> [1] "Sequential Processing with N =  10000  and K =  1000"
>   user  system elapsed
> 71.295   6.330 109.656
> [1] "Parallel Processing with N =  10000  and K =  1000"
>   user  system elapsed
> 50.943   6.347  89.115
>
> Or are the times negligible?
I would think that for most applications getting a gain of efficiency  
of 20% would be considered unworthy of the effort at setting up and  
maintaining. I suppose if a simulation ran for 18 hours in sequential  
mode and you would be happier if it were done in the morning after  
leaving overnight and  finding it had completed in 15 hours, it might  
be worth the effort.
> What happens if I use a supercomputer with several cores and much  
> more memory?
Or even a MacPro with 4 or 8 cores and 32-64 GB?. Generally you hope  
to see halving or quartering in times when you apply these techniques.

-- 
David.
>
> Thanks again!
>
> Simon
>
>
>
> On Aug 29, 2011, at 6:59 PM, David Winsemius wrote:
>
>>
>> On Aug 27, 2011, at 3:37 PM, Simon Zehnder wrote:
>>
>>> Dear R users,
>>>
>>> I am using R right now for a simulation of a model that needs a  
>>> lot of
>>> memory. Therefore I use the *bigmemory* package and - to make it  
>>> faster -
>>> the *doMC* package. See my code posted on
http://pastebin.com/dFRGdNrG
>>>
>>> Now, if I use the foreach loop with the addon %do% (for sequential
>>> run) I
>>> have no problems at all - only here and there some singularities in
>>> regressor matrices which should be ok.
>>> BUT if I run the loop on multiple cores I get very often a bad  
>>> exception. I
>>> have posted the exception on http://pastebin.com/eMWF4cu0 The  
>>> exception
>>> comes from the NeweyWest function loaded within the sandwich  
>>> library.
>>>
>>> I have no clue, what it want to say me and why it is so weirdly  
>>> printed to
>>> the terminal. I am used to receive here and there errors....but  
>>> the messages
>>> never look like this.
>>>
>>> Does anyone have a useful answer for me, where to look for the  
>>> cause of this
>>> weird error?
>>>
>>> Here some additional information:
>>>
>>> Hardware: MacBook Pro 2.66 GHz Intel Core Duo, 4 GB Memory 1067  
>>> MHz DDR3
>>> Software System: Mac Os X Lion 10.7.1 (11B26)
>>> Software App: R64 version 2.11.1 run via Mac terminal
>>
>> Using the R64 version in a 4GB environment will reduce the  
>> effective memory capacity since the larger pointers take up more  
>> space, and using parallel methods is unlikely to improve  
>> performance very much with only two cores. It also seems likely  
>> that there have been several bug fixes in the last couple of years  
>> since that version of R was released, so the package authors are  
>> unlikely to be very interested in segfault errors thrown by  
>> outdated software.
>>
>>> I hope someone has a good suggestion!
>>
>> Update R. Don't use features that only reduce performance and make
>> unstable a machine that has limited resources.
>>
>> -- 
>>
>> David Winsemius, MD
>> West Hartford, CT
>>
>
David Winsemius, MD
West Hartford, CT