R help - Jul 2020 - parallel number of cores according to memory?

ugghhh---apologies.  although in 2020, it would be nice if the mailing
list had an automatic html filter (or even bouncer!)

I am using macos.  alas, my experiments suggest that `mclapply()` on a
32-core intel system with 64GB of RAM, where the input data frame is
8GB and the output is about 500MB per core (to be stitched together
into about 16GB), the system starts swapping like crazy, comes to a
halt, and then usually crashes.

/iaw

The list does strip html, but the quality of what remains varies greatly.

Are you using tidyverse functions in your workers? Sounds to me like you are
doing something in the workers that is triggering making copies of the input
data frame.

On July 7, 2020 10:15:15 PM PDT, ivo welch <ivowel at gmail.com>
wrote:
>ugghhh---apologies.  although in 2020, it would be nice if the mailing
>list had an automatic html filter (or even bouncer!)
>
>I am using macos.  alas, my experiments suggest that `mclapply()` on a
>32-core intel system with 64GB of RAM, where the input data frame is
>8GB and the output is about 500MB per core (to be stitched together
>into about 16GB), the system starts swapping like crazy, comes to a
>halt, and then usually crashes.
>
>/iaw
-- 
Sent from my phone. Please excuse my brevity.

no, I'm not.   mostly conventional use afaik.  if this should not be
happening, I can trace it down to a small reproducible example to
figure it out.

--
Ivo Welch (ivo.welch at ucla.edu)

--
Ivo Welch (ivo.welch at ucla.edu)
http://www.ivo-welch.info/
J. Fred Weston Distinguished Professor of Finance, UCLA Anderson



On Tue, Jul 7, 2020 at 10:24 PM Jeff Newmiller <jdnewmil at
dcn.davis.ca.us> wrote:
>
> The list does strip html, but the quality of what remains varies greatly.
>
> Are you using tidyverse functions in your workers? Sounds to me like you
are doing something in the workers that is triggering making copies of the input
data frame.
>
> On July 7, 2020 10:15:15 PM PDT, ivo welch <ivowel at gmail.com>
wrote:
> >ugghhh---apologies.  although in 2020, it would be nice if the mailing
> >list had an automatic html filter (or even bouncer!)
> >
> >I am using macos.  alas, my experiments suggest that `mclapply()` on a
> >32-core intel system with 64GB of RAM, where the input data frame is
> >8GB and the output is about 500MB per core (to be stitched together
> >into about 16GB), the system starts swapping like crazy, comes to a
> >halt, and then usually crashes.
> >
> >/iaw
>
> --
> Sent from my phone. Please excuse my brevity.