R devel - Apr 2019 - SUGGESTION: Settings to disable forked processing in R, e.g. parallel::mclapply()

On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com>
wrote:
>
> I think it's worth saying that mclapply() works as documented
Mostly, yes. But it says nothing about fork's copy-on-write and memory
overcommitment, and that this means that it may work nicely or fail
spectacularly depending on whether, e.g., you operate on a long
vector.

-- 
I?aki ?car

Sure, but that a completely bogus argument because in that case it would fail
even more spectacularly with any other method like PSOCK because you would *have
to* allocate n times as much memory so unlike mclapply it is guaranteed to fail.
With mclapply it is simply much more efficient as it will share memory as long
as possible. It is rather obvious that any new objects you create can no longer
be shared as they now exist separately in each process.

Cheers,
Simon


> On Apr 13, 2019, at 06:05, I?aki Ucar <iucar at fedoraproject.org>
wrote:
> 
> On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com>
wrote:
>> 
>> I think it's worth saying that mclapply() works as documented
> 
> Mostly, yes. But it says nothing about fork's copy-on-write and memory
> overcommitment, and that this means that it may work nicely or fail
> spectacularly depending on whether, e.g., you operate on a long
> vector.
> 
> -- 
> I?aki ?car
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

On Sat, 13 Apr 2019 at 18:41, Simon Urbanek <simon.urbanek at
r-project.org> wrote:
>
> Sure, but that a completely bogus argument because in that case it would
fail even more spectacularly with any other method like PSOCK because you would
*have to* allocate n times as much memory so unlike mclapply it is guaranteed to
fail. With mclapply it is simply much more efficient as it will share memory as
long as possible. It is rather obvious that any new objects you create can no
longer be shared as they now exist separately in each process.
The point was that PSOCK fails and succeeds *consistently*,
independently of what you do with the input in the function provided.
I think that's a good property.

-- 
I?aki ?car

On 4/13/19 12:05 PM, I?aki Ucar wrote:
> On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at gmail.com>
wrote:
>> I think it's worth saying that mclapply() works as documented
> Mostly, yes. But it says nothing about fork's copy-on-write and memory
> overcommitment, and that this means that it may work nicely or fail
> spectacularly depending on whether, e.g., you operate on a long
> vector.
R cannot possibly replicate documentation of the underlying operating 
systems. It clearly says that fork() is used and readers who may not 
know what fork() is need to learn it from external sources. 
Copy-on-write is an elementary property of fork().

Reimplementing mclapply to use PSOCK does not make sense -- if someone 
wants to write code that can be used both with PSOCK and FORK, there is 
the cluster API in parallel for that.

Tomas

On Mon, 15 Apr 2019 at 08:44, Tomas Kalibera <tomas.kalibera at gmail.com>
wrote:
>
> On 4/13/19 12:05 PM, I?aki Ucar wrote:
> > On Sat, 13 Apr 2019 at 03:51, Kevin Ushey <kevinushey at
gmail.com> wrote:
> >> I think it's worth saying that mclapply() works as documented
> > Mostly, yes. But it says nothing about fork's copy-on-write and
memory
> > overcommitment, and that this means that it may work nicely or fail
> > spectacularly depending on whether, e.g., you operate on a long
> > vector.
>
> R cannot possibly replicate documentation of the underlying operating
> systems. It clearly says that fork() is used and readers who may not
> know what fork() is need to learn it from external sources.
> Copy-on-write is an elementary property of fork().
Just to be precise, copy-on-write is an optimization widely deployed
in most modern *nixes, particularly for the architectures in which R
usually runs. But it is not an elementary property; it is not even
possible without an MMU.

-- 
I?aki ?car