I am trying to implement a global counter in a library. This counter
resides in an environment local to that package, and is incremented by
calling a function within this package.
Problems arise when I use parallelization. I protected the increment
operation with a lock so that this operation effectively should be
atomic. However, the environment local to the package seems not to be
shared among the threads that use the functions in that package (not in
the sense that a change would be visible in the other threads).
How is such a global counter best implemented?
Here is a basic sketch of what I tried:
library(parallel)
library(flock)
## begin of code in package
tmp <- tempfile()
.localstuff <- new.env()
.localstuff$N <- 0
.localstuff$tmp <- tempfile()
incr <- function()
{
??? h <- lock(.localstuff$tmp)
??? .localstuff$N <- .localstuff$N + 1
??? unlock(h)
}
## end code in library
invisible(mclapply(1:10, function(i) incr()))
cat("N =", .localstuff$N)
On 11/03/2019 2:57 p.m., Pascal A. Niklaus wrote:> I am trying to implement a global counter in a library. This counter > resides in an environment local to that package, and is incremented by > calling a function within this package. > > Problems arise when I use parallelization. I protected the increment > operation with a lock so that this operation effectively should be > atomic. However, the environment local to the package seems not to be > shared among the threads that use the functions in that package (not in > the sense that a change would be visible in the other threads). > > How is such a global counter best implemented?Use the Rdsm package. See ?Rdsm::Rdsm for help. Duncan Murdoch> > Here is a basic sketch of what I tried: > > > library(parallel) > > library(flock) > > ## begin of code in package > > tmp <- tempfile() > .localstuff <- new.env() > .localstuff$N <- 0 > .localstuff$tmp <- tempfile() > > incr <- function() > { > ??? h <- lock(.localstuff$tmp) > ??? .localstuff$N <- .localstuff$N + 1 > ??? unlock(h) > } > > ## end code in library > > invisible(mclapply(1:10, function(i) incr())) > > cat("N =", .localstuff$N)
Thanks for the hint. I think I will use a big.matrix [with 1 element only ;-) ] and a mutex to ensure access is thread-safe. Pascal On 12.03.19 17:35, Duncan Murdoch wrote:> On 11/03/2019 2:57 p.m., Pascal A. Niklaus wrote: >> I am trying to implement a global counter in a library. This counter >> resides in an environment local to that package, and is incremented by >> calling a function within this package. >> >> Problems arise when I use parallelization. I protected the increment >> operation with a lock so that this operation effectively should be >> atomic. However, the environment local to the package seems not to be >> shared among the threads that use the functions in that package (not in >> the sense that a change would be visible in the other threads). >> >> How is such a global counter best implemented? > > Use the Rdsm package.? See ?Rdsm::Rdsm for help. > > Duncan Murdoch > >> >> Here is a basic sketch of what I tried: >> >> >> library(parallel) >> >> library(flock) >> >> ## begin of code in package >> >> tmp <- tempfile() >> .localstuff <- new.env() >> .localstuff$N <- 0 >> .localstuff$tmp <- tempfile() >> >> incr <- function() >> { >> ? ??? h <- lock(.localstuff$tmp) >> ? ??? .localstuff$N <- .localstuff$N + 1 >> ? ??? unlock(h) >> } >> >> ## end code in library >> >> invisible(mclapply(1:10, function(i) incr())) >> >> cat("N =", .localstuff$N)