Heads up: I posted this question on Stackoverflow yesterday
(http://stackoverflow.com/questions/26484103/using-a-custom-memory-allocation-function-in-r).
So far I haven?t gotten a response and I thought this could be an even
better place to ask such a question.
I would like to be able to use my own memory allocation function for
certain data structures (real valued vectors and arrays) in R. The
reason for this is that I need my data to be 64bit aligned and I would
like to use the numa library for having control over which memory node
is used (I'm working on compute nodes with four 12-core AMD Opteron
6174 CPUs).
Now I have two functions for allocating and freeing memory:
numa_alloc_onnode and numa_free(courtesy of
http://stackoverflow.com/questions/8154162/numa-aware-cache-aligned-memory-allocation).
I'm using R version 3.1.1, so I have access to the function
allocVector3(src/main/memory.c), which seems to me as the intended way
of adding a custom memory allocator. I also found the struct
R_allocator in src/include/R_ext
However it is not clear to me how to put these pieces together. Let's
say, in R, I want the result res of an evaluation such as
res <- Y - mean(Y)
to be saved in a memory area allocated with my own function, how would
I do this? Can I integrate allocVector3 directly at the R level? I
assume I have to go through the R-C interface. As far as I know, I
cannot just return a pointer to the allocated area, but have to pass
the result as an argument. So in R I call something like
n <- length(Y)
res <- numeric(length=1)
.Call("R_allocate_using_myalloc", n, res)
res <- Y - mean(Y)
and in C
#include <R.h>
#include <Rinternals.h>
#include <numa.h>
SEXP R_allocate_using_myalloc(SEXP R_n, SEXP R_res){
PROTECT(R_n = coerceVector(R_n, INTSXP));
PROTECT(R_res = coerceVector(R_res, REALSXP));
int *restrict n = INTEGER(R_n);
R_allocator_t myAllocator;
myAllocator.mem_alloc = numa_alloc_onnode;
myAllocator.mem_free = numa_free;
myAllocator.res = NULL;
myAllocator.data = ???;
R_res = allocVector3(REALSXP, n, myAllocator);
UNPROTECT(2);
return R_res;
}
Unfortunately I cannot get beyond a variable has incomplete type
'R_allocator_t' compilation error (I had to remove the .data line
since I have no clue as to what I should put there). Also the function
signature for numa_alloc_onnode is (size_t size, int node). How do I
pass the number of the memory node to the numa_alloc_onnode? Is that
somehow done through the .data field? Does any of the above code make
sense? Is there an easier way of achieving what I want to? It seems a
bit odd to have to allocate a small vector in R and the change its
location in memory in C just to be able to both control the memory
allocation and have the vector available in R...
I'm trying to avoid using Rcpp, as I'm modifying a fairly large
package and do not want to convert all C calls and thought that mixing
different C interfaces could perform sub-optimally.
Any help is greatly appreciated.
hello, I do not understand your aim... it not helpful in malloc_hook? example when I used hugepege is right here. http://prs.ism.ac.jp/~nakama/ISM/ism.c However, I think that you can more experiments in numaclt(see man numactl). 2014-10-22 23:53 GMT+09:00 SO User <user3855417 at gmail.com>:> Heads up: I posted this question on Stackoverflow yesterday > (http://stackoverflow.com/questions/26484103/using-a-custom-memory-allocation-function-in-r). > So far I haven't gotten a response and I thought this could be an even > better place to ask such a question. > > I would like to be able to use my own memory allocation function for > certain data structures (real valued vectors and arrays) in R. The > reason for this is that I need my data to be 64bit aligned and I would > like to use the numa library for having control over which memory node > is used (I'm working on compute nodes with four 12-core AMD Opteron > 6174 CPUs). > > Now I have two functions for allocating and freeing memory: > numa_alloc_onnode and numa_free(courtesy of > http://stackoverflow.com/questions/8154162/numa-aware-cache-aligned-memory-allocation). > I'm using R version 3.1.1, so I have access to the function > allocVector3(src/main/memory.c), which seems to me as the intended way > of adding a custom memory allocator. I also found the struct > R_allocator in src/include/R_ext > > However it is not clear to me how to put these pieces together. Let's > say, in R, I want the result res of an evaluation such as > > res <- Y - mean(Y) > > to be saved in a memory area allocated with my own function, how would > I do this? Can I integrate allocVector3 directly at the R level? I > assume I have to go through the R-C interface. As far as I know, I > cannot just return a pointer to the allocated area, but have to pass > the result as an argument. So in R I call something like > > n <- length(Y) > res <- numeric(length=1) > .Call("R_allocate_using_myalloc", n, res) > res <- Y - mean(Y) > > and in C > > #include <R.h> > #include <Rinternals.h> > #include <numa.h> > > SEXP R_allocate_using_myalloc(SEXP R_n, SEXP R_res){ > > PROTECT(R_n = coerceVector(R_n, INTSXP)); > PROTECT(R_res = coerceVector(R_res, REALSXP)); > int *restrict n = INTEGER(R_n); > > R_allocator_t myAllocator; > myAllocator.mem_alloc = numa_alloc_onnode; > myAllocator.mem_free = numa_free; > myAllocator.res = NULL; > myAllocator.data = ???; > > R_res = allocVector3(REALSXP, n, myAllocator); > > UNPROTECT(2); > return R_res; > } > > Unfortunately I cannot get beyond a variable has incomplete type > 'R_allocator_t' compilation error (I had to remove the .data line > since I have no clue as to what I should put there). Also the function > signature for numa_alloc_onnode is (size_t size, int node). How do I > pass the number of the memory node to the numa_alloc_onnode? Is that > somehow done through the .data field? Does any of the above code make > sense? Is there an easier way of achieving what I want to? It seems a > bit odd to have to allocate a small vector in R and the change its > location in memory in C just to be able to both control the memory > allocation and have the vector available in R... > > I'm trying to avoid using Rcpp, as I'm modifying a fairly large > package and do not want to convert all C calls and thought that mixing > different C interfaces could perform sub-optimally. > > Any help is greatly appreciated. > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel-- Best Regards, -- Eiji NAKAMA <nakama (a) ki.rim.or.jp> "\u4e2d\u9593\u6804\u6cbb" <nakama (a) ki.rim.or.jp>
SO, On Oct 22, 2014, at 10:53 AM, SO User <user3855417 at gmail.com> wrote:> > Heads up: I posted this question on Stackoverflow yesterday > (http://stackoverflow.com/questions/26484103/using-a-custom-memory-allocation-function-in-r). > So far I haven?t gotten a response and I thought this could be an even > better place to ask such a question. > > I would like to be able to use my own memory allocation function for > certain data structures (real valued vectors and arrays) in R. The > reason for this is that I need my data to be 64bit aligned and I would > like to use the numa library for having control over which memory node > is used (I'm working on compute nodes with four 12-core AMD Opteron > 6174 CPUs). > > Now I have two functions for allocating and freeing memory: > numa_alloc_onnode and numa_free(courtesy of > http://stackoverflow.com/questions/8154162/numa-aware-cache-aligned-memory-allocation). > I'm using R version 3.1.1, so I have access to the function > allocVector3(src/main/memory.c), which seems to me as the intended way > of adding a custom memory allocator. I also found the struct > R_allocator in src/include/R_ext > > However it is not clear to me how to put these pieces together. Let's > say, in R, I want the result res of an evaluation such as > > res <- Y - mean(Y) > > to be saved in a memory area allocated with my own function, how would > I do this? Can I integrate allocVector3 directly at the R level? I > assume I have to go through the R-C interface. As far as I know, I > cannot just return a pointer to the allocated area, but have to pass > the result as an argument. So in R I call something like > > n <- length(Y) > res <- numeric(length=1) > .Call("R_allocate_using_myalloc", n, res) > res <- Y - mean(Y) >That obviously won't work, because you're replacing your "res" with another binding - effectively removing anything you did. You could use something like res <- .Call("R_allocate_using_myalloc", Y - mean(y)) if you wanted to convert "regular" R object into your own backed store. However, the intent is typically to create objects with special backing store in your C code. One example are mmaped R objects - see https://gist.github.com/s-u/6712c97ca74181f5a1a5 for an example.> and in C > > #include <R.h> > #include <Rinternals.h> > #include <numa.h> >You are missing #include <R_ext/Rallocators.h>> SEXP R_allocate_using_myalloc(SEXP R_n, SEXP R_res){ > > PROTECT(R_n = coerceVector(R_n, INTSXP)); > PROTECT(R_res = coerceVector(R_res, REALSXP)); > int *restrict n = INTEGER(R_n); > > R_allocator_t myAllocator; > myAllocator.mem_alloc = numa_alloc_onnode; > myAllocator.mem_free = numa_free; > myAllocator.res = NULL; > myAllocator.data = ???; > > R_res = allocVector3(REALSXP, n, myAllocator); > > UNPROTECT(2); > return R_res; > } > > Unfortunately I cannot get beyond a variable has incomplete type > 'R_allocator_t' compilation errorMissed include above.> (I had to remove the .data line > since I have no clue as to what I should put there).data is an opaque pointer that you can use for anything you want - R doesn't use it, it just passes it through so you can have auxiliary information attached to your allocation.> Also the function > signature for numa_alloc_onnode is (size_t size, int node). How do I > pass the number of the memory node to the numa_alloc_onnode? Is that > somehow done through the .data field?Yes, you can supply it from your initiator function (just like the filename is passed in the mmap example).> Does any of the above code make > sense? Is there an easier way of achieving what I want to? It seems a > bit odd to have to allocate a small vector in R and the change its > location in memory in C just to be able to both control the memory > allocation and have the vector available in R... > > I'm trying to avoid using Rcpp, as I'm modifying a fairly large > package and do not want to convert all C calls and thought that mixing > different C interfaces could perform sub-optimally. >AFAICT Rcpp wouldn't help you there - since it still has to go back to R for allocation of R objects. Cheers, Simon> Any help is greatly appreciated. > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel