Simone Giannerini
2005-Sep-09 17:04 UTC
[Rd] A question on R memory management in .Fortran() calls under Windows
Dear R community,
I have a question on how R manages memory allocation in .Fortran()
calls under Windows.
In brief, apparently, it is not possible to allocate large matrices
inside a Fortran subroutine
unless you pass them as arguments. If you do not act in this way
RGUI crashes with a stack overflow error and acting on memory through
vsize nsize ppsize and memory.limit does not help at all.
****************************************************************************************
Details of the configurations on which I performed testing follow:
R 2.1.1 on WinXP Pro SP2 ITA
PC1: AMD 64 3700+ 1GB RAM
PC2: AMD AthlonXP 2400+ 512Mb RAM
Compaq Visual Fortran pro 6.6C
****************************************************************************************
To give an idea I attach a brief example on how to reproduce this:
Create two simple subroutines 'foo' and 'foobis' that, say, give
the
sum of the elements of a matrix:
*** file foo.f90 starts
***********************************************************************
SUBROUTINE foo(X,M,N,S)
!DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foo_' :: FOO
IMPLICIT NONE
integer:: M,N
real*8:: X(M,N),S
S = sum(X);
END SUBROUTINE foo
SUBROUTINE foobis(M,N,S)
!DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foobis_' :: FOOBIS
IMPLICIT NONE
integer:: M,N
real*8:: X(M,N),S
X = 1;
S = sum(X);
END SUBROUTINE foobis
*** file foo.f90 ends
***********************************************************************
Notice that the matrix X is an input argument in foo and an internal
variable in foobis.
After compiling and linking turn to R:
**************************************************************************> dyn.load("foo.dll");
> is.loaded(symbol.For("foo"));
[1] TRUE> is.loaded(symbol.For("foobis"));
[1] TRUE> M <- 10;
> N <- 10;
> X <- matrix(1,M,N);
> .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S;
[1] 100 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S;
[1] 100
## no problem here with small matrices, let's increase the size
> M <- 3000;
> N <- 100;
> X <- matrix(1,M,N);
> .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S;
[1] 3e+05 ## OK
.Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S;
## *** R GUI CRASHES WITH A STACK OVERFLOW ERROR ***
****************************************************************************
Any suggestion would be greatly appreciated, I apologize if this
problem has already been addressed previously, I did not notice it.
Many thanks in advance,
Simone
______________________________________________________
Simone Giannerini
Dipartimento di Scienze Statistiche "Paolo Fortunati"
Universita' di Bologna
Via delle belle arti 41 - 40126 Bologna, ITALY
Tel: +39 051 2098248 Fax: +39 051 232153
E-mail: giannerini at stat.unibo.it
Duncan Murdoch
2005-Sep-09 17:51 UTC
[Rd] A question on R memory management in .Fortran() calls under Windows
On 9/9/2005 1:04 PM, Simone Giannerini wrote:> Dear R community, > > I have a question on how R manages memory allocation in .Fortran() > calls under Windows. > In brief, apparently, it is not possible to allocate large matrices > inside a Fortran subroutine > unless you pass them as arguments. If you do not act in this way > RGUI crashes with a stack overflow error and acting on memory through > vsize nsize ppsize and memory.limit does not help at all.It looks as though your Fortran compiler is allocating the new matrix on the stack. R doesn't give you a huge stack, and that's causing the overflow. When you get R to do the allocation, it does it on the heap, which has no artificial limits. Only a pointer to the object ends up on the stack. I'd say your only reasonable workarounds are to tell your compiler to use the heap for the local matrix allocation (if that's possible), or do your allocations in R. There might be some utility somewhere which can modify Rgui.exe to tell it to start with a larger stack, and there is probably some linker option to put one in place, but it's not really reasonable to increase R's stack, because in almost all other situations, writing that much data to the stack is a sign of an infinite recursion. Duncan Murdoch> > **************************************************************************************** > Details of the configurations on which I performed testing follow: > R 2.1.1 on WinXP Pro SP2 ITA > PC1: AMD 64 3700+ 1GB RAM > PC2: AMD AthlonXP 2400+ 512Mb RAM > Compaq Visual Fortran pro 6.6C > **************************************************************************************** > > To give an idea I attach a brief example on how to reproduce this: > Create two simple subroutines 'foo' and 'foobis' that, say, give the > sum of the elements of a matrix: > > *** file foo.f90 starts > *********************************************************************** > SUBROUTINE foo(X,M,N,S) > !DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foo_' :: FOO > > IMPLICIT NONE > integer:: M,N > real*8:: X(M,N),S > > S = sum(X); > > END SUBROUTINE foo > > SUBROUTINE foobis(M,N,S) > !DEC$ ATTRIBUTES DLLEXPORT,C,REFERENCE,ALIAS:'foobis_' :: FOOBIS > > IMPLICIT NONE > integer:: M,N > real*8:: X(M,N),S > X = 1; > S = sum(X); > > END SUBROUTINE foobis > > *** file foo.f90 ends > *********************************************************************** > > Notice that the matrix X is an input argument in foo and an internal > variable in foobis. > After compiling and linking turn to R: > ************************************************************************** >> dyn.load("foo.dll"); >> is.loaded(symbol.For("foo")); > [1] TRUE >> is.loaded(symbol.For("foobis")); > [1] TRUE >> M <- 10; >> N <- 10; >> X <- matrix(1,M,N); >> .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S; > [1] 100 >> .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S; > [1] 100 > > ## no problem here with small matrices, let's increase the size > >> M <- 3000; >> N <- 100; >> X <- matrix(1,M,N); >> .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S; > [1] 3e+05 ## OK > > > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S; > ## *** R GUI CRASHES WITH A STACK OVERFLOW ERROR *** > **************************************************************************** > Any suggestion would be greatly appreciated, I apologize if this > problem has already been addressed previously, I did not notice it. > Many thanks in advance, > > Simone > ______________________________________________________ > > Simone Giannerini > Dipartimento di Scienze Statistiche "Paolo Fortunati" > Universita' di Bologna > Via delle belle arti 41 - 40126 Bologna, ITALY > Tel: +39 051 2098248 Fax: +39 051 232153 > E-mail: giannerini at stat.unibo.it > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel
Simon Urbanek
2005-Sep-09 18:00 UTC
[Rd] A question on R memory management in .Fortran() calls under Windows
Simone, On Sep 9, 2005, at 1:04 PM, Simone Giannerini wrote:> Dear R community, > > I have a question on how R manages memory allocation in .Fortran() > calls under Windows. > In brief, apparently, it is not possible to allocate large matrices > inside a Fortran subroutineI suspect that this is a problem of your compiler, not R, because it works without problems for me: > dyn.load("foo.so") > M=10 > N=10 > X=matrix(1,M,N) > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S [1] 100 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S [1] 100 > M=3000 > N=100 > X=matrix(1,M,N) > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S [1] 3e+05 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S [1] 3e+05 > M=10000 > N=10000 > X=matrix(1,M,N) > .Fortran("foo",X,as.integer(M),as.integer(N),S=as.double(0))$S [1] 1e+08 > .Fortran("foobis",as.integer(M),as.integer(N),S=as.double(0))$S [1] 1e+08 Tested on PC1: Win XP SP2, AMD64 3000+, 1GB RAM, gfortran 4.1.0 (20050902) PC2: OS X, G5 1.8, 1GB RAM, gfortran 4.0.1 Cheers, Simon