Dear all, I'm trying to create a virtual S4 class with some subclasses. I noticed that adding slots to this class increases the memory use and slows the functions down. Note that I'm adding very small slots (integer or character both of length 1). I've made a reproducible example at https://github.com/ThierryO/testvirtualclass. The R CMD check --as-cran fails on the tests. Some of the output of R CMD check * using R version 3.2.0 (2015-04-16) * using platform: i386-w64-mingw32 (32-bit) * using session charset: ISO8859-1 * using option '--as-cran' * checking tests ... Running 'testthat.R' [125s] ERROR Running the tests in 'tests/testthat.R' failed. Last 13 lines of output: Execution halted Error: C stack usage 16583636 is too close to the limit at that point R crashes and uses about 3.7 GB RAM Any ideas on what is going wrong? Am I using the virual classes in the wrong way? Best regards, ir. Thierry Onkelinx Instituut voor natuur- en bosonderzoek / Research Institute for Nature and Forest team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance Kliniekstraat 25 1070 Anderlecht Belgium To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of. ~ Sir Ronald Aylmer Fisher The plural of anecdote is not data. ~ Roger Brinner The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data. ~ John Tukey [[alternative HTML version deleted]]
Dear all,
I did some more work in this.
- put all the metadata in a dedicated subclass (see metadataclass branch in
github repository). This doesn't solve the problem.
- test the code under R 3.1.2. This gives the same problem. So this is NOT
a bug introduced by R 3.2.0 :-)
- Profiling the code. Below the summary (code is on GitHub). It seems like
most of the time is spent by paste(). Note that my code doesn't use paste()
so it used in the internals of R. Could someone with more understanding of
S4 enlight me on what is happening. And on what I am doing wrong?
Best regards,
Thierry
$by.self
self.time self.pct total.time total.pct
"paste" 166.06 90.98 170.54 93.44
".findInheritedMethods" 9.30 5.10 182.38 99.92
"rep.int" 2.24 1.23 4.48 2.45
":" 2.24 1.23 2.24 1.23
"unique.default" 1.42 0.78 1.42 0.78
"match" 0.58 0.32 0.58 0.32
"[[" 0.54 0.30 0.54 0.30
".getClassFromCache" 0.08 0.04 0.08 0.04
"loadMethod" 0.06 0.03 0.06 0.03
$by.total
total.time total.pct self.time self.pct
"eval" 182.52 100.00 0.00 0.00
"my_lmer" 182.52 100.00 0.00 0.00
"source" 182.52 100.00 0.00 0.00
"standardGeneric" 182.52 100.00 0.00 0.00
"withVisible" 182.52 100.00 0.00 0.00
".findInheritedMethods" 182.38 99.92 9.30 5.10
"<Anonymous>" 182.38 99.92 0.00
0.00
"paste" 170.54 93.44 166.06 90.98
"outerLabels" 170.54 93.44 0.00 0.00
"rep.int" 4.48 2.45 2.24 1.23
":" 2.24 1.23 2.24 1.23
"unique.default" 1.42 0.78 1.42 0.78
"unique" 1.42 0.78 0.00 0.00
"match" 0.58 0.32 0.58 0.32
"[[" 0.54 0.30 0.54 0.30
".inheritedArgsExpression" 0.54 0.30 0.00 0.00
"extends" 0.54 0.30 0.00 0.00
".getClassFromCache" 0.08 0.04 0.08 0.04
"getClassDef" 0.08 0.04 0.00 0.00
"initialize" 0.08 0.04 0.00 0.00
"new" 0.08 0.04 0.00 0.00
"loadMethod" 0.06 0.03 0.06 0.03
$sample.interval
[1] 0.02
$sampling.time
[1] 182.52
>
ir. Thierry Onkelinx
Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
Forest
team Biometrie & Kwaliteitszorg / team Biometrics & Quality Assurance
Kliniekstraat 25
1070 Anderlecht
Belgium
To call in the statistician after the experiment is done may be no more
than asking him to perform a post-mortem examination: he may be able to say
what the experiment died of. ~ Sir Ronald Aylmer Fisher
The plural of anecdote is not data. ~ Roger Brinner
The combination of some data and an aching desire for an answer does not
ensure that a reasonable answer can be extracted from a given body of data.
~ John Tukey
2015-05-04 17:10 GMT+02:00 Thierry Onkelinx <thierry.onkelinx at inbo.be>:
> Dear all,
>
> I'm trying to create a virtual S4 class with some subclasses. I noticed
> that adding slots to this class increases the memory use and slows the
> functions down. Note that I'm adding very small slots (integer or
character
> both of length 1).
>
> I've made a reproducible example at
> https://github.com/ThierryO/testvirtualclass. The R CMD check --as-cran
> fails on the tests.
>
> Some of the output of R CMD check
>
> * using R version 3.2.0 (2015-04-16)
> * using platform: i386-w64-mingw32 (32-bit)
> * using session charset: ISO8859-1
> * using option '--as-cran'
> * checking tests ...
> Running 'testthat.R' [125s]
> ERROR
> Running the tests in 'tests/testthat.R' failed.
> Last 13 lines of output:
> Execution halted
> Error: C stack usage 16583636 is too close to the limit
>
> at that point R crashes and uses about 3.7 GB RAM
>
> Any ideas on what is going wrong? Am I using the virual classes in the
> wrong way?
>
> Best regards,
>
> ir. Thierry Onkelinx
> Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
> Forest
> team Biometrie & Kwaliteitszorg / team Biometrics & Quality
Assurance
> Kliniekstraat 25
> 1070 Anderlecht
> Belgium
>
> To call in the statistician after the experiment is done may be no more
> than asking him to perform a post-mortem examination: he may be able to say
> what the experiment died of. ~ Sir Ronald Aylmer Fisher
> The plural of anecdote is not data. ~ Roger Brinner
> The combination of some data and an aching desire for an answer does not
> ensure that a reasonable answer can be extracted from a given body of data.
> ~ John Tukey
>
[[alternative HTML version deleted]]
There are too many arguments in the signature of the my_inla generic. The
signature defaults to every argument in the formals, which are probably
already over-specified. Typically, one defines a generic with the formals
(x, ...), or perhaps in this case (x, model, ...), but more on that below
Unrelated to this issue, the my_inla generic is defined incorrectly. It
should call standardGeneric("my_inla"), not standard.generic(my_inla).
When
you pass a generic definition that does not contain a call to
standardGeneric(), it is taken as the default (ANY) method, so the generic
still works in this case, but the ANY method is broken.
The ANY method that is separately defined should probably be defined on
"data.frame,missing", since otherwise it will break. If you really
want an
ANY method, consider coercing the input to data.frame, but you can do that
without a generic.
Instead of having the my_inla and my_inla_model classes, you could have
model.fit be an optional slot in my_inla. Then my_inla() does not have to
be a generic anymore. To make an optional slot, use a class union of the
slot class with NULL.
Also, I think it's sort of the convention that class names be upper camel
case.
The reason it calls paste() so many times is that the methods package
searches for every possible combination of classes (and their parent
classes), which in this case, is many, since the generic signature is so
long, and the Matrix package adds several classes above "integer".
Obviously, this is not very efficient, however, thanks to caching it only
happens once for each target signature, so in practice (and with more
reasonable generic signatures), this is not an issue.
Hope this helps and sorry for the lack of S4 documentation.
Michael
On Tue, May 5, 2015 at 1:04 AM, Thierry Onkelinx <thierry.onkelinx at
inbo.be>
wrote:
> Dear all,
>
> I did some more work in this.
>
> - put all the metadata in a dedicated subclass (see metadataclass branch in
> github repository). This doesn't solve the problem.
> - test the code under R 3.1.2. This gives the same problem. So this is NOT
> a bug introduced by R 3.2.0 :-)
> - Profiling the code. Below the summary (code is on GitHub). It seems like
> most of the time is spent by paste(). Note that my code doesn't use
paste()
> so it used in the internals of R. Could someone with more understanding of
> S4 enlight me on what is happening. And on what I am doing wrong?
>
> Best regards,
>
> Thierry
>
> $by.self
> self.time self.pct total.time total.pct
> "paste" 166.06 90.98 170.54 93.44
> ".findInheritedMethods" 9.30 5.10 182.38 99.92
> "rep.int" 2.24 1.23 4.48 2.45
> ":" 2.24 1.23 2.24 1.23
> "unique.default" 1.42 0.78 1.42 0.78
> "match" 0.58 0.32 0.58 0.32
> "[[" 0.54 0.30 0.54 0.30
> ".getClassFromCache" 0.08 0.04 0.08 0.04
> "loadMethod" 0.06 0.03 0.06 0.03
>
> $by.total
> total.time total.pct self.time self.pct
> "eval" 182.52 100.00 0.00
0.00
> "my_lmer" 182.52 100.00 0.00
0.00
> "source" 182.52 100.00 0.00
0.00
> "standardGeneric" 182.52 100.00 0.00
0.00
> "withVisible" 182.52 100.00 0.00
0.00
> ".findInheritedMethods" 182.38 99.92 9.30
5.10
> "<Anonymous>" 182.38 99.92 0.00
0.00
> "paste" 170.54 93.44 166.06
90.98
> "outerLabels" 170.54 93.44 0.00
0.00
> "rep.int" 4.48 2.45 2.24
1.23
> ":" 2.24 1.23 2.24
1.23
> "unique.default" 1.42 0.78 1.42
0.78
> "unique" 1.42 0.78 0.00
0.00
> "match" 0.58 0.32 0.58
0.32
> "[[" 0.54 0.30 0.54
0.30
> ".inheritedArgsExpression" 0.54 0.30 0.00
0.00
> "extends" 0.54 0.30 0.00
0.00
> ".getClassFromCache" 0.08 0.04 0.08
0.04
> "getClassDef" 0.08 0.04 0.00
0.00
> "initialize" 0.08 0.04 0.00
0.00
> "new" 0.08 0.04 0.00
0.00
> "loadMethod" 0.06 0.03 0.06
0.03
>
> $sample.interval
> [1] 0.02
>
> $sampling.time
> [1] 182.52
>
> >
>
> ir. Thierry Onkelinx
> Instituut voor natuur- en bosonderzoek / Research Institute for Nature and
> Forest
> team Biometrie & Kwaliteitszorg / team Biometrics & Quality
Assurance
> Kliniekstraat 25
> 1070 Anderlecht
> Belgium
>
> To call in the statistician after the experiment is done may be no more
> than asking him to perform a post-mortem examination: he may be able to say
> what the experiment died of. ~ Sir Ronald Aylmer Fisher
> The plural of anecdote is not data. ~ Roger Brinner
> The combination of some data and an aching desire for an answer does not
> ensure that a reasonable answer can be extracted from a given body of data.
> ~ John Tukey
>
> 2015-05-04 17:10 GMT+02:00 Thierry Onkelinx <thierry.onkelinx at
inbo.be>:
>
> > Dear all,
> >
> > I'm trying to create a virtual S4 class with some subclasses. I
noticed
> > that adding slots to this class increases the memory use and slows the
> > functions down. Note that I'm adding very small slots (integer or
> character
> > both of length 1).
> >
> > I've made a reproducible example at
> > https://github.com/ThierryO/testvirtualclass. The R CMD check
--as-cran
> > fails on the tests.
> >
> > Some of the output of R CMD check
> >
> > * using R version 3.2.0 (2015-04-16)
> > * using platform: i386-w64-mingw32 (32-bit)
> > * using session charset: ISO8859-1
> > * using option '--as-cran'
> > * checking tests ...
> > Running 'testthat.R' [125s]
> > ERROR
> > Running the tests in 'tests/testthat.R' failed.
> > Last 13 lines of output:
> > Execution halted
> > Error: C stack usage 16583636 is too close to the limit
> >
> > at that point R crashes and uses about 3.7 GB RAM
> >
> > Any ideas on what is going wrong? Am I using the virual classes in the
> > wrong way?
> >
> > Best regards,
> >
> > ir. Thierry Onkelinx
> > Instituut voor natuur- en bosonderzoek / Research Institute for Nature
> and
> > Forest
> > team Biometrie & Kwaliteitszorg / team Biometrics & Quality
Assurance
> > Kliniekstraat 25
> > 1070 Anderlecht
> > Belgium
> >
> > To call in the statistician after the experiment is done may be no
more
> > than asking him to perform a post-mortem examination: he may be able
to
> say
> > what the experiment died of. ~ Sir Ronald Aylmer Fisher
> > The plural of anecdote is not data. ~ Roger Brinner
> > The combination of some data and an aching desire for an answer does
not
> > ensure that a reasonable answer can be extracted from a given body of
> data.
> > ~ John Tukey
> >
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
[[alternative HTML version deleted]]