Sander Maijers
2012-Jun-05 13:03 UTC
[Rd] Memory leak during instantiation of a reference class?
Hi,
I have a problem with a large data set wrapped in a reference class. I do that
to access the data by reference from within various functions in a script.
# The class definition:
setRefClass("data",
fields = list(h5_df = "list",
training_data = "array",
true_class = "vector"));
# The class instantiation:
data_1 <- new("data",
h5_df = as.list(h5_df),
training_data = training_data,
true_class = true_class);
The objects assigned to the "data" fields during instantiation:
Browse[1]> str(training_data)
num [1:501, 1:62, 1:579] -10.4 -11.3 -12.1 -12.8 -13.3 ...
- attr(*, "dimnames")=List of 3
..$ : NULL
..$ : NULL
..$ : NULL
Browse[1]> str(true_class)
num [1:1002] 1 1 1 1 1 1 1 1 1 1 ...
Browse[1]> str(h5_df)
Dotted pair list of 11
$ avg.txt.lp :Garbage collection 135 = 44+18+73 (level 2) ...
49.4 Mbytes of cons cells used (63%)
2840.0 Mbytes of vectors used (30%)
num [1:501, 1:62, 1:607] -10.4 -11.3 -12.1 -12.8 -13.3 ...
$ avg.tls.txt.lp: num [1:501, 1:62, 1:281] -10.4 -11.3 -12.1 -12.8 -13.3 ...
$ avg.tls.pic.lp: num [1:501, 1:62, 1:305] -2.81 -4.95 -6.98 -8.79 -10.32 ...
$ avg.tls.lp : num [1:501, 1:62, 1:884] -10.4 -11.3 -12.1 -12.8 -13.3 ...
$ avg.tls.aud.lp: num [1:501, 1:62, 1:298] -13.9 -11.36 -8.91 -6.61 -4.55 ...
$ avg.pic.lp : num [1:501, 1:62, 1:653] -2.81 -4.95 -6.98 -8.79 -10.32 ...
$ avg.aud.lp : num [1:501, 1:62, 1:662] -13.9 -11.36 -8.91 -6.61 -4.55 ...
$ avg.anm.txt.lp: num [1:501, 1:62, 1:273] -3.37 -2.96 -2.59 -2.27 -2 ...
$ avg.anm.pic.lp: num [1:501, 1:62, 1:285] 11.7 12.3 12.8 13.2 13.5 ...
$ avg.anm.lp : num [1:501, 1:62, 1:855] -3.37 -2.96 -2.59 -2.27 -2 ...
$ avg.anm.aud.lp:Garbage collection 136 = 44+18+74 (level 2) ...
49.4 Mbytes of cons cells used (63%)
2980.8 Mbytes of vectors used (32%)
num [1:501, 1:62, 1:297] -13.57 -11.96 -10.36 -8.77 -7.2 ...
The procedure that gives rise to the problem:
> rm(list=ls())
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 921231 49.2 1476915 78.9 1166886 62.4
Vcells 792872 6.1 565291490 4312.9 690075184 5264.9> load_data()
Processing object: avg_anm_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332028
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 297 / 501 62 297
501 62 297 Tidying name avg_anm_aud_lp .. to avg.anm.aud.lp
...Finished dataset
Processing object: avg_anm_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332031
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 855 / 501 62 855
501 62 855 Tidying name avg_anm_lp .. to avg.anm.lp
...Finished dataset
Processing object: avg_anm_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332034
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 285 / 501 62 285
501 62 285 Tidying name avg_anm_pic_lp .. to avg.anm.pic.lp
...Finished dataset
Processing object: avg_anm_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332037
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 273 / 501 62 273
501 62 273 Tidying name avg_anm_txt_lp .. to avg.anm.txt.lp
...Finished dataset
Processing object: avg_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332040
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 662 / 501 62 662
501 62 662 Tidying name avg_aud_lp .. to avg.aud.lp
...Finished dataset
Processing object: avg_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332043
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 653 / 501 62 653
501 62 653 Tidying name avg_pic_lp .. to avg.pic.lp
...Finished dataset
Processing object: avg_tls_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332046
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 298 / 501 62 298
501 62 298 Tidying name avg_tls_aud_lp .. to avg.tls.aud.lp
...Finished dataset
Processing object: avg_tls_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332049
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 884 / 501 62 884
501 62 884 Tidying name avg_tls_lp .. to avg.tls.lp
...Finished dataset
Processing object: avg_tls_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332052
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 305 / 501 62 305
501 62 305 Tidying name avg_tls_pic_lp .. to avg.tls.pic.lp
...Finished dataset
Processing object: avg_tls_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332055
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 281 / 501 62 281
501 62 281 Tidying name avg_tls_txt_lp .. to avg.tls.txt.lp
...Finished dataset
Processing object: avg_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332058
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501 62 607 / 501 62 607
501 62 607 Tidying name avg_txt_lp .. to avg.txt.lp
...Finished dataset
Garbage collection 118 = 44+17+57 (level 0) ...
49.4 Mbytes of cons cells used (63%)
2277.9 Mbytes of vectors used (53%)
Called from: load_data()
Browse[1]> str(data_1)
Garbage collection 119 = 44+18+57 (level 1) ...
49.4 Mbytes of cons cells used (63%)
4119.7 Mbytes of vectors used (96%)
Reference class 'data' [package ".GlobalEnv"] with 3 fields
Garbage collection 120 = 44+18+58 (level 2) ...
49.3 Mbytes of cons cells used (62%)
4256.9 Mbytes of vectors used (92%)
Garbage collection 121 = 44+18+59 (level 2) ...
49.3 Mbytes of cons cells used (62%)
4539.6 Mbytes of vectors used (91%)
Garbage collection 122 = 44+18+60 (level 2) ...
49.3 Mbytes of cons cells used (62%)
4974.5 Mbytes of vectors used (92%)
Garbage collection 123 = 44+18+61 (level 2) ...
49.3 Mbytes of cons cells used (62%)
5263.6 Mbytes of vectors used (92%)
Garbage collection 124 = 44+18+62 (level 2) ...
49.3 Mbytes of cons cells used (62%)
5673.9 Mbytes of vectors used (93%)
Garbage collection 125 = 44+18+63 (level 2) ...
49.3 Mbytes of cons cells used (62%)
5956.6 Mbytes of vectors used (92%)
Garbage collection 126 = 44+18+64 (level 2) ...
49.3 Mbytes of cons cells used (62%)
6391.4 Mbytes of vectors used (93%)
Garbage collection 127 = 44+18+65 (level 2) ...
49.3 Mbytes of cons cells used (62%)
6680.6 Mbytes of vectors used (92%)
Garbage collection 128 = 44+18+66 (level 2) ...
49.3 Mbytes of cons cells used (62%)
5673.9 Mbytes of vectors used (75%)
Garbage collection 129 = 44+18+67 (level 2) ...
49.3 Mbytes of cons cells used (62%)
7583.0 Mbytes of vectors used (94%)
Garbage collection 130 = 44+18+68 (level 2) ...
49.3 Mbytes of cons cells used (62%)
8030.0 Mbytes of vectors used (94%)
Garbage collection 131 = 44+18+69 (level 2) ...
49.3 Mbytes of cons cells used (62%)
8370.5 Mbytes of vectors used (94%)
Garbage collection 132 = 44+18+70 (level 2) ...
49.3 Mbytes of cons cells used (62%)
7373.5 Mbytes of vectors used (79%)
$ h5_df :Garbage collection 133 = 44+18+71 (level 2) ...
49.3 Mbytes of cons cells used (62%)
9070.6 Mbytes of vectors used (92%)
Error: cannot allocate vector of size 154.8 Mb
Garbage collection 134 = 44+18+72 (level 2) ...
49.3 Mbytes of cons cells used (62%)
9070.6 Mbytes of vectors used (92%)
Error during wrapup: cannot allocate vector of size 143.8 Mb
The computer system that runs this code is has a 64-bit CPU and 12 GiB RAM.
It seems that there is a memory leak somewhere. Any suggestions on how to solve
this problem?
Regards,
Sander Maijers
Possibly Parallel Threads
Wisdom of the Ancients
