Frank van Berkum
2019-Mar-15 09:09 UTC
[R] [mgcv] Memory issues with bam() on computer cluster
Dear Community, In our current research we are trying to fit Generalized Additive Models to a large dataset. We are using the package mgcv in R. Our dataset contains about 22 million records with less than 20 risk factors for each observation, so in our case n>>p. The dataset covers the period 2006 until 2011, and we analyse both the complete dataset and datasets in which we leave out a single year. The latter part is done to analyse robustness of the results. We understand k-fold cross validation may seem more appropriate, but out approach is closer to what is done in practice (how will one additional year of information affect your estimates?). We use the function bam as advocated in Wood et al. (2017), and we apply the following options: bam(?, discrete=TRUE, chunk.size=10000, gc.level=1). We run these analyses on a computer cluster (see https://userinfo.surfsara.nl/systems/lisa/description for details), and the job is allocated to a node within the computer cluster. A node has at least 16 cores and 64Gb memory. We had expected 64Gb of memory to be sufficient for these analyses, especially since the bam function is built specifically for large datasets. However, when applying this function to the different datasets described above with different regression specifications (different risk factors included in the linear predictor), we sometimes obtain errors of the following form. Error in XWyd(G$Xd, w, z, G$kd, G$ks, G$ts, G$dt, G$v, G$qc, G$drop, ar.stop, : 'Calloc' could not allocate memory (22624897 of 8 bytes) Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> XWyd Execution halted Warning message: system call failed: Cannot allocate memory Error in Xbd(G$Xd, coef, G$kd, G$ks, G$ts, G$dt, G$v, G$qc, G$drop) : 'Calloc' could not allocate memory (18590685 of 8 bytes) Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> Xbd Execution halted Warning message: system call failed: Cannot allocate memory Error: cannot allocate vector of size 1.7 Gb Timing stopped at: 2 0.556 4.831 Error in system.time(oo <- .C(C_XWXd0, XWX = as.double(rep(0, (pt + nt)^2)), : 'Calloc' could not allocate memory (55315650 of 24 bytes) Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> XWXd -> system.time -> .C Timing stopped at: 1.056 1.396 2.459 Execution halted Warning message: system call failed: Cannot allocate memory The errors seem to arise at different stages in the optimization process. We have analysed whether these errors disappear if different settings are used (different chunk.size, different gc.level), but this does not resolve our problem. Also, the errors occur on different datasets when using different settings, and even when using the same settings it is possible that an error that occurred on dataset X in one run it does not necessarily occur on dataset X in a different run. When using the discrete=TRUE option, optimization can be parallelized, but we have chosen to not employ this feature to ensure memory does not have to be shared between parallel processes. Naturally I cannot share our dataset with you which makes the problem difficult to analyse. However, based on your collective knowledge, could you pinpoint us to where the problem may occur? Is it something within the C-code used within the package (as the last error seems to indicate), or is it related to the computer cluster? Any help or insights is much appreciated. Kind regards, Frank [[alternative HTML version deleted]]
Can you supply the results of sessionInfo() please, and the full bam call that causes this. best, Simon (mgcv maintainer) On 15/03/2019 09:09, Frank van Berkum wrote:> Dear Community, > > In our current research we are trying to fit Generalized Additive Models to a large dataset. We are using the package mgcv in R. > > Our dataset contains about 22 million records with less than 20 risk factors for each observation, so in our case n>>p. The dataset covers the period 2006 until 2011, and we analyse both the complete dataset and datasets in which we leave out a single year. The latter part is done to analyse robustness of the results. We understand k-fold cross validation may seem more appropriate, but out approach is closer to what is done in practice (how will one additional year of information affect your estimates?). > > We use the function bam as advocated in Wood et al. (2017), and we apply the following options: bam(?, discrete=TRUE, chunk.size=10000, gc.level=1). We run these analyses on a computer cluster (see https://userinfo.surfsara.nl/systems/lisa/description for details), and the job is allocated to a node within the computer cluster. A node has at least 16 cores and 64Gb memory. > > We had expected 64Gb of memory to be sufficient for these analyses, especially since the bam function is built specifically for large datasets. However, when applying this function to the different datasets described above with different regression specifications (different risk factors included in the linear predictor), we sometimes obtain errors of the following form. > > Error in XWyd(G$Xd, w, z, G$kd, G$ks, G$ts, G$dt, G$v, G$qc, G$drop, ar.stop, : > > 'Calloc' could not allocate memory (22624897 of 8 bytes) > > Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> XWyd > > Execution halted > > Warning message: > > system call failed: Cannot allocate memory > > Error in Xbd(G$Xd, coef, G$kd, G$ks, G$ts, G$dt, G$v, G$qc, G$drop) : > > 'Calloc' could not allocate memory (18590685 of 8 bytes) > > Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> Xbd > > Execution halted > > Warning message: > > system call failed: Cannot allocate memory > > Error: cannot allocate vector of size 1.7 Gb > > Timing stopped at: 2 0.556 4.831 > > Error in system.time(oo <- .C(C_XWXd0, XWX = as.double(rep(0, (pt + nt)^2)), : > > 'Calloc' could not allocate memory (55315650 of 24 bytes) > > Calls: fnEstimateModel_bam -> bam -> bgam.fitd -> XWXd -> system.time -> .C > > Timing stopped at: 1.056 1.396 2.459 > > Execution halted > > Warning message: > > system call failed: Cannot allocate memory > > The errors seem to arise at different stages in the optimization process. We have analysed whether these errors disappear if different settings are used (different chunk.size, different gc.level), but this does not resolve our problem. Also, the errors occur on different datasets when using different settings, and even when using the same settings it is possible that an error that occurred on dataset X in one run it does not necessarily occur on dataset X in a different run. When using the discrete=TRUE option, optimization can be parallelized, but we have chosen to not employ this feature to ensure memory does not have to be shared between parallel processes. > > Naturally I cannot share our dataset with you which makes the problem difficult to analyse. However, based on your collective knowledge, could you pinpoint us to where the problem may occur? Is it something within the C-code used within the package (as the last error seems to indicate), or is it related to the computer cluster? > > Any help or insights is much appreciated. > > Kind regards, > > Frank > > [[alternative HTML version deleted]] > > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Simon Wood, School of Mathematics, University of Bristol, BS8 1TW UK https://people.maths.bris.ac.uk/~sw15190/ [[alternative HTML version deleted]]