I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu
14.04.4 LTS*). I am running *var model *on a matrix with 199 columns and
604800 rows. The server has 12 core and 32GB of memory. When the model is
running, i checked CPU and Memory consumption using 'htop' linux
command. I
observe all the cores are being used and memory memory usage is on average
17GB out of 32GB. After running the model for about an hour, the system
kills the R process. Following the output:
*> library("vars")*
*> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")*
*Killed*
Please help.
Regards,
Vivek Kumar Singh
PhD student,
Information Systems Decision Sciences,
MUMA College of Business,
USF
Phone- (813) 5809131
Web: http://vivek4.myweb.usf.edu/
On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges <ligges at
statistik.tu-dortmund.de> wrote:
> Wild guess: You have huge and high dimensional VAR models, i.e. the
> matrices get huge and you use huge amounts of memory and you use more than
> what is available physically. The operating system protects itself by
> killing processes in such a case...
>
> Best,
> Uwe Ligges
>
>
>
> On 31.05.2016 20:29, Vivek Singh wrote:
>
>> Hi,
>>
>> I am using VARS (vector autoregressive model). The process gets killed
>> after running for sometime. Following is the output of R.
>>
>> vivek at isds-research:~/cloudAuction/padding/panel$ cat var.Rout
>>
>> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing"
>> Copyright (C) 2013 The R Foundation for Statistical Computing
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> R is free software and comes with ABSOLUTELY NO WARRANTY.
>> You are welcome to redistribute it under certain conditions.
>> Type 'license()' or 'licence()' for distribution
details.
>>
>> Natural language support but running in an English locale
>>
>> R is a collaborative project with many contributors.
>> Type 'contributors()' for more information and
>> 'citation()' on how to cite R or R packages in publications.
>>
>> Type 'demo()' for some demos, 'help()' for on-line
help, or
>> 'help.start()' for an HTML browser interface to help.
>> Type 'q()' to quit R.
>>
>> [Previously saved workspace restored]
>>
>> data=read.csv("output1.csv")
>>> attach(data)
>>> only_variables= subset(data, select=c(-date,-hour,-minute,-sec))
>>>
>>> library("vars")
>>>
>> Loading required package: MASS
>> Loading required package: strucchange
>> Loading required package: zoo
>>
>> Attaching package: ?zoo?
>>
>> The following objects are masked from ?package:base?:
>>
>> as.Date, as.Date.numeric
>>
>> Loading required package: sandwich
>> Loading required package: urca
>> Loading required package: lmtest
>>
>>> summary(VAR(only_variables, p = 1, type ="both"))
>>>
>> *Killed*
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>>
[[alternative HTML version deleted]]
I checked the issue on different forums like stackoverflow. The issue is related to Out Of Memory (OOM) linux feature which kills processes that consume large memory and swap. I started to monitor the memory and swap consumption while VAR model was running. The R consumed all 32 GB of RAM memory and then 20 GB of swap. I have only 32GB of RAM and 20GB of swap in my server. Then the R was killed. So, i have two queries: 1. Is there a way to limit the amount of memory used by R so that it runs for longer time? 2. Is it some sort of inefficiency of garbage collector in R? Regards, Vivek Kumar Singh PhD student, Information Systems Decision Sciences, MUMA College of Business, USF Phone- (813) 5809131 Web: http://vivek4.myweb.usf.edu/ On Wed, Jun 8, 2016 at 10:13 AM, Vivek Singh <vivek4 at mail.usf.edu> wrote:> I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu > 14.04.4 LTS*). I am running *var model *on a matrix with 199 columns and > 604800 rows. The server has 12 core and 32GB of memory. When the model is > running, i checked CPU and Memory consumption using 'htop' linux command. I > observe all the cores are being used and memory memory usage is on average > 17GB out of 32GB. After running the model for about an hour, the system > kills the R process. Following the output: > > *> library("vars")* > *> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")* > *Killed* > > Please help. > > > > Regards, > > Vivek Kumar Singh > > PhD student, > Information Systems Decision Sciences, > MUMA College of Business, > USF > Phone- (813) 5809131 > Web: http://vivek4.myweb.usf.edu/ > > On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges < > ligges at statistik.tu-dortmund.de> wrote: > >> Wild guess: You have huge and high dimensional VAR models, i.e. the >> matrices get huge and you use huge amounts of memory and you use more than >> what is available physically. The operating system protects itself by >> killing processes in such a case... >> >> Best, >> Uwe Ligges >> >> >> >> On 31.05.2016 20:29, Vivek Singh wrote: >> >>> Hi, >>> >>> I am using VARS (vector autoregressive model). The process gets killed >>> after running for sometime. Following is the output of R. >>> >>> vivek at isds-research:~/cloudAuction/padding/panel$ cat var.Rout >>> >>> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing" >>> Copyright (C) 2013 The R Foundation for Statistical Computing >>> Platform: x86_64-pc-linux-gnu (64-bit) >>> >>> R is free software and comes with ABSOLUTELY NO WARRANTY. >>> You are welcome to redistribute it under certain conditions. >>> Type 'license()' or 'licence()' for distribution details. >>> >>> Natural language support but running in an English locale >>> >>> R is a collaborative project with many contributors. >>> Type 'contributors()' for more information and >>> 'citation()' on how to cite R or R packages in publications. >>> >>> Type 'demo()' for some demos, 'help()' for on-line help, or >>> 'help.start()' for an HTML browser interface to help. >>> Type 'q()' to quit R. >>> >>> [Previously saved workspace restored] >>> >>> data=read.csv("output1.csv") >>>> attach(data) >>>> only_variables= subset(data, select=c(-date,-hour,-minute,-sec)) >>>> >>>> library("vars") >>>> >>> Loading required package: MASS >>> Loading required package: strucchange >>> Loading required package: zoo >>> >>> Attaching package: ?zoo? >>> >>> The following objects are masked from ?package:base?: >>> >>> as.Date, as.Date.numeric >>> >>> Loading required package: sandwich >>> Loading required package: urca >>> Loading required package: lmtest >>> >>>> summary(VAR(only_variables, p = 1, type ="both")) >>>> >>> *Killed* >>> >>> [[alternative HTML version deleted]] >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide >>> http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >>> >[[alternative HTML version deleted]]
1. Don't allocate it. 2. If it was, would it make a difference? Seriously, some algorithms need more memory than others, and some packages are more wasteful than others. R is not monolithic... sometimes you just have to roll up your sleeves or buy more memory. -- Sent from my phone. Please excuse my brevity. On June 8, 2016 7:51:00 AM PDT, Vivek Singh <vivek4 at mail.usf.edu> wrote:>I checked the issue on different forums like stackoverflow. The issue >is >related to Out Of Memory (OOM) linux feature which kills processes that >consume large memory and swap. I started to monitor the memory and swap >consumption while VAR model was running. The R consumed all 32 GB of >RAM >memory and then 20 GB of swap. I have only 32GB of RAM and 20GB of swap >in >my server. Then the R was killed. > >So, i have two queries: > >1. Is there a way to limit the amount of memory used by R so that it >runs >for longer time? > >2. Is it some sort of inefficiency of garbage collector in R? > > > >Regards, > >Vivek Kumar Singh > >PhD student, >Information Systems Decision Sciences, >MUMA College of Business, >USF >Phone- (813) 5809131 >Web: http://vivek4.myweb.usf.edu/ > >On Wed, Jun 8, 2016 at 10:13 AM, Vivek Singh <vivek4 at mail.usf.edu> >wrote: > >> I am using *R version 3.0.2* (2013-09-25) on Ubuntu desktop (*Ubuntu >> 14.04.4 LTS*). I am running *var model *on a matrix with 199 columns >and >> 604800 rows. The server has 12 core and 32GB of memory. When the >model is >> running, i checked CPU and Memory consumption using 'htop' linux >command. I >> observe all the cores are being used and memory memory usage is on >average >> 17GB out of 32GB. After running the model for about an hour, the >system >> kills the R process. Following the output: >> >> *> library("vars")* >> *> vmodel=VAR(only_variables_without_missing, p = 1, type ="both")* >> *Killed* >> >> Please help. >> >> >> >> Regards, >> >> Vivek Kumar Singh >> >> PhD student, >> Information Systems Decision Sciences, >> MUMA College of Business, >> USF >> Phone- (813) 5809131 >> Web: http://vivek4.myweb.usf.edu/ >> >> On Tue, May 31, 2016 at 4:41 PM, Uwe Ligges < >> ligges at statistik.tu-dortmund.de> wrote: >> >>> Wild guess: You have huge and high dimensional VAR models, i.e. the >>> matrices get huge and you use huge amounts of memory and you use >more than >>> what is available physically. The operating system protects itself >by >>> killing processes in such a case... >>> >>> Best, >>> Uwe Ligges >>> >>> >>> >>> On 31.05.2016 20:29, Vivek Singh wrote: >>> >>>> Hi, >>>> >>>> I am using VARS (vector autoregressive model). The process gets >killed >>>> after running for sometime. Following is the output of R. >>>> >>>> vivek at isds-research:~/cloudAuction/padding/panel$ cat var.Rout >>>> >>>> R version 3.0.2 (2013-09-25) -- "Frisbee Sailing" >>>> Copyright (C) 2013 The R Foundation for Statistical Computing >>>> Platform: x86_64-pc-linux-gnu (64-bit) >>>> >>>> R is free software and comes with ABSOLUTELY NO WARRANTY. >>>> You are welcome to redistribute it under certain conditions. >>>> Type 'license()' or 'licence()' for distribution details. >>>> >>>> Natural language support but running in an English locale >>>> >>>> R is a collaborative project with many contributors. >>>> Type 'contributors()' for more information and >>>> 'citation()' on how to cite R or R packages in publications. >>>> >>>> Type 'demo()' for some demos, 'help()' for on-line help, or >>>> 'help.start()' for an HTML browser interface to help. >>>> Type 'q()' to quit R. >>>> >>>> [Previously saved workspace restored] >>>> >>>> data=read.csv("output1.csv") >>>>> attach(data) >>>>> only_variables= subset(data, select=c(-date,-hour,-minute,-sec)) >>>>> >>>>> library("vars") >>>>> >>>> Loading required package: MASS >>>> Loading required package: strucchange >>>> Loading required package: zoo >>>> >>>> Attaching package: ?zoo? >>>> >>>> The following objects are masked from ?package:base?: >>>> >>>> as.Date, as.Date.numeric >>>> >>>> Loading required package: sandwich >>>> Loading required package: urca >>>> Loading required package: lmtest >>>> >>>>> summary(VAR(only_variables, p = 1, type ="both")) >>>>> >>>> *Killed* >>>> >>>> [[alternative HTML version deleted]] >>>> >>>> ______________________________________________ >>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>>> https://stat.ethz.ch/mailman/listinfo/r-help >>>> PLEASE do read the posting guide >>>> http://www.R-project.org/posting-guide.html >>>> and provide commented, minimal, self-contained, reproducible code. >>>> >>>> >> > > [[alternative HTML version deleted]] > >______________________________________________ >R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code.[[alternative HTML version deleted]]