Dear R Forum, I have about 2000+ FX forward transactions and I am trying to run 1000 simulations. If I use less no of simulations, I am able to get the desired results. However, when I try to use more than 1000 simulations, I get following error.> sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate, change_in_mtm_bc = mtm_bc - mtm_bc[1])Error: cannot allocate vector of size 15.6 Mb In addition: Warning messages: 1: Reached total allocation of 3583Mb: see help(memory.size) 2: Reached total allocation of 3583Mb: see help(memory.size) 3: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) 4: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) 5: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) 6: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) 7: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) 8: In output[[var]][rng] <- df[[var]] : Reached total allocation of 3583Mb: see help(memory.size) When I checked -> memory.size()[1] 846.83> memory.limit()[1] 3583 The code is bit lengthy and unfortunately can't be shared. Kindly guide how this memory probelm can be tackled? I am using R x64 3.2.0 Regards Amelia
It is hard to tell from the information that you have provided. Do you have a list of the sizes of all the objects that you have in memory? Are you releasing large objects at the end of each simulation run? Are you using 'gc' to garbage collect any memory after deallocating objects? Collect some additional information with a simple function like below: f_mem_stats <- function(memo='') cat(memo, proc.time(), memory.size(), '\n')> f_mem_stats(2)2 2.85 11.59 85444.93 NA NA 39.08 This will print out what you pass in as a parameter, e.g., the iteration number, and then outputs the amount of CPU and memory used so far. I use this all the time to keep track of resource consumption in long running scripts. Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. On Wed, Apr 6, 2016 at 7:39 AM, Amelia Marsh via R-help < r-help at r-project.org> wrote:> Dear R Forum, > > I have about 2000+ FX forward transactions and I am trying to run 1000 > simulations. If I use less no of simulations, I am able to get the desired > results. However, when I try to use more than 1000 simulations, I get > following error. > > > sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate, > change_in_mtm_bc = mtm_bc - mtm_bc[1]) > > Error: cannot allocate vector of size 15.6 Mb > > > In addition: Warning messages: > 1: Reached total allocation of 3583Mb: see help(memory.size) > 2: Reached total allocation of 3583Mb: see help(memory.size) > 3: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > 4: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > 5: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > 6: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > 7: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > 8: In output[[var]][rng] <- df[[var]] : > Reached total allocation of 3583Mb: see help(memory.size) > > > When I checked - > > > memory.size() > [1] 846.83 > > memory.limit() > [1] 3583 > > > The code is bit lengthy and unfortunately can't be shared. > > Kindly guide how this memory probelm can be tackled? I am using R x64 3.2.0 > > Regards > > Amelia > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >[[alternative HTML version deleted]]
Dear Sir,
Thanks for the guidance. Will check. And yes, at the end of each simulation, a
large result is getting stored.?
Regards
Amelia
On Wednesday, 6 April 2016 5:48 PM, jim holtman <jholtman at
gmail.com> wrote:
It is hard to tell from the information that you have provided.? Do you have a
list of the sizes of all the objects that you have in memory?? Are you releasing
large objects at the end of each simulation run?? Are you using 'gc' to
garbage collect any memory after deallocating objects?? Collect some additional
information with a simple function like below:
f_mem_stats <- function(memo='') cat(memo, proc.time(),
memory.size(), '\n')
> f_mem_stats(2)2 2.85 11.59 85444.93 NA NA 39.08?
This will print out what you pass in as a parameter, e.g., the iteration number,
and then outputs the amount of CPU and memory used so far.? I use this all the
time to keep track of resource consumption in long running scripts.
Jim Holtman
Data Munger Guru
?
What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.
On Wed, Apr 6, 2016 at 7:39 AM, Amelia Marsh via R-help <r-help at
r-project.org> wrote:
Dear R Forum,
I have about 2000+ FX forward transactions and I am trying to run 1000
simulations. If I use less no of simulations, I am able to get the desired
results. However, when I try to use more than 1000 simulations, I get following
error.
> sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate,
change_in_mtm_bc = mtm_bc - mtm_bc[1])
Error: cannot allocate vector of size 15.6 Mb
In addition: Warning messages:
1: Reached total allocation of 3583Mb: see help(memory.size)
2: Reached total allocation of 3583Mb: see help(memory.size)
3: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
4: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
5: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
6: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
7: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
8: In output[[var]][rng] <- df[[var]] :
Reached total allocation of 3583Mb: see help(memory.size)
When I checked -
> memory.size()
[1] 846.83> memory.limit()
[1] 3583
The code is bit lengthy and unfortunately can't be shared.
Kindly guide how this memory probelm can be tackled? I am using R x64 3.2.0
Regards
Amelia
______________________________________________
R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
As Jim has indicated, memory usage problems can require very specific diagnostics and code changes, so generic help is tough to give. However, in most cases I have found the dplyr package to be more memory efficient than plyr, so you could consider that. Also, you can be explicit about only saving the minimum results you want to keep rather than making a list of complete results and extracting results later. -- Sent from my phone. Please excuse my brevity. On April 6, 2016 4:39:59 AM PDT, Amelia Marsh via R-help <r-help at r-project.org> wrote:>Dear R Forum, > >I have about 2000+ FX forward transactions and I am trying to run 1000 >simulations. If I use less no of simulations, I am able to get the >desired results. However, when I try to use more than 1000 simulations, >I get following error. > >> sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate, >change_in_mtm_bc = mtm_bc - mtm_bc[1]) > >Error: cannot allocate vector of size 15.6 Mb > > >In addition: Warning messages: >1: Reached total allocation of 3583Mb: see help(memory.size) >2: Reached total allocation of 3583Mb: see help(memory.size) >3: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) >4: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) >5: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) >6: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) >7: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) >8: In output[[var]][rng] <- df[[var]] : >Reached total allocation of 3583Mb: see help(memory.size) > > >When I checked - > >> memory.size() >[1] 846.83 >> memory.limit() >[1] 3583 > > >The code is bit lengthy and unfortunately can't be shared. > >Kindly guide how this memory probelm can be tackled? I am using R x64 >3.2.0 > >Regards > >Amelia > >______________________________________________ >R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code.[[alternative HTML version deleted]]
Dear Sir,
Yes I am using the plyr and in the end I am writing the output to the
data.frame. Earlier I had the problem of process time and hence I made some
changes in the code and now I am fetching all the required inputs needed for
valuation purpose using ddply, store the results in a data.frame and once that
is over, I am carrying out the calculations.
Here is part of my R code-
library(plyr)
library(reshape)
tx <- read.csv('transaction_fxdeal.csv')
tx$id <- as.character(tx$id)
n <- max(unique(simulated_exchange$id))
result <- NULL
current <- 1
rcount <- 0
current1 <- 1
rcount1 <- 0
current2 <- 1
rcount2 <- 0
for (env in 0:n) {
if (rcount == 0) rcount <- nrow(subset(simulated_interest, id==env))
temp <- current+rcount-1
env_rates <- simulated_interest[current:temp,]
env_rates <- env_rates[order(env_rates$curve, env_rates$day_count), ]
if (rcount1 == 0)rcount1 <- nrow(subset(simulated_exchange, id==env))
temp <- current1+rcount1-1
exch_rates <- simulated_exchange[current1:temp,]
if (rcount2 == 0)rcount2 <- nrow(subset(simulated_instruments, id==env))
temp <- current2+rcount2-1
instr_rates<- simulated_instruments[current2:temp,]
current <- current+rcount
current1 <- current1+rcount1
current2 <- current2+rcount2
curve <- daply(env_rates, 'curve', function(x) {
return(approxfun(x$day_count, x$rate, rule = 2))
})
result <- rbind(result, ddply(tx, 'id', function(x) {
intrate_from <- curve[[x$currency_from]](x$maturity_from)
intrate_to <- curve[[x$currency_to]](x$maturity_to)
cross_rate <- subset(exch_rates, key==paste(x$currency_from_exch,
x$currency_to_exch, sep='_'))$rate
base_rate <- subset(exch_rates, key==paste(x$currency_to_exch,
x$currency_base, sep='_'))$rate
return(data.frame(env=env, intrate_from=intrate_from, intrate_to=intrate_to,
cross_rate=cross_rate, base_rate=base_rate))
}))
}
sorted <- result[order(result$id, result$env),]
sorted$currency_from_exch <- rep(tx$currency_from_exch, each =
length(unique(sorted$env)))
sorted$currency_to_exch <- rep(tx$currency_to_exch, each =
length(unique(sorted$env)))
sorted$currency_base <- rep(tx$currency_base, each =
length(unique(sorted$env)))
sorted$transaction_type <- rep(tx$transaction_type, each =
length(unique(sorted$env)))
sorted$amount_fromccy <- rep(tx$amount_fromccy, each =
length(unique(sorted$env)))
sorted$amount_toccy <- rep(tx$amount_toccy, each =
length(unique(sorted$env)))
sorted$intbasis_fromccy <- rep(tx$intbasis_fromccy, each =
length(unique(sorted$env)))
sorted$intbasis_toccy <- rep(tx$intbasis_toccy, each =
length(unique(sorted$env)))
sorted$maturity_from <- rep(tx$maturity_from, each =
length(unique(sorted$env)))
sorted$maturity_to <- rep(tx$maturity_to, each = length(unique(sorted$env)))
sorted$currency_from <- rep(tx$currency_from, each =
length(unique(sorted$env)))
sorted$currency_to <- rep(tx$currency_to, each = length(unique(sorted$env)))
sorted$from_mtm <- sorted$cross_rate * (sorted$amount_fromccy / ((1 +
(sorted$intrate_from/100))^(sorted$maturity_from / sorted$intbasis_fromccy)))
sorted$to_mtm <- (sorted$amount_toccy / ((1 +
(sorted$intrate_to/100))^(sorted$maturity_to / sorted$intbasis_toccy)))
mtm_base <- function(from_mtm, to_mtm, base_rate)
{
mtm <- (from_mtm + to_mtm)
mtm_bc = mtm*base_rate[1]
return(data.frame(mtm_bc = mtm_bc))
}
sorted1 <- ddply(.data=sorted, .variables = "id", .fun=function(x)
mtm_base(from_mtm = x$from_mtm, to_mtm = x$to_mtm, base_rate = x$base_rate))
sorted$mtm <- sorted1$mtm
sorted$mtm_bc <- sorted1$mtm_bc
sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate, change_in_mtm_bc
= mtm_bc - mtm_bc[1])
sorted$change_in_mtm_bc <- sorted2$change_in_mtm_bc
sorted <- sorted[order(sorted$id, sorted$env),]
write.csv(data.frame(sorted), file='MC_result_fxdeal.csv',
row.names=FALSE)
# ________________________________________________________
# END of Code
With regards
Amelia
On Wednesday, 6 April 2016 7:43 PM, Jeff Newmiller <jdnewmil at
dcn.davis.ca.us> wrote:
As Jim has indicated, memory usage problems can require very specific
diagnostics and code changes, so generic help is tough to give.
However, in most cases I have found the dplyr package to be more memory
efficient than plyr, so you could consider that. Also, you can be explicit about
only saving the minimum results you want to keep rather than making a list of
complete results and extracting results later.
--
Sent from my phone. Please excuse my brevity.
On April 6, 2016 4:39:59 AM PDT, Amelia Marsh via R-help <r-help at
r-project.org> wrote:
Dear R Forum,>
>I have about 2000+ FX forward transactions and I am trying to run 1000
simulations. If I use less no of simulations, I am able to get the desired
results. However, when I try to use more than 1000 simulations, I get following
error.
>
>
>sorted2 <- ddply(sorted, .(currency_from_exch, id), mutate,
change_in_mtm_bc = mtm_bc - mtm_bc[1])
>>
>Error: cannot allocate vector of size 15.6 Mb
>
>
>In addition: Warning messages:
>1: Reached total allocation of 3583Mb: see help(memory.size)
>2: Reached total allocation of 3583Mb: see help(memory.size)
>3: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>4: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>5: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>6: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>7: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>8: In output[[var]][rng] <- df[[var]] :
>Reached total allocation of 3583Mb: see help(memory.size)
>
>
>When I checked -
>
>
>memory.size()
>>[1] 846.83
>
>memory.limit()
>>[1] 3583
>
>
>The code is bit lengthy and unfortunately can't be shared.
>
>Kindly guide how this memory probelm can be tackled? I am using R x64 3.2.0
>
>Regards
>
>Amelia
>
>>________________________________
>
>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.
>