Prof Brian Ripley
2004-Mar-25 21:51 UTC
[R] Aggregating frequency of irregular time series
R itself has no support for irregular time series, but it does have an aggregate method for regular ones. You need to look into whichever package is handling irregular time series. However, it seems to me that this is not a time series problem at all: you have a set of observations whose indices are data-times, and tapply() will do the job. On Thu, 25 Mar 2004, Ivan Alves wrote:> S-Plus has the function AggregateSeries() whose name is self > explanatory. For instance one can derive monthly series from daily > ones by specifying end-of-period, averages, sums, etc. I looked for a > similar function in the packages "its" and "tseries", but found > nothing. I also help.searched() for aggregate to no avail. Would > anybody be so kind to point me in the right direction?-- Brian D. Ripley, ripley at stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UK Fax: +44 1865 272595
Thank you very much Brian. Indeed, by looking at ?tapply() this would do the job for regular time series (whose data-time INDEX renders itself naturally for " bundling data" in regular groups). However, with irregular time series the story is different, as some careful "bundling" is necessary prior to applying tapply(). Whereas this may not be a problem for months (creating strings month-year), it would be for weeks, for instance. Also the object returned would need to be converted again to a time series object with additional function calls. Furthermore, the function AggregateSeries() provides additional functionality, such as creating moving averages, rolling variances, minima and maxima, all with options to the same function call. I take from your response that there is no easy way out (an already existing function), and that some programming will be required. Kind regards, Ivan _______________________ Ivan Alves mailto://papucho at mac.com On 25 Mar 2004, at 22:51, Prof Brian Ripley wrote:> R itself has no support for irregular time series, but it does have an > aggregate method for regular ones. You need to look into whichever > package is handling irregular time series. However, it seems to me > that > this is not a time series problem at all: you have a set of > observations > whose indices are data-times, and tapply() will do the job. > > On Thu, 25 Mar 2004, Ivan Alves wrote: > >> S-Plus has the function AggregateSeries() whose name is self >> explanatory. For instance one can derive monthly series from daily >> ones by specifying end-of-period, averages, sums, etc. I looked for a >> similar function in the packages "its" and "tseries", but found >> nothing. I also help.searched() for aggregate to no avail. Would >> anybody be so kind to point me in the right direction? > > -- > Brian D. Ripley, ripley at stats.ox.ac.uk > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > University of Oxford, Tel: +44 1865 272861 (self) > 1 South Parks Road, +44 1865 272866 (PA) > Oxford OX1 3TG, UK Fax: +44 1865 272595 >
> From: Ivan Alves > > Thank you very much Brian. Indeed, by looking at ?tapply() > this would > do the job for regular time series (whose data-time INDEX renders > itself naturally for " bundling data" in regular groups). > However, with > irregular time series the story is different, as some careful > "bundling" is necessary prior to applying tapply(). Whereas this may > not be a problem for months (creating strings month-year), it > would be > for weeks, for instance. Also the object returned would need to be > converted again to a time series object with additional > function calls. > Furthermore, the function AggregateSeries() provides additional > functionality, such as creating moving averages, rolling variances, > minima and maxima, all with options to the same function call. I take > from your response that there is no easy way out (an already existing > function), and that some programming will be required.So can we count on you to contribute this (and the cointegration that you requested in another post)? You did read the message at the R startup screen, didn't you? Best, Andy> Kind regards, > > Ivan > _______________________ > Ivan Alves > mailto://papucho at mac.com > On 25 Mar 2004, at 22:51, Prof Brian Ripley wrote: > > > R itself has no support for irregular time series, but it > does have an > > aggregate method for regular ones. You need to look into whichever > > package is handling irregular time series. However, it seems to me > > that > > this is not a time series problem at all: you have a set of > > observations > > whose indices are data-times, and tapply() will do the job. > > > > On Thu, 25 Mar 2004, Ivan Alves wrote: > > > >> S-Plus has the function AggregateSeries() whose name is self > >> explanatory. For instance one can derive monthly series from daily > >> ones by specifying end-of-period, averages, sums, etc. I > looked for a > >> similar function in the packages "its" and "tseries", but found > >> nothing. I also help.searched() for aggregate to no avail. Would > >> anybody be so kind to point me in the right direction? > > > > -- > > Brian D. Ripley, ripley at stats.ox.ac.uk > > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ > > University of Oxford, Tel: +44 1865 272861 (self) > > 1 South Parks Road, +44 1865 272866 (PA) > > Oxford OX1 3TG, UK Fax: +44 1865 272595 > > > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html > >
Gabor Grothendieck
2004-Mar-26 04:54 UTC
[R] Aggregating frequency of irregular time series
Assume r.its is an its time series and assume that chron has been loaded. chron represents dates as days since an origin so the week number can be obtained by dividing by 7. Then tapply the mean or other function. Finally, recreate an its. week <- 7 * (as.numeric( chron( format( r.its at dates, "%m/%d/%Y" ) ) ) %/% 7 ) z <- tapply( r.its, week, mean ) z <- its( c(z), date = as.POSIXct( chron( as.numeric( names( z ) ) ) ) ) There is a new Dates class in R 1.9.0 that could be used in place of chron. Ivan Alves <papucho <at> mac.com> writes: : : Thank you very much Brian. Indeed, by looking at ?tapply() this would : do the job for regular time series (whose data-time INDEX renders : itself naturally for " bundling data" in regular groups). However, with : irregular time series the story is different, as some careful : "bundling" is necessary prior to applying tapply(). Whereas this may : not be a problem for months (creating strings month-year), it would be : for weeks, for instance. Also the object returned would need to be : converted again to a time series object with additional function calls. : Furthermore, the function AggregateSeries() provides additional : functionality, such as creating moving averages, rolling variances, : minima and maxima, all with options to the same function call. I take : from your response that there is no easy way out (an already existing : function), and that some programming will be required. : : Kind regards, : : Ivan : _______________________ : Ivan Alves : mailto://papucho <at> mac.com : On 25 Mar 2004, at 22:51, Prof Brian Ripley wrote: : : > R itself has no support for irregular time series, but it does have an : > aggregate method for regular ones. You need to look into whichever : > package is handling irregular time series. However, it seems to me : > that : > this is not a time series problem at all: you have a set of : > observations : > whose indices are data-times, and tapply() will do the job. : > : > On Thu, 25 Mar 2004, Ivan Alves wrote: : > : >> S-Plus has the function AggregateSeries() whose name is self : >> explanatory. For instance one can derive monthly series from daily : >> ones by specifying end-of-period, averages, sums, etc. I looked for a : >> similar function in the packages "its" and "tseries", but found : >> nothing. I also help.searched() for aggregate to no avail. Would : >> anybody be so kind to point me in the right direction? : > : > -- : > Brian D. Ripley, ripley <at> stats.ox.ac.uk : > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ : > University of Oxford, Tel: +44 1865 272861 (self) : > 1 South Parks Road, +44 1865 272866 (PA) : > Oxford OX1 3TG, UK Fax: +44 1865 272595 : > : : ______________________________________________ : R-help <at> stat.math.ethz.ch mailing list : https://www.stat.math.ethz.ch/mailman/listinfo/r-help : PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html : :
Thanks to all once again for the very useful suggestions. I will have to teach myself some real S before tackling the writting of the AggregateSeries-like function! I brace myself. Kind regards, Ivan _______________________ Ivan Alves mailto://papucho at mac.com
> S-Plus has the function AggregateSeries() whose name is self > explanatory. For instance one can derive monthly series from daily > ones by specifying end-of-period, averages, sums, etc. I looked for > a similar function in the packages "its" and "tseries", but found > nothing. I also help.searched() for aggregate to no avail. Would > anybody be so kind to point me in the right direction?I once needed a function which would convert daily stock prices into weekly returns. I know, the code is pretty bad (it is all loops and it's very slow), but my knowledge of R is weak and I really needed it, so I just used brute force. See EOF for the function. Gabor and Dirk and Brian Ripley and others on the list were very helpful to me in getting to the point where I could write this, though obviously they should not be blamed for my bad code! :-) I would be very happy if listers could give me ideas on how to do this better. Daily to monthly is innately easier since months are 'more normal' than weeks. I have perl code which does this, which supports 2 cases: Reporting the last traded price (LTP) of the month versus reporting the average of the month. If this is useful to you, ask me. What are the specialised finance libraries available with S-Plus? Can one marry R with commercial S libraries? I don't like having a dependence on commercial code, but I might be willing to compromise and buy libraries that work with R. Thanks, -- Ajay Shah Consultant ajayshah at mayin.org Department of Economic Affairs http://www.mayin.org/ajayshah Ministry of Finance, New Delhi ### --------------------------------------------------------------------------- ### Function levels2WeeklyRet ### ### Purpose To convert a vector of daily levels (e.g. a stock price, an ### index, a currency) into a vector of weekly returns. ### How We focus on friday-to-friday returns. If a friday was ### not traded, we take the most-recent available price. ### We compute 100*log(p2/p1) where p1 and p2 are the prices ### as of 2 consecutive fridays. ### Inputs lastfriday A chron object showing the last friday which ### is NOT in the dataset. ### dates A vector of dates (chron objects) ### levels A vector of prices. ### Daily levels data is the list of points (dates[i], levels[i]) ### CAREFUL This function could break if you feed it really perverse data, ### e.g. with numerous consecutive days of missing data. ### It _must_ be the case that there is atleast 1 traded date ### in the data after lastfriday but before lastfriday+7. ### Depends library(chron) levels2WeeklyRet <- function(lastfriday, dates, levels) { ## First do a "dates canonicalise" operation; i.e. lay out a big table with ## all dates and populate it. Uninitialised values are left at 0. T = length(dates); all.dates = seq.dates(lastfriday, dates[T]); all.values = numeric(length(all.dates)); j=1; for (i in 1:T) { while (dates[i] != all.dates[j]) j = j+1; all.values[j] = levels[i]; } ## Walk through all.values[] making a vector of LTPs for all fridays. f.ltp = numeric(trunc((dates[T]-lastfriday)/7)) i = 8; # The 1st data friday in all.values for (j in 1:length(f.ltp)) { # Walk in friday's vector. k=i; while (0 == all.values[k]) k = k-1; # Go as far back as needed f.ltp[j] = all.values[k]; # This is the LTP as of this friday. i = i+7; # Get ready for the next time -- } return(100*diff(log(f.ltp), 1)); # Pretty computation of returns vector! :-) } ### --------------------------------------------------------------------------- # Compute the friday before the stated date. prevFriday <- function(x) { repeat { x <- x - 1; if (5 == with(month.day.year(x), day.of.week(month,day,year))) break; } return(x); } # Other tricks which could have worked also : # do.call( "day.of.week", month.day.year(x) ) # as.numeric(x-3)%%7 # uses fact that chron(3) is Sunday
Assuming that you are using log returns, one approach (probably not the best) is to convert the dates to julian days, find all of the Sundays spanning the dates in the series, then do something along the lines of: tapply(the.log.returns, cut(julian.days, sundays), sum) Patrick Burns Burns Statistics patrick at burns-stat.com +44 (0)20 8525 0696 http://www.burns-stat.com (home of S Poetry and "A Guide for the Unwilling S User") Ajay Shah wrote:>>S-Plus has the function AggregateSeries() whose name is self >>explanatory. For instance one can derive monthly series from daily >>ones by specifying end-of-period, averages, sums, etc. I looked for >>a similar function in the packages "its" and "tseries", but found >>nothing. I also help.searched() for aggregate to no avail. Would >>anybody be so kind to point me in the right direction? >> >> > >I once needed a function which would convert daily stock prices into >weekly returns. I know, the code is pretty bad (it is all loops and >it's very slow), but my knowledge of R is weak and I really needed it, >so I just used brute force. See EOF for the function. Gabor and Dirk >and Brian Ripley and others on the list were very helpful to me in >getting to the point where I could write this, though obviously they >should not be blamed for my bad code! :-) > >I would be very happy if listers could give me ideas on how to do this >better. > >Daily to monthly is innately easier since months are 'more normal' >than weeks. I have perl code which does this, which supports 2 cases: >Reporting the last traded price (LTP) of the month versus reporting >the average of the month. If this is useful to you, ask me. > >What are the specialised finance libraries available with S-Plus? Can >one marry R with commercial S libraries? I don't like having a >dependence on commercial code, but I might be willing to compromise >and buy libraries that work with R. > >Thanks, > > >
This function will probably work for you if your dates are in the format yyyymmdd. Use it to convert the series from daily to monthly, then use a 1 period difference to calc the returns of the series. Feel free to contact me off list with any questions. Dates <- function(x) rownames(x) Timebase <- function(x,monthly=T,first.day=T) { if(length(dim(x))!=2) stop("can only take 2 dimensional data.") seriesDates <- as.integer(Dates(x)) # year and Month seriesYM <- as.integer(seriesDates/100) seriesYM.unique <- sort(unique(seriesYM)) ans.nrow <- length(seriesYM.unique) # loop through all the unique months and grab the max day of month ans <- matrix(NA,nrow=ans.nrow,ncol=ncol(x)) ans.dates <- vector("numeric",ans.nrow) for (i in 1:ans.nrow) { this.month <- seriesYM == seriesYM.unique[i] max.day <- max(seriesDates[this.month]) ans.dates[i] <- max.day } colnames(ans) <- colnames(x) ans[] <- x[as.character(ans.dates),] if(first.day==T) { rownames(ans) <- paste(substring(as.character(ans.dates),1,6),"01",sep="") } else { rownames(ans) <- ans.dates } as.tseries(ans) } -----Original Message----- From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Patrick Burns Sent: Tuesday, March 30, 2004 2:37 PM To: Ajay Shah Cc: r-help at stat.math.ethz.ch; Ivan Alves Subject: Re: [R] Aggregating frequency of irregular time series Assuming that you are using log returns, one approach (probably not the best) is to convert the dates to julian days, find all of the Sundays spanning the dates in the series, then do something along the lines of: tapply(the.log.returns, cut(julian.days, sundays), sum) Patrick Burns Burns Statistics patrick at burns-stat.com +44 (0)20 8525 0696 http://www.burns-stat.com (home of S Poetry and "A Guide for the Unwilling S User") Ajay Shah wrote:>>S-Plus has the function AggregateSeries() whose name is self >>explanatory. For instance one can derive monthly series from daily >>ones by specifying end-of-period, averages, sums, etc. I looked for a >>similar function in the packages "its" and "tseries", but found >>nothing. I also help.searched() for aggregate to no avail. Would >>anybody be so kind to point me in the right direction? >> >> > >I once needed a function which would convert daily stock prices into >weekly returns. I know, the code is pretty bad (it is all loops and >it's very slow), but my knowledge of R is weak and I really needed it, >so I just used brute force. See EOF for the function. Gabor and Dirk >and Brian Ripley and others on the list were very helpful to me in >getting to the point where I could write this, though obviously they >should not be blamed for my bad code! :-) > >I would be very happy if listers could give me ideas on how to do this >better. > >Daily to monthly is innately easier since months are 'more normal' than >weeks. I have perl code which does this, which supports 2 cases: >Reporting the last traded price (LTP) of the month versus reporting the >average of the month. If this is useful to you, ask me. > >What are the specialised finance libraries available with S-Plus? Can >one marry R with commercial S libraries? I don't like having a >dependence on commercial code, but I might be willing to compromise and >buy libraries that work with R. > >Thanks, > > >______________________________________________ R-help at stat.math.ethz.ch mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
You are quite right. One can see from the original function I wrote that I am not very familiar with the apply family. Thanks for pointing out this improvement. Below is the new version of the function; any other comments are welcome. Regards, Whit Timebase <- function(x,monthly=T,first.day=T) { if(length(dim(x))!=2) stop("can only take 2 dimensional data.") seriesDates <- as.integer(Dates(x)) # year and Month seriesYM <- as.integer(seriesDates/100) # for each seriesYM grab the max day of month ans.dates <- as.numeric(tapply( seriesDates, seriesYM, max )) ans <- matrix(NA,nrow=length(ans.dates),ncol=ncol(x)) colnames(ans) <- colnames(x) ans[] <- x[as.character(ans.dates),] if(first.day==T) { rownames(ans) <- paste(substring(as.character(ans.dates),1,6),"01",sep="") } else { rownames(ans) <- ans.dates } as.tseries(ans) } -----Original Message----- From: r-help-bounces at stat.math.ethz.ch [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Gabor Grothendieck Sent: Tuesday, March 30, 2004 8:59 PM To: r-help at stat.math.ethz.ch Subject: Re: [R] Aggregating frequency of irregular time series I am not 100% sure I understand what you are trying to accomplish but if its to calculate the highest date in seriesDates for each value of seriesYM then you can do it without a loop like this (untested): tapply( seriesDates, seriesYM, max ) Whit Armstrong <Whit.Armstrong <at> tudor.com> writes: : : This function will probably work for you if your dates are in the format : yyyymmdd. Use it to convert the series from daily to monthly, then use a 1 : period difference to calc the returns of the series. : : Feel free to contact me off list with any questions. : : Dates <- function(x) rownames(x) : : Timebase <- function(x,monthly=T,first.day=T) { : if(length(dim(x))!=2) stop("can only take 2 dimensional data.") : : seriesDates <- as.integer(Dates(x)) : : # year and Month : seriesYM <- as.integer(seriesDates/100) : seriesYM.unique <- sort(unique(seriesYM)) : ans.nrow <- length(seriesYM.unique) : : # loop through all the unique months and grab the max day of month : ans <- matrix(NA,nrow=ans.nrow,ncol=ncol(x)) : ans.dates <- vector("numeric",ans.nrow) : : for (i in 1:ans.nrow) { : : this.month <- seriesYM == seriesYM.unique[i] : max.day <- max(seriesDates[this.month]) : ans.dates[i] <- max.day : } : : : colnames(ans) <- colnames(x) : ans[] <- x[as.character(ans.dates),] : : if(first.day==T) { : rownames(ans) <- : paste(substring(as.character(ans.dates),1,6),"01",sep="") : } else { : rownames(ans) <- ans.dates : } : : as.tseries(ans) : } : : -----Original Message----- : From: r-help-bounces <at> stat.math.ethz.ch : [mailto:r-help-bounces <at> stat.math.ethz.ch] On Behalf Of Patrick Burns : Sent: Tuesday, March 30, 2004 2:37 PM : To: Ajay Shah : Cc: r-help <at> stat.math.ethz.ch; Ivan Alves : Subject: Re: [R] Aggregating frequency of irregular time series : : : Assuming that you are using log returns, one approach (probably not the : best) is to convert the dates to julian days, find all of the Sundays : spanning the dates in the series, then do something along the lines of: : : tapply(the.log.returns, cut(julian.days, sundays), sum) : : Patrick Burns : : Burns Statistics : patrick <at> burns-stat.com : +44 (0)20 8525 0696 : http://www.burns-stat.com : (home of S Poetry and "A Guide for the Unwilling S User") : : Ajay Shah wrote: : : >>S-Plus has the function AggregateSeries() whose name is self : >>explanatory. For instance one can derive monthly series from daily : >>ones by specifying end-of-period, averages, sums, etc. I looked for a : >>similar function in the packages "its" and "tseries", but found : >>nothing. I also help.searched() for aggregate to no avail. Would : >>anybody be so kind to point me in the right direction? : >> : >> : > : >I once needed a function which would convert daily stock prices into : >weekly returns. I know, the code is pretty bad (it is all loops and : >it's very slow), but my knowledge of R is weak and I really needed it, : >so I just used brute force. See EOF for the function. Gabor and Dirk : >and Brian Ripley and others on the list were very helpful to me in : >getting to the point where I could write this, though obviously they : >should not be blamed for my bad code! : > : >I would be very happy if listers could give me ideas on how to do this : >better. : > : >Daily to monthly is innately easier since months are 'more normal' than : >weeks. I have perl code which does this, which supports 2 cases: : >Reporting the last traded price (LTP) of the month versus reporting the : >average of the month. If this is useful to you, ask me. : > : >What are the specialised finance libraries available with S-Plus? Can : >one marry R with commercial S libraries? I don't like having a : >dependence on commercial code, but I might be willing to compromise and : >buy libraries that work with R. : > : >Thanks, : > : > : > : : ______________________________________________ : R-help <at> stat.math.ethz.ch mailing list : https://www.stat.math.ethz.ch/mailman/listinfo/r-help : PLEASE do read the posting guide! : http://www.R-project.org/posting-guide.html : : ______________________________________________ : R-help <at> stat.math.ethz.ch mailing list : https://www.stat.math.ethz.ch/mailman/listinfo/r-help : PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html : : ______________________________________________ R-help at stat.math.ethz.ch mailing list https://www.stat.math.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Although this thread might be considered closed by some, I'd like to make a late contribution, since I contributed the extractIts() function in the 'its' package. See ?itsSubset, which gives details of: extractIts(x,weekday=FALSE,find=c("all","last","first"),period=c("week","mon th"),partials=TRUE,select) Using this on its own just 'extracts', using it with cumsum() and diff() will give you 'sum' aggregation albeit with loss of precision. As for 'average', it would depend how you want to average... - Giles> -----Original Message----- > From: r-help-bounces at stat.math.ethz.ch > [mailto:r-help-bounces at stat.math.ethz.ch]On Behalf Of Ivan Alves > Sent: 26 March 2004 21:51 > To: r-help at stat.math.ethz.ch > Subject: [R] Aggregating frequency of irregular time series > > > Thanks to all once again for the very useful suggestions. I will have > to teach myself some real S before tackling the writting of the > AggregateSeries-like function! I brace myself. > > Kind regards, > > Ivan > _______________________ > Ivan Alves > mailto://papucho at mac.com > > ______________________________________________ > R-help at stat.math.ethz.ch mailing list > https://www.stat.math.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide! > http://www.R-project.org/posting-guide.html >********************************************************************** This is a commercial communication from Commerzbank AG.\ \ T...{{dropped}}
Reasonably Related Threads
- aggregating along bins and bin-quantiles
- S+Finmetrics cointegration functions
- Stuck in trying to convert repetitive code into a function
- Manipulation in timeSeries object:how to use the function "applySeries" by daily?
- aggregating values at discreet irregular time intervals into hourly values