dear R experts---I would like to update OLS regressions with new observations on the front of the data, and delete some old observations from the rear. my goal is to have a "flexible" moving-window regression, with a minimum number of observations and a maximum number of observations. I can keep (X' X) and (X' y), and add or subtract observations from these two quantities myself, and then use crossprod. strucchange does recursive residuals, which is closely related, but it is not designed for such flexible movable windows, nor primarily designed to produce standard errors of coefficients. before I get started on this, I just wanted to inquire whether someone has already written such a function. regards, /iaw ---- Ivo Welch (ivo.welch at gmail.com)
Ivo: 1. You should not be fitting linear models as you describe. For why not and how they should be fit, consult a suitable text on numerical methods (e.g. Givens and Hoeting). 2. In R, I suggest using lm() and ?update, feeding update() data modified as you like. This is, after all, the reason for update(). -- Bert On Mon, May 27, 2013 at 8:12 AM, ivo welch <ivo.welch at anderson.ucla.edu> wrote:> dear R experts---I would like to update OLS regressions with new > observations on the front of the data, and delete some old > observations from the rear. my goal is to have a "flexible" > moving-window regression, with a minimum number of observations and a > maximum number of observations. I can keep (X' X) and (X' y), and add > or subtract observations from these two quantities myself, and then > use crossprod. > > strucchange does recursive residuals, which is closely related, but it > is not designed for such flexible movable windows, nor primarily > designed to produce standard errors of coefficients. > > before I get started on this, I just wanted to inquire whether someone > has already written such a function. > > regards, > > /iaw > ---- > Ivo Welch (ivo.welch at gmail.com) > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Bert Gunter Genentech Nonclinical Biostatistics Internal Contact Info: Phone: 467-7374 Website: http://pharmadevelopment.roche.com/index/pdb/pdb-functional-groups/pdb-biostatistics/pdb-ncb-home.htm
Look at the biglm package. It does 2 of the 3 things that you asked for: Construct an initial lm fit and add a new block of data to update that fit. It does not remove data, but you may be able to look at the code and figure out a way to modify it to do the final piece. On Mon, May 27, 2013 at 9:12 AM, ivo welch <ivo.welch@anderson.ucla.edu>wrote:> dear R experts---I would like to update OLS regressions with new > observations on the front of the data, and delete some old > observations from the rear. my goal is to have a "flexible" > moving-window regression, with a minimum number of observations and a > maximum number of observations. I can keep (X' X) and (X' y), and add > or subtract observations from these two quantities myself, and then > use crossprod. > > strucchange does recursive residuals, which is closely related, but it > is not designed for such flexible movable windows, nor primarily > designed to produce standard errors of coefficients. > > before I get started on this, I just wanted to inquire whether someone > has already written such a function. > > regards, > > /iaw > ---- > Ivo Welch (ivo.welch@gmail.com) > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Gregory (Greg) L. Snow Ph.D. 538280@gmail.com [[alternative HTML version deleted]]
The essential trick here is the Sherman-Morrison-Woodbury formula. My quantreg package has a lm.fit.recursive function that implements a fortran version for adding observations, but like biglm I don't remove observations at the other end either. Roger Koenker rkoenker at illinois.edu On May 27, 2013, at 2:07 PM, Greg Snow wrote:> Look at the biglm package. It does 2 of the 3 things that you asked for: > Construct an initial lm fit and add a new block of data to update that > fit. It does not remove data, but you may be able to look at the code and > figure out a way to modify it to do the final piece. > > > On Mon, May 27, 2013 at 9:12 AM, ivo welch <ivo.welch at anderson.ucla.edu>wrote: > >> dear R experts---I would like to update OLS regressions with new >> observations on the front of the data, and delete some old >> observations from the rear. my goal is to have a "flexible" >> moving-window regression, with a minimum number of observations and a >> maximum number of observations. I can keep (X' X) and (X' y), and add >> or subtract observations from these two quantities myself, and then >> use crossprod. >> >> strucchange does recursive residuals, which is closely related, but it >> is not designed for such flexible movable windows, nor primarily >> designed to produce standard errors of coefficients. >> >> before I get started on this, I just wanted to inquire whether someone >> has already written such a function. >> >> regards, >> >> /iaw >> ---- >> Ivo Welch (ivo.welch at gmail.com) >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > > > -- > Gregory (Greg) L. Snow Ph.D. > 538280 at gmail.com > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
On 27-05-2013, at 17:12, ivo welch <ivo.welch at anderson.ucla.edu> wrote:> dear R experts---I would like to update OLS regressions with new > observations on the front of the data, and delete some old > observations from the rear. my goal is to have a "flexible" > moving-window regression, with a minimum number of observations and a > maximum number of observations. I can keep (X' X) and (X' y), and add > or subtract observations from these two quantities myself, and then > use crossprod. > > strucchange does recursive residuals, which is closely related, but it > is not designed for such flexible movable windows, nor primarily > designed to produce standard errors of coefficients. > > before I get started on this, I just wanted to inquire whether someone > has already written such a function. >For regression one would use a QR decomposition. There is an opensource Fortran library qrupdate (http://sourceforge.net/projects/qrupdate/) that can update an unpivoted QR decomposition for the case of deleting rows/columns and inserting rows/columns. It could be used to make an R package, which could be used for doing a moving window regression. Quite a lot of work. Berend