Dr. D. P. Kreil (Boku)
2011-Jun-24 15:03 UTC
[Rd] Is there an implementation of loess with more than 3 parametric ...
Dear John,> I suggest that you look at the abilities of the mgcv package. > There are notes of mine at > > http://www.maths.anu.edu.au/%7Ejohnm/r-book/xtras/autosmooth.pdf > > that may help you get started.Thank?you very much for the suggestion and the link to your write-up, it was indeed very helpful! I have experimented with this library for a while now and am really happy about its flexibility. For my immediate applied problem, I will now go with a gam fit ("z~te(x,y)+fa-1"). I note, however, that this is much, much slower than loess, and is thus limited to smaller numbers of data points. (I could not fit the full model to 50,000 data points in a reasonable time.) I am therefore wondering if you knew of a way of also fixing the implementation of loess in R?>From the error message (recompile with larger d2MAX) it seems that theunderlying Fortran library was perfectly happy to fit a larger number of parametric variables. So is there a way one could remove the restriction to 4 parameters in the R interface/compilation? I have not found an obvious place where d2MAX is defined or configured, I suspect it might be hard-coded... With best regards, David.
Dr. D. P. Kreil (Boku)
2011-Jun-25 01:17 UTC
[Rd] Is there an implementation of loess with more than 3 parametric ...
Dear John,> With loess, I regret that I cannot help. I have not looked > into the code. ?I'd not been aware that there was such a > speed difference. ? Are you sure that you are comparing > like with like? ?loess does not optimise the choice of 'span', > whereas (unless you fix it), mgcv does optimise the > smoothing parameter.I am aware that mgcv is the much more flexible package, so it is hard to do a fair comparison but I have run loess in a cross-validation loop to set the span parameter, and this runs in a few minutes on 50k data points, while I aborted the gam attempt after several hours.> You might at some point resubmit your question re the > loess limitation to 4 parameters to CRAN, now with more > background information.Ok, what additional information can I provide that might be helpful? Many thanks again for your help and support, David.
Ben Bolker
2011-Jun-27 15:19 UTC
[Rd] Is there an implementation of loess with more than 3 parametric ...
Dr. D. P. Kreil (Boku <David.Kreil <at> boku.ac.at> writes:> > Dear John, > > > I suggest that you look at the abilities of the mgcv package. > > There are notes of mine at > > > > http://www.maths.anu.edu.au/%7Ejohnm/r-book/xtras/autosmooth.pdf > > > > that may help you get started. > > Thank?you very much for the suggestion and the link to your write-up, > it was indeed very helpful! > > I have experimented with this library for a while now and am really > happy about its flexibility. For my immediate applied problem, I will > now go with a gam fit ("z~te(x,y)+fa-1"). > > I note, however, that this is much, much slower than loess, and is > thus limited to smaller numbers of data points. (I could not fit the > full model to 50,000 data points in a reasonable time.) > I am therefore wondering if you knew of a way of also fixing the > implementation of loess in R? > > >From the error message (recompile with larger d2MAX) it seems that the > underlying Fortran library was perfectly happy to fit a larger number > of parametric variables. So is there a way one could remove the > restriction to 4 parameters in the R interface/compilation? I have not > found an obvious place where d2MAX is defined or configured, I suspect > it might be hard-coded... >If you go looking for d2MAX in the R code (i.e. download the entire source or get a copy of the SVN tree) you will proceed as follows: find . -name "*.c" -exec grep -l d2MAX {} \; takes you to ./src/library/stats/src/loessc.c Looking therein finds: case 105:MSG("k>d2MAX in ehg136. Need to recompile with increased dimensions.") grep 105 *.f finds loessf.f within which we find two lines (1240 and 1361) if(k .gt. 15) call ehg182(105) Searching for "15" then finds a *lot* of places where "15" is hardcoded in the file. To modify the file you could either blindly change all the hard-coded "15" dimensions to something larger and hope that it worked (i.e., use a good test suite to make sure you didn't screw something up) or dig through the code and understand it in detail to know what should be changed ...
Reasonably Related Threads
- Is there an implementation loess with more than 4 parametric predictors or a trick to similar effect?
- Update: Is there an implementation of loess with more than 3 parametric predictors or a trick to a similar effect?
- Is there an implementation of loess with more than 3 parametric predictors or a trick to a similar effect? [re-posting as plain text to pass char-set filter]
- test for readline installation fails if installation location non-standard (PR#8104)
- loess crash