R devel - Jun 2011 - Is there an implementation of loess with more than 3 parametric ...

Dear John,
> I suggest that you look at the abilities of the mgcv package.
> There are notes of mine at
>
> http://www.maths.anu.edu.au/%7Ejohnm/r-book/xtras/autosmooth.pdf
>
> that may help you get started.
Thank?you very much for the suggestion and the link to your write-up,
it was indeed very helpful!

I have experimented with this library for a while now and am really
happy about its flexibility. For my immediate applied problem, I will
now go with a gam fit ("z~te(x,y)+fa-1").

I note, however, that this is much, much slower than loess, and is
thus limited to smaller numbers of data points. (I could not fit the
full model to 50,000 data points in a reasonable time.)
I am therefore wondering if you knew of a way of also fixing the
implementation of loess in R?
>From the error message (recompile with larger d2MAX) it seems that the
underlying Fortran library was perfectly happy to fit a larger number
of parametric variables. So is there a way one could remove the
restriction to 4 parameters in the R interface/compilation? I have not
found an obvious place where d2MAX is defined or configured, I suspect
it might be hard-coded...

With best regards,
David.

Dear John,
> With loess, I regret that I cannot help. I have not looked
> into the code. ?I'd not been aware that there was such a
> speed difference. ? Are you sure that you are comparing
> like with like? ?loess does not optimise the choice of 'span',
> whereas (unless you fix it), mgcv does optimise the
> smoothing parameter.
I am aware that mgcv is the much more flexible package, so it is hard
to do a fair comparison but I have run loess in a cross-validation
loop to set the span parameter, and this runs in a few minutes on 50k
data points, while I aborted the gam attempt after several hours.
> You might at some point resubmit your question re the
> loess limitation to 4 parameters to CRAN, now with more
> background information.
Ok, what additional information can I provide that might be helpful?

Many thanks again for
your help and support,

David.

Dr. D. P. Kreil (Boku <David.Kreil <at> boku.ac.at> writes:
> 
> Dear John,
> 
> > I suggest that you look at the abilities of the mgcv package.
> > There are notes of mine at
> >
> > http://www.maths.anu.edu.au/%7Ejohnm/r-book/xtras/autosmooth.pdf
> >
> > that may help you get started.
> 
> Thank?you very much for the suggestion and the link to your write-up,
> it was indeed very helpful!
> 
> I have experimented with this library for a while now and am really
> happy about its flexibility. For my immediate applied problem, I will
> now go with a gam fit ("z~te(x,y)+fa-1").
> 
> I note, however, that this is much, much slower than loess, and is
> thus limited to smaller numbers of data points. (I could not fit the
> full model to 50,000 data points in a reasonable time.)
> I am therefore wondering if you knew of a way of also fixing the
> implementation of loess in R?
> 
> >From the error message (recompile with larger d2MAX) it seems that the
> underlying Fortran library was perfectly happy to fit a larger number
> of parametric variables. So is there a way one could remove the
> restriction to 4 parameters in the R interface/compilation? I have not
> found an obvious place where d2MAX is defined or configured, I suspect
> it might be hard-coded...
> 
  If you go looking for d2MAX in the R code (i.e. download the entire
source or get a copy of the SVN tree) you will proceed as follows:

 find . -name "*.c" -exec grep -l d2MAX {} \;


takes you to ./src/library/stats/src/loessc.c

Looking therein finds:

 case 105:MSG("k>d2MAX in ehg136.  Need to recompile with increased
dimensions.")

  grep 105 *.f finds loessf.f

within which we find two lines (1240 and 1361)

      if(k .gt. 15)   call ehg182(105)

Searching for "15" then finds a *lot* of places where "15"
is
hardcoded in the file.

 To modify the file you could either blindly change all the hard-coded
"15" dimensions to something larger and hope that it worked (i.e., use
a good test suite to make sure you didn't screw something up) or dig
through the code and understand it in detail to know what should
be changed ...