R help - Mar 2010 - Error "singular gradient matrix at initial parameter estimates" in nls

I am using nls to fit a non linear function to some data.

The non linear function is:

y= 1- exp(-(k0+k1*p1+ .... + kn*pn))

I have chosen algorithm "port", with lower boundary is 0 for all of
the
ki parameters, and I have tried many start values for the parameters ki 
(including generating them at random).

If I fit the non linear function to the same data using an external 
algorithm, it fits perfectly and finds the parameters.

As soon as I come to my R installation (2.10.1 on Kubuntu Linux 910 64 
bit), I keep getting the error:

Error in nlsModel(formula, mf, start, wts, upper) :   singular gradient 
matrix at initial parameter estimates

I have read all the previous postings and the documentation, but to no 
avail: the error is there to stay. I am sure the problem is with nls, 
because the external fitting algorithm perfectly fits it in less than a 
second. Also, if my n is 4, then the nls works perfectly (but that 
excludes all the k5 .... kn).

Can anyone help me with suggestions? Thanks in advance.

Alternatively, what do you suggest I should do? Shall I abandon nls in 
favour of optim?

Regards

-- 
Corrado Topi
PhD Researcher
Global Climate Change and Biodiversity
Area 18,Department of Biology
University of York, York, YO10 5YW, UK
Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk

You could try method="brute-force" in the nls2 package to find
starting values.

On Tue, Mar 30, 2010 at 7:03 AM, Corrado <ct529 at york.ac.uk>
wrote:
> I am using nls to fit a non linear function to some data.
>
> The non linear function is:
>
> y= 1- exp(-(k0+k1*p1+ .... + kn*pn))
>
> I have chosen algorithm "port", with lower boundary is 0 for all
of the ki
> parameters, and I have tried many start values for the parameters ki
> (including generating them at random).
>
> If I fit the non linear function to the same data using an external
> algorithm, it fits perfectly and finds the parameters.
>
> As soon as I come to my R installation (2.10.1 on Kubuntu Linux 910 64
bit),
> I keep getting the error:
>
> Error in nlsModel(formula, mf, start, wts, upper) : ? singular gradient
> matrix at initial parameter estimates
>
> I have read all the previous postings and the documentation, but to no
> avail: the error is there to stay. I am sure the problem is with nls,
> because the external fitting algorithm perfectly fits it in less than a
> second. Also, if my n is 4, then the nls works perfectly (but that excludes
> all the k5 .... kn).
>
> Can anyone help me with suggestions? Thanks in advance.
>
> Alternatively, what do you suggest I should do? Shall I abandon nls in
> favour of optim?
>
> Regards
>
> --
> Corrado Topi
> PhD Researcher
> Global Climate Change and Biodiversity
> Area 18,Department of Biology
> University of York, York, YO10 5YW, UK
> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

Hi Gabor,

same problem even using nls2 with method=brute-force to calculate the 
initial parameters.

Best,

Gabor Grothendieck wrote:
> You could try method="brute-force" in the nls2 package to find
starting values.
>
> On Tue, Mar 30, 2010 at 7:03 AM, Corrado <ct529 at york.ac.uk> wrote:
>   
>> I am using nls to fit a non linear function to some data.
>>
>> The non linear function is:
>>
>> y= 1- exp(-(k0+k1*p1+ .... + kn*pn))
>>
>> I have chosen algorithm "port", with lower boundary is 0 for
all of the ki
>> parameters, and I have tried many start values for the parameters ki
>> (including generating them at random).
>>
>> If I fit the non linear function to the same data using an external
>> algorithm, it fits perfectly and finds the parameters.
>>
>> As soon as I come to my R installation (2.10.1 on Kubuntu Linux 910 64
bit),
>> I keep getting the error:
>>
>> Error in nlsModel(formula, mf, start, wts, upper) :   singular gradient
>> matrix at initial parameter estimates
>>
>> I have read all the previous postings and the documentation, but to no
>> avail: the error is there to stay. I am sure the problem is with nls,
>> because the external fitting algorithm perfectly fits it in less than a
>> second. Also, if my n is 4, then the nls works perfectly (but that
excludes
>> all the k5 .... kn).
>>
>> Can anyone help me with suggestions? Thanks in advance.
>>
>> Alternatively, what do you suggest I should do? Shall I abandon nls in
>> favour of optim?
>>
>> Regards
>>
>> --
>> Corrado Topi
>> PhD Researcher
>> Global Climate Change and Biodiversity
>> Area 18,Department of Biology
>> University of York, York, YO10 5YW, UK
>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>>     

-- 
Corrado Topi
PhD Researcher
Global Climate Change and Biodiversity
Area 18,Department of Biology
University of York, York, YO10 5YW, UK
Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk

Sorry, its algorithm="brute-force"

On Tue, Mar 30, 2010 at 10:29 AM, Corrado <ct529 at york.ac.uk>
wrote:
> Hi Gabor,
>
> same problem even using nls2 with method=brute-force to calculate the
> initial parameters.
>
> Best,
>
> Gabor Grothendieck wrote:
>>
>> You could try method="brute-force" in the nls2 package to
find starting
>> values.
>>
>> On Tue, Mar 30, 2010 at 7:03 AM, Corrado <ct529 at york.ac.uk>
wrote:
>>
>>>
>>> I am using nls to fit a non linear function to some data.
>>>
>>> The non linear function is:
>>>
>>> y= 1- exp(-(k0+k1*p1+ .... + kn*pn))
>>>
>>> I have chosen algorithm "port", with lower boundary is 0
for all of the
>>> ki
>>> parameters, and I have tried many start values for the parameters
ki
>>> (including generating them at random).
>>>
>>> If I fit the non linear function to the same data using an external
>>> algorithm, it fits perfectly and finds the parameters.
>>>
>>> As soon as I come to my R installation (2.10.1 on Kubuntu Linux 910
64
>>> bit),
>>> I keep getting the error:
>>>
>>> Error in nlsModel(formula, mf, start, wts, upper) : ? singular
gradient
>>> matrix at initial parameter estimates
>>>
>>> I have read all the previous postings and the documentation, but to
no
>>> avail: the error is there to stay. I am sure the problem is with
nls,
>>> because the external fitting algorithm perfectly fits it in less
than a
>>> second. Also, if my n is 4, then the nls works perfectly (but that
>>> excludes
>>> all the k5 .... kn).
>>>
>>> Can anyone help me with suggestions? Thanks in advance.
>>>
>>> Alternatively, what do you suggest I should do? Shall I abandon nls
in
>>> favour of optim?
>>>
>>> Regards
>>>
>>> --
>>> Corrado Topi
>>> PhD Researcher
>>> Global Climate Change and Biodiversity
>>> Area 18,Department of Biology
>>> University of York, York, YO10 5YW, UK
>>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>
>
> --
> Corrado Topi
> PhD Researcher
> Global Climate Change and Biodiversity
> Area 18,Department of Biology
> University of York, York, YO10 5YW, UK
> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>
>

Yes, of course. The problem still stays.

Gabor Grothendieck wrote:
> Sorry, its algorithm="brute-force"
>
> On Tue, Mar 30, 2010 at 10:29 AM, Corrado <ct529 at york.ac.uk>
wrote:
>   
>> Hi Gabor,
>>
>> same problem even using nls2 with method=brute-force to calculate the
>> initial parameters.
>>
>> Best,
>>
>> Gabor Grothendieck wrote:
>>     
>>> You could try method="brute-force" in the nls2 package to
find starting
>>> values.
>>>
>>> On Tue, Mar 30, 2010 at 7:03 AM, Corrado <ct529 at
york.ac.uk> wrote:
>>>
>>>       
>>>> I am using nls to fit a non linear function to some data.
>>>>
>>>> The non linear function is:
>>>>
>>>> y= 1- exp(-(k0+k1*p1+ .... + kn*pn))
>>>>
>>>> I have chosen algorithm "port", with lower boundary
is 0 for all of the
>>>> ki
>>>> parameters, and I have tried many start values for the
parameters ki
>>>> (including generating them at random).
>>>>
>>>> If I fit the non linear function to the same data using an
external
>>>> algorithm, it fits perfectly and finds the parameters.
>>>>
>>>> As soon as I come to my R installation (2.10.1 on Kubuntu Linux
910 64
>>>> bit),
>>>> I keep getting the error:
>>>>
>>>> Error in nlsModel(formula, mf, start, wts, upper) :   singular
gradient
>>>> matrix at initial parameter estimates
>>>>
>>>> I have read all the previous postings and the documentation,
but to no
>>>> avail: the error is there to stay. I am sure the problem is
with nls,
>>>> because the external fitting algorithm perfectly fits it in
less than a
>>>> second. Also, if my n is 4, then the nls works perfectly (but
that
>>>> excludes
>>>> all the k5 .... kn).
>>>>
>>>> Can anyone help me with suggestions? Thanks in advance.
>>>>
>>>> Alternatively, what do you suggest I should do? Shall I abandon
nls in
>>>> favour of optim?
>>>>
>>>> Regards
>>>>
>>>> --
>>>> Corrado Topi
>>>> PhD Researcher
>>>> Global Climate Change and Biodiversity
>>>> Area 18,Department of Biology
>>>> University of York, York, YO10 5YW, UK
>>>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible
code.
>>>>
>>>>
>>>>         
>> --
>> Corrado Topi
>> PhD Researcher
>> Global Climate Change and Biodiversity
>> Area 18,Department of Biology
>> University of York, York, YO10 5YW, UK
>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>
>>
>>     

-- 
Corrado Topi
PhD Researcher
Global Climate Change and Biodiversity
Area 18,Department of Biology
University of York, York, YO10 5YW, UK
Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk

What do you mean the problem still stays?   If you are using brute
force its not a problem to have it fail on some of the evaluations
since each one is separate.  How large a grid are you using?  Are you
claiming that every single point on the grid fails? Please provide
reproducible code showing what you are doing.

On Tue, Mar 30, 2010 at 10:56 AM, Corrado <ct529 at york.ac.uk>
wrote:
> Yes, of course. The problem still stays.
>
> Gabor Grothendieck wrote:
>>
>> Sorry, its algorithm="brute-force"
>>
>> On Tue, Mar 30, 2010 at 10:29 AM, Corrado <ct529 at york.ac.uk>
wrote:
>>
>>>
>>> Hi Gabor,
>>>
>>> same problem even using nls2 with method=brute-force to calculate
the
>>> initial parameters.
>>>
>>> Best,
>>>
>>> Gabor Grothendieck wrote:
>>>
>>>>
>>>> You could try method="brute-force" in the nls2
package to find starting
>>>> values.
>>>>
>>>> On Tue, Mar 30, 2010 at 7:03 AM, Corrado <ct529 at
york.ac.uk> wrote:
>>>>
>>>>
>>>>>
>>>>> I am using nls to fit a non linear function to some data.
>>>>>
>>>>> The non linear function is:
>>>>>
>>>>> y= 1- exp(-(k0+k1*p1+ .... + kn*pn))
>>>>>
>>>>> I have chosen algorithm "port", with lower
boundary is 0 for all of the
>>>>> ki
>>>>> parameters, and I have tried many start values for the
parameters ki
>>>>> (including generating them at random).
>>>>>
>>>>> If I fit the non linear function to the same data using an
external
>>>>> algorithm, it fits perfectly and finds the parameters.
>>>>>
>>>>> As soon as I come to my R installation (2.10.1 on Kubuntu
Linux 910 64
>>>>> bit),
>>>>> I keep getting the error:
>>>>>
>>>>> Error in nlsModel(formula, mf, start, wts, upper) : ?
singular gradient
>>>>> matrix at initial parameter estimates
>>>>>
>>>>> I have read all the previous postings and the
documentation, but to no
>>>>> avail: the error is there to stay. I am sure the problem is
with nls,
>>>>> because the external fitting algorithm perfectly fits it in
less than a
>>>>> second. Also, if my n is 4, then the nls works perfectly
(but that
>>>>> excludes
>>>>> all the k5 .... kn).
>>>>>
>>>>> Can anyone help me with suggestions? Thanks in advance.
>>>>>
>>>>> Alternatively, what do you suggest I should do? Shall I
abandon nls in
>>>>> favour of optim?
>>>>>
>>>>> Regards
>>>>>
>>>>> --
>>>>> Corrado Topi
>>>>> PhD Researcher
>>>>> Global Climate Change and Biodiversity
>>>>> Area 18,Department of Biology
>>>>> University of York, York, YO10 5YW, UK
>>>>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>>>>
>>>>> ______________________________________________
>>>>> R-help at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>> PLEASE do read the posting guide
>>>>> http://www.R-project.org/posting-guide.html
>>>>> and provide commented, minimal, self-contained,
reproducible code.
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Corrado Topi
>>> PhD Researcher
>>> Global Climate Change and Biodiversity
>>> Area 18,Department of Biology
>>> University of York, York, YO10 5YW, UK
>>> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>>>
>>>
>>>
>
>
> --
> Corrado Topi
> PhD Researcher
> Global Climate Change and Biodiversity
> Area 18,Department of Biology
> University of York, York, YO10 5YW, UK
> Phone: + 44 (0) 1904 328645, E-mail: ct529 at york.ac.uk
>
>

If you have a perfect fit, you have zero residuals. But in the nls manual page
we have:
> Warning:
> 
>      *Do not use ?nls? on artificial "zero-residual" data.*
So this is a case of complaining that your diesel car is broken because you
ignored the
"Diesel fuel only" sign on the filler cap and put in gasoline.

However I've not been happy with this choice in the code of nls -- it's
been there a long
time -- and my own codes from 1974 onwards have always handled zero residual
cases. I do
believe that the code could at least give a better diagnostic message. Zero
residuals --
perfect fits -- arise when one is interested more or less in an interpolating
function
rather than doing statistics, and I can understand the reluctance of
statisticians to
countenance such a use of nls.

And Bert's comment on overparametrization is almost certainly valid also.

JN