R devel - Mar 2009 - lsfit w/ rank-deficient x

Dear R-devel,

It seems that lsfit incorrectly reports coefficients when the input matrix
'x' is rank-deficient, see the example below:

## here values of 'b' and 'c' are incorrectly
swapped
> x <- cbind(a=rnorm(100), b=0, c=rnorm(100)); y <- rnorm(100);
lsfit(x, y)$coef
 Intercept          a          b          c
-0.0227787  0.1042860 -0.1729261  0.0000000
Warning message:
In lsfit(x, y) : 'X' matrix was collinear

## correct values
> lsfit(x[,-2], y)$coef
 Intercept          a          c
-0.0227787  0.1042860 -0.1729261


I looked inside the lsfit code and it appears that even though rank-deficiency
is detected there is no attempt to patch the coefficients. Why is that?

Taking clues from the code it appears that the following trick might do the
work:
> foo <- lsfit(x, y)
Warning message:
In lsfit(x, y) : 'X' matrix was collinear
> structure(foo$coefficients[foo$qr$pivot], names=names(foo$coefficients))
  Intercept           a           b           c
 0.14857345 -0.07473099  0.00000000  0.12835155


Is this reliable or there are cases when it may fail?

Thanks,
Vadim

P.S.
> version
               _
platform       i386-pc-mingw32
arch           i386
os             mingw32
system         i386, mingw32
status
major          2
minor          7.1
year           2008
month          06
day            23
svn rev        45970
language       R
version.string R version 2.7.1 (2008-06-23)
>
Note: This email is for the confidential use of the named addressee(s) only and
may contain proprietary, confidential or privileged information. If you are not
the intended recipient, you are hereby notified that any review, dissemination
or copying of this email is strictly prohibited, and to please notify the sender
immediately and destroy this email and any attachments.  Email transmission
cannot be guaranteed to be secure or error-free.  Jump Trading, therefore, does
not make any guarantees as to the completeness or accuracy of this email or any
attachments.  This email is for informational purposes only and does not
constitute a recommendation, offer, request or solicitation of any kind to buy,
sell, subscribe, redeem or perform any type of transaction of a financial
product.

Actually, the correct permutation is given by the inverse of qr$pivot:

foo$coefficients[foo$qr$pivot] <- foo$coefficients

Here foo is an object returned by lsfit, see below.



-----Original Message-----
From: Vadim Ogranovich
Sent: Friday, March 13, 2009 5:25 PM
To: 'r-devel at r-project.org'
Subject: lsfit w/ rank-deficient x

Dear R-devel,

It seems that lsfit incorrectly reports coefficients when the input matrix
'x' is rank-deficient, see the example below:

## here values of 'b' and 'c' are incorrectly
swapped
> x <- cbind(a=rnorm(100), b=0, c=rnorm(100)); y <- rnorm(100);
lsfit(x, y)$coef
 Intercept          a          b          c
-0.0227787  0.1042860 -0.1729261  0.0000000
Warning message:
In lsfit(x, y) : 'X' matrix was collinear

## correct values
> lsfit(x[,-2], y)$coef
 Intercept          a          c
-0.0227787  0.1042860 -0.1729261


I looked inside the lsfit code and it appears that even though rank-deficiency
is detected there is no attempt to patch the coefficients. Why is that?

Taking clues from the code it appears that the following trick might do the
work:
> foo <- lsfit(x, y)
Warning message:
In lsfit(x, y) : 'X' matrix was collinear
> structure(foo$coefficients[foo$qr$pivot], names=names(foo$coefficients))
  Intercept           a           b           c
 0.14857345 -0.07473099  0.00000000  0.12835155


Is this reliable or there are cases when it may fail?

Thanks,
Vadim

P.S.
> version
               _
platform       i386-pc-mingw32
arch           i386
os             mingw32
system         i386, mingw32
status
major          2
minor          7.1
year           2008
month          06
day            23
svn rev        45970
language       R
version.string R version 2.7.1 (2008-06-23)
>
Note: This email is for the confidential use of the named addressee(s) only and
may contain proprietary, confidential or privileged information. If you are not
the intended recipient, you are hereby notified that any review, dissemination
or copying of this email is strictly prohibited, and to please notify the sender
immediately and destroy this email and any attachments.  Email transmission
cannot be guaranteed to be secure or error-free.  Jump Trading, therefore, does
not make any guarantees as to the completeness or accuracy of this email or any
attachments.  This email is for informational purposes only and does not
constitute a recommendation, offer, request or solicitation of any kind to buy,
sell, subscribe, redeem or perform any type of transaction of a financial
product.