R help - Jun 2002 - how to skip NA columns ?

R-helpers!

na.omit() can be used to remove rows with NA's
but how can I remove columns ? and remember, which columns have been removed
?

I guess I can do t(na.omit(t(o))) as shown below, but this probably creates
a lot of overhead and I do not know which columns
have been removed.

Yours,

R
> o  
     [,1] [,2] [,3]
[1,]    1   NA    7
[2,]    2   NA    8
[3,]    3   NA    9
> t(na.omit(t(o)))
     [,1] [,2]
[1,]    1    7
[2,]    2    8
[3,]    3    9

Ryszard Czerminski   phone: (781)994-0479
ArQule, Inc.         email:ryszard at arqule.com
19 Presidential Way  http://www.arqule.com
Woburn, MA 01801     fax: (781)994-0679

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Columns of what?  A data frame or a matrix?

For a data frame DF

hasna <- sapply(DF, function(x) any(is.na(x)))
DF <- DF[!hasna]

and `hasna' tells which you removed.  Use apply and matrix indexing for a
matrix.

On Thu, 20 Jun 2002, Czerminski, Ryszard wrote:
> R-helpers!
>
> na.omit() can be used to remove rows with NA's
> but how can I remove columns ? and remember, which columns have been
removed
> ?
>
> I guess I can do t(na.omit(t(o))) as shown below, but this probably creates
> a lot of overhead and I do not know which columns
> have been removed.
It only works for matrices, too.
> Yours,
>
> R
>
> > o
>      [,1] [,2] [,3]
> [1,]    1   NA    7
> [2,]    2   NA    8
> [3,]    3   NA    9
> > t(na.omit(t(o)))
>      [,1] [,2]
> [1,]    1    7
> [2,]    2    8
> [3,]    3    9
>
> Ryszard Czerminski   phone: (781)994-0479
> ArQule, Inc.         email:ryszard at arqule.com
> 19 Presidential Way  http://www.arqule.com
> Woburn, MA 01801     fax: (781)994-0679
>
>
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
> r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
> Send "info", "help", or "[un]subscribe"
> (in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
>
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Ryszard,

1) with your example, I get:
>t(na.omit(t(o)))
     [,1] [,2]
[1,]    1    7
[2,]    2    8
[3,]    3    9
attr(,"na.action")
[1] 2
attr(,"na.action")attr(,"class")
[1] "omit"
>
in which the first attr statement seems to say that the old column 2 has
been removed
( and this information can be recovered with things like:
   > t<-t(na.omit(t(o)))
   > attr(t,"na.action")[1]
)

2) are you sure your solution is so bad? I did:
> o<-matrix(c(1,2,3,rep(NA,3),7,8,9),3,3)
> ot<-t(o)
> date();for(i in 1:10000) t(na.omit(t(o)));date()
[1] "Thu Jun 20 14:48:53 2002"
[1] "Thu Jun 20 14:49:02 2002"
> date();for(i in 1:10000) na.omit(ot);date()
[1] "Thu Jun 20 14:49:13 2002"
[1] "Thu Jun 20 14:49:20 2002"
>
which seems to suggest that (for this toy example at least) the transposes
are taking around 30% of the time. Similarly for a slightly bigger example:
> o100<-matrix(c(1,NA),100,100,byrow=TRUE)
> ot100<-t(o100)
> date();for(i in 1:1000) t(na.omit(t(o100)));date()
[1] "Thu Jun 20 14:54:40 2002"
[1] "Thu Jun 20 14:54:52 2002"
> date();for(i in 1:1000) na.omit(o100);date()
[1] "Thu Jun 20 14:55:10 2002"
[1] "Thu Jun 20 14:55:19 2002"
>
While the exact times will be machine dependent, I'd guess the ratios are
likely to be fairly consistent. So, unless your code does lots of this and
little else, I'd suggest finding something else to fret about!

Cheers,

Mike.

     > -----Original Message-----
     > From: owner-r-help at stat.math.ethz.ch
     > [mailto:owner-r-help at stat.math.ethz.ch]On Behalf Of
     > Czerminski, Ryszard
     > Sent: 20 June 2002 12:58
     > To: r-help at stat.math.ethz.ch
     > Subject: [R] how to skip NA columns ?
     >
     >
     > R-helpers!
     >
     > na.omit() can be used to remove rows with NA's
     > but how can I remove columns ? and remember, which columns
     > have been removed
     > ?
     >
     > I guess I can do t(na.omit(t(o))) as shown below, but this
     > probably creates
     > a lot of overhead and I do not know which columns
     > have been removed.
     >
     > Yours,
     >
     > R
     >
     > > o
     >      [,1] [,2] [,3]
     > [1,]    1   NA    7
     > [2,]    2   NA    8
     > [3,]    3   NA    9
     > > t(na.omit(t(o)))
     >      [,1] [,2]
     > [1,]    1    7
     > [2,]    2    8
     > [3,]    3    9
     >
     > Ryszard Czerminski   phone: (781)994-0479
     > ArQule, Inc.         email:ryszard at arqule.com
     > 19 Presidential Way  http://www.arqule.com
     > Woburn, MA 01801     fax: (781)994-0679
     >
     > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
     > .-.-.-.-.-.-.-.-.-
     > r-help mailing list -- Read
http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.
_._

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

It works for me for data frame very nicely, but I am not quite sure how to
get it to work for a matrix ?

R
> df
  V1 V2 V3
1  1 NA  7
2  2 NA  8
3  3 NA  9
> mask <- sapply(df, function(x) any(is.na(x)))
> df[!mask]
  V1 V3
1  1  7
2  2  8
3  3  9
> m <- as.matrix(df)
> m
  V1 V2 V3
1  1 NA  7
2  2 NA  8
3  3 NA  9
> mask <- apply(m, 2, function(x) any(is.na(x)))
> m[!mask]
   1    3 <NA> <NA> <NA> <NA> 
   1    3   NA   NA    7    9 
> dim(m[!mask])
NULL
>


Ryszard Czerminski   phone: (781)994-0479
ArQule, Inc.         email:ryszard at arqule.com
19 Presidential Way  http://www.arqule.com
Woburn, MA 01801     fax: (781)994-0679


-----Original Message-----
From: Prof Brian D Ripley [mailto:ripley at stats.ox.ac.uk]
Sent: Thursday, June 20, 2002 8:14 AM
To: Czerminski, Ryszard
Cc: r-help at stat.math.ethz.ch
Subject: Re: [R] how to skip NA columns ?


Columns of what?  A data frame or a matrix?

For a data frame DF

hasna <- sapply(DF, function(x) any(is.na(x)))
DF <- DF[!hasna]

and `hasna' tells which you removed.  Use apply and matrix indexing for a
matrix.

On Thu, 20 Jun 2002, Czerminski, Ryszard wrote:
> R-helpers!
>
> na.omit() can be used to remove rows with NA's
> but how can I remove columns ? and remember, which columns have been
removed
> ?
>
> I guess I can do t(na.omit(t(o))) as shown below, but this probably
creates
> a lot of overhead and I do not know which columns
> have been removed.
It only works for matrices, too.
> Yours,
>
> R
>
> > o
>      [,1] [,2] [,3]
> [1,]    1   NA    7
> [2,]    2   NA    8
> [3,]    3   NA    9
> > t(na.omit(t(o)))
>      [,1] [,2]
> [1,]    1    7
> [2,]    2    8
> [3,]    3    9
>
> Ryszard Czerminski   phone: (781)994-0479
> ArQule, Inc.         email:ryszard at arqule.com
> 19 Presidential Way  http://www.arqule.com
> Woburn, MA 01801     fax: (781)994-0679
>
>
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.
-.-
> r-help mailing list -- Read
http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
> Send "info", "help", or "[un]subscribe"
> (in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
>
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.
_._
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.
-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._.
_._

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

"Czerminski, Ryszard" wrote:
> 
> It works for me for data frame very nicely, but I am not quite sure how to
> get it to work for a matrix ?
> 
> R
> 
> > df
>   V1 V2 V3
> 1  1 NA  7
> 2  2 NA  8
> 3  3 NA  9
> > mask <- sapply(df, function(x) any(is.na(x)))
> > df[!mask]
>   V1 V3
> 1  1  7
> 2  2  8
> 3  3  9
> > m <- as.matrix(df)
> > m
>   V1 V2 V3
> 1  1 NA  7
> 2  2 NA  8
> 3  3 NA  9
> > mask <- apply(m, 2, function(x) any(is.na(x)))
> > m[!mask]
>    1    3 <NA> <NA> <NA> <NA>
>    1    3   NA   NA    7    9
> > dim(m[!mask])
> NULL
A comma will help:

  m[ , !mask]  

Uwe
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._