sparandekar
2011-Dec-26 10:44 UTC
[R] Summary tables of large datasets including character and numerical variables
Hello ! I am attempting to switch from being a long time SAS user to R, and would really appreciate a bit of help ! The first thing I do in getting a large dataset (thousands of obervations and hundreds of variables) is to run a SAS command PROC CONTENTS VARNUM command - this provides me a table with the name of each variable, its type and length; then I run a PROC MEANS - for numerical variables it gives me a table with the number of non-missing values, min, max, mean and std. dev. My data usually has errors and this first step helps me to spot the errors and 'clean' the dataset. The 'summary' function in R and other function as part of Hmisc or Psych package do not work for me. How can I get a table from an R data.frame that has the following structure (header row and example). Rowname Character/Integer Length Non-Missing Minimum Maximum Mean SD HHID Integer 12 32,344 114455007701 514756007812 2.345 x 10^10 1.456 x 10^10 Head Character 38 24,566 - - - - Thank you very much. -- View this message in context: http://r.789695.n4.nabble.com/Summary-tables-of-large-datasets-including-character-and-numerical-variables-tp4234296p4234296.html Sent from the R help mailing list archive at Nabble.com.
John Kane
2011-Dec-26 23:01 UTC
[R] Summary tables of large datasets including character and numerical variables
It would be? very helpful to have an actual sample of your data. As usual in R there are probably several different ways to approach the problem but a small sample of the data or a mock-up would be most helpful. Probably the easiest way to supply some data would be something like df1 <- mydata[, 100] dput(df1) and paste the resulting output into a message.?? Sorry I'm not more helpful but seeing real data makes life a lot easier. ----- Original Message ----- From: sparandekar <sparandekar at yahoo.com.ar> To: r-help at r-project.org Cc: Sent: Monday, December 26, 2011 5:44:53 AM Subject: [R] Summary tables of large datasets including character and numerical variables Hello ! I am attempting to switch from being a long time SAS user to R, and would really appreciate a bit of help ! The first thing I do in getting a large dataset (thousands of obervations and hundreds of variables) is to run a SAS command PROC CONTENTS VARNUM command - this provides me a table with the name of each variable, its type and length;? then I run a PROC MEANS - for numerical variables it gives me a table with the number of non-missing values, min, max, mean and std. dev.? My data usually has errors and this first step helps me to spot the errors and 'clean' the dataset. The 'summary' function in R and other function as part of Hmisc or Psych package do not work for me. How can I get a table from an R data.frame that has the following structure (header row and example). Rowname? Character/Integer? Length?? Non-Missing? ? Minimum? ? ? ? ? Maximum? ? ? ? ? ? ? Mean? ? ? ? ? ? ? ? ?? SD HHID? ? ? ? ? ? Integer? ? ? ? ? ? ? ? ? ? ?? 12? ? ? ? ? ? 32,344? ? ? ? ? ? ? 114455007701?? 514756007812? ? ?? 2.345 x 10^10? ? 1.456 x 10^10 Head? ? ? ? ? ? Character? ? ? ? ? ? ? ? ?? 38? ? ? ? ? ? 24,566? ? ? ? ? ? ? ? ? ? ? -? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? -? ? ? ? ? ? ? ? ? ? ? ?? -? ? ? ? ? ? ? ? ? ? ? ? ? - Thank you very much. -- View this message in context: http://r.789695.n4.nabble.com/Summary-tables-of-large-datasets-including-character-and-numerical-variables-tp4234296p4234296.html Sent from the R help mailing list archive at Nabble.com. ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Duncan Murdoch
2011-Dec-27 02:56 UTC
[R] Summary tables of large datasets including character and numerical variables
On 11-12-26 5:44 AM, sparandekar wrote:> Hello ! > > I am attempting to switch from being a long time SAS user to R, and would > really appreciate a bit of help ! The first thing I do in getting a large > dataset (thousands of obervations and hundreds of variables) is to run a SAS > command PROC CONTENTS VARNUM command - this provides me a table with the > name of each variable, its type and length; then I run a PROC MEANS - for > numerical variables it gives me a table with the number of non-missing > values, min, max, mean and std. dev. My data usually has errors and this > first step helps me to spot the errors and 'clean' the dataset. > > The 'summary' function in R and other function as part of Hmisc or Psych > package do not work for me. > > How can I get a table from an R data.frame that has the following structure > (header row and example). > > Rowname Character/Integer Length Non-Missing Minimum > Maximum Mean SD > > HHID Integer 12 32,344 > 114455007701 514756007812 2.345 x 10^10 1.456 x 10^10 > Head Character 38 24,566 > - - - > -Using the tables package, you can get something like that as follows, assuming that "df" is your dataframe: nonmissing <- function(x) sum(!is.na(x)) tabular(All(df, character=TRUE) ~ (typeof + length + nonmissing + min + max + mean + sd)) It isn't perfect: it will skip anything that isn't numeric or character (e.g. factors). There are ways to work around that, but they aren't as simple as you might like. You can also use sapply(df, class) to see the classes of all the columns. Duncan Murdoch
David Winsemius
2011-Dec-27 03:38 UTC
[R] Summary tables of large datasets including character and numerical variables
On Dec 26, 2011, at 5:44 AM, sparandekar wrote:> Hello ! > > I am attempting to switch from being a long time SAS user to R, and > would > really appreciate a bit of help ! The first thing I do in getting a > large > dataset (thousands of obervations and hundreds of variables) is to > run a SAS > command PROC CONTENTS VARNUM command - this provides me a table with > the > name of each variable, its type and length; then I run a PROC MEANS > - for > numerical variables it gives me a table with the number of non-missing > values, min, max, mean and std. dev. My data usually has errors and > this > first step helps me to spot the errors and 'clean' the dataset. > > The 'summary' function in R and other function as part of Hmisc or > Psych > package do not work for me. > > How can I get a table from an R data.frame that has the following > structure > (header row and example). > > Rowname Character/Integer Length Non-Missing Minimum > Maximum Mean SD > > HHID Integer 12 32,344 > 114455007701 514756007812 2.345 x 10^10 1.456 x 10^10 > Head Character 38 24,566 > - - - > -I generally use ( in order of increasing information content and increasing length of output): names(dfrm) str(dfrm) Hmisc::describe(dfrm) (Several other packages have their own versions of 'describe'.) -- David Winsemius, MD West Hartford, CT
John Kane
2011-Dec-27 12:36 UTC
[R] Summary tables of large datasets including character and numerical variables
A function like the one below will give you the class and number of valid
entries for a dataset.? At sample data set would help determine if it works.
It works on a simple data set I created and one from the ggplot2 package but it
is
not really tested.
With your data set as df1 something like
xx <- msum(df1)
should supply you with the information.
Other than that I really don't see why Hmisc :: describe() would not give
you
most or all the information you seek with the exception of the
"length"thing which
really don't understand. Actually I believe describe() will give you
the number of non-NA entries for a variable.
=========================================================msum <-
function(dat1) {
nams <- c("class", "non-missing")
cl <- lapply(dat1, class)
nas <-colSums(!is.na(dat1))
mysum <- data.frame(unlist(cl), unlist(nas))
names(mysum) <- nams
mysum <- mysum
}
==============================================================
----- Original Message -----
From: sparandekar <sparandekar at yahoo.com.ar>
To: r-help at r-project.org
Cc:
Sent: Monday, December 26, 2011 5:44:53 AM
Subject: [R] Summary tables of large datasets including character and numerical
variables
Hello !
I am attempting to switch from being a long time SAS user to R, and would
really appreciate a bit of help ! The first thing I do in getting a large
dataset (thousands of obervations and hundreds of variables) is to run a SAS
command PROC CONTENTS VARNUM command - this provides me a table with the
name of each variable, its type and length;? then I run a PROC MEANS - for
numerical variables it gives me a table with the number of non-missing
values, min, max, mean and std. dev.? My data usually has errors and this
first step helps me to spot the errors and 'clean' the dataset.
The 'summary' function in R and other function as part of Hmisc or Psych
package do not work for me.
How can I get a table from an R data.frame that has the following structure
(header row and example).
Rowname? Character/Integer? Length? Non-Missing? ? Minimum? ? ? ? ?
Maximum? ? ? ? ? ? ? Mean? ? ? ? ? ? ? ? ? SD
HHID? ? ? ? ? ? Integer? ? ? ? ? ? ? ? ? ? ? 12? ? ? ? ? ? 32,344? ? ? ? ? ? ?
114455007701? 514756007812? ? ? 2.345 x 10^10? ? 1.456 x 10^10
Head? ? ? ? ? ? Character? ? ? ? ? ? ? ? ? 38? ? ? ? ? ? 24,566? ? ? ? ? ? ? ?
? ? ?
-? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? -? ? ? ? ? ? ? ? ? ? ? ? -? ? ? ? ? ? ? ? ?
? ? ? ?
-
Thank you very much.
--
View this message in context:
http://r.789695.n4.nabble.com/Summary-tables-of-large-datasets-including-character-and-numerical-variables-tp4234296p4234296.html
Sent from the R help mailing list archive at Nabble.com.
______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.