Jeff Newmiller
2017-Mar-31 21:49 UTC
[R] Taking the sum of only some columns of a data frame
You can also look at the knitr-RMarkdown work flow, or the knitr-latex work flow. In both of these it is reasonable to convert your data frame to a temporary character-only form purely for output purposes. However, one can usually use an existing function to push your results out without damaging your working data. It is important to separate your data from your output because mixing results (totals) with data makes using the data further extremely difficult. Mixing them is one of the major flaws of the spreadsheet model of computation, and it causes problems there as well as in R. -- Sent from my phone. Please excuse my brevity. On March 31, 2017 1:05:09 PM PDT, William Michels via R-help <r-help at r-project.org> wrote:>Again, you should always copy the R-help list on replies to your OP. > >The short answer is you **shouldn't** replace NAs with blanks in your >matrix or dataframe. NA is the proper designation for those cell >positions. Replacing NA with a "blank" in a dataframe will convert >that column to a "character" mode, precluding further numeric >manipulation of those columns. > >Consider your workflow: are you tying to export a table? If so, take >a look at installing pander (see 'missing' argument on webpage below): > >https://cran.r-project.org/web/packages/pander/README.html > >Finally, please review the Introductory PDF, available here: > >https://cran.r-project.org/doc/manuals/R-intro.pdf > >HTH, Bill. > >William Michels, Ph.D. > > > >On Fri, Mar 31, 2017 at 11:21 AM, BR_email <br at dmstat1.com> wrote: >> William: >> How can I replace the "NAs" with blanks? >> Bruce >> >> Bruce Ratner, Ph.D. >> The Significant Statistician? >> >> >> William Michels wrote: >>> >>> I'm sure there are more efficient ways, but this works: >>> >>>> test1 <- matrix(runif(50), nrow=10, ncol=5) >>>> ## test1 <- as.data.frame(test1) >>>> test1 <- rbind(test1, NA) >>>> test1[11, c(1,3)] <- colSums(test1[1:10,c(1,3)]) >>>> test1 >>> >>> >>> HTH, >>> >>> Bill. >>> >>> William Michels, Ph.D. >>> >>> >>> >>> On Fri, Mar 31, 2017 at 9:20 AM, Bruce Ratner PhD <br at dmstat1.com> >wrote: >>>> >>>> Hi R'ers: >>>> Given a data.frame of five columns and ten rows. >>>> I would like to take the sum of, say, the first and third columns >only. >>>> For the remaining columns, I do not want any calculations, thus >rending >>>> their "values" on the "total" row blank. The sum/total row is to be >combined >>>> to the original data.frame, yielding a data.frame with five columns >and >>>> eleven rows. >>>> >>>> Thanks, in advance. >>>> Bruce >>>> >>>> >>>> ______________ >>>> Bruce Ratner PhD >>>> The Significant Statistician? >>>> >>>> >>>> >>>> >>>> [[alternative HTML version deleted]] >>>> >>>> ______________________________________________ >>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>>> https://stat.ethz.ch/mailman/listinfo/r-help >>>> PLEASE do read the posting guide >>>> http://www.R-project.org/posting-guide.html >>>> and provide commented, minimal, self-contained, reproducible code. >>> >>> >>> >> > >______________________________________________ >R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >https://stat.ethz.ch/mailman/listinfo/r-help >PLEASE do read the posting guide >http://www.R-project.org/posting-guide.html >and provide commented, minimal, self-contained, reproducible code.
All: 1. I agree wholeheartedly with prior responses. 2. But let's suppose that for some reason, you *did* want to carry around some "calculated values" with the data frame. Then one way to do it is to add them as attributes to the data frame. This way they cannot "pollute" the data in the way Jeff warned against; e.g. attr(your_frame,"colsums") <- colSums(your_frame) This of course calculates them all, but you can of course just attach some (e.g. colSums(your_frame[,c(1,3)] ) 3. This, of course, has the disadvantage of requiring recalculation of the attribute if the data changes, which is an invitation to problems. A better approach might be to attach the *function* that does the calculation as an attribute, which when invoked always uses the current data: attr(your_frame,"colsums") <- function(x)colSums(x) For example: df <- data.frame(x=1:5,y=21:25) attr(df,"colsums")<- function(x)colSums(x) ## then:> attr(df,"colsums")(df)x y 15 115 ## add a row> df[6,] <- rep(100,2) > attr(df,"colsums")(df)x y 115 215 This survives changing the name of df:> dat <- df > attr(dat,"colsums")(dat)x y 115 215 As it stands, the call: attr(df,"colsums")(df) is a bit clumsy; one could easily write a function that does this sort of thing more cleanly, as, for example, is done via the "selfStart" functionality for nonlinear models. But all this presupposes that the OP is familiar with R programming paradigms, especially the use of functions as first class objects, and the language in general. While I may have missed this, his posts do not seem to me to indicate such familiarity, so as others have suggested, perhaps the best answer is to first spend some time with an R tutorial or two and *not* try to mimic bad spreadsheet practices in R. Cheers, Bert Bert Gunter "The trouble with having an open mind is that people keep coming along and sticking things into it." -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip ) On Fri, Mar 31, 2017 at 2:49 PM, Jeff Newmiller <jdnewmil at dcn.davis.ca.us> wrote:> You can also look at the knitr-RMarkdown work flow, or the knitr-latex work flow. In both of these it is reasonable to convert your data frame to a temporary character-only form purely for output purposes. However, one can usually use an existing function to push your results out without damaging your working data. > > It is important to separate your data from your output because mixing results (totals) with data makes using the data further extremely difficult. Mixing them is one of the major flaws of the spreadsheet model of computation, and it causes problems there as well as in R. > -- > Sent from my phone. Please excuse my brevity. > > On March 31, 2017 1:05:09 PM PDT, William Michels via R-help <r-help at r-project.org> wrote: >>Again, you should always copy the R-help list on replies to your OP. >> >>The short answer is you **shouldn't** replace NAs with blanks in your >>matrix or dataframe. NA is the proper designation for those cell >>positions. Replacing NA with a "blank" in a dataframe will convert >>that column to a "character" mode, precluding further numeric >>manipulation of those columns. >> >>Consider your workflow: are you tying to export a table? If so, take >>a look at installing pander (see 'missing' argument on webpage below): >> >>https://cran.r-project.org/web/packages/pander/README.html >> >>Finally, please review the Introductory PDF, available here: >> >>https://cran.r-project.org/doc/manuals/R-intro.pdf >> >>HTH, Bill. >> >>William Michels, Ph.D. >> >> >> >>On Fri, Mar 31, 2017 at 11:21 AM, BR_email <br at dmstat1.com> wrote: >>> William: >>> How can I replace the "NAs" with blanks? >>> Bruce >>> >>> Bruce Ratner, Ph.D. >>> The Significant Statistician? >>> >>> >>> William Michels wrote: >>>> >>>> I'm sure there are more efficient ways, but this works: >>>> >>>>> test1 <- matrix(runif(50), nrow=10, ncol=5) >>>>> ## test1 <- as.data.frame(test1) >>>>> test1 <- rbind(test1, NA) >>>>> test1[11, c(1,3)] <- colSums(test1[1:10,c(1,3)]) >>>>> test1 >>>> >>>> >>>> HTH, >>>> >>>> Bill. >>>> >>>> William Michels, Ph.D. >>>> >>>> >>>> >>>> On Fri, Mar 31, 2017 at 9:20 AM, Bruce Ratner PhD <br at dmstat1.com> >>wrote: >>>>> >>>>> Hi R'ers: >>>>> Given a data.frame of five columns and ten rows. >>>>> I would like to take the sum of, say, the first and third columns >>only. >>>>> For the remaining columns, I do not want any calculations, thus >>rending >>>>> their "values" on the "total" row blank. The sum/total row is to be >>combined >>>>> to the original data.frame, yielding a data.frame with five columns >>and >>>>> eleven rows. >>>>> >>>>> Thanks, in advance. >>>>> Bruce >>>>> >>>>> >>>>> ______________ >>>>> Bruce Ratner PhD >>>>> The Significant Statistician? >>>>> >>>>> >>>>> >>>>> >>>>> [[alternative HTML version deleted]] >>>>> >>>>> ______________________________________________ >>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>>>> https://stat.ethz.ch/mailman/listinfo/r-help >>>>> PLEASE do read the posting guide >>>>> http://www.R-project.org/posting-guide.html >>>>> and provide commented, minimal, self-contained, reproducible code. >>>> >>>> >>>> >>> >> >>______________________________________________ >>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>https://stat.ethz.ch/mailman/listinfo/r-help >>PLEASE do read the posting guide >>http://www.R-project.org/posting-guide.html >>and provide commented, minimal, self-contained, reproducible code. > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.
William Michels
2017-Apr-01 01:52 UTC
[R] Taking the sum of only some columns of a data frame
Thank you Jeff for pointing out bad spreadsheet practices in R,
seconded by Mathew and Bert.
I should have considered creating a second dataframe ("test1_summary")
to distinguish raw from processed data. Those who want to address
memory issues caused by unnecessary duplication, feel free to chime
in.
Finally, thank you Bert for your most informative post on adding
attributes to dataframes. I really learned a lot!
Best Regards,
Bill.
William Michels, Ph.D.
On Fri, Mar 31, 2017 at 4:59 PM, Bert Gunter <bgunter.4567 at gmail.com>
wrote:> All:
>
> 1. I agree wholeheartedly with prior responses.
>
> 2. But let's suppose that for some reason, you *did* want to carry
> around some "calculated values" with the data frame. Then one way
to
> do it is to add them as attributes to the data frame. This way they
> cannot "pollute" the data in the way Jeff warned against; e.g.
>
> attr(your_frame,"colsums") <- colSums(your_frame)
>
> This of course calculates them all, but you can of course just attach
> some (e.g. colSums(your_frame[,c(1,3)] )
>
> 3. This, of course, has the disadvantage of requiring recalculation of
> the attribute if the data changes, which is an invitation to problems.
> A better approach might be to attach the *function* that does the
> calculation as an attribute, which when invoked always uses the
> current data:
>
> attr(your_frame,"colsums") <- function(x)colSums(x)
>
> For example:
>
> df <- data.frame(x=1:5,y=21:25)
> attr(df,"colsums")<- function(x)colSums(x)
>
> ## then:
>> attr(df,"colsums")(df)
> x y
> 15 115
>
> ## add a row
>> df[6,] <- rep(100,2)
>> attr(df,"colsums")(df)
> x y
> 115 215
>
>
> This survives changing the name of df:
>
>> dat <- df
>> attr(dat,"colsums")(dat)
> x y
> 115 215
>
> As it stands, the call: attr(df,"colsums")(df) is a bit clumsy;
one
> could easily write a function that does this sort of thing more
> cleanly, as, for example, is done via the "selfStart"
functionality
> for nonlinear models.
>
> But all this presupposes that the OP is familiar with R programming
> paradigms, especially the use of functions as first class objects, and
> the language in general. While I may have missed this, his posts do
> not seem to me to indicate such familiarity, so as others have
> suggested, perhaps the best answer is to first spend some time with an
> R tutorial or two and *not* try to mimic bad spreadsheet practices in
> R.
>
> Cheers,
> Bert
>
>
> Bert Gunter
>
> "The trouble with having an open mind is that people keep coming along
> and sticking things into it."
> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip
)
>
>
> On Fri, Mar 31, 2017 at 2:49 PM, Jeff Newmiller
> <jdnewmil at dcn.davis.ca.us> wrote:
>> You can also look at the knitr-RMarkdown work flow, or the knitr-latex
work flow. In both of these it is reasonable to convert your data frame to a
temporary character-only form purely for output purposes. However, one can
usually use an existing function to push your results out without damaging your
working data.
>>
>> It is important to separate your data from your output because mixing
results (totals) with data makes using the data further extremely difficult.
Mixing them is one of the major flaws of the spreadsheet model of computation,
and it causes problems there as well as in R.
>> --
>> Sent from my phone. Please excuse my brevity.
>>
>> On March 31, 2017 1:05:09 PM PDT, William Michels via R-help <r-help
at r-project.org> wrote:
>>>Again, you should always copy the R-help list on replies to your OP.
>>>
>>>The short answer is you **shouldn't** replace NAs with blanks in
your
>>>matrix or dataframe. NA is the proper designation for those cell
>>>positions. Replacing NA with a "blank" in a dataframe will
convert
>>>that column to a "character" mode, precluding further
numeric
>>>manipulation of those columns.
>>>
>>>Consider your workflow: are you tying to export a table? If so,
take
>>>a look at installing pander (see 'missing' argument on
webpage below):
>>>
>>>https://cran.r-project.org/web/packages/pander/README.html
>>>
>>>Finally, please review the Introductory PDF, available here:
>>>
>>>https://cran.r-project.org/doc/manuals/R-intro.pdf
>>>
>>>HTH, Bill.
>>>
>>>William Michels, Ph.D.
>>>
>>>
>>>
>>>On Fri, Mar 31, 2017 at 11:21 AM, BR_email <br at dmstat1.com>
wrote:
>>>> William:
>>>> How can I replace the "NAs" with blanks?
>>>> Bruce
>>>>
>>>> Bruce Ratner, Ph.D.
>>>> The Significant Statistician?
>>>>
>>>>
>>>> William Michels wrote:
>>>>>
>>>>> I'm sure there are more efficient ways, but this works:
>>>>>
>>>>>> test1 <- matrix(runif(50), nrow=10, ncol=5)
>>>>>> ## test1 <- as.data.frame(test1)
>>>>>> test1 <- rbind(test1, NA)
>>>>>> test1[11, c(1,3)] <- colSums(test1[1:10,c(1,3)])
>>>>>> test1
>>>>>
>>>>>
>>>>> HTH,
>>>>>
>>>>> Bill.
>>>>>
>>>>> William Michels, Ph.D.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 31, 2017 at 9:20 AM, Bruce Ratner PhD <br at
dmstat1.com>
>>>wrote:
>>>>>>
>>>>>> Hi R'ers:
>>>>>> Given a data.frame of five columns and ten rows.
>>>>>> I would like to take the sum of, say, the first and
third columns
>>>only.
>>>>>> For the remaining columns, I do not want any
calculations, thus
>>>rending
>>>>>> their "values" on the "total" row
blank. The sum/total row is to be
>>>combined
>>>>>> to the original data.frame, yielding a data.frame with
five columns
>>>and
>>>>>> eleven rows.
>>>>>>
>>>>>> Thanks, in advance.
>>>>>> Bruce
>>>>>>
>>>>>>
>>>>>> ______________
>>>>>> Bruce Ratner PhD
>>>>>> The Significant Statistician?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> [[alternative HTML version deleted]]
>>>>>>
>>>>>> ______________________________________________
>>>>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE
and more, see
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>> PLEASE do read the posting guide
>>>>>> http://www.R-project.org/posting-guide.html
>>>>>> and provide commented, minimal, self-contained,
reproducible code.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>______________________________________________
>>>R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>>>https://stat.ethz.ch/mailman/listinfo/r-help
>>>PLEASE do read the posting guide
>>>http://www.R-project.org/posting-guide.html
>>>and provide commented, minimal, self-contained, reproducible code.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.