R help - Jul 2014 - Best practice: to factor or not to factor for float variables

Hi,

I would like to ask for best practice advice on the design of data
structure and the connected analysis techniques.

In my particular case, I have measurements of several variables at
several, sometimes equal, heights. Following the tidy data approach of
Hadley Wickham, I want to put all data in one data frame. In principle,
the height variable is something like a category. For example, I want to
average over time for every height. Using dplyr this works very well
when my height variable is a factor. However, if it is not a factor the
grouping sometimes will not work probably due to numerical issues:

http://stackoverflow.com/questions/24555010/dplyr-and-group-by-factor-vs-no-factor
https://github.com/hadley/dplyr/issues/482

Even if the behaviour described in the links above is a bug, on can
easily create other numerical issues in R:
> (0.1+0.2) == 0.3
[1] FALSE

Thus, it seems one should avoid grouping by float values and, in my
case, use factors. However, from time to time, I need the numerical
character of the heights: compare heights, find the maximum height, etc.
Here, the ordered factor approach might help. However, I have to combine
(via rbind or merge) different data sets quite often so keeping the
order of the different ordered factor heights also seem to be difficult.

Is there any general approach which reduces the work or do I have to
switch between approaches as needed?

Thanks a lot for any input,
Sebastian

Hi

I would keep height as numeric and created height.f as factor, maybe ordered.
> hh<-runif(50)
> hh
 [1] 0.116060220 0.447546370 0.433749570 0.006548963 0.425710667 0.328972894
 [7] 0.091274539 0.271797166 0.007669982 0.208922146 0.168174196 0.227466231
...
hh.f<-cut(hh, seq(0,1,.1))
> hh.f
 [1] (0.1,0.2] (0.4,0.5] (0.4,0.5] (0,0.1]   (0.4,0.5] (0.3,0.4] (0,0.1]
 [8] (0.2,0.3] (0,0.1]   (0.2,0.3] (0.1,0.2] (0.2,0.3] (0.8,0.9] (0.2,0.3]
...
10 Levels: (0,0.1] (0.1,0.2] (0.2,0.3] (0.3,0.4] (0.4,0.5] ...
(0.9,1]
> boxplot(split(hh, hh.f))
> aggregate(hh, list(hh.f), mean)
     Group.1          x
1    (0,0.1] 0.04679132
2  (0.1,0.2] 0.14659980
3  (0.2,0.3] 0.24458350
4  (0.3,0.4] 0.34881489
5  (0.4,0.5] 0.45444531
6  (0.5,0.6] 0.50291886
7  (0.6,0.7] 0.66860900
8  (0.7,0.8] 0.76984008
9  (0.8,0.9] 0.85753777
10   (0.9,1] 0.95682747
>
Regards
Petr

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Sebastian Schubert
> Sent: Friday, July 04, 2014 1:04 PM
> To: r-help at r-project.org
> Subject: [R] Best practice: to factor or not to factor for float
> variables
>
> Hi,
>
> I would like to ask for best practice advice on the design of data
> structure and the connected analysis techniques.
>
> In my particular case, I have measurements of several variables at
> several, sometimes equal, heights. Following the tidy data approach of
> Hadley Wickham, I want to put all data in one data frame. In principle,
> the height variable is something like a category. For example, I want
> to average over time for every height. Using dplyr this works very well
> when my height variable is a factor. However, if it is not a factor the
> grouping sometimes will not work probably due to numerical issues:
>
> http://stackoverflow.com/questions/24555010/dplyr-and-group-by-factor-
> vs-no-factor
> https://github.com/hadley/dplyr/issues/482
>
> Even if the behaviour described in the links above is a bug, on can
> easily create other numerical issues in R:
> > (0.1+0.2) == 0.3
> [1] FALSE
>
> Thus, it seems one should avoid grouping by float values and, in my
> case, use factors. However, from time to time, I need the numerical
> character of the heights: compare heights, find the maximum height,
> etc.
> Here, the ordered factor approach might help. However, I have to
> combine (via rbind or merge) different data sets quite often so keeping
> the order of the different ordered factor heights also seem to be
> difficult.
>
> Is there any general approach which reduces the work or do I have to
> switch between approaches as needed?
>
> Thanks a lot for any input,
> Sebastian
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.
________________________________
Tento e-mail a jak?koliv k n?mu p?ipojen? dokumenty jsou d?v?rn? a jsou ur?eny
pouze jeho adres?t?m.
Jestli?e jste obdr?el(a) tento e-mail omylem, informujte laskav? neprodlen? jeho
odes?latele. Obsah tohoto emailu i s p??lohami a jeho kopie vyma?te ze sv?ho
syst?mu.
Nejste-li zam??len?m adres?tem tohoto emailu, nejste opr?vn?ni tento email
jakkoliv u??vat, roz?i?ovat, kop?rovat ?i zve?ej?ovat.
Odes?latel e-mailu neodpov?d? za eventu?ln? ?kodu zp?sobenou modifikacemi ?i
zpo?d?n?m p?enosu e-mailu.

V p??pad?, ?e je tento e-mail sou??st? obchodn?ho jedn?n?:
- vyhrazuje si odes?latel pr?vo ukon?it kdykoliv jedn?n? o uzav?en? smlouvy, a
to z jak?hokoliv d?vodu i bez uveden? d?vodu.
- a obsahuje-li nab?dku, je adres?t opr?vn?n nab?dku bezodkladn? p?ijmout;
Odes?latel tohoto e-mailu (nab?dky) vylu?uje p?ijet? nab?dky ze strany p??jemce
s dodatkem ?i odchylkou.
- trv? odes?latel na tom, ?e p??slu?n? smlouva je uzav?ena teprve v?slovn?m
dosa?en?m shody na v?ech jej?ch n?le?itostech.
- odes?latel tohoto emailu informuje, ?e nen? opr?vn?n uzav?rat za spole?nost
??dn? smlouvy s v?jimkou p??pad?, kdy k tomu byl p?semn? zmocn?n nebo p?semn?
pov??en a takov? pov??en? nebo pln? moc byly adres?tovi tohoto emailu p??padn?
osob?, kterou adres?t zastupuje, p?edlo?eny nebo jejich existence je adres?tovi
?i osob? j?m zastoupen? zn?m?.

This e-mail and any documents attached to it may be confidential and are
intended only for its intended recipients.
If you received this e-mail by mistake, please immediately inform its sender.
Delete the contents of this e-mail with all attachments and its copies from your
system.
If you are not the intended recipient of this e-mail, you are not authorized to
use, disseminate, copy or disclose this e-mail in any manner.
The sender of this e-mail shall not be liable for any possible damage caused by
modifications of the e-mail or by delay with transfer of the email.

In case that this e-mail forms part of business dealings:
- the sender reserves the right to end negotiations about entering into a
contract in any time, for any reason, and without stating any reasoning.
- if the e-mail contains an offer, the recipient is entitled to immediately
accept such offer; The sender of this e-mail (offer) excludes any acceptance of
the offer on the part of the recipient containing any amendment or variation.
- the sender insists on that the respective contract is concluded only upon an
express mutual agreement on all its aspects.
- the sender of this e-mail informs that he/she is not authorized to enter into
any contracts on behalf of the company except for cases in which he/she is
expressly authorized to do so in writing, and such authorization or power of
attorney is submitted to the recipient or the person represented by the
recipient, or the existence of such authorization is known to the recipient of
the person represented by the recipient.

Why not just round the floating point numbers to ensure they're equal
with zapsmall, round or signif?

Hadley

On Fri, Jul 4, 2014 at 4:04 AM, Sebastian Schubert
<schubert.seb at gmail.com> wrote:
> Hi,
>
> I would like to ask for best practice advice on the design of data
> structure and the connected analysis techniques.
>
> In my particular case, I have measurements of several variables at
> several, sometimes equal, heights. Following the tidy data approach of
> Hadley Wickham, I want to put all data in one data frame. In principle,
> the height variable is something like a category. For example, I want to
> average over time for every height. Using dplyr this works very well
> when my height variable is a factor. However, if it is not a factor the
> grouping sometimes will not work probably due to numerical issues:
>
>
http://stackoverflow.com/questions/24555010/dplyr-and-group-by-factor-vs-no-factor
> https://github.com/hadley/dplyr/issues/482
>
> Even if the behaviour described in the links above is a bug, on can
> easily create other numerical issues in R:
>> (0.1+0.2) == 0.3
> [1] FALSE
>
> Thus, it seems one should avoid grouping by float values and, in my
> case, use factors. However, from time to time, I need the numerical
> character of the heights: compare heights, find the maximum height, etc.
> Here, the ordered factor approach might help. However, I have to combine
> (via rbind or merge) different data sets quite often so keeping the
> order of the different ordered factor heights also seem to be difficult.
>
> Is there any general approach which reduces the work or do I have to
> switch between approaches as needed?
>
> Thanks a lot for any input,
> Sebastian
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


-- 
http://had.co.nz/

However,
> format((0.1+0.2)) == format(0.3)
[1] TRUE

Which suggests that if you want to treat measured variables as categories,
one way to do it is to format them first.

Of course, one may have to control the format more carefully than above
(if necessary, see for example ?formatC).

merge() on carefully formatted float values may be more reliable than
merging on the floats themselves.

-Don


-- 
Don MacQueen

Lawrence Livermore National Laboratory
7000 East Ave., L-627
Livermore, CA 94550
925-423-1062





On 7/4/14 4:04 AM, "Sebastian Schubert" <schubert.seb at
gmail.com> wrote:
>Hi,
>
>I would like to ask for best practice advice on the design of data
>structure and the connected analysis techniques.
>
>In my particular case, I have measurements of several variables at
>several, sometimes equal, heights. Following the tidy data approach of
>Hadley Wickham, I want to put all data in one data frame. In principle,
>the height variable is something like a category. For example, I want to
>average over time for every height. Using dplyr this works very well
>when my height variable is a factor. However, if it is not a factor the
>grouping sometimes will not work probably due to numerical issues:
>
>http://stackoverflow.com/questions/24555010/dplyr-and-group-by-factor-vs-n
>o-factor
>https://github.com/hadley/dplyr/issues/482
>
>Even if the behaviour described in the links above is a bug, on can
>easily create other numerical issues in R:
>> (0.1+0.2) == 0.3
>[1] FALSE
>
>Thus, it seems one should avoid grouping by float values and, in my
>case, use factors. However, from time to time, I need the numerical
>character of the heights: compare heights, find the maximum height, etc.
>Here, the ordered factor approach might help. However, I have to combine
>(via rbind or merge) different data sets quite often so keeping the
>order of the different ordered factor heights also seem to be difficult.
>
>Is there any general approach which reduces the work or do I have to
>switch between approaches as needed?
>
>Thanks a lot for any input,
>Sebastian
>
>______________________________________________
>R-help at r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.