On 03/02/2019 12:04 p.m., Radford Neal wrote:> While doing some performance testing with the new version of pqR (see
> pqR-project.org), I've encountered an extreme, and quite unnecessary,
> inefficiency in the current R Core implementation of R, which I think
> you might want to correct.
>
> The inefficiency is in access to columns of a data frame, as in
> expressions such as df$col[i], which I think are very common (the
> alternatives of df[i,"col"] and df[["col"]][i] are, I
think, less
> common).
>
> Here is the setup for an example showing the issue:
>
> L <- list (abc=1:9, xyz=11:19)
> Lc <- L; class(Lc) <- "glub"
> df <- data.frame(L)
>
> And here are some times for R-3.5.2 (r-devel of 2019-02-01 is much
> the same):
>
> > system.time (for (i in 1:1000000) r <- L$xyz)
> user system elapsed
> 0.086 0.004 0.089
> > system.time (for (i in 1:1000000) r <- Lc$xyz)
> user system elapsed
> 0.494 0.000 0.495
> > system.time (for (i in 1:1000000) r <- df$xyz)
> user system elapsed
> 3.425 0.000 3.426
>
> So accessing a column of a data frame is 38 times slower than
> accessing a list element (which is what happens in the underlying
> implementation of a data frame), and 7 times slower than accessing an
> element of a list with a class attribute (for which it's necessary to
> check whether there is a $.glub method, which there isn't here).
>
> For comparison, here are the times for pqR-2019-01-25:
>
> > system.time (for (i in 1:1000000) r <- L$xyz)
> user system elapsed
> 0.057 0.000 0.058
> > system.time (for (i in 1:1000000) r <- Lc$xyz)
> user system elapsed
> 0.251 0.000 0.251
> > system.time (for (i in 1:1000000) r <- df$xyz)
> user system elapsed
> 0.247 0.000 0.247
>
> So when accessing df$xyz, R-3.5.2 is 14 times slower than pqR-2019-01-25.
> (For a partial match, like df$xy, R-3.5.2 is 34 times slower.)
>
> I wasn't surprised that pqR was faster, but I didn't expect this
big a
> difference. Then I remembered having seen a NEWS item from R-3.1.0:
>
> * Partial matching when using the $ operator _on data frames_ now
> throws a warning and may become defunct in the future. If partial
> matching is intended, replace foo$bar by foo[["bar", exact
> FALSE]].
>
> and having looked at the code then:
>
> `$.data.frame` <- function(x,name) {
> a <- x[[name]]
> if (!is.null(a)) return(a)
>
> a <- x[[name, exact=FALSE]]
> if (!is.null(a)) warning("Name partially matched in data
frame")
> return(a)
> }
>
> I recall thinking at the time that this involved a pretty big
> performance hit, compared to letting the primitive $ operator do it,
> just to produce a warning. But it wasn't until now that I noticed
> this NEWS in R-3.1.1:
>
> * The warning when using partial matching with the $ operator on
> data frames is now only given when
> options("warnPartialMatchDollar") is TRUE.
>
> for which the code was changed to:
>
> `$.data.frame` <- function(x,name) {
> a <- x[[name]]
> if (!is.null(a)) return(a)
>
> a <- x[[name, exact=FALSE]]
> if (!is.null(a) &&
getOption("warnPartialMatchDollar", default=FALSE)) {
> names <- names(x)
> warning(gettextf("Partial match of '%s' to
'%s' in data frame",
> name, names[pmatch(name, names)]))
> }
> return(a)
> }
>
> One can see the effect now when warnPartialMatchDollar is enabled:
>
> > options(warnPartialMatchDollar=TRUE)
> > Lc$xy
> [1] 11 12 13 14 15 16 17 18 19
> Warning message:
> In Lc$xy : partial match of 'xy' to 'xyz'
> > df$xy
> [1] 11 12 13 14 15 16 17 18 19
> Warning message:
> In `$.data.frame`(df, xy) : Partial match of 'xy' to
'xyz' in data frame
>
> So the only thing that slowing down acesses like df$xyz by a factor of
> seven achieves now is to add the words "in data frame" to the
warning
> message (while making the earlier part of the message less intelligible).
>
> I think you might want to just delete the definition of $.data.frame,
> reverting to the situation before R-3.1.0.
I imagine the cause is that the list version is done in C code rather
than R code (i.e. there's no R function `$.list`). So an alternative
solution would be to also implement `$.data.frame` in the underlying C
code. This won't be quite as fast (it needs that test for NULL), but
should be close in the full match case.
Duncan Murdoch