R help - Jun 2015 - unlisting dplyr do output

I used the dplyr do function to apply a kernel regression smoother to a 3
column data table (grouping index, x, y) with about 7 M rows and 45000
groups.

This runs quickly, about 1-2 minutes.

It creates an data table (44,326 by 2) - grouping index, kernel smoothing
output.

The kernel smoothing output is a list of two element lists (x, smoothed y).

I used a for loop to unlist this into a data table.

  for (i in 1:nrow(do object)) {
df <- bind_rows(list(df,
                                  data.frame(grouping index = do object[i],
                                             x = do object[[i]]$x,
                                             y = do object[[i]]$smoothed
y)))
}


This takes about 100 minutes.

Any guidance for a faster (more elegant?) solution will be appreciated.

Nathan

If I read you code right you have in every loop iteration a reallocation of
memory which gets progressively larger.  If you allocate the whole data
frame
at the beginning and only set the values in the loop it should speed up
lots.

On Wed, Jun 24, 2015 at 10:39 PM, Nathan Pace <n.l.pace at utah.edu>
wrote:
> I used the dplyr do function to apply a kernel regression smoother to a 3
> column data table (grouping index, x, y) with about 7 M rows and 45000
> groups.
>
> This runs quickly, about 1-2 minutes.
>
> It creates an data table (44,326 by 2) - grouping index, kernel smoothing
> output.
>
> The kernel smoothing output is a list of two element lists (x, smoothed y).
>
> I used a for loop to unlist this into a data table.
>
>   for (i in 1:nrow(do object)) {
> df <- bind_rows(list(df,
>                                   data.frame(grouping index = do object[i],
>                                              x = do object[[i]]$x,
>                                              y = do object[[i]]$smoothed
> y)))
> }
>
>
> This takes about 100 minutes.
>
> Any guidance for a faster (more elegant?) solution will be appreciated.
>
> Nathan
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]