R help - Jan 2010 - combining data frames in a list - how do I add breaks?

I'm a week-old R user, and have become stuck trying to create usable CSV
outputs for post-processing. I am using the package Rioja, which provides
small datasets of results. I am running several analyses in a loop and
iteratively adding the results to a *list* ("combined"). Within each
iteration I use the following:
> combined[[i]] <- performance(fit)
With two iterations I get the following list, as output to a CSV:

           object.RMSE object.R2   object.RMSE.1object.R2.1
WA.inv     0.321543059 0.86388897  0.350494954  0.882600618
WA.cla     0.345947482 0.86388897  0.373078015  0.882600618
WA.inv.tol 0.308649941 0.874585589 0.336975196  0.89148291
WA.cla.tol 0.330038831 0.874585589 0.356895789  0.89148291

Obviously if I run thousands of iterations this continues on to the right,
and the CSV files end up being too big for post-processing in most
spreadsheet programs. Instead I want it to look like this...

1
           object.RMSE object.R2
WA.inv     0.321543059 0.86388897
WA.cla     0.345947482 0.86388897
WA.inv.tol 0.308649941 0.874585589
WA.cla.tol 0.330038831 0.874585589
2
           object.RMSE.1 object.R2.1
WA.inv     0.350494954   0.882600618
WA.cla     0.373078015   0.882600618
WA.inv.tol 0.336975196   0.89148291
WA.cla.tol 0.356895789   0.89148291

...but I have had no luck getting the list to respond to a break in the
data. I have tried inserting the iteration number between results using
"combined[[i]]<-i" and even trying to insert an arbitrary text
string
between results. It seems to ignore any such insertions, and yet gives no
errors.

I'd appreciate any thoughts on ways to create a more usable output.
Best - Euan.

	[[alternative HTML version deleted]]

How about posting your complete set of code that is manipulating the
list.  Normally when I am using a list, each list element is the
result from a test/iteration and then I can use something like 'rbind'
at the end.  I would not expect the output you are getting with the
results extending to the right, so it must have something to do with
the code that you are using.

On Sun, Jan 31, 2010 at 2:56 PM, Euan Reavie <euan.reavie at gmail.com>
wrote:
> I'm a week-old R user, and have become stuck trying to create usable
CSV
> outputs for post-processing. I am using the package Rioja, which provides
> small datasets of results. I am running several analyses in a loop and
> iteratively adding the results to a *list* ("combined"). Within
each
> iteration I use the following:
>
>> combined[[i]] <- performance(fit)
>
> With two iterations I get the following list, as output to a CSV:
>
> ? ? ? ? ? object.RMSE object.R2 ? object.RMSE.1object.R2.1
> WA.inv ? ? 0.321543059 0.86388897 ?0.350494954 ?0.882600618
> WA.cla ? ? 0.345947482 0.86388897 ?0.373078015 ?0.882600618
> WA.inv.tol 0.308649941 0.874585589 0.336975196 ?0.89148291
> WA.cla.tol 0.330038831 0.874585589 0.356895789 ?0.89148291
>
> Obviously if I run thousands of iterations this continues on to the right,
> and the CSV files end up being too big for post-processing in most
> spreadsheet programs. Instead I want it to look like this...
>
> 1
> ? ? ? ? ? object.RMSE object.R2
> WA.inv ? ? 0.321543059 0.86388897
> WA.cla ? ? 0.345947482 0.86388897
> WA.inv.tol 0.308649941 0.874585589
> WA.cla.tol 0.330038831 0.874585589
> 2
> ? ? ? ? ? object.RMSE.1 object.R2.1
> WA.inv ? ? 0.350494954 ? 0.882600618
> WA.cla ? ? 0.373078015 ? 0.882600618
> WA.inv.tol 0.336975196 ? 0.89148291
> WA.cla.tol 0.356895789 ? 0.89148291
>
> ...but I have had no luck getting the list to respond to a break in the
> data. I have tried inserting the iteration number between results using
> "combined[[i]]<-i" and even trying to insert an arbitrary text
string
> between results. It seems to ignore any such insertions, and yet gives no
> errors.
>
> I'd appreciate any thoughts on ways to create a more usable output.
> Best - Euan.
>
> ? ? ? ?[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem that you are trying to solve?

I have several questions on your goals.

Why are you planning to post-processing outside of R?

Why use CSV files when there are usually better ways to maintain
the structure of the data?

Why do you want to flatten your tables?  It looks like a three-dimensional array
would better capture the information.

Since spreadsheets seem to be part of your style, please look at RExcel
rcom.univie.ac.at, which allows for smooth integration of R and Excel.

Rich

With Jim's help, the solution is as follows...

-----Original Message-----
From: jim holtman [mailto:jholtman * at sign * gmail.com] 
Sent: Sunday, January 31, 2010 5:41 PM
To: Euan Reavie
Subject: Re: [R] combining data frames in a list - how do I add breaks?

Your 'combined' was a 'list of a list'.  I think the object
returned by:

performance(fit)

was actually a list and you probably should have done:

combined[[i]] <- performance(fit)$object

Here is what I did to get your data combined:
> str(combined)
List of 2
 $ :List of 1
  ..$ object: num [1:4, 1:4] 0.394 0.454 0.352 0.392 0.754 ...
  .. ..- attr(*, "dimnames")=List of 2
  .. .. ..$ : chr [1:4] "WA.inv" "WA.cla"
"WA.inv.tol" "WA.cla.tol"
  .. .. ..$ : chr [1:4] "RMSE" "R2" "Avg.Bias"
"Max.Bias"
 $ :List of 1
  ..$ object: num [1:4, 1:4] 0.324 0.364 0.319 0.357 0.793 ...
  .. ..- attr(*, "dimnames")=List of 2
  .. .. ..$ : chr [1:4] "WA.inv" "WA.cla"
"WA.inv.tol" "WA.cla.tol"
  .. .. ..$ : chr [1:4] "RMSE" "R2" "Avg.Bias"
"Max.Bias"
> do.call(rbind, lapply(combined, '[[', 1))
                RMSE        R2      Avg.Bias   Max.Bias
WA.inv     0.3944042 0.7542192 -1.421083e-15 -0.3926396
WA.cla     0.4541431 0.7542192  1.421091e-15 -0.1854335
WA.inv.tol 0.3517808 0.8044719  0.000000e+00 -0.3588896
WA.cla.tol 0.3922082 0.8044719 -6.394885e-15 -0.2735517
WA.inv     0.3242771 0.7930415  0.000000e+00  0.6145005
WA.cla     0.3641399 0.7930415 -7.105536e-16  0.6170610
WA.inv.tol 0.3194655 0.7991376  3.552714e-15  0.6267316
WA.cla.tol 0.3573659 0.7991376  4.973807e-15  0.6096722
>
Here is some code that will put the trial number as the first column
to make it easier to process:
> # add the trial number as the first column to make it easier to process
> trial <- 0
> result <- do.call(rbind, lapply(combined, function(.mat){
+     trial <<- trial + 1  # need the "<<-" because we are
with an 'lapply'
+     cbind(trial=trial, .mat$object)
+ }))
>
>
> result
           trial      RMSE        R2      Avg.Bias   Max.Bias
WA.inv         1 0.3944042 0.7542192 -1.421083e-15 -0.3926396
WA.cla         1 0.4541431 0.7542192  1.421091e-15 -0.1854335
WA.inv.tol     1 0.3517808 0.8044719  0.000000e+00 -0.3588896
WA.cla.tol     1 0.3922082 0.8044719 -6.394885e-15 -0.2735517
WA.inv         2 0.3242771 0.7930415  0.000000e+00  0.6145005
WA.cla         2 0.3641399 0.7930415 -7.105536e-16  0.6170610
WA.inv.tol     2 0.3194655 0.7991376  3.552714e-15  0.6267316
WA.cla.tol     2 0.3573659 0.7991376  4.973807e-15  0.6096722

>> On Sun, Jan 31, 2010 at 3:25 PM, Euan Reavie <euan.reavie * at sign
*
gmail.com>
> wrote:
>>> Thanks for your feedback Jim. The complete code is below (showing
10
>>> iterations), and if you think I should re-post this to the list
please
> let
>>> me know.
>>> Best - Euan.
>>>
>>> library(rioja, pos=4)
>>> TSSstuff <-
read.table("C:/Users/Owner/Desktop/Rfiles/salinity.txt",
>>> header=TRUE, sep="", na.strings="NA",
dec=".", strip.white=TRUE)
>>> TSSspecies <-
read.table("C:/Users/Owner/Desktop/Rfiles/sppdata.txt",
>>> header=TRUE, sep="", na.strings="NA",
dec=".", strip.white=TRUE)
>>>
>>> # make a "combined" list for results
>>> combined <- list()
>>>
>>> # run iterations
>>> for(i in 1:10) {
>>> spec <- TSSspecies
>>> TSS <- TSSstuff
>>>
>>> # get 10 random numbers (20 samples total, no repeats)
>>> x1 <- sample(1:20, 10, replace=F)
>>>
>>> # remove the random samples from both datasets
>>> TSS <- TSS[-x1,]
>>> spec <- spec[-x1,]
>>>
>>> # remove all-zero species columns
>>> spec <- spec[, colSums(spec) > 0]
>>>
>>> # get performance results - jackknife
>>> fit <- WA(spec, TSS, tolDW=TRUE)
>>>
>>> # put iterative results into "combined" list
>>> combined[[i]] <- i ?# seems to do nothing for some reason
>>> combined[[i]] <- performance(fit)
>>>
>>> }
>>> write.csv(combined, file = "output.csv")
>>> # End
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: jim holtman [mailto:jholtman * at sign * gmail.com]
>>> Sent: Sunday, January 31, 2010 2:16 PM
>>> To: Euan Reavie
>>> Cc: r-help * at sign * r-project.org
>>> Subject: Re: [R] combining data frames in a list - how do I add
breaks?
>>>
>>> How about posting your complete set of code that is manipulating
the
>>> list. ?Normally when I am using a list, each list element is the
>>> result from a test/iteration and then I can use something like
'rbind'
>>> at the end. ?I would not expect the output you are getting with the
>>> results extending to the right, so it must have something to do
with
>>> the code that you are using.
>>>
>>> On Sun, Jan 31, 2010 at 2:56 PM, Euan Reavie <euan.reavie * at
sign *
gmail.com>
>> wrote:
>>>> I'm a week-old R user, and have become stuck trying to
create usable
CSV
>>>> outputs for post-processing. I am using the package Rioja,
which
> provides
>>>> small datasets of results. I am running several analyses in a
loop and
>>>> iteratively adding the results to a *list*
("combined"). Within each
>>>> iteration I use the following:
>>>>
>>>>> combined[[i]] <- performance(fit)
>>>>
>>>> With two iterations I get the following list, as output to a
CSV:
>>>>
>>>> ? ? ? ? ? object.RMSE object.R2 ? object.RMSE.1object.R2.1
>>>> WA.inv ? ? 0.321543059 0.86388897 ?0.350494954 ?0.882600618
>>>> WA.cla ? ? 0.345947482 0.86388897 ?0.373078015 ?0.882600618
>>>> WA.inv.tol 0.308649941 0.874585589 0.336975196 ?0.89148291
>>>> WA.cla.tol 0.330038831 0.874585589 0.356895789 ?0.89148291
>>>>
>>>> Obviously if I run thousands of iterations this continues on to
the
>> right,
>>>> and the CSV files end up being too big for post-processing in
most
>>>> spreadsheet programs. Instead I want it to look like this...
>>>>
>>>> 1
>>>> ? ? ? ? ? object.RMSE object.R2
>>>> WA.inv ? ? 0.321543059 0.86388897
>>>> WA.cla ? ? 0.345947482 0.86388897
>>>> WA.inv.tol 0.308649941 0.874585589
>>>> WA.cla.tol 0.330038831 0.874585589
>>>> 2
>>>> ? ? ? ? ? object.RMSE.1 object.R2.1
>>>> WA.inv ? ? 0.350494954 ? 0.882600618
>>>> WA.cla ? ? 0.373078015 ? 0.882600618
>>>> WA.inv.tol 0.336975196 ? 0.89148291
>>>> WA.cla.tol 0.356895789 ? 0.89148291
>>>>
>>>> ...but I have had no luck getting the list to respond to a
break in the
>>>> data. I have tried inserting the iteration number between
results using
>>>> "combined[[i]]<-i" and even trying to insert an
arbitrary text string
>>>> between results. It seems to ignore any such insertions, and
yet gives
> no
>>>> errors.
>>>>
>>>> I'd appreciate any thoughts on ways to create a more usable
output.
>>>> Best - Euan.
>>>>
>>>
>>> --
>>> Jim Holtman
>>> Cincinnati, OH
>>> +1 513 646 9390
>>>
>>> What is the problem that you are trying to solve?
>>>