R help - Nov 2007 - Sweaving single master file to get multiple individualised reports

Hi

Apologies in advance if I've missed something obvious. I have read the
Sweave manual, the first article in R News, looked at the Help pages,
googled Sweave and words like loop, output, files, multiple, done much
the same on R site search (in case I missed something on Google) and I
couldn't find exactly what I'm after.

What I'm trying to do
?----------------------
Make individualised school (or regional or ...) reports combining
generic words, overall totals and individual results (in tables and
graphs), typically reporting student assessment data.

System info
?------------
Windows XP
R 2-5-1 (always wait for the R x-x-1 version)
Latest Miktex
Working in latest Emacs with the help of ESS & Auctex
Have (but seldom use) cygwin

What I think the problem is (I'm hoping it's not)
?------------------------------
I've used odfWeave to do something similar, and was then able to
specify both the infile and the outfile (and so could go something like
infile = template.odf and outfile = 
paste("ReportName",schoolname,".odt",sep = ""). 

[Why stop using odfWeave? a) runs slowly with all the zipping and
unzipping b) very cryptic error messages c) using eps to get high
quality graphics results in a somewhat flaky OpenOffice file where
scrolling through it to check pagination and table widths is really hard
d) turning the OpenOffice file into a pdf to send to the schools
requires an extra step or two that is both time-consuming and a nuisance
e) computer got very hot and bothered running a number of multipage
reports - to the point of having to be left to cool down before trying
again.]

This involved an R file that had

# getting packages, reading data etc stuff
:
:
# doing some pre-processing on project-wide totals- easier here where
it can be debugged bit by bit
:
:
for (i in 1:Nschools){
# set indicator variable for school i or otherwise set up subset for a
school
:
# more pre-processing for school/regional summaries
:
:
infile <- "templateName.odt"      # same for all schools/regions
outfile <- paste("ReportName",schoolname,".odt",sep =
"")  # unique to
school/region
odfWeave(infile,outfile)
}

However, as far as I can see (admittedly, seldom very far) the
equivalent Sweave call is just Sweave(SnwFile) which would create
SnwFile.tex.

What I'm hoping for
?--------------------
Any one or more of:

- A blasting for lack of understanding that tells me it's all easy,
really (plus a "how to" or a "where to find and actually
see")
- A work-around
- An idiot's guide to how to do it using a shell script or equivalent
(which I suspect to be the way to go)
- Help with automatically LaTeXing (actually pdfLaTeXing) all the 
*.tex files generated as a batch.

One advantage of the within-a-loop way of working is that when the
report for one or more schools needs re-running this can be done without
re-running the whole lot. This flexibility needs to be part of the final
solution.

Many thanks for any expertise and experience offered


Edith Hodgen
Statistician and Data Manager
New Zealand Council for Educational Research
Postal address: P.O. Box 3237, Wellington 6140
Street address: Level 10, Education House, 178-182 Willis Street,
                        Te Aro, Wellington 6011
Phone: +64-4-802 1433
Fax:      +64-4-384 7933
edith.hodgen at nzcer.org.nz

Web site http://www.nzcer.org.nz

Here are a couple of thoughts:

You can use 'file.rename' to rename the .tex file created by Sweave to
your pattern (within the loop if you stick with that approach).

I did something similar (using odfWeave) where the template file always
accessed a given dataframe (mydata1 for example) then when I wanted to
do the report for the different subsets of the data (full data in
mydata) I would use something like:
> mydata1 <- subset(mydata, ...
> odfWeave(...
> file.rename( ...
You could do something similar in your loop, or even better,
The Sweave manual has as one of the faqs (#3) how to run sweave as a
batch process, what you need to do is use make and makefiles (will be
well worth your effort to learn if you don't know about them already,
there are versions of make available for windows).  You can set up a
rule for making a .tex file from a .nws (or whatever file) including the
subsetting and renaming, then also set up rules for making your final
pdf file from the .tex file.  Then at the command line you can just type
"make" or "make all" to generate all the reports and
"make
myreport-1.pdf" to only regenerate a single report (and if you can get
things set up in just the right way, typing "make" will only
regenerate
those reports that have had data change since the last time you ran
them).

Hope this helps,

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at intermountainmail.org
(801) 408-8111
 
 
> -----Original Message-----
> From: r-help-bounces at r-project.org 
> [mailto:r-help-bounces at r-project.org] On Behalf Of Edith Hodgen
> Sent: Thursday, November 08, 2007 3:43 PM
> To: r-help at stat.math.ethz.ch
> Subject: [R] Sweaving single master file to get 
> multipleindividualised reports
> 
> Hi
> 
> Apologies in advance if I've missed something obvious. I have 
> read the Sweave manual, the first article in R News, looked 
> at the Help pages, googled Sweave and words like loop, 
> output, files, multiple, done much the same on R site search 
> (in case I missed something on Google) and I couldn't find 
> exactly what I'm after.
> 
> What I'm trying to do
> -----------------------
> Make individualised school (or regional or ...) reports 
> combining generic words, overall totals and individual 
> results (in tables and graphs), typically reporting student 
> assessment data.
> 
> System info
> -------------
> Windows XP
> R 2-5-1 (always wait for the R x-x-1 version) Latest Miktex 
> Working in latest Emacs with the help of ESS & Auctex Have 
> (but seldom use) cygwin
> 
> What I think the problem is (I'm hoping it's not)
> -------------------------------
> I've used odfWeave to do something similar, and was then able 
> to specify both the infile and the outfile (and so could go 
> something like infile = template.odf and outfile = 
> paste("ReportName",schoolname,".odt",sep =
"").
> 
> [Why stop using odfWeave? a) runs slowly with all the zipping 
> and unzipping b) very cryptic error messages c) using eps to 
> get high quality graphics results in a somewhat flaky 
> OpenOffice file where scrolling through it to check 
> pagination and table widths is really hard
> d) turning the OpenOffice file into a pdf to send to the 
> schools requires an extra step or two that is both 
> time-consuming and a nuisance
> e) computer got very hot and bothered running a number of 
> multipage reports - to the point of having to be left to cool 
> down before trying again.]
> 
> This involved an R file that had
> 
> # getting packages, reading data etc stuff
> :
> :
> # doing some pre-processing on project-wide totals- easier 
> here where it can be debugged bit by bit
> :
> :
> for (i in 1:Nschools){
> # set indicator variable for school i or otherwise set up 
> subset for a school
> :
> # more pre-processing for school/regional summaries
> :
> :
> infile <- "templateName.odt"      # same for all
schools/regions
> outfile <- paste("ReportName",schoolname,".odt",sep
= "")  #
> unique to school/region
> odfWeave(infile,outfile)
> }
> 
> However, as far as I can see (admittedly, seldom very far) 
> the equivalent Sweave call is just Sweave(SnwFile) which 
> would create SnwFile.tex.
> 
> What I'm hoping for
> ---------------------
> Any one or more of:
> 
> - A blasting for lack of understanding that tells me it's all 
> easy, really (plus a "how to" or a "where to find and
actually see")
> - A work-around
> - An idiot's guide to how to do it using a shell script or 
> equivalent (which I suspect to be the way to go)
> - Help with automatically LaTeXing (actually pdfLaTeXing) all 
> the *.tex files generated as a batch.
> 
> One advantage of the within-a-loop way of working is that 
> when the report for one or more schools needs re-running this 
> can be done without re-running the whole lot. This 
> flexibility needs to be part of the final solution.
> 
> Many thanks for any expertise and experience offered
> 
> 
> Edith Hodgen
> Statistician and Data Manager
> New Zealand Council for Educational Research Postal address: 
> P.O. Box 3237, Wellington 6140 Street address: Level 10, 
> Education House, 178-182 Willis Street,
>                         Te Aro, Wellington 6011
> Phone: +64-4-802 1433
> Fax:      +64-4-384 7933
> edith.hodgen at nzcer.org.nz
> 
> Web site http://www.nzcer.org.nz
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

On Nov 8, 2007, at 5:42 PM, Edith Hodgen wrote:
> Hi
>
>
[snip]
> What I think the problem is (I'm hoping it's not)
> ?------------------------------
> I've used odfWeave to do something similar, and was then able to
> specify both the infile and the outfile (and so could go something  
> like
> infile = template.odf and outfile >
paste("ReportName",schoolname,".odt",sep = "").
> This involved an R file that had
>
> # getting packages, reading data etc stuff
> :
> :
> # doing some pre-processing on project-wide totals- easier here where
> it can be debugged bit by bit
> :
> :
> for (i in 1:Nschools){
> # set indicator variable for school i or otherwise set up subset for a
> school
> :
> # more pre-processing for school/regional summaries
> :
> :
> infile <- "templateName.odt"      # same for all
schools/regions
> outfile <- paste("ReportName",schoolname,".odt",sep
= "")  # unique to
> school/region
> odfWeave(infile,outfile)
> }
>
> However, as far as I can see (admittedly, seldom very far) the
> equivalent Sweave call is just Sweave(SnwFile) which would create
> SnwFile.tex.
Sweave accepts outfiles as well, but you have to look at ?RweaveLatex  
to find that out. Sweave just passes its extra parameters to that. I  
use code like the following in an "analysis.r" file:

for (trs in unique(pulse$transmitter)) {
	data <- subset(pulse, transmitter==trs)
	fname <-
paste("analysis",gsub("\\.","-",trs),".tex",
sep="")
	range.IPI <- cover(range(data$IPI))
	Sweave("analysis.rnw", output=fname)
}

You can probably use "system" in that loop to also run pdflatex on  
the resulting files. I had a hard time redirecting the output of  
those calls where I wanted it, but it did work I think. What I ended  
up doing instead was to create the following basic shell script instead:

echo "Creating TeX files ..."
R CMD BATCH --vanilla --quiet analysis.r
echo "Compiling TeX files ..."
for file in *.tex
do
	pdflatex $file >>logs.log
done

(This is all on MacOSX, and I am guessing Linux would work as well.  
Don't know what goes on in the Windows world I'm afraid.)

Hope this helps.
> What I'm hoping for
> ?--------------------
> Any one or more of:
>
> - A blasting for lack of understanding that tells me it's all easy,
> really (plus a "how to" or a "where to find and actually
see")
> - A work-around
> - An idiot's guide to how to do it using a shell script or equivalent
> (which I suspect to be the way to go)
> - Help with automatically LaTeXing (actually pdfLaTeXing) all the
> *.tex files generated as a batch.
>
> One advantage of the within-a-loop way of working is that when the
> report for one or more schools needs re-running this can be done  
> without
> re-running the whole lot. This flexibility needs to be part of the  
> final
> solution.
>
> Many thanks for any expertise and experience offered
>
>
> Edith Hodgen
> Statistician and Data Manager
> New Zealand Council for Educational Research
> Postal address: P.O. Box 3237, Wellington 6140
> Street address: Level 10, Education House, 178-182 Willis Street,
>                         Te Aro, Wellington 6011
> Phone: +64-4-802 1433
> Fax:      +64-4-384 7933
> edith.hodgen at nzcer.org.nz
>
> Web site http://www.nzcer.org.nz
>
Haris Skiadas
Department of Mathematics and Computer Science
Hanover College