R devel - Apr 2014 - R 3.1.0: 'R CMD Sweave' deletes non tex files created upon batch mode exit

>>>>> Marc Schwartz <marc_schwartz at me.com>
>>>>>     on Sun, 13 Apr 2014 10:22:55 -0500 writes:
[on the R-SIG-Mac  mailing list] :	  

    > Hi all,
    > With R version 3.1.0 on OSX, using either the Snow Leopard or the
Mavericks binary installation on a Mac with fully updated Mavericks, there has
been a change in behavior since 3.0.3.

I've just written to R-core about this:

    R CMD Sweave

has *serious* problems also in the very simple case when it
should produce figures: They are not available after the
completion of R CMD Sweave,
confirming your  'deletes non tex files created upon batch mode exit'

So this is not related to OSX only, but also a big problem 
at least on other *nix descendent platforms, such as Linux.

In short, failure (no graphic produce in the example below) by

   R CMD Sweave foo.Rnw

but all is fine with using Sweave(), as you Marc, noted, and hence with  

   Rscript -e 'Sweave("foo.Rnw")'

... and to answer your question:  
No this was not intended and is probably one of the bigger / 
most embarrassing bugs in a newly released version of R
in my view: 

Basically  'R CMD Sweave' is partly broken in R 3.1.0.
Yes, this should never have happened.

I had to partially revert the R 3.1.0 installation here (our
statistics dept), by making R 3.0.3 the default 'R'
for now, as we are relying on 'R CMD Sweave ..' in many places.

Personally I've never noticed the problem, as I seem to always
Sweave from ESS (Emacs speaks statistics) which calls  Sweave()
in R, and that works fine, also in  R 3.1.0, as you've already
noted 

Martin Maechler, ETH Zurich

%------------------------------------------------------------
\documentclass[12pt]{article}
\usepackage{Sweave}
\begin{document}  Just a simple graphic 

<<qqnorm, fig=TRUE>>= 
qqnorm(rnorm(20))
@ 

and that's all, folks!
\end{document}
%------------------------------------------------------------

    > I have a master .Rnw file which runs a series of outputs from multiple
R code files, each called in BATCH mode using system() from within the master
.Rnw file. The output of the R code files go to separate text files in order to
catch some of the function call output that would not otherwise be included in
the resultant .tex file due to output redirection.

    > Those text files are then included in the resultant .tex file using,
for example:

    > \lstinputlisting[caption={}]{test.out}

    > directives which are included in the .Rnw source file.

    > A simple example to replicate the observed behaviors.

    > The test.Rnw file content:

    > %% R CMD Sweave test.Rnw
    > <<results=tex,echo=false>>    > system("R CMD BATCH
test.R test.out")
    > @ 


    > The test.R file content:

    > options(echo = FALSE)
    > options(useFancyQuotes = FALSE)
    > installed.packages()


    > On version 3.0.3, the file test.out is created, along with test.tex.
test.out contains the output of installed.packages(). I did not include the
aforementioned listing directive in test.Rnw here for simplicity.

    > On version 3.1.0, the file test.out is created, but when the R CMD
Sweave command exits and returns to the CLI in the console, test.out is deleted,
presumably as part of a post R batch session clean up process. The file test.tex
is retained.

    > I uninstalled 3.1.0 and reinstalled 3.0.3 and observed the prior
behavior. I then tried clean installs of both the Snow Leopard and Mavericks
3.1.0 binaries, with the new behavior observed in both cases.

    > In reading the NEWS file for 3.1.0, there are multiple references to
Sweave, but there is nothing explicit about this new behavior.

    > I should note that when the .Rnw file is run from within an R 3.1.0
interactive session using:

    > Sweave("test.Rnw")

    > the test.out file is created and not deleted upon the function exit, or
when exiting the R session back to the console.

    > Thus, this new behavior seems to be limited to running Sweave from the
CLI using R CMD. It is not clear to me if this new behavior is by design or
perhaps an unintended consequence of changes elsewhere in Sweave processing or
in the handling of R in BATCH mode.

    > When watching the folder where the file output activity takes place, I
note that a file .build.timestamp is created and then deleted, which leads me to
believe that the new functions fileSnapshot() and changedFiles() are being used
in the course of processing R in BATCH mode. If that is correct and there is a
clean up step that is occurring upon BATCH mode exit, which deletes all files
and folders other than .tex files, is it possible that this process is being
overly aggressive? Or if not, is there a way to not have these files deleted or
have a protected folder where these files can be retained?

    > Thanks for any insights.

    > Regards,

    > Marc Schwartz

Hi Martin,

Thanks for your confirmation on this.

I normally do not use R CMD Sweave, as I too run under ESS in normal day to day
operations. This finding was a quirk of having a particular Rnw document that I
occasionally run using R CMD Sweave and I had done so over the weekend,
realizing this behavior.

Thanks again.

Regards,

Marc


On Apr 14, 2014, at 7:28 AM, Martin Maechler <maechler at
stat.math.ethz.ch> wrote:
>>>>>> Marc Schwartz <marc_schwartz at me.com>
>>>>>>    on Sun, 13 Apr 2014 10:22:55 -0500 writes:
> 
> [on the R-SIG-Mac  mailing list] :	  
> 
>> Hi all,
>> With R version 3.1.0 on OSX, using either the Snow Leopard or the
Mavericks binary installation on a Mac with fully updated Mavericks, there has
been a change in behavior since 3.0.3.
> 
> I've just written to R-core about this:
> 
>    R CMD Sweave
> 
> has *serious* problems also in the very simple case when it
> should produce figures: They are not available after the
> completion of R CMD Sweave,
> confirming your  'deletes non tex files created upon batch mode
exit'
> 
> So this is not related to OSX only, but also a big problem 
> at least on other *nix descendent platforms, such as Linux.
> 
> In short, failure (no graphic produce in the example below) by
> 
>   R CMD Sweave foo.Rnw
> 
> but all is fine with using Sweave(), as you Marc, noted, and hence with  
> 
>   Rscript -e 'Sweave("foo.Rnw")'
> 
> ... and to answer your question:  
> No this was not intended and is probably one of the bigger / 
> most embarrassing bugs in a newly released version of R
> in my view: 
> 
> Basically  'R CMD Sweave' is partly broken in R 3.1.0.
> Yes, this should never have happened.
> 
> I had to partially revert the R 3.1.0 installation here (our
> statistics dept), by making R 3.0.3 the default 'R'
> for now, as we are relying on 'R CMD Sweave ..' in many places.
> 
> Personally I've never noticed the problem, as I seem to always
> Sweave from ESS (Emacs speaks statistics) which calls  Sweave()
> in R, and that works fine, also in  R 3.1.0, as you've already
> noted 
> 
> Martin Maechler, ETH Zurich
> 
> %------------------------------------------------------------
> \documentclass[12pt]{article}
> \usepackage{Sweave}
> \begin{document}  Just a simple graphic 
> 
> <<qqnorm, fig=TRUE>>= 
> qqnorm(rnorm(20))
> @ 
> 
> and that's all, folks!
> \end{document}
> %------------------------------------------------------------
> 
>> I have a master .Rnw file which runs a series of outputs from multiple
R code files, each called in BATCH mode using system() from within the master
.Rnw file. The output of the R code files go to separate text files in order to
catch some of the function call output that would not otherwise be included in
the resultant .tex file due to output redirection.
> 
>> Those text files are then included in the resultant .tex file using,
for example:
> 
>> \lstinputlisting[caption={}]{test.out}
> 
>> directives which are included in the .Rnw source file.
> 
>> A simple example to replicate the observed behaviors.
> 
>> The test.Rnw file content:
> 
>> %% R CMD Sweave test.Rnw
>> <<results=tex,echo=false>>>> system("R CMD BATCH
test.R test.out")
>> @ 
> 
> 
>> The test.R file content:
> 
>> options(echo = FALSE)
>> options(useFancyQuotes = FALSE)
>> installed.packages()
> 
> 
>> On version 3.0.3, the file test.out is created, along with test.tex.
test.out contains the output of installed.packages(). I did not include the
aforementioned listing directive in test.Rnw here for simplicity.
> 
>> On version 3.1.0, the file test.out is created, but when the R CMD
Sweave command exits and returns to the CLI in the console, test.out is deleted,
presumably as part of a post R batch session clean up process. The file test.tex
is retained.
> 
>> I uninstalled 3.1.0 and reinstalled 3.0.3 and observed the prior
behavior. I then tried clean installs of both the Snow Leopard and Mavericks
3.1.0 binaries, with the new behavior observed in both cases.
> 
>> In reading the NEWS file for 3.1.0, there are multiple references to
Sweave, but there is nothing explicit about this new behavior.
> 
>> I should note that when the .Rnw file is run from within an R 3.1.0
interactive session using:
> 
>> Sweave("test.Rnw")
> 
>> the test.out file is created and not deleted upon the function exit, or
when exiting the R session back to the console.
> 
>> Thus, this new behavior seems to be limited to running Sweave from the
CLI using R CMD. It is not clear to me if this new behavior is by design or
perhaps an unintended consequence of changes elsewhere in Sweave processing or
in the handling of R in BATCH mode.
> 
>> When watching the folder where the file output activity takes place, I
note that a file .build.timestamp is created and then deleted, which leads me to
believe that the new functions fileSnapshot() and changedFiles() are being used
in the course of processing R in BATCH mode. If that is correct and there is a
clean up step that is occurring upon BATCH mode exit, which deletes all files
and folders other than .tex files, is it possible that this process is being
overly aggressive? Or if not, is there a way to not have these files deleted or
have a protected folder where these files can be retained?
> 
>> Thanks for any insights.
> 
>> Regards,
> 
>> Marc Schwartz

Until fixed, one way to fool R here without changing the vignette or
anything else seems to be (verified on Windows):

mkdir figures
set SWEAVE_OPTIONS=prefix.string=figures/fig
R CMD Sweave foo.Rnw

That obviously has some limitations.

/Henrik


On Mon, Apr 14, 2014 at 5:28 AM, Martin Maechler
<maechler at stat.math.ethz.ch> wrote:
>>>>>> Marc Schwartz <marc_schwartz at me.com>
>>>>>>     on Sun, 13 Apr 2014 10:22:55 -0500 writes:
>
> [on the R-SIG-Mac  mailing list] :
>
>     > Hi all,
>     > With R version 3.1.0 on OSX, using either the Snow Leopard or the
Mavericks binary installation on a Mac with fully updated Mavericks, there has
been a change in behavior since 3.0.3.
>
> I've just written to R-core about this:
>
>     R CMD Sweave
>
> has *serious* problems also in the very simple case when it
> should produce figures: They are not available after the
> completion of R CMD Sweave,
> confirming your  'deletes non tex files created upon batch mode
exit'
>
> So this is not related to OSX only, but also a big problem
> at least on other *nix descendent platforms, such as Linux.
>
> In short, failure (no graphic produce in the example below) by
>
>    R CMD Sweave foo.Rnw
>
> but all is fine with using Sweave(), as you Marc, noted, and hence with
>
>    Rscript -e 'Sweave("foo.Rnw")'
>
> ... and to answer your question:
> No this was not intended and is probably one of the bigger /
> most embarrassing bugs in a newly released version of R
> in my view:
>
> Basically  'R CMD Sweave' is partly broken in R 3.1.0.
> Yes, this should never have happened.
>
> I had to partially revert the R 3.1.0 installation here (our
> statistics dept), by making R 3.0.3 the default 'R'
> for now, as we are relying on 'R CMD Sweave ..' in many places.
>
> Personally I've never noticed the problem, as I seem to always
> Sweave from ESS (Emacs speaks statistics) which calls  Sweave()
> in R, and that works fine, also in  R 3.1.0, as you've already
> noted
>
> Martin Maechler, ETH Zurich
>
> %------------------------------------------------------------
> \documentclass[12pt]{article}
> \usepackage{Sweave}
> \begin{document}  Just a simple graphic
>
> <<qqnorm, fig=TRUE>>> qqnorm(rnorm(20))
> @
>
> and that's all, folks!
> \end{document}
> %------------------------------------------------------------
>
>     > I have a master .Rnw file which runs a series of outputs from
multiple R code files, each called in BATCH mode using system() from within the
master .Rnw file. The output of the R code files go to separate text files in
order to catch some of the function call output that would not otherwise be
included in the resultant .tex file due to output redirection.
>
>     > Those text files are then included in the resultant .tex file
using, for example:
>
>     > \lstinputlisting[caption={}]{test.out}
>
>     > directives which are included in the .Rnw source file.
>
>     > A simple example to replicate the observed behaviors.
>
>     > The test.Rnw file content:
>
>     > %% R CMD Sweave test.Rnw
>     > <<results=tex,echo=false>>>     > system("R
CMD BATCH test.R test.out")
>     > @
>
>
>     > The test.R file content:
>
>     > options(echo = FALSE)
>     > options(useFancyQuotes = FALSE)
>     > installed.packages()
>
>
>     > On version 3.0.3, the file test.out is created, along with
test.tex. test.out contains the output of installed.packages(). I did not
include the aforementioned listing directive in test.Rnw here for simplicity.
>
>     > On version 3.1.0, the file test.out is created, but when the R CMD
Sweave command exits and returns to the CLI in the console, test.out is deleted,
presumably as part of a post R batch session clean up process. The file test.tex
is retained.
>
>     > I uninstalled 3.1.0 and reinstalled 3.0.3 and observed the prior
behavior. I then tried clean installs of both the Snow Leopard and Mavericks
3.1.0 binaries, with the new behavior observed in both cases.
>
>     > In reading the NEWS file for 3.1.0, there are multiple references
to Sweave, but there is nothing explicit about this new behavior.
>
>     > I should note that when the .Rnw file is run from within an R
3.1.0 interactive session using:
>
>     > Sweave("test.Rnw")
>
>     > the test.out file is created and not deleted upon the function
exit, or when exiting the R session back to the console.
>
>     > Thus, this new behavior seems to be limited to running Sweave from
the CLI using R CMD. It is not clear to me if this new behavior is by design or
perhaps an unintended consequence of changes elsewhere in Sweave processing or
in the handling of R in BATCH mode.
>
>     > When watching the folder where the file output activity takes
place, I note that a file .build.timestamp is created and then deleted, which
leads me to believe that the new functions fileSnapshot() and changedFiles() are
being used in the course of processing R in BATCH mode. If that is correct and
there is a clean up step that is occurring upon BATCH mode exit, which deletes
all files and folders other than .tex files, is it possible that this process is
being overly aggressive? Or if not, is there a way to not have these files
deleted or have a protected folder where these files can be retained?
>
>     > Thanks for any insights.
>
>     > Regards,
>
>     > Marc Schwartz
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel