R devel - Apr 2017 - tempdir() may be deleted during long-running R session

>>>>> Dirk Eddelbuettel <edd at debian.org>
>>>>>     on Wed, 26 Apr 2017 08:40:38 -0500 writes:
    > On 26 April 2017 at 08:29, Duncan Murdoch wrote:
    > | This seems like the wrong approach.  The problem occurs as soon as
the
    > | tempdir() gets cleaned up:  there could be information in temp files 
    > | that gets lost at that point.  So the solution should be to prevent
the
    > | cleanup, not to continue on after it has occurred (as "check =
TRUE"
    > | does).  This follows the principle that it's better for the
process to
    > | always die than to sometimes silently produce incorrect results.

    > That is generally true, but also "hard" as we don't have
a handle on the OS
.

Indeed...
and that was the reason I've proposed the simple platform
agnostic tool which does not entirely solve the problem (in this sense I
agree with "wrong approach") but allows to mitigate it and (by
followup changes) to work around many use case problems.

    > | Frederick posted the way to do this in systems using systemd.  We
should

    > While that was a very helpful post yet it may only apply to Arch Linux
as
    > stated.  My Ubuntu systems at home and work all run systemd too, but do
_not_
    > automatically remove tempfiles. 

    > Yet what he suggested is quite right: we should define a proper config
file
    > for this facility and then possibly also use the /run directory as many
other
    > services now and (of course) also either TEMPDIR or later the code to
have
    > /run be another fallback if TMP, TEMP, TMPDIR, ... are unset.

    > Distribution maintainers such as yours truly could then include this
    > configuration.

    > | be putting that in place, or the equivalent on systems using other 
    > | tempfile cleanups.  This looks to me like something that "make
install"
    > | should do, or perhaps it should be done by people putting together 
    > | packages for specific systems.

    > Doesn't 'make install' only write to $RHOME/ and below,
plus $PREFIX/bin ?

Also, 'make install' is optional for good reasons.
E.g., I never ever run 'make install': I typically always have many R
versions, all available in the shell and ESS (Emacs Speaks
Statistics) via symbolic links into a directory on PATH.

Dirk mentioned (as well) that this is all very platform specific
which I do think is important. From my typical OS point of view:
  Why should the user who runs R not have the right to delete the
  tempdir which was created by the process that she runs and hence owns ?

I agree it would be an improvement if we made such deletion much
harder than it is now, and yes, there may be great (almost)
cross-platform tools available to manage this much better than
we do now, e.g., via open files.

Before we are there, I would find it useful to have a new
'tempdir' (i.e. folder/directory for R's temporary files) to be
re-created manually or automagically in those cases it has
disappeared, and that is within easy reach via the proposed
tempdir() functionality.

OTOH, I typically live very well by quickly killing
and restarting R (from inside ESS).  

The OP issue was to help newbies and computer-non-experts, the
latter nowadays comprising more than 90% of R users (I'd guess ~
98% looking at our otherwise smart students).

These are typically "slightly" confused when they ask for help and
get a pretty severe error message:

  > ?lm
  Error in file(out, "wt") : cannot open the connection
  In addition: Warning message:
  In file(out, "wt") :
    cannot open file '/tmp/RtmpztK6f7/Rtxt36972b91938': No such file or
directory


Martin

The Ubuntu machine I use a lot (along with others) must not be cleaning
/tmp as it has a fair number of Rtmp* directories in /tmp, even when there
are no R sessions running on the machine.  I would like to automate their
removal but there is no obvious way to see if the R process that created
the tempdir is still around.   If there were a way to associate R tempdir's
with R processes then one could make an R-specific tool to get rid of the
unused tempdirs.


Bill Dunlap
TIBCO Software
wdunlap tibco.com

On Wed, Apr 26, 2017 at 8:09 AM, Martin Maechler <maechler at
stat.math.ethz.ch
> wrote:
> >>>>> Dirk Eddelbuettel <edd at debian.org>
> >>>>>     on Wed, 26 Apr 2017 08:40:38 -0500 writes:
>
>     > On 26 April 2017 at 08:29, Duncan Murdoch wrote:
>     > | This seems like the wrong approach.  The problem occurs as soon
as
> the
>     > | tempdir() gets cleaned up:  there could be information in temp
> files
>     > | that gets lost at that point.  So the solution should be to
> prevent the
>     > | cleanup, not to continue on after it has occurred (as
"check > TRUE"
>     > | does).  This follows the principle that it's better for the
> process to
>     > | always die than to sometimes silently produce incorrect results.
>
>     > That is generally true, but also "hard" as we don't
have a handle on
> the OS
> .
>
> Indeed...
> and that was the reason I've proposed the simple platform
> agnostic tool which does not entirely solve the problem (in this sense I
> agree with "wrong approach") but allows to mitigate it and (by
> followup changes) to work around many use case problems.
>
>     > | Frederick posted the way to do this in systems using systemd. 
We
> should
>
>     > While that was a very helpful post yet it may only apply to Arch
> Linux as
>     > stated.  My Ubuntu systems at home and work all run systemd too,
but
> do _not_
>     > automatically remove tempfiles.
>
>     > Yet what he suggested is quite right: we should define a proper
> config file
>     > for this facility and then possibly also use the /run directory as
> many other
>     > services now and (of course) also either TEMPDIR or later the code
> to have
>     > /run be another fallback if TMP, TEMP, TMPDIR, ... are unset.
>
>     > Distribution maintainers such as yours truly could then include
this
>     > configuration.
>
>     > | be putting that in place, or the equivalent on systems using
other
>     > | tempfile cleanups.  This looks to me like something that
"make
> install"
>     > | should do, or perhaps it should be done by people putting
together
>     > | packages for specific systems.
>
>     > Doesn't 'make install' only write to $RHOME/ and
below, plus
> $PREFIX/bin ?
>
> Also, 'make install' is optional for good reasons.
> E.g., I never ever run 'make install': I typically always have many
R
> versions, all available in the shell and ESS (Emacs Speaks
> Statistics) via symbolic links into a directory on PATH.
>
> Dirk mentioned (as well) that this is all very platform specific
> which I do think is important. From my typical OS point of view:
>   Why should the user who runs R not have the right to delete the
>   tempdir which was created by the process that she runs and hence owns ?
>
> I agree it would be an improvement if we made such deletion much
> harder than it is now, and yes, there may be great (almost)
> cross-platform tools available to manage this much better than
> we do now, e.g., via open files.
>
> Before we are there, I would find it useful to have a new
> 'tempdir' (i.e. folder/directory for R's temporary files) to be
> re-created manually or automagically in those cases it has
> disappeared, and that is within easy reach via the proposed
> tempdir() functionality.
>
> OTOH, I typically live very well by quickly killing
> and restarting R (from inside ESS).
>
> The OP issue was to help newbies and computer-non-experts, the
> latter nowadays comprising more than 90% of R users (I'd guess ~
> 98% looking at our otherwise smart students).
>
> These are typically "slightly" confused when they ask for help
and
> get a pretty severe error message:
>
>   > ?lm
>   Error in file(out, "wt") : cannot open the connection
>   In addition: Warning message:
>   In file(out, "wt") :
>     cannot open file '/tmp/RtmpztK6f7/Rtxt36972b91938': No such
file or
> directory
>
>
> Martin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]

The main point of this discussion has been that deleting the tempdir() a
fixed number of days after its creation is a problem.  It should be deleted
when the process that created it is done.  R attempts to do this, but there
are cases in which it does not so a backup is needed.

Bill Dunlap
TIBCO Software
wdunlap tibco.com

On Wed, Apr 26, 2017 at 8:46 AM, Brian G. Peterson <brian at
braverock.com>
wrote:
> Bill,
>
> It seems that tmpreaper, tmpwatch, or systemd-tmpfiles should be able
> to handle this for you.
>
> set the atime to some number of days rather than a short duration.
>
> I think I set mine to three days, and I can';t recall ever having a
> problem that was more than a minor annoyance (help breaking) on old R
> processes.
>
> Cheers,
>
> Brian
>
> --
> Brian G. Peterson
> http://braverock.com/brian/
> Ph: 773-459-4973
> IM: bgpbraverock
>
> On Wed, 2017-04-26 at 08:25 -0700, William Dunlap via R-devel wrote:
> > The Ubuntu machine I use a lot (along with others) must not be
> > cleaning
> > /tmp as it has a fair number of Rtmp* directories in /tmp, even when
> > there
> > are no R sessions running on the machine.  I would like to automate
> > their
> > removal but there is no obvious way to see if the R process that
> > created
> > the tempdir is still around.   If there were a way to associate R
> > tempdir's
> > with R processes then one could make an R-specific tool to get rid of
> > the
> > unused tempdirs.
> >
> >
> > Bill Dunlap
> > TIBCO Software
> > wdunlap tibco.com
> >
> > On Wed, Apr 26, 2017 at 8:09 AM, Martin Maechler <maechler at
stat.math.
> > ethz.ch
> > > wrote:
> > > > > > > > Dirk Eddelbuettel <edd at
debian.org>
> > > > > > > >     on Wed, 26 Apr 2017 08:40:38 -0500
writes:
> > >
> > >     > On 26 April 2017 at 08:29, Duncan Murdoch wrote:
> > >     > | This seems like the wrong approach.  The problem
occurs as
> > > soon as
> > > the
> > >     > | tempdir() gets cleaned up:  there could be information
in
> > > temp
> > > files
> > >     > | that gets lost at that point.  So the solution should
be to
> > > prevent the
> > >     > | cleanup, not to continue on after it has occurred (as
> > > "check > > > TRUE"
> > >     > | does).  This follows the principle that it's
better for the
> > > process to
> > >     > | always die than to sometimes silently produce
incorrect
> > > results.
> > >
> > >     > That is generally true, but also "hard" as we
don't have a
> > > handle on
> > > the OS
> > > .
> > >
> > > Indeed...
> > > and that was the reason I've proposed the simple platform
> > > agnostic tool which does not entirely solve the problem (in this
> > > sense I
> > > agree with "wrong approach") but allows to mitigate it
and (by
> > > followup changes) to work around many use case problems.
> > >
> > >     > | Frederick posted the way to do this in systems using
> > > systemd.  We
> > > should
> > >
> > >     > While that was a very helpful post yet it may only apply
to
> > > Arch
> > > Linux as
> > >     > stated.  My Ubuntu systems at home and work all run
systemd
> > > too, but
> > > do _not_
> > >     > automatically remove tempfiles.
> > >
> > >     > Yet what he suggested is quite right: we should define a
> > > proper
> > > config file
> > >     > for this facility and then possibly also use the /run
> > > directory as
> > > many other
> > >     > services now and (of course) also either TEMPDIR or
later the
> > > code
> > > to have
> > >     > /run be another fallback if TMP, TEMP, TMPDIR, ... are
unset.
> > >
> > >     > Distribution maintainers such as yours truly could then
> > > include this
> > >     > configuration.
> > >
> > >     > | be putting that in place, or the equivalent on systems
> > > using other
> > >     > | tempfile cleanups.  This looks to me like something
that
> > > "make
> > > install"
> > >     > | should do, or perhaps it should be done by people
putting
> > > together
> > >     > | packages for specific systems.
> > >
> > >     > Doesn't 'make install' only write to $RHOME/
and below, plus
> > > $PREFIX/bin ?
> > >
> > > Also, 'make install' is optional for good reasons.
> > > E.g., I never ever run 'make install': I typically always
have many
> > > R
> > > versions, all available in the shell and ESS (Emacs Speaks
> > > Statistics) via symbolic links into a directory on PATH.
> > >
> > > Dirk mentioned (as well) that this is all very platform specific
> > > which I do think is important. From my typical OS point of view:
> > >   Why should the user who runs R not have the right to delete the
> > >   tempdir which was created by the process that she runs and
hence
> > > owns ?
> > >
> > > I agree it would be an improvement if we made such deletion much
> > > harder than it is now, and yes, there may be great (almost)
> > > cross-platform tools available to manage this much better than
> > > we do now, e.g., via open files.
> > >
> > > Before we are there, I would find it useful to have a new
> > > 'tempdir' (i.e. folder/directory for R's temporary
files) to be
> > > re-created manually or automagically in those cases it has
> > > disappeared, and that is within easy reach via the proposed
> > > tempdir() functionality.
> > >
> > > OTOH, I typically live very well by quickly killing
> > > and restarting R (from inside ESS).
> > >
> > > The OP issue was to help newbies and computer-non-experts, the
> > > latter nowadays comprising more than 90% of R users (I'd
guess ~
> > > 98% looking at our otherwise smart students).
> > >
> > > These are typically "slightly" confused when they ask
for help and
> > > get a pretty severe error message:
> > >
> > >   > ?lm
> > >   Error in file(out, "wt") : cannot open the connection
> > >   In addition: Warning message:
> > >   In file(out, "wt") :
> > >     cannot open file '/tmp/RtmpztK6f7/Rtxt36972b91938':
No such
> > > file or
> > > directory
> > >
> > >
> > > Martin
> > >
> > > ______________________________________________
> > > R-devel at r-project.org mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-devel
> > >
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
	[[alternative HTML version deleted]]