R devel - Apr 2017 - tempdir() may be deleted during long-running R session

>>>>> Jeroen Ooms <jeroenooms at gmail.com>
>>>>>     on Tue, 25 Apr 2017 15:05:51 +0200 writes:
    > On Tue, Apr 25, 2017 at 1:00 PM, Martin Maechler
    > <maechler at stat.math.ethz.ch> wrote:
    >> As I've found it is not at all hard to add an option
    >> which checks the existence and if the directory is no
    >> longer "valid", tries to recreate it (and if it fails
    >> doing that it calls the famous R_Suicide(), as it does
    >> when R starts up and tempdir() cannot be initialized
    >> correctly).

    > Perhaps this can also fix the problem with mcparallel
    > deleting the tempdir() when one of its children dies:

   >   file.exists(tempdir()) #TRUE
   >   parallel::mcparallel(q('no'))
   >   file.exists(tempdir()) # FALSE

Thank you, Jeroen, for the extra example.

I now have comitted the new feature... (completely back
compatible: in R's code tempdir() is not yet called with an
argument and the default is  check = FALSE ),
actually in a "suicide-free" way ...  which needed only slightly
more code.

In the worst case, one could save the R session by
   Sys.setenv(TEMPDIR = "<something writable>")
if for instance /tmp/ suddenly became unwritable for the user.

What we could consider is making the default of 'check' settable
by an option, and experiment with setting the option to TRUE, so
all such problems would be auto-solved (says the incurable optimist ...).

Martin

Martin,

Thanks for your work on this.

One thing that seems to be missing from the conversation is that recreating
the temp directory will prevent future failures when R wants to write a
temp file, but the files will, of course, not be there. Any code written
assuming the contract is that the temporary directory, and thus temporary
files, will not be cleaned up before the R process exits (which was my
naive assumption before this thread, and is the behavior AFAICT on all the
systems I regularly use) will still break.

I'm not saying that's necessarily fixable (though the R keeping a
permanent
pointer to a file in the dir suggested by Malcom might? fix it.), but I
would argue if it IS fixable, a fix that includes that would be preferable.

Best,
~G

On Tue, Apr 25, 2017 at 8:53 AM, Martin Maechler <maechler at
stat.math.ethz.ch
> wrote:
> >>>>> Jeroen Ooms <jeroenooms at gmail.com>
> >>>>>     on Tue, 25 Apr 2017 15:05:51 +0200 writes:
>
>     > On Tue, Apr 25, 2017 at 1:00 PM, Martin Maechler
>     > <maechler at stat.math.ethz.ch> wrote:
>     >> As I've found it is not at all hard to add an option
>     >> which checks the existence and if the directory is no
>     >> longer "valid", tries to recreate it (and if it
fails
>     >> doing that it calls the famous R_Suicide(), as it does
>     >> when R starts up and tempdir() cannot be initialized
>     >> correctly).
>
>     > Perhaps this can also fix the problem with mcparallel
>     > deleting the tempdir() when one of its children dies:
>
>    >   file.exists(tempdir()) #TRUE
>    >   parallel::mcparallel(q('no'))
>    >   file.exists(tempdir()) # FALSE
>
> Thank you, Jeroen, for the extra example.
>
> I now have comitted the new feature... (completely back
> compatible: in R's code tempdir() is not yet called with an
> argument and the default is  check = FALSE ),
> actually in a "suicide-free" way ...  which needed only slightly
> more code.
>
> In the worst case, one could save the R session by
>    Sys.setenv(TEMPDIR = "<something writable>")
> if for instance /tmp/ suddenly became unwritable for the user.
>
> What we could consider is making the default of 'check' settable
> by an option, and experiment with setting the option to TRUE, so
> all such problems would be auto-solved (says the incurable optimist ...).
>
> Martin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


-- 
Gabriel Becker, PhD
Associate Scientist (Bioinformatics)
Genentech Research

	[[alternative HTML version deleted]]

> Martin,
 > 
 > Thanks for your work on this.
 > 
 > One thing that seems to be missing from the conversation is that
recreating
 > the temp directory will prevent future failures when R wants to write a
 > temp file, but the files will, of course, not be there. Any code written
 > assuming the contract is that the temporary directory, and thus temporary
 > files, will not be cleaned up before the R process exits (which was my
 > naive assumption before this thread, and is the behavior AFAICT on all the
 > systems I regularly use) will still break.
 > 

That is the kind of scenario I was hoping to obviate with my suggestion...

 > I'm not saying that's necessarily fixable (though the R keeping a
permanent
 > pointer to a file in the dir suggested by Malcom might? fix it.), 

(and, FWIW, that's "Malcolm" with two "l"s.  I think all
those missing "l"s are flattened out versions of all the extra close
parens I typed in the 80s that somehow got lost on the nets...)))
> but I
 > would argue if it IS fixable, a fix that includes that would be
preferable.

Agreed!

 > 
 > Best,
 > ~G
 > 
 > On Tue, Apr 25, 2017 at 8:53 AM, Martin Maechler
 > <maechler at stat.math.ethz.ch
 > > wrote:
 > 
 > > >>>>> Jeroen Ooms <jeroenooms at gmail.com>
 > > >>>>>     on Tue, 25 Apr 2017 15:05:51 +0200 writes:
 > >
 > >     > On Tue, Apr 25, 2017 at 1:00 PM, Martin Maechler
 > >     > <maechler at stat.math.ethz.ch> wrote:
 > >     >> As I've found it is not at all hard to add an option
 > >     >> which checks the existence and if the directory is no
 > >     >> longer "valid", tries to recreate it (and if
it fails
 > >     >> doing that it calls the famous R_Suicide(), as it does
 > >     >> when R starts up and tempdir() cannot be initialized
 > >     >> correctly).
 > >
 > >     > Perhaps this can also fix the problem with mcparallel
 > >     > deleting the tempdir() when one of its children dies:
 > >
 > >    >   file.exists(tempdir()) #TRUE
 > >    >   parallel::mcparallel(q('no'))
 > >    >   file.exists(tempdir()) # FALSE
 > >
 > > Thank you, Jeroen, for the extra example.
 > >
 > > I now have comitted the new feature... (completely back
 > > compatible: in R's code tempdir() is not yet called with an
 > > argument and the default is  check = FALSE ),
 > > actually in a "suicide-free" way ...  which needed only
slightly
 > > more code.
 > >
 > > In the worst case, one could save the R session by
 > >    Sys.setenv(TEMPDIR = "<something writable>")
 > > if for instance /tmp/ suddenly became unwritable for the user.
 > >
 > > What we could consider is making the default of 'check'
settable
 > > by an option, and experiment with setting the option to TRUE, so
 > > all such problems would be auto-solved (says the incurable optimist
...).
 > >
 > > Martin
 > >
 > > ______________________________________________
 > > R-devel at r-project.org mailing list
 > > https://stat.ethz.ch/mailman/listinfo/r-devel
 > >
 > 
 > 
 > 
 > --
 > Gabriel Becker, PhD
 > Associate Scientist (Bioinformatics)
 > Genentech Research
 > 
 > 	[[alternative HTML version deleted]]
 > 
 > ______________________________________________
 > R-devel at r-project.org mailing list
 > https://stat.ethz.ch/mailman/listinfo/r-devel

Hi Gabriel,

Thanks for asking for a better solution, as far as actually preventing
temporary files from getting deleted in the first place.

I still don't know very much about other peoples' distributions, but
Arch uses Systemd which is the culprit on my system. Systemd's
'tmpfiles.d(5)' man page says we can create configuration files in
locations like

    /usr/lib/tmpfiles.d/*.conf
    /etc/tmpfiles.d/*.conf

which control when temporary files are deleted. There is an 'x'
specifier which accepts glob paths and can protect everything in
/tmp/Rtmp* ...:

    $ mkdir /tmp/Rtmpaoeu
    $ touch -d "12 days ago" /tmp/Rtmpaoeu
    $ sudo systemd-tmpfiles --clean
    $ ls /tmp/Rtmpaoeu
    ls: cannot access '/tmp/Rtmpaoeu': No such file or directory

    $ sudo sh -c "echo 'x /tmp/Rtmp*' >
/etc/tmpfiles.d/R.conf"
    $ mkdir /tmp/Rtmpaoeu
    $ touch -d "12 days ago" /tmp/Rtmpaoeu
    $ sudo systemd-tmpfiles --clean
    $ ls /tmp/Rtmpaoeu
    (still there)

I guess installing such a file is something that would be done by the
various distribution-specific R packages. Even though I run R from a
home-directory compiled version, I have my distribution's binary
package installed globally, and so I would get the benefit of this
protection from the distribution package. If this sounds like it makes
sense then I can ask the Arch package maintainer to do it. Of course I
don't need permission but it would be good to hear if I'm missing or
forgetting something.

Based on what other packages are doing the file should probably be
named:

    /usr/lib/tmpfiles.d/R.conf

and contain:

    x /tmp/Rtmp*

(For example on my system I have stuff like this owned by various
packages:

    $ pacman -Qo /usr/lib/tmpfiles.d/*
    /usr/lib/tmpfiles.d/apache.conf is owned by apache 2.4.25-1
    /usr/lib/tmpfiles.d/bind.conf is owned by bind 9.11.0.P3-3
    /usr/lib/tmpfiles.d/colord.conf is owned by colord 1.3.4-1
    /usr/lib/tmpfiles.d/etc.conf is owned by systemd 232-8
    /usr/lib/tmpfiles.d/gvfsd-fuse-tmpfiles.conf is owned by gvfs 1.30.3-1
    ...

)

Thanks!

Frederick

On Tue, Apr 25, 2017 at 09:03:01AM -0700, Gabriel Becker
wrote:
> Martin,
> 
> Thanks for your work on this.
> 
> One thing that seems to be missing from the conversation is that recreating
> the temp directory will prevent future failures when R wants to write a
> temp file, but the files will, of course, not be there. Any code written
> assuming the contract is that the temporary directory, and thus temporary
> files, will not be cleaned up before the R process exits (which was my
> naive assumption before this thread, and is the behavior AFAICT on all the
> systems I regularly use) will still break.
> 
> I'm not saying that's necessarily fixable (though the R keeping a
permanent
> pointer to a file in the dir suggested by Malcom might? fix it.), but I
> would argue if it IS fixable, a fix that includes that would be preferable.
> 
> Best,
> ~G
> 
> On Tue, Apr 25, 2017 at 8:53 AM, Martin Maechler <maechler at
stat.math.ethz.ch
> > wrote:
> 
> > >>>>> Jeroen Ooms <jeroenooms at gmail.com>
> > >>>>>     on Tue, 25 Apr 2017 15:05:51 +0200 writes:
> >
> >     > On Tue, Apr 25, 2017 at 1:00 PM, Martin Maechler
> >     > <maechler at stat.math.ethz.ch> wrote:
> >     >> As I've found it is not at all hard to add an option
> >     >> which checks the existence and if the directory is no
> >     >> longer "valid", tries to recreate it (and if it
fails
> >     >> doing that it calls the famous R_Suicide(), as it does
> >     >> when R starts up and tempdir() cannot be initialized
> >     >> correctly).
> >
> >     > Perhaps this can also fix the problem with mcparallel
> >     > deleting the tempdir() when one of its children dies:
> >
> >    >   file.exists(tempdir()) #TRUE
> >    >   parallel::mcparallel(q('no'))
> >    >   file.exists(tempdir()) # FALSE
> >
> > Thank you, Jeroen, for the extra example.
> >
> > I now have comitted the new feature... (completely back
> > compatible: in R's code tempdir() is not yet called with an
> > argument and the default is  check = FALSE ),
> > actually in a "suicide-free" way ...  which needed only
slightly
> > more code.
> >
> > In the worst case, one could save the R session by
> >    Sys.setenv(TEMPDIR = "<something writable>")
> > if for instance /tmp/ suddenly became unwritable for the user.
> >
> > What we could consider is making the default of 'check'
settable
> > by an option, and experiment with setting the option to TRUE, so
> > all such problems would be auto-solved (says the incurable optimist
...).
> >
> > Martin
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
> 
> 
> 
> -- 
> Gabriel Becker, PhD
> Associate Scientist (Bioinformatics)
> Genentech Research
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>