I am running into a problem with the 00LOCK file and .nfs files. It
seemed to be caused by the following
R is installed on NFS
user A has loaded pkg_A containing a dynamically loaded library
user B (the administrator) runs install.packages("pkg_A")
as the final part of the installation process the 00LOCK directory is removed
this creates a .nfs file because user A has the dynamic library open
this .nfs file cannot be removed until user A exits R
because of this, 00LOCK cannot be removed, and if the
install.packages call is going to install packages after pkg_A, they
fail
Now, while there are certainly open questions concerning this, I have
a pretty good idea of the problem, and I even have some ideas on how
to fix it.
But before I spent more time on this, I would like to hear how other
people handle this. In my analysis, this is a problem whenever you
have R installed on NFS, this R is used by multiple users and you
often run update.packages to keep the installation current. There
must be plenty of people out there who do this. In other words:
someone else must have had this problem before (and presumably fixed
it). I don't know much (anything) about NFS, so perhaps this is
caused by our NFS setup.
Comments, before I spent more hours looking into this?
Thanks,
Kasper
On Mon, Apr 26, 2010 at 11:15 AM, Kasper Daniel Hansen < kasperdanielhansen@gmail.com> wrote:> I am running into a problem with the 00LOCK file and .nfs files. It > seemed to be caused by the following > R is installed on NFS > user A has loaded pkg_A containing a dynamically loaded library > user B (the administrator) runs install.packages("pkg_A") > as the final part of the installation process the 00LOCK directory is > removed > this creates a .nfs file because user A has the dynamic library open > this .nfs file cannot be removed until user A exits R > because of this, 00LOCK cannot be removed, and if the > install.packages call is going to install packages after pkg_A, they > fail >Yup, I've seen that. The program "lsof" can be useful to find which process has the library open. User A does not need to exit R; it's enough to detach(package:pkg_A); library(pkg_A);. Now, while there are certainly open questions concerning this, I have> a pretty good idea of the problem, and I even have some ideas on how > to fix it. > > But before I spent more time on this, I would like to hear how other > people handle this. In my analysis, this is a problem whenever you > have R installed on NFS, this R is used by multiple users and you > often run update.packages to keep the installation current.To nitpick: You don't even need multiple users. It could be one user with multiple sessions (one running R interactively, the other installing). Cheers, /Christian There must be plenty of people out there who do this. In other words:> someone else must have had this problem before (and presumably fixed > it). I don't know much (anything) about NFS, so perhaps this is > caused by our NFS setup. > > Comments, before I spent more hours looking into this? > > Thanks, > Kasper > > ______________________________________________ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel >[[alternative HTML version deleted]]
Further experimentation using this patch to install.packages shows that I sometimes have remaining 00LOCK-pkgname directories after doing a massive update on a multi-user system with R installed on NFS. However, I have never had install.packages/update.packages stop midway because of an unremovable 00LOCK directory. I therefore consider the patch to be big improvement for people like me, having a multi-user R installed on NFS. Private followups to my original email shows that I am not the only one with this problem. I would very much like the patch (or some variant hereof) to be considered for inclusion in R-devel. Kasper On Tue, May 18, 2010 at 11:59 AM, Kasper Daniel Hansen <kasperdanielhansen at gmail.com> wrote:> This is a follow-up to an old thread with kind of solution to the > 00LOCK problem on NFS. > > I have made a patch to install.packages to accept a new argument > ?locktype = c("lock", "no-lock", "pkglock") > which is passed to R CMD INSTALL. ?This addition might have > independent interest aside from the NFS problem, as it exposes > functionality from R CMD INSTALL to install.packages and the very > convenient update.packages. ?Patches are at > ?http://www.biostat.jhsph.edu/~khansen/packages2.R-patch > ?http://www.biostat.jhsph.edu/~khansen/install.packages.Rd-patch > (patches to files in the utils package) and both > ?R-devel (R version 2.12.0 Under development (unstable) (2010-05-17 r52025)) > and > ?R-2.11 (R version 2.11.0 Patched (2010-05-17 r52025)) > passed make check-all with these two patches applied. ?I thought about > adding a note describing my findings below to the details section, but > decided against it. > > Regarding the 00LOCK problem. ?In my testing, using the patches above > and setting locktype = "pkglock", makes it possible to deal with the > NFS problem. ?Specifically, I have not been able to make > update.packages() fail midway, due to a un-removable 00LOCK file > (which is not too surprising, as I now have a per-package lock). > > However, sometimes (but far less frequently than before), a > 00LOCK-pkgname directory remains after update/install.packages. > Sometimes this 00LOCK-pkgname directory does not contain any .nfs* > files (!?) and sometimes it does. For this reason, I still precede any > install/update.packages with a check for the existence of a > 00LOCK-pkgname directory and an attempt to remove it. > > The difference between using locktype = "pkglock" and not is > specifically that without, it was possible for update.packages to fail > midway even though there were no 00LOCK* files at the start of the > update process. > > Originally I hypothesized that the presence of the .nfs* files in the > 00LOCK directory had to do with synchronization issues between the > file server and the compute node. ?In order to approach this I tried > to insert a > ?system("sleep 10") > at the beginning of > ?do_cleanup > in > ?tools/R/install.R > but that did not work. > > Since the pkglock approach described above seems to solve this issue > for me, I have not pursued the synchronization issue further. > > Kasper >