thr3ads.net - R devel - [Rd] Questions/suggestions about new staged installation [Apr 2019]

If this information is useful, please help other people find it:
Share via:

Pages, Herve

2019-Apr-25 01:11 UTC

[Rd] Questions/suggestions about new staged installation

Hi,

I was playing around with inotifywait (great tool!) to see the new 
staged installation of source packages in action. In one terminal I'm 
monitoring the create/delete/move events of the installation library with:

 ? inotifywait -m --timefmt '%F %T' --format '%T -- %w %e %f' -e
create
-e delete -e move path/to/R/library/

While in another terminal I install CRAN package abc with:

 ? R CMD INSTALL abc_2.1.tar.gz

All deps are already installed.

## With R 3.5

When installing abc the first time with R 3.5 inotifywait reports the 
following events:

2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR 
00LOCK-abc
2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR 
00LOCK-

Then on subsequent times:

2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR 
00LOCK-abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ 
MOVED_FROM,ISDIR abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR 
00LOCK-abc

IIUC the additional 'MOVED_FROM,ISDIR abc'? event on subsequent installs
is triggered by the backing up of the earlier installation in case the 
new installation fails.

## With R 3.6

First installation:

2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR 
00LOCK-abc
2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/ 
MOVED_TO,ISDIR abc
2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR 
00LOCK-abc

Subsequent installations:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR 
00LOCK-abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ 
MOVED_FROM,ISDIR abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/ 
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR 
00LOCK-abc

IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package 
installation folder being moved from the temporary location to the final 
location.

However I wonder about the CREATE,ISDIR abc event. It looks like even 
for a staged installation tools:::.install_packages() still creates the 
empty abc dir. This should no longer be necessary because this empty 
folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I missing
something?

Another thing: the times indicate that the 'MOVED_TO,ISDIR abc' event 
(move from temp to final location) happens about 3-4 seconds after the 
'MOVED_FROM,ISDIR abc' event (backing up). In this interval of time, the
abc package is missing. Wouldn't it be safer to leave the earlier 
installation where it is and to replace it when the new installation 
folder is moved from the temporary location to the final location? Said 
otherwise, the backing up step no longer seems necessary with staged 
installs. So inotifywait would report something like this:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR 
00LOCK-abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/ 
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR 
00LOCK-abc

That is, only 3 events. On first and subsequent installation.

Cheers,

H.


 > sessionInfo()
R version 3.6.0 beta (2019-04-12 r76385)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS:?? /home/hpages/R/R-3.6.r76385/lib/libRblas.so
LAPACK: /home/hpages/R/R-3.6.r76385/lib/libRlapack.so

locale:
 ?[1] LC_CTYPE=en_US.UTF-8?????? LC_NUMERIC=C
 ?[3] LC_TIME=en_US.UTF-8??????? LC_COLLATE=en_US.UTF-8
 ?[5] LC_MONETARY=en_US.UTF-8??? LC_MESSAGES=en_US.UTF-8
 ?[7] LC_PAPER=en_US.UTF-8?????? LC_NAME=C
 ?[9] LC_ADDRESS=C?????????????? LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats???? graphics? grDevices utils???? datasets? methods base

loaded via a namespace (and not attached):
[1] compiler_3.6.0


-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

Tomas Kalibera

2019-Apr-25 11:57 UTC

head link

[Rd] Questions/suggestions about new staged installation

On 4/25/19 3:11 AM, Pages, Herve wrote:> Hi,
>
> I was playing around with inotifywait (great tool!) to see the new
> staged installation of source packages in action. In one terminal I'm
> monitoring the create/delete/move events of the installation library with:
>
>   ? inotifywait -m --timefmt '%F %T' --format '%T -- %w %e
%f' -e create
> -e delete -e move path/to/R/library/
>
> While in another terminal I install CRAN package abc with:
>
>   ? R CMD INSTALL abc_2.1.tar.gz
>
> All deps are already installed.
>
> ## With R 3.5
>
> When installing abc the first time with R 3.5 inotifywait reports the
> following events:
>
> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
abc
> 2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
> 00LOCK-
>
> Then on subsequent times:
>
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
> MOVED_FROM,ISDIR abc
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
abc
> 2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
> 00LOCK-abc
>
> IIUC the additional 'MOVED_FROM,ISDIR abc'? event on subsequent
installs
> is triggered by the backing up of the earlier installation in case the
> new installation fails.
>
> ## With R 3.6
>
> First installation:
>
> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
abc
> 2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> Subsequent installations:
>
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_FROM,ISDIR abc
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
abc
> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package
> installation folder being moved from the temporary location to the final
> location.
>
> However I wonder about the CREATE,ISDIR abc event. It looks like even
> for a staged installation tools:::.install_packages() still creates the
> empty abc dir. This should no longer be necessary because this empty
> folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I
missing
> somethin
Yes, the final directory location gets created before the installation. 
I did this to minimize the added cognitive complexity to the code. I 
could explore if this could be changed, if the installation could work 
without that directory (which would mean implementing and testing on all 
CRAN/BIOC, installing in different modes, so certainly some effort).? Is 
the current implementation causing any problem?
> Another thing: the times indicate that the 'MOVED_TO,ISDIR abc'
event
> (move from temp to final location) happens about 3-4 seconds after the
> 'MOVED_FROM,ISDIR abc' event (backing up). In this interval of
time, the
> abc package is missing. Wouldn't it be safer to leave the earlier
> installation where it is and to replace it when the new installation
> folder is moved from the temporary location to the final location? Said
> otherwise, the backing up step no longer seems necessary with staged
> installs. So inotifywait would report something like this:
>
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> That is, only 3 events. On first and subsequent installation.
I don't think that would be possible. The package is test-loaded also 
from the final installation directory, and if that fails, restored from 
a back-up. This test-loading is useful as it can detect some problems 
with hard-coded directories possibly not detected before on test-loading 
from the temporary installation directory.

Best,
Tomas
>
> Cheers,
>
> H.
>
>
>   > sessionInfo()
> R version 3.6.0 beta (2019-04-12 r76385)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 16.04.5 LTS
>
> Matrix products: default
> BLAS:?? /home/hpages/R/R-3.6.r76385/lib/libRblas.so
> LAPACK: /home/hpages/R/R-3.6.r76385/lib/libRlapack.so
>
> locale:
>   ?[1] LC_CTYPE=en_US.UTF-8?????? LC_NUMERIC=C
>   ?[3] LC_TIME=en_US.UTF-8??????? LC_COLLATE=en_US.UTF-8
>   ?[5] LC_MONETARY=en_US.UTF-8??? LC_MESSAGES=en_US.UTF-8
>   ?[7] LC_PAPER=en_US.UTF-8?????? LC_NAME=C
>   ?[9] LC_ADDRESS=C?????????????? LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats???? graphics? grDevices utils???? datasets? methods base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.6.0
>
>

Pages, Herve

2019-Apr-25 12:20 UTC

head link

[Rd] Questions/suggestions about new staged installation

On 4/25/19 04:57, Tomas Kalibera wrote:
> On 4/25/19 3:11 AM, Pages, Herve wrote:
>> Hi,
>>
>> I was playing around with inotifywait (great tool!) to see the new
>> staged installation of source packages in action. In one terminal
I'm
>> monitoring the create/delete/move events of the installation library 
>> with:
>>
>> ? ? inotifywait -m --timefmt '%F %T' --format '%T -- %w %e
%f' -e create
>> -e delete -e move path/to/R/library/
>>
>> While in another terminal I install CRAN package abc with:
>>
>> ? ? R CMD INSTALL abc_2.1.tar.gz
>>
>> All deps are already installed.
>>
>> ## With R 3.5
>>
>> When installing abc the first time with R 3.5 inotifywait reports the
>> following events:
>>
>> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/
CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/
DELETE,ISDIR
>> 00LOCK-
>>
>> Then on subsequent times:
>>
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
>> MOVED_FROM,ISDIR abc
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/
DELETE,ISDIR
>> 00LOCK-abc
>>
>> IIUC the additional 'MOVED_FROM,ISDIR abc'? event on subsequent
installs
>> is triggered by the backing up of the earlier installation in case the
>> new installation fails.
>>
>> ## With R 3.6
>>
>> First installation:
>>
>> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/
CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/
DELETE,ISDIR
>> 00LOCK-abc
>>
>> Subsequent installations:
>>
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_FROM,ISDIR abc
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/
DELETE,ISDIR
>> 00LOCK-abc
>>
>> IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the
package
>> installation folder being moved from the temporary location to the
final
>> location.
>>
>> However I wonder about the CREATE,ISDIR abc event. It looks like even
>> for a staged installation tools:::.install_packages() still creates the
>> empty abc dir. This should no longer be necessary because this empty
>> folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am
I missing
>> somethin
>
> Yes, the final directory location gets created before the 
> installation. I did this to minimize the added cognitive complexity to 
> the code. I could explore if this could be changed, if the 
> installation could work without that directory (which would mean 
> implementing and testing on all CRAN/BIOC, installing in different 
> modes, so certainly some effort).? Is the current implementation 
> causing any problem?
>
>> Another thing: the times indicate that the 'MOVED_TO,ISDIR abc'
event
>> (move from temp to final location) happens about 3-4 seconds after the
>> 'MOVED_FROM,ISDIR abc' event (backing up). In this interval of
time, the
>> abc package is missing. Wouldn't it be safer to leave the earlier
>> installation where it is and to replace it when the new installation
>> folder is moved from the temporary location to the final location? Said
>> otherwise, the backing up step no longer seems necessary with staged
>> installs. So inotifywait would report something like this:
>>
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/
DELETE,ISDIR
>> 00LOCK-abc
>>
>> That is, only 3 events. On first and subsequent installation.
>
> I don't think that would be possible. The package is test-loaded also 
> from the final installation directory, and if that fails, restored 
> from a back-up. This test-loading is useful as it can detect some 
> problems with hard-coded directories possibly not detected before on 
> test-loading from the temporary installation directory.
Makes sense.

So backing up is still needed.

But couldn't the gap of 3-4 seconds where the abc package is missing be 
reduced by backing up right before moving the newly installed package 
from the temporary location to the final location?

In other words, instead of:

 ??? back up, then install to temp loc, then move to final loc

do:

 ??? install to temp loc, then back up, then move to final loc

This would reduce the gap to almost 0 seconds on a Unix file system 
where moving a folder is very cheap (even if the folder is big), at 
least when moving doesn't cross partition borders.

Not a big deal for a light package like abc but some Bioconductor 
packages take about 20 min. to compile!

Thanks,

H.

>
>
> Best,
> Tomas
>
>>
>> Cheers,
>>
>> H.
>>
>>
>> ? > sessionInfo()
>> R version 3.6.0 beta (2019-04-12 r76385)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>> Running under: Ubuntu 16.04.5 LTS
>>
>> Matrix products: default
>> BLAS:?? /home/hpages/R/R-3.6.r76385/lib/libRblas.so
>> LAPACK: /home/hpages/R/R-3.6.r76385/lib/libRlapack.so
>>
>> locale:
>> ? ?[1] LC_CTYPE=en_US.UTF-8?????? LC_NUMERIC=C
>> ? ?[3] LC_TIME=en_US.UTF-8??????? LC_COLLATE=en_US.UTF-8
>> ? ?[5] LC_MONETARY=en_US.UTF-8??? LC_MESSAGES=en_US.UTF-8
>> ? ?[7] LC_PAPER=en_US.UTF-8?????? LC_NAME=C
>> ? ?[9] LC_ADDRESS=C?????????????? LC_TELEPHONE=C
>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats???? graphics? grDevices utils???? datasets? methods base
>>
>> loaded via a namespace (and not attached):
>> [1] compiler_3.6.0
>>
>>
>-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319

Apparently Analagous Threads

Search for more reasonably related threads

R devel - Apr 2019 - Questions/suggestions about new staged installation

[Rd] Questions/suggestions about new staged installation

[Rd] Questions/suggestions about new staged installation

[Rd] Questions/suggestions about new staged installation

Apparently Analagous Threads