R devel - Aug 2019 - New lazyload rdx key type: list(eagerKey=, lazyKeys=)

Prior to R-3.6.0 the keys in the lazyload key files, e.g.
pkg/data/Rdata.rdx or pkg/R/pkg.rdx, seemed to all be 2-long integer
vectors.  Now they can be lists.  The ones I have seen have two components,
"eagerKey" is a 2-long integer vector and "lazyKeys" is a
named list of
2-long integer vectors.
> rdx <- readRDS(system.file(package="survival",
"data", "Rdata.rdx"))
> str(Filter(is.list, rdx$references))
List of 2
 $ env::1:List of 2
  ..$ eagerKey: int [1:2] 273691 183
  ..$ lazyKeys:List of 1
  .. ..$ lines: int [1:2] 273874 284
 $ env::2:List of 2
  ..$ eagerKey: int [1:2] 473142 166
  ..$ lazyKeys:List of 1
  .. ..$ lines: int [1:2] 473308 310

or
>  rdx <- readRDS(system.file(package="lambda.r", "R",
"lambda.r.rdx"))
> length(Filter(is.integer, rdx$references))
[1] 4
> str(Filter(Negate(is.integer), rdx$references))
List of 5
 $ env::5:List of 2
  ..$ eagerKey: int [1:2] 28278 328
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 28606 80
  .. ..$ parseData: int [1:2] 28686 389
 $ env::6:List of 2
  ..$ eagerKey: int [1:2] 29075 327
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 29402 71
  .. ..$ parseData: int [1:2] 29473 321
 $ env::7:List of 2
  ..$ eagerKey: int [1:2] 29794 325
  ..$ lazyKeys:List of 2
  .. ..$ lines    : int [1:2] 30119 117
  .. ..$ parseData: int [1:2] 30236 752
... many more ...

All the ones I've seen involve the environment in srcref attributes and
most packages do not keep that.  Will these be used for more sorts of
environments in the future?

What is the meaning of the lazyKeys?  Are these stored as promises until
needed or is there some special option to never or always load them?

Bill Dunlap
TIBCO Software
wdunlap tibco.com

	[[alternative HTML version deleted]]

On 8/31/19 12:35 AM, William Dunlap via R-devel wrote:
> Prior to R-3.6.0 the keys in the lazyload key files, e.g.
> pkg/data/Rdata.rdx or pkg/R/pkg.rdx, seemed to all be 2-long integer
> vectors.  Now they can be lists.  The ones I have seen have two components,
> "eagerKey" is a 2-long integer vector and "lazyKeys" is
a named list of
> 2-long integer vectors.
>
>> rdx <- readRDS(system.file(package="survival",
"data", "Rdata.rdx"))
>> str(Filter(is.list, rdx$references))
> List of 2
>   $ env::1:List of 2
>    ..$ eagerKey: int [1:2] 273691 183
>    ..$ lazyKeys:List of 1
>    .. ..$ lines: int [1:2] 273874 284
>   $ env::2:List of 2
>    ..$ eagerKey: int [1:2] 473142 166
>    ..$ lazyKeys:List of 1
>    .. ..$ lines: int [1:2] 473308 310
>
> or
>
>>   rdx <- readRDS(system.file(package="lambda.r",
"R", "lambda.r.rdx"))
>> length(Filter(is.integer, rdx$references))
> [1] 4
>> str(Filter(Negate(is.integer), rdx$references))
> List of 5
>   $ env::5:List of 2
>    ..$ eagerKey: int [1:2] 28278 328
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 28606 80
>    .. ..$ parseData: int [1:2] 28686 389
>   $ env::6:List of 2
>    ..$ eagerKey: int [1:2] 29075 327
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 29402 71
>    .. ..$ parseData: int [1:2] 29473 321
>   $ env::7:List of 2
>    ..$ eagerKey: int [1:2] 29794 325
>    ..$ lazyKeys:List of 2
>    .. ..$ lines    : int [1:2] 30119 117
>    .. ..$ parseData: int [1:2] 30236 752
> ... many more ...
>
> All the ones I've seen involve the environment in srcref attributes and
> most packages do not keep that.  Will these be used for more sorts of
> environments in the future?
The new mechanism allows to load lazily selected bindings from an 
environment - previously only the bindings from the package/namespace 
environment were loaded lazily. The mechanism is currently used only for 
"lines" and "parseData" bindings of "srcfile"
environments. Packages
that include parse data will benefit from the latter and all packages 
that include source references will benefit from both. This change made 
it feasible to run package checks with source references enabled, so now 
one can get more useful error traces with execution time overhead 
reasonable for check runs.

The format of the lazy loading database is specific to R release, it can 
change at any time. Specifically this mechanism could be used for more 
environments, but I am not aware of any that would benefit at the moment.
> What is the meaning of the lazyKeys?  Are these stored as promises until
> needed or is there some special option to never or always load them?
Bindings listed in lazyKeys are treated the same way as bindings in the 
package/namespace environment, so they become promises after loading the 
environment. Those promises load the individual objects for the bindings 
when forced (details in makeLazyLoad.R and lazyLoad.R and now also 
explicitly in R-ints).

Best
Tomas
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel