Thank you Gabriel,
Just for future readers. Below is a base R way to address this common
problem, as instructed by you (+stopifnot to suppress print).
Rscript -e 'stopifnot(file.copy("DESCRIPTION",
file.path(tdir<-tempdir(), "PACKAGES")));
db<-available.packages(paste0("file://", tdir));
install.packages(setdiff(tools::package_dependencies(read.dcf("DESCRIPTION",
fields="Package")[[1L]], db, which="most")[[1L]],
installed.packages(priority="high")[,"Package"]))'
3 liner, 310 chars long command, far from ideal, but does work.
Best,
Jan
On Fri, Oct 28, 2022 at 10:42 PM Gabriel Becker <gabembecker at gmail.com>
wrote:>
> Hi Jan,
>
>
> On Fri, Oct 28, 2022 at 1:57 PM Jan Gorecki <j.gorecki at wit.edu.pl>
wrote:
>>
>> Gabriel,
>>
>> It is the most basic CI use case. One wants to install only
>> dependencies only of the package, and run R CMD check on the package.
>
>
> Really what you're looking for though, is to install all the
dependencies which aren't present right? Excluding base packages is just a
particular way to do that under certain assumptions about the CI environment.
>
> So
>
>
> needed_pkgs <- setdiff(package_dependencies(...),
installed.packages()[,"Package"])
> install.packages(needed_pkgs, repos = fancyrepos)
>
>
> will do what you want without installing the package itself, if that is
important. This will filter out base and recommended packages (which will be
already installed in your CI container, since R is).
>
>
> Now this does not take into account versioned dependencies, so it's not
actually fully correct (whereas installing the package is), but it gets you
where you're trying to go. And in a clean CI container without cached
package installation for the deps, its equivalent.
>
>
> Also, as an aside, if you need to get the base packages, you can do
>
> installed.packages(priority="base")[,"Package"]
>
> base compiler datasets graphics grDevices grid
>
> "base" "compiler" "datasets"
"graphics" "grDevices" "grid"
>
> methods parallel splines stats stats4 tcltk
>
> "methods" "parallel" "splines"
"stats" "stats4" "tcltk"
>
> tools utils
>
> "tools" "utils"
>
>
> (to get base and recommended packages use 'high' instead of
'base')
>
> No need to be reaching down into unexported functions. So if you *really*
only want to exclude base functions (which likely will give you some protection
from versioned dep issues), you can change the code above to
>
> needed_pkgs <- setdiff(package_dependencies(...),
installed.packages(priority = "high")[,"Package"])
> install.packages(needed_pkgs, repos = fancyrepos)
>
> Best,
> ~G
>
>>
>> On Fri, Oct 28, 2022 at 8:42 PM Gabriel Becker <gabembecker at
gmail.com> wrote:
>> >
>> > Hi Jan,
>> >
>> > The reason, I suspect without speaking for R-core, is that by
design you should not be specifying package dependencies as additional packages
to install. install.packages already does this for you, as it did in the
construct of a repository code that I provided previously in the thread. You
should be *only* doing
>> >
>> > install.packages(<pkg in question>, repos = *)
>> >
>> > Then everything happens automatically via extremely well tested
very mature code.
>> >
>> > I (still) don't understand why you'd need to pass
install.packages the vector of dependencies yourself, as that is counter to
install.packages' core design.
>> >
>> > Does that make sense?
>> >
>> > Best,
>> > ~G
>> >
>> > On Fri, Oct 28, 2022 at 12:18 PM Jan Gorecki <j.gorecki at
wit.edu.pl> wrote:
>> >>
>> >> Gabriel,
>> >>
>> >> I am trying to design generic solution that could be applied
to
>> >> arbitrary package. Therefore I went with the latter solution
you
>> >> proposed.
>> >> If we wouldn't have to exclude base packages, then its a 3
liner
>> >>
>> >> file.copy("DESCRIPTION",
file.path(tdir<-tempdir(), "PACKAGES"));
>> >> db<-available.packages(paste0("file://", tdir));
>> >>
utils::install.packages(tools::package_dependencies("pkgname", db,
>> >> which="most")[[1L]])
>> >>
>> >> As you noticed, we still have to filter out base packages.
Otherwise
>> >> it won't be a robust utility that can be used in CI.
Therefore we have
>> >> to add a call to tools:::.get_standard_package_names() which
is an
>> >> internal function (as of now). Not only complicating the call
but also
>> >> putting the functionality outside of safe use.
>> >>
>> >> Considering above, don't you agree that the following one
liner could
>> >> nicely address the problem? The problem that
hundreds/thousands of
>> >> packages are now addressing in their CI scripts by using a
third party
>> >> packages.
>> >>
>> >> utils::install.packages(packages.dcf("DESCRIPTION",
which="most"))
>> >>
>> >> It is hard to me to understand why R members don't
consider this basic
>> >> functionality to be part of base R. Possibly they just
don't need it
>> >> themselves. Yet isn't this sufficient that
hundreds/thousands of
>> >> packages does need this functionality?
>> >>
>> >> Best regards,
>> >> Jan
>> >>
>> >> On Mon, Oct 17, 2022 at 8:39 AM Jan Gorecki <j.gorecki at
wit.edu.pl> wrote:
>> >> >
>> >> > Gabriel and Simon
>> >> >
>> >> > I completely agree with what you are saying.
>> >> > The thing is that obtaining recursive deps, all/most
whatever, is already well supported in core R. What is missing is just this
single functionality I am requesting.
>> >> >
>> >> > If you will look into the branch you can see there is
mirror.packages function meant to mirror a slice of CRAN. It is doing exactly
what you described: package_dependencies; to obtain recursive deps, then
download all, etc.
>> >> > I would love to have this function provided by core R as
well, but we need to start somewhere.
>> >> >
>> >> > There are other use cases as well.
>> >> > For example CI, where one wants to install all/most
dependencies and then run R CMD check. Then we don't worry about recursive
deps are they will be resolved automatically.
>> >> > I don't think it's reasonable to force users to
use 3rd party packages to handle such a common and simple use case. Otherwise
one has to hard code deps in CI script. Not robust at all.
>> >> >
>> >> > packages.dcf and repos.dcf makes all that way easier, and
are solid base for building customized orchestration like mirroring slice of
CRAN.
>> >> >
>> >> > Best regards
>> >> > Jan
>> >> >
>> >> > On Sun, Oct 16, 2022, 01:31 Simon Urbanek
<simon.urbanek at r-project.org> wrote:
>> >> >>
>> >> >> Jan,
>> >> >>
>> >> >> I think using a single DCF as input is not very
practical and would not be useful in the context you describe (creating self
contained repos) since they typically concern a list of packages, but
essentially splitting out the part of install.packages() which determines which
files will be pulled from where would be very useful as it would be trivial to
use it to create repository (what we always do in corporate environments)
instead of installing the packages. I suspect that install packages is already
too complex so instead of adding a flag to install.packages one could move that
functionality into a separate function - we all do that constantly for the sites
we manage, so it would be certainly something worthwhile.
>> >> >>
>> >> >> Cheers,
>> >> >> Simon
>> >> >>
>> >> >>
>> >> >> > On Oct 15, 2022, at 7:14 PM, Jan Gorecki
<j.gorecki at wit.edu.pl> wrote:
>> >> >> >
>> >> >> > Hi Gabriel,
>> >> >> >
>> >> >> > It's very nice usage you provided here.
Maybe instead of adding new
>> >> >> > function we could extend packages_depenedncies
then? To accept file path to
>> >> >> > dsc file.
>> >> >> >
>> >> >> > What about repos.dcf? Maybe additional
repositories could be an attribute
>> >> >> > attached to returned character vector.
>> >> >> >
>> >> >> > The use case is to, for a given package sources,
obtain its dependencies,
>> >> >> > so one can use that for installing
them/mirroring CRAN subset, or whatever.
>> >> >> > The later is especially important for a
production environment where one
>> >> >> > wants to have fixed version of packages, and
mirroring relevant subset of
>> >> >> > CRAN is the most simple, and IMO reliable, way
to manage such environment.
>> >> >> >
>> >> >> > Regards
>> >> >> > Jan
>> >> >> >
>> >> >> > On Fri, Oct 14, 2022, 23:34 Gabriel Becker
<gabembecker at gmail.com> wrote:
>> >> >> >
>> >> >> >> Hi Jan and Jan,
>> >> >> >>
>> >> >> >> Can you explain a little more what exactly
you want the non-recursive,
>> >> >> >> non-version aware dependencies from an
individual package for?
>> >> >> >>
>> >> >> >> Either way package_dependencies will do this
for you* with a little
>> >> >> >> "aggressive convincing". It wants
output from available.packages, but who
>> >> >> >> really cares what it wants? It's a
function and we are people :)
>> >> >> >>
>> >> >> >>> library(tools)
>> >> >> >>> db <-
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION")
>> >> >> >>>
package_dependencies("rtables", db, which =
intersect(c("Depends",
>> >> >> >> "Suggests", "Imports",
"LinkingTo"), colnames(db)))
>> >> >> >> $rtables
>> >> >> >> [1] "methods"
"magrittr" "formatters" "dplyr"
"tibble"
>> >> >> >> [6] "tidyr"
"testthat" "xml2" "knitr"
"rmarkdown"
>> >> >> >> [11] "flextable"
"officer" "stats" "htmltools"
"grid"
>> >> >> >>
>> >> >> >>
>> >> >> >> The only gotcha that I see immediately is
that "LinkingTo" isn't always
>> >> >> >> there (whereas it is with real output from
available.packages). If you
>> >> >> >> know your package doesn't have that (or
that it does) at call time , this
>> >> >> >> becomes a one-liner:
>> >> >> >>
>> >> >> >> package_dependencies("rtables", db
>> >> >> >>
read.dcf("~/gabe/checkedout/rtables_clean/DESCRIPTION"), which
>> >> >> >> c("Depends", "Suggests",
"Imports"))
>> >> >> >> $rtables
>> >> >> >> [1] "methods"
"magrittr" "formatters" "dplyr"
"tibble"
>> >> >> >> [6] "tidyr"
"testthat" "xml2" "knitr"
"rmarkdown"
>> >> >> >> [11] "flextable"
"officer" "stats" "htmltools"
"grid"
>> >> >> >>
>> >> >> >> You can also trick it a slightly different
way by giving it what it
>> >> >> >> actually wants
>> >> >> >>
>> >> >> >>> tdir <- tempdir()
>> >> >> >>>
file.copy("~/gabe/checkedout/rtables_clean/DESCRIPTION",
file.path(tdir,
>> >> >> >> "PACKAGES"))
>> >> >> >> [1] TRUE
>> >> >> >>> avl <-
available.packages(paste0("file://", tdir))
>> >> >> >>> library(tools)
>> >> >> >>>
package_dependencies("rtables", avl)
>> >> >> >> $rtables
>> >> >> >> [1] "methods"
"magrittr" "formatters" "stats"
"htmltools"
>> >> >> >> [6] "grid"
>> >> >> >>
>> >> >> >>>
package_dependencies("rtables", avl, which = "all")
>> >> >> >> $rtables
>> >> >> >> [1] "methods"
"magrittr" "formatters" "stats"
"htmltools"
>> >> >> >> [6] "grid" "dplyr"
"tibble" "tidyr" "testthat"
>> >> >> >> [11] "xml2"
"knitr" "rmarkdown" "flextable"
"officer"
>> >> >> >>
>> >> >> >> So the only real benefits I see that
we'd be picking up here is automatic
>> >> >> >> filtering by priority, and automatic
extraction of the package name from
>> >> >> >> the DESCRIPTION file. I'm not sure
either of those warrant a new exported
>> >> >> >> function that R-core has to maintain
forever.
>> >> >> >>
>> >> >> >> Best,
>> >> >> >> ~G
>> >> >> >>
>> >> >> >> * I haven't tested this across all OSes,
but I dont' know of any reason it
>> >> >> >> wouldn't work generally.
>> >> >> >>
>> >> >> >> On Fri, Oct 14, 2022 at 2:33 PM Jan Gorecki
<j.gorecki at wit.edu.pl> wrote:
>> >> >> >>
>> >> >> >>> Hello Jan,
>> >> >> >>>
>> >> >> >>> Thanks for confirming about many
packages reinventing this missing
>> >> >> >>> functionality.
>> >> >> >>> packages.dcf was not meant handle
versions. It just extracts names of
>> >> >> >>> dependencies... Yes, such a simple
thing, yet missing in base R.
>> >> >> >>>
>> >> >> >>> Versions of packages can be controlled
when setting up R pkgs repo. This
>> >> >> >>> is
>> >> >> >>> how I used to handle it. Making a CRAN
subset mirror of fixed version
>> >> >> >>> pkgs.
>> >> >> >>> BTW. function for that is also included
in mentioned branch. I am just not
>> >> >> >>> proposing it, to increase the chance of
having at least this simple,
>> >> >> >>> missing, functionality merged.
>> >> >> >>>
>> >> >> >>> Best
>> >> >> >>> Jan
>> >> >> >>>
>> >> >> >>> On Fri, Oct 14, 2022, 15:14 Jan Net?k
<netikja at gmail.com> wrote:
>> >> >> >>>
>> >> >> >>>> Hello Jan,
>> >> >> >>>>
>> >> >> >>>> I have seen many packages that
implemented dependencies "extraction" on
>> >> >> >>>> their own for internal purposes and
today I was doing exactly that for
>> >> >> >>>> mine. It's not a big deal using
read.dcf on DESCRIPTION. It was
>> >> >> >>> sufficient
>> >> >> >>>> for me, but I had to take care of
some \n chars (the overall returned
>> >> >> >>> value
>> >> >> >>>> has some rough edges, in my
opinion). However, the function from the
>> >> >> >>> branch
>> >> >> >>>> seems to not care about version
requirements, which are crucial for me.
>> >> >> >>>> Maybe that is something to
reconsider before merging.
>> >> >> >>>>
>> >> >> >>>> Best,
>> >> >> >>>> Jan
>> >> >> >>>>
>> >> >> >>>> p? 14. 10. 2022 v 2:27 odes?latel
Jan Gorecki <j.gorecki at wit.edu.pl>
>> >> >> >>>> napsal:
>> >> >> >>>>
>> >> >> >>>>> Dear R devs,
>> >> >> >>>>>
>> >> >> >>>>> I would like to raise a request
for a simple helper function.
>> >> >> >>>>> Utility function to extract
package dependencies from DESCRIPTION file.
>> >> >> >>>>>
>> >> >> >>>>> I do think that tools package is
better place, for such a fundamental
>> >> >> >>>>> functionality, than community
packages.
>> >> >> >>>>>
>> >> >> >>>>> tools pkg seems perfect fit
(having already great function
>> >> >> >>>>> write_PACKAGES).
>> >> >> >>>>>
>> >> >> >>>>> Functionality I am asking for is
already in R svn repository since
>> >> >> >>> 2016,
>> >> >> >>>>> in
>> >> >> >>>>> a branch tools4pkgs. Function is
called 'packages.dcf'.
>> >> >> >>>>> Another one 'repos.dcf'
would be a good functional complementary to it.
>> >> >> >>>>>
>> >> >> >>>>> Those two simple helper
functions really makes it easier for
>> >> >> >>> organizations
>> >> >> >>>>> to glue together usage of their
own R packages repos and CRAN repo in a
>> >> >> >>>>> smooth way. That could possibly
help to offload CRAN from new
>> >> >> >>> submissions.
>> >> >> >>>>>
>> >> >> >>>>> gh mirror link for easy preview:
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>
https://github.com/wch/r-source/blob/tools4pkgs/src/library/tools/R/packages.R#L419
>> >> >> >>>>>
>> >> >> >>>>> Regards
>> >> >> >>>>> Jan Gorecki
>> >> >> >>>>>
>> >> >> >>>>> [[alternative HTML
version deleted]]
>> >> >> >>>>>
>> >> >> >>>>>
______________________________________________
>> >> >> >>>>> R-devel at r-project.org mailing
list
>> >> >> >>>>>
https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>>
>> >> >> >>> [[alternative HTML version
deleted]]
>> >> >> >>>
>> >> >> >>>
______________________________________________
>> >> >> >>> R-devel at r-project.org mailing list
>> >> >> >>>
https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> > [[alternative HTML version deleted]]
>> >> >> >
>> >> >> > ______________________________________________
>> >> >> > R-devel at r-project.org mailing list
>> >> >> > https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >> >
>> >> >>