On Fri, 3 Jul 2020 at 19:03, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote: > > But the mass rebuild process is very different, because releng doesn't > > follow any particular order (they don't need to, because nothing > > really changed). > > > > The question is whether there is any tool to resolve dependencies and > > generate a list of builds, as I do with the CRAN database, for an > > arbitrary list of RPMs. That would be perfect. In theory, this should > > be possible with dnf, but when I tried, I failed (or my script would > > theoretically work, but it would take ages to complete). I also took a > > look at releng's repos on Pagure, but I didn't find any tool to do > > this either. > > > > We should bring this to devel, maybe ask Miro and other Python folks > > who may have The Method(TM), as they are rebuilding stuff on a daily > > basis. > > I think that I meant the way that the python packages are rebuilt for each new > python version. I should have said Python mass rebuild. > > I took your work as the basis, together with Tom's help and I used dnf to get > a list of R packages (assuming that their name starts with R- ). The advantage > of this approach is that it works for CRAN and Bioconductor. > > $ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail - > n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt > > That gave the list of the (source) packages to rebuild. > > In order to get the list of dependencies I used a similar strategy: > $ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery -- > disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/ > \n/,/g'; done) > r-packages-dependencies.csv > > The resulting file is a csv file with the first column as the package and the > remaining columns as the dependencies. > > Now the idea was to process this file using a python script to create a direct > graph of dependencies (using networkx). In the process non R packages are > removed and some dependencies are cleaned (e.g. -devel packages direct to the > source package).Nice! What if we create a group "R" on Pagure and a repo "fedora-scripts" or something like that?> The advantage of this is that it allows to identify cycles. > > My only surprise in this process was to find that several packages require R- > rpm-macros. > > That causes some dependency cycles because R-rpm-macros also requires > R-rprintf,R-devel,R-knitr,R-stringi,R-testthat.Are these the packages that require R-rpm-macros or others?> Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the > dependency. Do you agree?The packages are most probably maintained by Elliott, right? He would know better why they have this dependency. -- I?aki ?car
On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote:> Nice! What if we create a group "R" on Pagure and a repo > "fedora-scripts" or something like that?I would like to improve the scripts but FWIW here it comes a rough version of the script I used. The python script loads the csv file and constructs a directed graph and it starts to remove the leave packages (pruning or trimming if you take a gardening analogy). The csv files has the dependency of the packages. It is always possible to go to the packages that remain after the first stage and using a bootstrap mode to remove most of the dependencies and then to run the code again. This procedure was enough to process all the packages. -- Jos? Ab?lio -------------- next part -------------- A non-text attachment was scrubbed... Name: clean-deps.py Type: text/x-python Size: 1675 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-sig-fedora/attachments/20200721/705e2cb4/attachment-0001.py> -------------- next part -------------- A non-text attachment was scrubbed... Name: r-packages-dependencies.csv Type: text/csv Size: 33479 bytes Desc: not available URL: <https://stat.ethz.ch/pipermail/r-sig-fedora/attachments/20200721/705e2cb4/attachment-0001.csv>
On Tue, 21 Jul 2020 at 17:05, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote: > > Nice! What if we create a group "R" on Pagure and a repo > > "fedora-scripts" or something like that? > > I would like to improve the scripts but FWIW here it comes a rough version of > the script I used. > > The python script loads the csv file and constructs a directed graph and it > starts to remove the leave packages (pruning or trimming if you take a > gardening analogy). > > The csv files has the dependency of the packages. It is always possible to go > to the packages that remain after the first stage and using a bootstrap mode > to remove most of the dependencies and then to run the code again. > > This procedure was enough to process all the packages.Looks great! I've created an R group on Pagure and added you all (jamatos, qulogic, spot) to it (feel free to add anyone else that may be interested). Please, create a repo there with these tools. I'll transfer https://pagure.io/fedora-r-packages there too (if that's possible; or just mirror it if not). -- I?aki ?car
On Tue, 21 Jul 2020 at 11:05, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > On Friday, 3 July 2020 18.36.17 WEST I?aki Ucar wrote: > > Nice! What if we create a group "R" on Pagure and a repo > > "fedora-scripts" or something like that? > > I would like to improve the scripts but FWIW here it comes a rough version of > the script I used. > > The python script loads the csv file and constructs a directed graph and it > starts to remove the leave packages (pruning or trimming if you take a > gardening analogy). >I'm not exactly fluent in Haskell, but Jens Petersen has a tool called fbrnch [1], which is able to calculate the build order, and then submit them, and even handles override for non-Rawhide builds. It does need you to handle some of the bootstrap conditions yourself, but with a defined workflow, it might be possible to request that it handle that too.> The csv files has the dependency of the packages. It is always possible to go > to the packages that remain after the first stage and using a bootstrap mode > to remove most of the dependencies and then to run the code again. > > This procedure was enough to process all the packages. > -- > Jos? Ab?lio[1] https://github.com/juhp/fbrnch/ -- Elliott