On Mon, 29 Jun 2020 at 14:25, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > Again you guessed right, that was the main idea. :-) > As I told above and to reiterate it, the idea of this work is to make it easier to automate the > process. Similarly to how we do the mass builds for all the packages.But the mass rebuild process is very different, because releng doesn't follow any particular order (they don't need to, because nothing really changed). The question is whether there is any tool to resolve dependencies and generate a list of builds, as I do with the CRAN database, for an arbitrary list of RPMs. That would be perfect. In theory, this should be possible with dnf, but when I tried, I failed (or my script would theoretically work, but it would take ages to complete). I also took a look at releng's repos on Pagure, but I didn't find any tool to do this either. We should bring this to devel, maybe ask Miro and other Python folks who may have The Method(TM), as they are rebuilding stuff on a daily basis. -- I?aki ?car
On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote:> But the mass rebuild process is very different, because releng doesn't > follow any particular order (they don't need to, because nothing > really changed). > > The question is whether there is any tool to resolve dependencies and > generate a list of builds, as I do with the CRAN database, for an > arbitrary list of RPMs. That would be perfect. In theory, this should > be possible with dnf, but when I tried, I failed (or my script would > theoretically work, but it would take ages to complete). I also took a > look at releng's repos on Pagure, but I didn't find any tool to do > this either. > > We should bring this to devel, maybe ask Miro and other Python folks > who may have The Method(TM), as they are rebuilding stuff on a daily > basis.I think that I meant the way that the python packages are rebuilt for each new python version. I should have said Python mass rebuild. I took your work as the basis, together with Tom's help and I used dnf to get a list of R packages (assuming that their name starts with R- ). The advantage of this approach is that it works for CRAN and Bioconductor. $ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail - n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt That gave the list of the (source) packages to rebuild. In order to get the list of dependencies I used a similar strategy: $ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery -- disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/ \n/,/g'; done) > r-packages-dependencies.csv The resulting file is a csv file with the first column as the package and the remaining columns as the dependencies. Now the idea was to process this file using a python script to create a direct graph of dependencies (using networkx). In the process non R packages are removed and some dependencies are cleaned (e.g. -devel packages direct to the source package). The advantage of this is that it allows to identify cycles. My only surprise in this process was to find that several packages require R- rpm-macros. That causes some dependency cycles because R-rpm-macros also requires R-rprintf,R-devel,R-knitr,R-stringi,R-testthat. Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the dependency. Do you agree? -- Jos? Ab?lio
On Fri, 3 Jul 2020 at 19:03, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote: > > But the mass rebuild process is very different, because releng doesn't > > follow any particular order (they don't need to, because nothing > > really changed). > > > > The question is whether there is any tool to resolve dependencies and > > generate a list of builds, as I do with the CRAN database, for an > > arbitrary list of RPMs. That would be perfect. In theory, this should > > be possible with dnf, but when I tried, I failed (or my script would > > theoretically work, but it would take ages to complete). I also took a > > look at releng's repos on Pagure, but I didn't find any tool to do > > this either. > > > > We should bring this to devel, maybe ask Miro and other Python folks > > who may have The Method(TM), as they are rebuilding stuff on a daily > > basis. > > I think that I meant the way that the python packages are rebuilt for each new > python version. I should have said Python mass rebuild. > > I took your work as the basis, together with Tom's help and I used dnf to get > a list of R packages (assuming that their name starts with R- ). The advantage > of this approach is that it works for CRAN and Bioconductor. > > $ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail - > n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt > > That gave the list of the (source) packages to rebuild. > > In order to get the list of dependencies I used a similar strategy: > $ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery -- > disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/ > \n/,/g'; done) > r-packages-dependencies.csv > > The resulting file is a csv file with the first column as the package and the > remaining columns as the dependencies. > > Now the idea was to process this file using a python script to create a direct > graph of dependencies (using networkx). In the process non R packages are > removed and some dependencies are cleaned (e.g. -devel packages direct to the > source package).Nice! What if we create a group "R" on Pagure and a repo "fedora-scripts" or something like that?> The advantage of this is that it allows to identify cycles. > > My only surprise in this process was to find that several packages require R- > rpm-macros. > > That causes some dependency cycles because R-rpm-macros also requires > R-rprintf,R-devel,R-knitr,R-stringi,R-testthat.Are these the packages that require R-rpm-macros or others?> Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the > dependency. Do you agree?The packages are most probably maintained by Elliott, right? He would know better why they have this dependency. -- I?aki ?car
On Fri, 3 Jul 2020 at 13:03, Jos? Ab?lio Matos <jamatos at fc.up.pt> wrote:> > On Monday, 29 June 2020 13.46.02 WEST I?aki Ucar wrote: > > But the mass rebuild process is very different, because releng doesn't > > follow any particular order (they don't need to, because nothing > > really changed). > > > > The question is whether there is any tool to resolve dependencies and > > generate a list of builds, as I do with the CRAN database, for an > > arbitrary list of RPMs. That would be perfect. In theory, this should > > be possible with dnf, but when I tried, I failed (or my script would > > theoretically work, but it would take ages to complete). I also took a > > look at releng's repos on Pagure, but I didn't find any tool to do > > this either. > > > > We should bring this to devel, maybe ask Miro and other Python folks > > who may have The Method(TM), as they are rebuilding stuff on a daily > > basis. > > I think that I meant the way that the python packages are rebuilt for each new > python version. I should have said Python mass rebuild. > > I took your work as the basis, together with Tom's help and I used dnf to get > a list of R packages (assuming that their name starts with R- ). The advantage > of this approach is that it works for CRAN and Bioconductor. > > $ dnf list available --disablerepo=\* --enablerepo=rawhide-source R-* | tail - > n +3 | awk -e '{print $1};' | sed -e 's/\.src$//' > r-packages.txt > > That gave the list of the (source) packages to rebuild. > > In order to get the list of dependencies I used a similar strategy: > $ (for p in $(cat r-packages.txt); do echo -n $p, ; dnf -q repoquery -- > disablerepo=\* --enablerepo=rawhide-source --requires $p | sed ':a;N;$!ba;s/ > \n/,/g'; done) > r-packages-dependencies.csv > > The resulting file is a csv file with the first column as the package and the > remaining columns as the dependencies. > > Now the idea was to process this file using a python script to create a direct > graph of dependencies (using networkx). In the process non R packages are > removed and some dependencies are cleaned (e.g. -devel packages direct to the > source package). > > > The advantage of this is that it allows to identify cycles. > > My only surprise in this process was to find that several packages require R- > rpm-macros. > > That causes some dependency cycles because R-rpm-macros also requires > R-rprintf,R-devel,R-knitr,R-stringi,R-testthat. >Your process must have an error, because R-rpm-macros only Requires R-core. It also BuildRequires nothing. R-devel depends on R-rpm-macros, and this is intended, in order to pull in automated dependency generators. If anything else directly requires R-rpm-macros, that is an error, and these are all in packages I don't own.> Since R-rpm-macros is used for Fedora and epel 8 I suggest to remove the > dependency. Do you agree? > > > -- > Jos? Ab?lio >-- Elliott