Richard W.M. Jones
2021-Jun-13 10:38 UTC
[Libguestfs] virt-builder fails in fedora 34 container:
On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote:> I'm running this inside a fedora 34 container in ovirt-imageio CI: > > 'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', > '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', > '--root-password', 'password:', '-o', > '/var/tmp/imageio-images/cirros-0.3.5' > > The command fails in: > > supermin: ext2: copying kernel modules > supermin: ext2: creating minimal initrd > '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' > supermin: exception: > Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such > file or directory")We use this file in order to construct the minimal initrd with some virtio modules, which is sufficient to bring up the supermin root filesystem. The file itself is used to calculate the correct set of module dependencies (so that, for example, if a virtio kmod needs another kmod then the other kmod will be pulled in). See: https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13b1595b4f/src/format_ext2_initrd.ml I believe in Fedora the file is created via a %post script in kernel-core which runs /bin/kernel-install (from systemd-udev). This in turn is a shell script which runs snippets from /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied from all kinds of other packages. On my machine one of these is /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) which eventually runs depmod -a which creates this file. So those are some of the packages which might be missing. Does modules.dep get created anywhere inside the container? It's possible it moved. It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 which should show (in the dnf install output I guess?) if/when the depmod command runs.> I think I'm missing some package removed from the fedora 34 container, > but I don't have > any idea which one. > > I hope someone has more insight on this.It's curious that things have changed since F33. Could you build containers for both and diff the list of package names? Maybe a dependency is missing or has become a soft dep. rpm -qa --qf '%{name}\n' | sort Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones at redhat.com> wrote:> > On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote: > > I'm running this inside a fedora 34 container in ovirt-imageio CI: > > > > 'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', > > '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', > > '--root-password', 'password:', '-o', > > '/var/tmp/imageio-images/cirros-0.3.5' > > > > The command fails in: > > > > supermin: ext2: copying kernel modules > > supermin: ext2: creating minimal initrd > > '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' > > supermin: exception: > > Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such > > file or directory") > > We use this file in order to construct the minimal initrd with some > virtio modules, which is sufficient to bring up the supermin root > filesystem. The file itself is used to calculate the correct set of > module dependencies (so that, for example, if a virtio kmod needs > another kmod then the other kmod will be pulled in). See: > > https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13b1595b4f/src/format_ext2_initrd.ml > > I believe in Fedora the file is created via a %post script in > kernel-core which runs /bin/kernel-install (from systemd-udev). This > in turn is a shell script which runs snippets from > /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied > from all kinds of other packages. On my machine one of these is > /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) > which eventually runs depmod -a which creates this file. > > So those are some of the packages which might be missing. Does > modules.dep get created anywhere inside the container? It's possible > it moved. > > It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 > which should show (in the dnf install output I guess?) if/when the > depmod command runs. > > > I think I'm missing some package removed from the fedora 34 container, > > but I don't have > > any idea which one. > > > > I hope someone has more insight on this. > > It's curious that things have changed since F33. Could you build > containers for both and diff the list of package names? Maybe a > dependency is missing or has become a soft dep. > > rpm -qa --qf '%{name}\n' | sortI ran this: $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 rpm -qa --qf '%{name}\n' | sort > fedora-34.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 rpm -qa --qf '%{name}\n' | sort > fedora-33.out $ diff -u fedora-33.out fedora-34.out > fedora.diff $ egrep -- ^- fedora.diff > removed-in-fedora-34 $ egrep -- '^[+]' fedora.diff > added-in-fedora-34 $ cat removed-in-fedora-34 --- fedora-33.out 2021-06-13 22:51:13.156150245 +0300 -autogen-libopts -genisoimage -iptables -isl -kbd-legacy -libcbor -libdb-utils -libfido2 -libguestfs-tools-c -libmodman -libusal -nmap-ncat -pciutils -pciutils-libs -python3-idna -rdma-core -selinux-policy-minimum $ cat added-in-fedora-34 +++ fedora-34.out 2021-06-13 22:50:59.650016271 +0300 +guestfs-tools +inih +iptables-legacy +iptables-legacy-libs +langpacks-core-en_GB +langpacks-en_GB +libbsd +libcanberra +libcanberra-gtk3 +libcloudproviders +libpmemobj +libretls +libtdb +netcat +pcsc-lite-libs +selinux-policy-targeted +sound-theme-freedesktop I don't see anything that can explain the difference. I also compared contents of the /lib/modules/xxx/ directories: $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-33 ls -1 /lib/modules/5.12.9-200.fc33.x86_64 > fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out $ podman run -it --rm quay.io/ovirt/imageio-test-fedora-34 ls -1 /lib/modules/5.12.9-300.fc34.x86_64 > fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out $ diff -u fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out --- fedora-33-lib_modules_5.12.9-200.fc33.x86_64.out 2021-06-13 23:12:04.177539132 +0300 +++ fedora-34-lib_modules_5.12.9-300.fc34.x86_64.out 2021-06-13 23:12:18.072675988 +0300 @@ -3,23 +3,14 @@ build config kernel -modules.alias -modules.alias.bin modules.block modules.builtin modules.builtin.alias.bin -modules.builtin.bin modules.builtin.modinfo -modules.dep -modules.dep.bin -modules.devname modules.drm modules.modesetting modules.networking modules.order -modules.softdep -modules.symbols -modules.symbols.bin source symvers.gz updates Looks like some post install step is missing on the fedora 34 container.
On Sun, Jun 13, 2021 at 1:38 PM Richard W.M. Jones <rjones at redhat.com> wrote:> > On Sat, Jun 12, 2021 at 11:11:22PM +0300, Nir Soffer wrote: > > I'm running this inside a fedora 34 container in ovirt-imageio CI: > > > > 'virt-builder', '-v', '-x', 'cirros-0.3.5', '--write', > > '/etc/cirros-init/config:DATASOURCE_LIST="nocloud"', > > '--root-password', 'password:', '-o', > > '/var/tmp/imageio-images/cirros-0.3.5' > > > > The command fails in: > > > > supermin: ext2: copying kernel modules > > supermin: ext2: creating minimal initrd > > '/var/tmp/.guestfs-0/appliance.d.t7nny0l7/initrd' > > supermin: exception: > > Sys_error("/lib/modules/5.12.9-300.fc34.x86_64/modules.dep: No such > > file or directory") > > We use this file in order to construct the minimal initrd with some > virtio modules, which is sufficient to bring up the supermin root > filesystem. The file itself is used to calculate the correct set of > module dependencies (so that, for example, if a virtio kmod needs > another kmod then the other kmod will be pulled in). See:Is it correct to use /lib/modules/kernel-X/modules.dep when the file was created on the system building the container while running the container on kernel-Y? For example, we build the container in quay.io (no idea which kernel) and run the container in travis (Ubuntu xenial, kernel 5.4), or locally (Fedora 32, kernel 5.11). Should virt builder/supermin use "demod --show" to generate a temporary modules.dep, instead of depending on system modules.dep when running in a container?> https://github.com/libguestfs/supermin/blob/c89e4923909ef84541ac1c047bf64d13b1595b4f/src/format_ext2_initrd.ml > > I believe in Fedora the file is created via a %post script in > kernel-core which runs /bin/kernel-install (from systemd-udev). This > in turn is a shell script which runs snippets from > /etc/kernel/install.d and /usr/lib/kernel/install.d which are supplied > from all kinds of other packages. On my machine one of these is > /usr/lib/kernel/install.d/50-depmod.install (also from systemd-udev) > which eventually runs depmod -a which creates this file. > > So those are some of the packages which might be missing. Does > modules.dep get created anywhere inside the container? It's possible > it moved. > > It's also possible to set the environment variable KERNEL_INSTALL_VERBOSE=1 > which should show (in the dnf install output I guess?) if/when the > depmod command runs. > > > I think I'm missing some package removed from the fedora 34 container, > > but I don't have > > any idea which one. > > > > I hope someone has more insight on this. > > It's curious that things have changed since F33. Could you build > containers for both and diff the list of package names? Maybe a > dependency is missing or has become a soft dep. > > rpm -qa --qf '%{name}\n' | sort > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > libguestfs lets you edit virtual machines. Supports shell scripting, > bindings from many languages. http://libguestfs.org >