On Wed, Aug 24, 2022 at 05:25:50PM +0200, Laszlo Ersek
wrote:> On 08/24/22 16:13, Richard W.M. Jones wrote:
> > On Wed, Aug 24, 2022 at 04:12:04PM +0200, Laszlo Ersek wrote:
> >> I've started looking into RHBZ 1590721 for virt-p2v.
> >>
> >> For p2v development, quick local testing is helpful. "make
> >> run-virt-p2v-directly" seems to be working great; however,
the VM-based
> >> test methods seem to have developed problems, since I last looked.
> >>
> >> Namely:
> >>
> >> - "make run-virt-p2v-in-a-vm" boots quickly, but the GUI
does not come up.
> >>
> >> - "make run-virt-p2v-in-an-nvme-vm" boots *incredibly
slowly*. I
> >> checked, and the host CPU utilization during guest boot was around
20%
> >> (and KVM was enabled). I don't understand how or why, but
exposing the
> >> "physical machine" disk over NVMe slows guest boot to a
crawl -- it
> >> looks strangely "IO-bound". I don't recall this from
the time I added
> >> this Makefile target!
> >>
> >> Do these symptoms look familiar?
> >
> > Yes I had noticed this. I did look at it briefly but couldn't
work
> > out what was going wrong.
>
> Thanks for confirming!
>
> I think I've found something, at least in relation to the first issue
> (i.e., run-virt-p2v-in-a-vm booting quickly, but not launching the p2v
> service). The problem seems to be the following:
>
> - Both the "physical machine" disk (fedora.img) and the bootable
p2v
> disk (virt-p2v.img) are based on Fedora 35. The former because of commit
> 3b59d4acb82f ("Use fedora-35 as the test physical machine",
2022-05-10),
> the latter because my host system runs Fedora 35, and so
> virt-p2v-make-disk picks F35.
>
> - Consequently, both the "hd0" drive and the "usb0"
drive are based on
> the same Fedora 35 virt-builder template.
>
> - The grub command line (part of *both* images) specifies the *same*
> kernel parameter "root=UUID=fae2...".
>
> - This filesystem UUID exists *twice* in the virtual machine however,
> once in "hd0" and another time in "usb0" -- duplicate
UUID.
>
> - The guest kernel ends up mounting / loading the root filesystem from
> "hd0", that is, the "physical machine" Fedora 35 image,
which does not
> have anything injected that's related to p2v.
Ah yes, I remember being extremely confused by why it was booting the
wrong guest, and also I remember this exact bug having happened long
ago in the past. Can we re-randomize the file system UUID of one of
them when creating the test disks?
> I found this by attempting to log in to the guest. The "p2v"
password
> did not work (which is hardcoded in "virt-p2v-make-disk") , but
the
> password printed by virt-builder under the PHYSICAL_MACHINE target in
> "Makefile" did work. Then, out of curiosity, I tried mounting the
> "other" disk in the running VM, but the XFS driver rejected the
mount
> attempt, due to duplicate UUIDs.
>
> Now, virt-sysprep (and therefore virt-builder) has an operation called
> "fs-uuids", but it is not enabled by default:
> <https://libguestfs.org/virt-sysprep.1.html#notes-on-fs-uuids>, and
we
> also don't enable it manually for either "fedora.img" or
"virt-p2v.img".
ISTR virt-sysprep fs-uuids doesn't really work because it won't update
the grub config (or /etc/fstab). However it should be possible in
general to do this with libguestfs.
> I guess the issue was "masked" before commit 3b59d4acb82f
("Use
> fedora-35 as the test physical machine", 2022-05-10), at least on my
> workstation. I think the Fedora versions I've used on this workstation
> have been F34 and F35, so for "virt-p2v.img", virt-builder would
> download F34 and F35 templates, which would differ from the template
> used for "fedora.img" (F32) -- hence the ultimately different fs
UUIDs.
>
> ... I have no news regarding the NVMe slowness, but the above UUID
> collision exists in "run-virt-p2v-in-an-nvme-vm" just the same.
Once we
> fix the UUID conflict, the NVMe test might go back to normal too.
Thanks,
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines. Supports shell scripting,
bindings from many languages. http://libguestfs.org