On 11/22/22 16:47, Richard W.M. Jones wrote:> In Kubernetes and tools like Kubevirt, it's not possible to create > some disks and then attach to them (in order to populate them with > data) in one step. This makes virt-v2v conversions awkward because > ideally we would like the output mode (-o kubevirt) to both create the > target disks and populate them at the same time. > > So to work around this problem, we need a tool which can inspect the > virt-v2v source hypervisor before we do the conversion in order to > find out how many disks are needed and their sizes. Then we can > create the target disks, and then we can run a second container with > virt-v2v attached to the disks to do the conversion and populate the > output.So, is the population of the *pre-created* disk images (aka volumes) a feature of the new kubevirt output module?> n > This is a proposed tool to do this. It essentially uses the same -i* > options as virt-v2v (and no -o* options) and outputs various useful > metadata. Example: > > $ ./run virt-v2v-inspector --quiet -i disk /var/tmp/fedora-32.img > virt-v2v-inspector: The QEMU Guest Agent will be installed for this guest > at first boot. > virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references > unknown device "vda". You may have to fix this entry manually after > conversion. > <?xml version='1.0' encoding='utf-8'?> > <v2v-inspection> > <!-- generated by virt-v2v-inspector 2.1.9local,libvirt --> > <program>virt-v2v-inspector</program> > <package>virt-v2v</package> > <version>2.1.9</version> > <disks> > <disk index='0'> > <virtual-size>6442450944</virtual-size> > <allocated estimated='true'>1400897536</allocated> > </disk> > </disks> > <operatingsystem> > <name>linux</name> > <distro>fedora</distro> > <osinfo>fedora32</osinfo> > <arch>x86_64</arch> > <major_version>32</major_version> > <minor_version>0</minor_version> > <package_format>rpm</package_format> > <package_management>dnf</package_management> > <product_name>Fedora 32 (Thirty Two)</product_name> > </operatingsystem> > </v2v-inspection> > > There should be sufficient information in the <disks> section to > allocate target disks, plus additional information is printed which > might be useful. > > Note that we do a full conversion in order to generate this > information. In particular it's not possible to generate the > <allocated/> estimate without this. It's plausible we could have a > --no-convert option, but I'm not sure it's worthwhile: it would only > save a little time, but would make everything less accurate, plus > maybe it is a good idea to find out if conversion is going to work > before we create the target disks?I think this is a great approach. The current problem from the "recipient" (kubevirt) side is that temporary storage for the disk images "in the middle" is really not wanted. This approach prevents just that. All other logic from virt-v2v remains useful and should be kept IMO.> I chose XML instead of JSON for output. XML allows us to annotate > elements with attributes like "estimated='true'". It also lets us > represent 64 bit number accurately, where JSON cannot represent such > numbers. > > One obvious problem is that (without --quiet) the program mixes up > informational output with the final document, which is a bit of a > pain. Ideas here?I think *small* regular files (fitting under $TMPDIR for example) should be in order, as output. Add an "-o desc.xml" or "-O desc.xml" option to the new tool, for saving the XML in "desc.xml" at once? (I hope I understood the "mixing" issue correctly.) Laszlo
Richard W.M. Jones
2022-Nov-24 10:34 UTC
[Libguestfs] [PATCH v2v] New virt-v2v-inspector tool
On Thu, Nov 24, 2022 at 10:54:42AM +0100, Laszlo Ersek wrote:> On 11/22/22 16:47, Richard W.M. Jones wrote: > > In Kubernetes and tools like Kubevirt, it's not possible to create > > some disks and then attach to them (in order to populate them with > > data) in one step. This makes virt-v2v conversions awkward because > > ideally we would like the output mode (-o kubevirt) to both create the > > target disks and populate them at the same time. > > > > So to work around this problem, we need a tool which can inspect the > > virt-v2v source hypervisor before we do the conversion in order to > > find out how many disks are needed and their sizes. Then we can > > create the target disks, and then we can run a second container with > > virt-v2v attached to the disks to do the conversion and populate the > > output. > > So, is the population of the *pre-created* disk images (aka volumes) a > feature of the new kubevirt output module?When I get around to finishing it, yes. Basically they will be presented to virt-v2v in the container as /dev devices (PVC "volumeMode: Block"). When running virt-v2v you have to give the names of the devices to virt-v2v, since that is only known by whoever creates the virt-v2v container. Note they are pre-created but not pre-populated, ie. virt-v2v sees correctly sized but empty /dev devices initially. I should also note that raw block device PVCs are very sparsely documented. I wonder if they are not widely used? In particular every worked example I've found copies from the same example in the Kubernetes upstream documentation. Guess I'll find out if it works soon ...> > This is a proposed tool to do this. It essentially uses the same -i* > > options as virt-v2v (and no -o* options) and outputs various useful > > metadata. Example: > > > > $ ./run virt-v2v-inspector --quiet -i disk /var/tmp/fedora-32.img > > virt-v2v-inspector: The QEMU Guest Agent will be installed for this guest> > at first boot. > > virt-v2v-inspector: warning: /files/boot/grub2/device.map/hd0 references > > unknown device "vda". You may have to fix this entry manually after > > conversion. > > <?xml version='1.0' encoding='utf-8'?> > > <v2v-inspection> > > <!-- generated by virt-v2v-inspector 2.1.9local,libvirt --> > > <program>virt-v2v-inspector</program> > > <package>virt-v2v</package> > > <version>2.1.9</version> > > <disks> > > <disk index='0'> > > <virtual-size>6442450944</virtual-size> > > <allocated estimated='true'>1400897536</allocated> > > </disk> > > </disks> > > <operatingsystem> > > <name>linux</name> > > <distro>fedora</distro> > > <osinfo>fedora32</osinfo> > > <arch>x86_64</arch> > > <major_version>32</major_version> > > <minor_version>0</minor_version> > > <package_format>rpm</package_format> > > <package_management>dnf</package_management> > > <product_name>Fedora 32 (Thirty Two)</product_name> > > </operatingsystem> > > </v2v-inspection> > > > > There should be sufficient information in the <disks> section to > > allocate target disks, plus additional information is printed which > > might be useful. > > > > Note that we do a full conversion in order to generate this > > information. In particular it's not possible to generate the > > <allocated/> estimate without this. It's plausible we could have a > > --no-convert option, but I'm not sure it's worthwhile: it would only > > save a little time, but would make everything less accurate, plus > > maybe it is a good idea to find out if conversion is going to work > > before we create the target disks? > > I think this is a great approach. The current problem from the > "recipient" (kubevirt) side is that temporary storage for the disk > images "in the middle" is really not wanted. This approach prevents just > that. All other logic from virt-v2v remains useful and should be kept IMO. > > > I chose XML instead of JSON for output. XML allows us to annotate > > elements with attributes like "estimated='true'". It also lets us > > represent 64 bit number accurately, where JSON cannot represent such > > numbers. > > > > One obvious problem is that (without --quiet) the program mixes up > > informational output with the final document, which is a bit of a > > pain. Ideas here? > > I think *small* regular files (fitting under $TMPDIR for example) should > be in order, as output. Add an "-o desc.xml" or "-O desc.xml" option to > the new tool, for saving the XML in "desc.xml" at once? (I hope I > understood the "mixing" issue correctly.)Yeah, I was just hoping to avoid having to specify an output file, as outputing to stdout is quite nice. But perhaps it's unavoidable. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html