Nir Soffer
2021-Dec-18 22:40 UTC
[Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input
On Sat, Dec 18, 2021 at 11:33 PM Richard W.M. Jones <rjones at redhat.com> wrote:> > On Sat, Dec 18, 2021 at 10:36:30PM +0200, Nir Soffer wrote: > > After finalizing the transfer, virt-v2v try to connect to the output > > socket and query disk allocation. This may work for some outputs > > supporting block status, but for rhv_upload output this cannot work for > > 2 reasons: > > - The rhv-upload-plugin does not support extents > > - The transfer was finalized before this call, so the plugin lost access > > to the image. > > > > Here is an example failure log: > > > > [ 74.2] Creating output metadata > > python3 '/tmp/v2v.WMq8Tk/rhv-upload-finalize.py' '/tmp/v2v.WMq8Tk/params6.json' > > finalizing transfer b03fe3ba-a4ff-4634-a0a0-10b3daba3cc2 > > ... > > transfer b03fe3ba-a4ff-4634-a0a0-10b3daba3cc2 finalized in 2.118 seconds > > ... > > nbdkit: debug: accepted connection > > ... > > nbdkit: python[4]: debug: python: close > > virt-v2v: error: exception: NBD.Error("nbd_block_status: request out of > > bounds: Invalid argument", 22) > > > > Fix by using the input socket. > > --- > > lib/utils.ml | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/lib/utils.ml b/lib/utils.ml > > index d6861d08..f6c85543 100644 > > --- a/lib/utils.ml > > +++ b/lib/utils.ml > > @@ -171,21 +171,21 @@ let with_nbd_connect_unix ~socket ~meta_contexts ~f > > ~f:(fun () -> > > List.iter (NBD.add_meta_context nbd) meta_contexts; > > NBD.connect_unix nbd socket; > > protect > > ~f:(fun () -> f nbd) > > ~finally:(fun () -> NBD.shutdown nbd) > > ) > > ~finally:(fun () -> NBD.close nbd) > > > > let get_disk_allocated ~dir ~disknr > > - let socket = sprintf "%s/out%d" dir disknr > > + let socket = sprintf "%s/in%d" dir disknr > > This patch is definitely wrong - we need to get the allocation size > from the output disk. Options such as -oa preallocated, and just > general issues like block size, nbdcopy sparseness detection etc, > would affect this. > > It probably indicates a problem with rhv-upload-plugin (again) that > it's not really prepared to be an idempotent part of a disk image > pipeline.This patch also definitely works for rhv-upload, we create working vms. It can be fixed in another way, maybe skipping the check when using rhv-upload because it cannot provide the needed info. I'm not sure why this check is needed - RHV already has everything it needs at this point, and it does not care about disk allocation after the disk was uploaded. If RHV needs any info about the disk, it can get it from storage. Maybe you add stuff to the ovf that RHV does need?> > Rich. > > > and alloc_ctx = "base:allocation" in > > with_nbd_connect_unix ~socket ~meta_contexts:[alloc_ctx] > > ~f:(fun nbd -> > > if NBD.can_meta_context nbd alloc_ctx then ( > > (* Get the list of extents, using a 2GiB chunk size as hint. *) > > let size = NBD.get_size nbd > > and allocated = ref 0_L > > and fetch_offset = ref 0_L in > > while !fetch_offset < size do > > let remaining = size -^ !fetch_offset in > > -- > > 2.33.1 > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > virt-df lists disk usage of guests without needing to install any > software inside the virtual machine. Supports Linux and Windows. > http://people.redhat.com/~rjones/virt-df/ >
Nir Soffer
2021-Dec-18 23:30 UTC
[Libguestfs] [PATCH 2/5] v2v/lib/util.ml: Get disk allocation from input
On Sun, Dec 19, 2021 at 12:40 AM Nir Soffer <nsoffer at redhat.com> wrote:> > On Sat, Dec 18, 2021 at 11:33 PM Richard W.M. Jones <rjones at redhat.com> wrote: > > > > On Sat, Dec 18, 2021 at 10:36:30PM +0200, Nir Soffer wrote: > > > After finalizing the transfer, virt-v2v try to connect to the output > > > socket and query disk allocation. This may work for some outputs > > > supporting block status, but for rhv_upload output this cannot work for > > > 2 reasons: > > > - The rhv-upload-plugin does not support extents > > > - The transfer was finalized before this call, so the plugin lost access > > > to the image. > > > > > > Here is an example failure log: > > > > > > [ 74.2] Creating output metadata > > > python3 '/tmp/v2v.WMq8Tk/rhv-upload-finalize.py' '/tmp/v2v.WMq8Tk/params6.json' > > > finalizing transfer b03fe3ba-a4ff-4634-a0a0-10b3daba3cc2 > > > ... > > > transfer b03fe3ba-a4ff-4634-a0a0-10b3daba3cc2 finalized in 2.118 seconds > > > ... > > > nbdkit: debug: accepted connection > > > ... > > > nbdkit: python[4]: debug: python: close > > > virt-v2v: error: exception: NBD.Error("nbd_block_status: request out of > > > bounds: Invalid argument", 22) > > > > > > Fix by using the input socket. > > > --- > > > lib/utils.ml | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/lib/utils.ml b/lib/utils.ml > > > index d6861d08..f6c85543 100644 > > > --- a/lib/utils.ml > > > +++ b/lib/utils.ml > > > @@ -171,21 +171,21 @@ let with_nbd_connect_unix ~socket ~meta_contexts ~f > > > ~f:(fun () -> > > > List.iter (NBD.add_meta_context nbd) meta_contexts; > > > NBD.connect_unix nbd socket; > > > protect > > > ~f:(fun () -> f nbd) > > > ~finally:(fun () -> NBD.shutdown nbd) > > > ) > > > ~finally:(fun () -> NBD.close nbd) > > > > > > let get_disk_allocated ~dir ~disknr > > > - let socket = sprintf "%s/out%d" dir disknr > > > + let socket = sprintf "%s/in%d" dir disknr > > > > This patch is definitely wrong - we need to get the allocation size > > from the output disk. Options such as -oa preallocated, and just > > general issues like block size, nbdcopy sparseness detection etc, > > would affect this. > > > > It probably indicates a problem with rhv-upload-plugin (again) that > > it's not really prepared to be an idempotent part of a disk image > > pipeline. > > This patch also definitely works for rhv-upload, we create working > vms. > > It can be fixed in another way, maybe skipping the check when using > rhv-upload because it cannot provide the needed info. > > I'm not sure why this check is needed - RHV already has everything > it needs at this point, and it does not care about disk allocation after > the disk was uploaded. If RHV needs any info about the disk, it can > get it from storage. > > Maybe you add stuff to the ovf that RHV does need?I see that this code is not in rhel-9.0.0 branch, so this patch is not needed for https://bugzilla.redhat.com/2032324 Laszlo, can you explain why we need the number of allocated bytes after the disk was already converted to the target storage? Looking at the bug: https://bugzilla.redhat.com/2027598 This is a fix for the issue in the old rhv output, which is deprecated and should not be used in new code, and it is not compatible with rhv-upload which is the modern way to import into RHV. Also this cannot work for RHV now, so we need a way to disable this when importing to RHV. We can expose the block status in RHV plugin, but to use this we must call block status after the disk is converted, but before the output is finalized, since when you finalize RHV disconnect the disk from the host, and the RHV plugin cannot access it any more. The flow for rhv-upload should be: run nbdcopy query block status finalize output create ovf Nir