Libguestfs - Sep 2020 - Re: [PATCH v2v] v2v: Allow output to block devices (RHBZ#1868690).

On Tue, Sep 1, 2020 at 9:42 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Tue, Sep 01, 2020 at 07:37:42PM +0100, Richard W.M. Jones wrote:
> > On Tue, Sep 01, 2020 at 08:55:38PM +0300, Nir Soffer wrote:
> > ...
> > > > Note this commit intentionally does not prevent you from
writing qcow2
> > > > to a block device.  RHV uses this so it is a thing that
people do.
> > ...
> > > > +Note that you must create block devices of the correct
size, and you
> > > > +need to use I<-of raw> since other output formats
would not normally
> > > > +make sense on a block device.
> > >
> > > qcow2 on block device is useful normally, for example for
supporting
> > > incremental backup of the disk. Also using qcow2 avoids the
horrible issue
> > > of lvm on the host activating logical volumes inside the block
device while
> > > the guest is using the devices.
> > >
> > > So the statement "would not normally make sense" is too
extreme.
> > >
> > > Maybe something like:
> > >
> > >     Typically l<of raw> is used for block devices but other
formats such as
> > >     qcow2 can be useful in some cases.
> >
> > Sure, I will adjust this.  I knew that RHV was using it.
> >
> > > > @@ -713,27 +716,33 @@ and copy_targets cmdline targets input
output > > > >
> > > >        (match t.target_file with
> > > >         | TargetFile filename ->
> > > > -          (* It turns out that libguestfs's disk
creation code is
> > > > -           * considerably more flexible and easier to use
than
> > > > -           * qemu-img, so create the disk explicitly using
libguestfs
> > > > -           * then pass the 'qemu-img convert -n'
option so qemu reuses
> > > > -           * the disk.
> > > > -           *
> > > > -           * Also we allow the output mode to actually
create the disk
> > > > -           * image.  This lets the output mode set
ownership and
> > > > -           * permissions correctly if required.
> > > > +          (* As a special case, allow output to a block
device or
> > > > +           * symlink to a block device.  In this case we
don't
> > > > +           * create/overwrite the block device. 
(RHBZ#1868690).
> > > >             *)
> > > > -          (* What output preallocation mode should we use?
*)
> > > > -          let preallocation > > > > -          
match t.target_format, cmdline.output_alloc with
> > > > -            | ("raw"|"qcow2"), Sparse
-> Some "sparse"
> > > > -            | ("raw"|"qcow2"),
Preallocated -> Some "full"
> > > > -            | _ -> None (* ignore -oa flag for other
formats *) in
> > > > -          let compat > > > > -            match
t.target_format with "qcow2" -> Some "1.1" | _ -> None
in
> > > > -          output#disk_create filename t.target_format
> > > > -                            
t.target_overlay.ov_virtual_size
> > > > -                             ?preallocation ?compat
> > > > +          if not (is_block_device filename) then (
> > > > +            (* It turns out that libguestfs's disk
creation code is
> > > > +             * considerably more flexible and easier to use
than
> > > > +             * qemu-img, so create the disk explicitly
using libguestfs
> > > > +             * then pass the 'qemu-img convert -n'
option so qemu reuses
> > > > +             * the disk.
> > >
> > > How would -n work if you want to create qcow2 format?
> >
> > I'm completely sure what the question means, but virt-v2v
currently
> > does this:
> >
> >   qemu-img create -f FORMAT filename
> >   qemu-img convert -n overlay filename
This works, we also do this with file and block devices in most cases.

I was confused by:
> > > > +            (* It turns out that libguestfs's disk
creation code is
> > > > +             * considerably more flexible and easier to use
than
> > > > +             * qemu-img, so create the disk explicitly
using libguestfs
I guess you don't create qcow2 image without qemu-img?
> > The new code doesn't change that for files, but the proposal does
> > change it for devices to only this command:
> >
> >   qemu-img convert -n overlay device
This does not work for qcow2, since -n disable creation of the image.
> The actual command uses -O format so the format is set correctly in
> this case.
If -n is specified, qemu-img assumes that the target is already in qcow2 format.
If not, it will fail with:

    qemu-img: Could not open '/dev/loop0': Image is not in qcow2 format

But if you remove -n, convert will create the qcow2 header in the target.

There are some cases when using -n is bad, see here:
https://github.com/oVirt/vdsm/commit/2bbe66274524cc009f39d3a5e5577b84144f1171
>
> Rich.
>
> > I don't think that the format makes any difference AFAIK.
> >
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> >
> > _______________________________________________
> > Libguestfs mailing list
> > Libguestfs@redhat.com
> > https://www.redhat.com/mailman/listinfo/libguestfs
>
> --
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>

On Tue, Sep 01, 2020 at 10:08:24PM +0300, Nir Soffer
wrote:
> > > I'm completely sure what the question means, but virt-v2v
currently
> > > does this:
> > >
> > >   qemu-img create -f FORMAT filename
> > >   qemu-img convert -n overlay filename
> 
> This works, we also do this with file and block devices in most cases.
> 
> I was confused by:
> 
> > > > > +            (* It turns out that libguestfs's disk
creation code is
> > > > > +             * considerably more flexible and easier
to use than
> > > > > +             * qemu-img, so create the disk explicitly
using libguestfs
> 
> I guess you don't create qcow2 image without qemu-img?
> 
> > > The new code doesn't change that for files, but the proposal
does
> > > change it for devices to only this command:
> > >
> > >   qemu-img convert -n overlay device
> 
> This does not work for qcow2, since -n disable creation of the image.
I'm going to have to play with this.  It may be that we need
to drop -n when using devices.

Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Tue, Sep 01, 2020 at 08:38:32PM +0100, Richard W.M. Jones
wrote:
> On Tue, Sep 01, 2020 at 10:08:24PM +0300, Nir Soffer wrote:
> > > > I'm completely sure what the question means, but
virt-v2v currently
> > > > does this:
> > > >
> > > >   qemu-img create -f FORMAT filename
> > > >   qemu-img convert -n overlay filename
> > 
> > This works, we also do this with file and block devices in most cases.
> > 
> > I was confused by:
> > 
> > > > > > +            (* It turns out that libguestfs's
disk creation code is
> > > > > > +             * considerably more flexible and
easier to use than
> > > > > > +             * qemu-img, so create the disk
explicitly using libguestfs
> > 
> > I guess you don't create qcow2 image without qemu-img?
> > 
> > > > The new code doesn't change that for files, but the
proposal does
> > > > change it for devices to only this command:
> > > >
> > > >   qemu-img convert -n overlay device
> > 
> > This does not work for qcow2, since -n disable creation of the image.
> 
> I'm going to have to play with this.  It may be that we need
> to drop -n when using devices.
I had a look at this and you're right:

qemu-img: Could not open '/var/tmp/fedora-32-sda': Image is not in qcow2
format

but maybe if people use block devices they should have to format them
if they want to use qcow2?

One reason is that they would get control over the qcow2 parameters
like cluster size, which would be hard to do if we had to pass them
all through virt-v2v.  (Currently virt-v2v can only set qcow2
compression.)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW