Libguestfs - Sep 2020 - Re: [PATCH v2v] v2v: Allow output to block devices (RHBZ#1868690).

On Tue, Sep 01, 2020 at 08:55:38PM +0300, Nir Soffer wrote:
...
> > Note this commit intentionally does not prevent you from writing qcow2
> > to a block device.  RHV uses this so it is a thing that people do.
...
> > +Note that you must create block devices of the correct size, and you
> > +need to use I<-of raw> since other output formats would not
normally
> > +make sense on a block device.
> 
> qcow2 on block device is useful normally, for example for supporting
> incremental backup of the disk. Also using qcow2 avoids the horrible issue
> of lvm on the host activating logical volumes inside the block device while
> the guest is using the devices.
> 
> So the statement "would not normally make sense" is too extreme.
>
> Maybe something like:
> 
>     Typically l<of raw> is used for block devices but other formats
such as
>     qcow2 can be useful in some cases.
Sure, I will adjust this.  I knew that RHV was using it.
> > @@ -713,27 +716,33 @@ and copy_targets cmdline targets input output
> >
> >        (match t.target_file with
> >         | TargetFile filename ->
> > -          (* It turns out that libguestfs's disk creation code is
> > -           * considerably more flexible and easier to use than
> > -           * qemu-img, so create the disk explicitly using libguestfs
> > -           * then pass the 'qemu-img convert -n' option so
qemu reuses
> > -           * the disk.
> > -           *
> > -           * Also we allow the output mode to actually create the
disk
> > -           * image.  This lets the output mode set ownership and
> > -           * permissions correctly if required.
> > +          (* As a special case, allow output to a block device or
> > +           * symlink to a block device.  In this case we don't
> > +           * create/overwrite the block device.  (RHBZ#1868690).
> >             *)
> > -          (* What output preallocation mode should we use? *)
> > -          let preallocation > > -            match
t.target_format, cmdline.output_alloc with
> > -            | ("raw"|"qcow2"), Sparse -> Some
"sparse"
> > -            | ("raw"|"qcow2"), Preallocated ->
Some "full"
> > -            | _ -> None (* ignore -oa flag for other formats *) in
> > -          let compat > > -            match t.target_format
with "qcow2" -> Some "1.1" | _ -> None in
> > -          output#disk_create filename t.target_format
> > -                             t.target_overlay.ov_virtual_size
> > -                             ?preallocation ?compat
> > +          if not (is_block_device filename) then (
> > +            (* It turns out that libguestfs's disk creation code
is
> > +             * considerably more flexible and easier to use than
> > +             * qemu-img, so create the disk explicitly using
libguestfs
> > +             * then pass the 'qemu-img convert -n' option so
qemu reuses
> > +             * the disk.
> 
> How would -n work if you want to create qcow2 format?
I'm completely sure what the question means, but virt-v2v currently
does this:

  qemu-img create -f FORMAT filename
  qemu-img convert -n overlay filename

The new code doesn't change that for files, but the proposal does
change it for devices to only this command:

  qemu-img convert -n overlay device

I don't think that the format makes any difference AFAIK.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Tue, Sep 01, 2020 at 07:37:42PM +0100, Richard W.M. Jones
wrote:
> On Tue, Sep 01, 2020 at 08:55:38PM +0300, Nir Soffer wrote:
> ...
> > > Note this commit intentionally does not prevent you from writing
qcow2
> > > to a block device.  RHV uses this so it is a thing that people
do.
> ...
> > > +Note that you must create block devices of the correct size, and
you
> > > +need to use I<-of raw> since other output formats would
not normally
> > > +make sense on a block device.
> > 
> > qcow2 on block device is useful normally, for example for supporting
> > incremental backup of the disk. Also using qcow2 avoids the horrible
issue
> > of lvm on the host activating logical volumes inside the block device
while
> > the guest is using the devices.
> > 
> > So the statement "would not normally make sense" is too
extreme.
> >
> > Maybe something like:
> > 
> >     Typically l<of raw> is used for block devices but other
formats such as
> >     qcow2 can be useful in some cases.
> 
> Sure, I will adjust this.  I knew that RHV was using it.
> 
> > > @@ -713,27 +716,33 @@ and copy_targets cmdline targets input
output > > >
> > >        (match t.target_file with
> > >         | TargetFile filename ->
> > > -          (* It turns out that libguestfs's disk creation
code is
> > > -           * considerably more flexible and easier to use than
> > > -           * qemu-img, so create the disk explicitly using
libguestfs
> > > -           * then pass the 'qemu-img convert -n' option
so qemu reuses
> > > -           * the disk.
> > > -           *
> > > -           * Also we allow the output mode to actually create
the disk
> > > -           * image.  This lets the output mode set ownership and
> > > -           * permissions correctly if required.
> > > +          (* As a special case, allow output to a block device
or
> > > +           * symlink to a block device.  In this case we
don't
> > > +           * create/overwrite the block device.  (RHBZ#1868690).
> > >             *)
> > > -          (* What output preallocation mode should we use? *)
> > > -          let preallocation > > > -            match
t.target_format, cmdline.output_alloc with
> > > -            | ("raw"|"qcow2"), Sparse ->
Some "sparse"
> > > -            | ("raw"|"qcow2"), Preallocated
-> Some "full"
> > > -            | _ -> None (* ignore -oa flag for other formats
*) in
> > > -          let compat > > > -            match
t.target_format with "qcow2" -> Some "1.1" | _ -> None
in
> > > -          output#disk_create filename t.target_format
> > > -                             t.target_overlay.ov_virtual_size
> > > -                             ?preallocation ?compat
> > > +          if not (is_block_device filename) then (
> > > +            (* It turns out that libguestfs's disk creation
code is
> > > +             * considerably more flexible and easier to use than
> > > +             * qemu-img, so create the disk explicitly using
libguestfs
> > > +             * then pass the 'qemu-img convert -n'
option so qemu reuses
> > > +             * the disk.
> > 
> > How would -n work if you want to create qcow2 format?
> 
> I'm completely sure what the question means, but virt-v2v currently
> does this:
> 
>   qemu-img create -f FORMAT filename
>   qemu-img convert -n overlay filename
> 
> The new code doesn't change that for files, but the proposal does
> change it for devices to only this command:
> 
>   qemu-img convert -n overlay device
The actual command uses -O format so the format is set correctly in
this case.

Rich.
> I don't think that the format makes any difference AFAIK.
> 
> Rich.
> 
> -- 
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
> 
> _______________________________________________
> Libguestfs mailing list
> Libguestfs@redhat.com
> https://www.redhat.com/mailman/listinfo/libguestfs
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Tue, Sep 1, 2020 at 9:42 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
>
> On Tue, Sep 01, 2020 at 07:37:42PM +0100, Richard W.M. Jones wrote:
> > On Tue, Sep 01, 2020 at 08:55:38PM +0300, Nir Soffer wrote:
> > ...
> > > > Note this commit intentionally does not prevent you from
writing qcow2
> > > > to a block device.  RHV uses this so it is a thing that
people do.
> > ...
> > > > +Note that you must create block devices of the correct
size, and you
> > > > +need to use I<-of raw> since other output formats
would not normally
> > > > +make sense on a block device.
> > >
> > > qcow2 on block device is useful normally, for example for
supporting
> > > incremental backup of the disk. Also using qcow2 avoids the
horrible issue
> > > of lvm on the host activating logical volumes inside the block
device while
> > > the guest is using the devices.
> > >
> > > So the statement "would not normally make sense" is too
extreme.
> > >
> > > Maybe something like:
> > >
> > >     Typically l<of raw> is used for block devices but other
formats such as
> > >     qcow2 can be useful in some cases.
> >
> > Sure, I will adjust this.  I knew that RHV was using it.
> >
> > > > @@ -713,27 +716,33 @@ and copy_targets cmdline targets input
output > > > >
> > > >        (match t.target_file with
> > > >         | TargetFile filename ->
> > > > -          (* It turns out that libguestfs's disk
creation code is
> > > > -           * considerably more flexible and easier to use
than
> > > > -           * qemu-img, so create the disk explicitly using
libguestfs
> > > > -           * then pass the 'qemu-img convert -n'
option so qemu reuses
> > > > -           * the disk.
> > > > -           *
> > > > -           * Also we allow the output mode to actually
create the disk
> > > > -           * image.  This lets the output mode set
ownership and
> > > > -           * permissions correctly if required.
> > > > +          (* As a special case, allow output to a block
device or
> > > > +           * symlink to a block device.  In this case we
don't
> > > > +           * create/overwrite the block device. 
(RHBZ#1868690).
> > > >             *)
> > > > -          (* What output preallocation mode should we use?
*)
> > > > -          let preallocation > > > > -          
match t.target_format, cmdline.output_alloc with
> > > > -            | ("raw"|"qcow2"), Sparse
-> Some "sparse"
> > > > -            | ("raw"|"qcow2"),
Preallocated -> Some "full"
> > > > -            | _ -> None (* ignore -oa flag for other
formats *) in
> > > > -          let compat > > > > -            match
t.target_format with "qcow2" -> Some "1.1" | _ -> None
in
> > > > -          output#disk_create filename t.target_format
> > > > -                            
t.target_overlay.ov_virtual_size
> > > > -                             ?preallocation ?compat
> > > > +          if not (is_block_device filename) then (
> > > > +            (* It turns out that libguestfs's disk
creation code is
> > > > +             * considerably more flexible and easier to use
than
> > > > +             * qemu-img, so create the disk explicitly
using libguestfs
> > > > +             * then pass the 'qemu-img convert -n'
option so qemu reuses
> > > > +             * the disk.
> > >
> > > How would -n work if you want to create qcow2 format?
> >
> > I'm completely sure what the question means, but virt-v2v
currently
> > does this:
> >
> >   qemu-img create -f FORMAT filename
> >   qemu-img convert -n overlay filename
This works, we also do this with file and block devices in most cases.

I was confused by:
> > > > +            (* It turns out that libguestfs's disk
creation code is
> > > > +             * considerably more flexible and easier to use
than
> > > > +             * qemu-img, so create the disk explicitly
using libguestfs
I guess you don't create qcow2 image without qemu-img?
> > The new code doesn't change that for files, but the proposal does
> > change it for devices to only this command:
> >
> >   qemu-img convert -n overlay device
This does not work for qcow2, since -n disable creation of the image.
> The actual command uses -O format so the format is set correctly in
> this case.
If -n is specified, qemu-img assumes that the target is already in qcow2 format.
If not, it will fail with:

    qemu-img: Could not open '/dev/loop0': Image is not in qcow2 format

But if you remove -n, convert will create the qcow2 header in the target.

There are some cases when using -n is bad, see here:
https://github.com/oVirt/vdsm/commit/2bbe66274524cc009f39d3a5e5577b84144f1171
>
> Rich.
>
> > I don't think that the format makes any difference AFAIK.
> >
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> >
> > _______________________________________________
> > Libguestfs mailing list
> > Libguestfs@redhat.com
> > https://www.redhat.com/mailman/listinfo/libguestfs
>
> --
> Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
>