Libguestfs - Apr 2018 - Re: [Qemu-block] v2v: -o rhv-upload: Long time spent zeroing the disk

On Tue, Apr 10, 2018 at 03:25:47PM +0000, Nir Soffer
wrote:
> On Tue, Apr 10, 2018 at 5:50 PM Richard W.M. Jones
<rjones@redhat.com>
> wrote:
> 
> > On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote:
> > > This makes sense if the device is backed by a block device on
oVirt side,
> > > and the NBD support efficient zeroing. But in this case the
device is
> > backed
> > > by an empty sparse file on NFS, and oVirt does not support yet
efficient
> > > zeroing, we just write zeros manually.
> > >
> > > I think should be handled on virt-v2v plugin side. When zeroing a
file
> > raw
> > > image,
> > > you can ignore zero requests after the highest write offset,
since the
> > > plugin
> > > created a new image, and we know that the image is empty.
> > >
> > > When the destination is a block device we cannot avoid zeroing
since a
> > block
> > > device may contain junk data (we usually get dirty empty images
from our
> > > local
> > > xtremio server).
> >
> > (Off topic for qemu-block but ...)  We don't have enough
information
> > at our end to know about any of this.
> >
> 
> Can't use use this logic in the oVirt plugin?
> 
> file based storage -> skip initial zeroing
> block based storage -> use initial zeroing
>
> Do you think that publishing disk capabilities in the sdk will solve this?
The plugin would have to do some complicated gymnastics.  It would
have to keep track of which areas of the disk have been written and
ignore NBD_CMD_WRITE_ZEROES for other areas, except if block-based
storage is being used.  And so yes we'd also need the imageio API to
publish that information to the plugin.

So it's possible but not trivial.

By the way I think we're slowly reimplementing NBD in the imageio API.
Dan Berrange pointed out earlier on that it might be easier if imageio
just exposed NBD, or if we found a way to tunnel NBD requests over web
sockets (in the format case nbdkit would not be needed, in the latter
case nbdkit could act as a bridge).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Tue, Apr 10, 2018 at 6:53 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Tue, Apr 10, 2018 at 03:25:47PM +0000, Nir Soffer wrote:
> > On Tue, Apr 10, 2018 at 5:50 PM Richard W.M. Jones
<rjones@redhat.com>
> > wrote:
> >
> > > On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote:
> > > > This makes sense if the device is backed by a block device
on oVirt
> side,
> > > > and the NBD support efficient zeroing. But in this case the
device is
> > > backed
> > > > by an empty sparse file on NFS, and oVirt does not support
yet
> efficient
> > > > zeroing, we just write zeros manually.
> > > >
> > > > I think should be handled on virt-v2v plugin side. When
zeroing a
> file
> > > raw
> > > > image,
> > > > you can ignore zero requests after the highest write offset,
since
> the
> > > > plugin
> > > > created a new image, and we know that the image is empty.
> > > >
> > > > When the destination is a block device we cannot avoid
zeroing since
> a
> > > block
> > > > device may contain junk data (we usually get dirty empty
images from
> our
> > > > local
> > > > xtremio server).
> > >
> > > (Off topic for qemu-block but ...)  We don't have enough
information
> > > at our end to know about any of this.
> > >
> >
> > Can't use use this logic in the oVirt plugin?
> >
> > file based storage -> skip initial zeroing
> > block based storage -> use initial zeroing
> >
> > Do you think that publishing disk capabilities in the sdk will solve
> this?
>
> The plugin would have to do some complicated gymnastics.  It would
> have to keep track of which areas of the disk have been written and
> ignore NBD_CMD_WRITE_ZEROES for other areas, except if block-based
> storage is being used.  And so yes we'd also need the imageio API to
> publish that information to the plugin.
>
> So it's possible but not trivial.
>
I think this should be fixed in qemu-img. The current zero optimization may
have insignificant improvement when the backend supports fast zeroing
(e.g fallocate, blkzeroout) by minimizing number of syscalls, but it has
severe
performance issue when the desntiation does not support fast zero
(e.g. NFS < 4.2).

We plan to improve zero performance in 4.2.z, but this will not solve the
issue
on NFS < 4.2.

By the way I think we're slowly reimplementing NBD in the imageio
API.
>
You can also look at it as - slowly showing that HTTP can replace
NBD :-)

> Dan Berrange pointed out earlier on that it might be easier if imageio
> just exposed NBD, or if we found a way to tunnel NBD requests over web
> sockets (in the format case nbdkit would not be needed, in the latter
> case nbdkit could act as a bridge).
>
I'm not the more complex web sockets are needed for moving images
around. web sockets are good when the clients is a browser, and you need
to pass lot of small messages. If you have a real socket you don't need
web sockets. And the overhead of http headers is insignificant when you
move lot of data around.

Exposing NBD using the same authentication mechanism can be interesting,
but only if clients have an easy way to use this.

Will this allow client (e.g. backup vender) to download and upload images
using qemu-img?

    qemu-img <- nbd-> ovirt-imageio <- ndb -> qemu/ndb-server

We also need streaming support - with http you can stream the data from
anywhere, while qemu-img needs a file. I don't think that backup vendors
or users will be happy to implement nbdkit plugins.

If qemu-img would support input and ouptut from stdin/stdout instead of
files, this model sounds much more interesting.

Nir

On Thu, Apr 12, 2018 at 2:07 AM Nir Soffer <nirsof@gmail.com> wrote:
> On Tue, Apr 10, 2018 at 6:53 PM Richard W.M. Jones
<rjones@redhat.com>
> wrote:
>
...
> Dan Berrange pointed out earlier on that it might be easier if imageio
>>
> just exposed NBD, or if we found a way to tunnel NBD requests over web
>> sockets (in the format case nbdkit would not be needed, in the latter
>> case nbdkit could act as a bridge).
>
>
I think we can expose NBD using ndb-server and dynamic exports.
It can work like this:

0. Install nbd and enable nbd-server on a host, running
   as vdsm:kvm, not exporting anything.

1. User starts transfer session via oVirt API with protocol="nbd"
    (if not specified, use "https" for backward compatibility)

2. oVirt selects a host for the transfer and configures an export
   in that host - for example:

[bbff5ab7-2e8e-45b8-a497-423c2d2b3ff1]
exportname = /path/to/image
trim = yes
timeout = 300

nbd-server supports reloading configuration via SIGHUP.
Looks like it should work for adding export dynamically:
https://github.com/NetworkBlockDevice/nbd/blob/cb6e816c4d2c536cfff5ba9859f210aabe39539b/nbd-server.c#L3007

I could find code handling removing exports.

3. User gets back
- transfer_url: "nbd:hostname:port"
- export_name: bbff5ab7-2e8e-45b8-a497-423c2d2b3ff1

Assuming that if NBD_OPT_EXPORT_NAME is bbff5ab7-2e8e-45b8-a497-423c2d2b3ff1
ndb-server will export the /path/to/image mentioned in the config
exportname.

This seems to be the case based on:
- https://github.com/NetworkBlockDevice/nbd/blob/master/nbd-server.c#L998
-
https://github.com/NetworkBlockDevice/nbd/blob/cb6e816c4d2c536cfff5ba9859f210aabe39539b/nbd-server.c#L2074

4. User transmit data using nbd-client or qemu-img or maybe
   a wrapper script to make it easier

5. During the transfer, oVirt monitors progress and activity.

I don't see how we can do this with current nbd-server.

6. User ends the transfer session via oVirt API

7. oVirt removes the export and finalize the transfer.

I guess this can work in a similar way for KubeVirt. The way
the image is provisioned and attached to a host will be
different, but the way a user will transfer image data can be
the same.

Seems that nbd-server is missing:
- way to remove exports dynamically
- way to monitor export activity and progress

Nir