similar to: v2v: -o rhv-upload: Long time spent zeroing the disk

Am 10.04.2018 um 15:03 hat Nir Soffer geschrieben: > On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > We now have true zeroing support in oVirt imageio, thanks for that. > > > > However a problem is that ‘qemu-img convert’ issues zero requests for > > the whole disk before starting the transfer. It does this using 32 MB

On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote: > This makes sense if the device is backed by a block device on oVirt side, > and the NBD support efficient zeroing. But in this case the device is backed > by an empty sparse file on NFS, and oVirt does not support yet efficient > zeroing, we just write zeros manually. > > I think should be handled on virt-v2v plugin

On 04/10/2018 09:07 AM, Nir Soffer wrote: > On Tue, Apr 10, 2018 at 4:48 PM Kevin Wolf <kwolf@redhat.com> wrote: > >> Am 10.04.2018 um 15:03 hat Nir Soffer geschrieben: >>> On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> >>> wrote: >>> >>>> We now have true zeroing support in oVirt imageio, thanks for that.

On Tue, Apr 10, 2018 at 03:25:47PM +0000, Nir Soffer wrote: > On Tue, Apr 10, 2018 at 5:50 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote: > > > This makes sense if the device is backed by a block device on oVirt side, > > > and the NBD support efficient zeroing. But in this case the device

On Tue, Apr 10, 2018 at 4:48 PM Kevin Wolf <kwolf@redhat.com> wrote: > Am 10.04.2018 um 15:03 hat Nir Soffer geschrieben: > > On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> > > wrote: > > > > > We now have true zeroing support in oVirt imageio, thanks for that. > > > > > > However a problem is that ‘qemu-img

v1 was here: https://www.redhat.com/archives/libguestfs/2018-June/msg00015.html v2 fixes all the issues raised in Nir's review. I tested it against ovirt-engine-4.2.4.1-1.el7.noarch and it works, although I wasn't easily able to trigger the error path so that code is not really tested (except for syntax checking). Rich.

On Thu, Apr 12, 2018 at 2:07 AM Nir Soffer <nirsof@gmail.com> wrote: > On Tue, Apr 10, 2018 at 6:53 PM Richard W.M. Jones <rjones@redhat.com> > wrote: > ... > Dan Berrange pointed out earlier on that it might be easier if imageio >> > just exposed NBD, or if we found a way to tunnel NBD requests over web >> sockets (in the format case nbdkit would not be

When sending request with small or no payload, it is simpler and possibly more efficient to use the high level HTTPSConnection.request(), instead of the lower level APIs. The only reason to use the lower level APIs is to avoid copying the payload, or on python 2, to use a bigger buffer size when streaming a file-like object. --- v2v/rhv-upload-plugin.py | 35 ++++++++++++++++-------------------

On Tue, Apr 10, 2018 at 5:50 PM Richard W.M. Jones <rjones@redhat.com> wrote: > On Tue, Apr 10, 2018 at 02:07:33PM +0000, Nir Soffer wrote: > > This makes sense if the device is backed by a block device on oVirt side, > > and the NBD support efficient zeroing. But in this case the device is > backed > > by an empty sparse file on NFS, and oVirt does not support yet

On 04/10/2018 09:40 AM, Richard W.M. Jones wrote: >> When the destination is a block device we cannot avoid zeroing since a block >> device may contain junk data (we usually get dirty empty images from our >> local >> xtremio server). > > (Off topic for qemu-block but ...) We don't have enough information > at our end to know about any of this. Yep, see my

On Thu, Mar 08, 2018 at 12:13:01PM +0000, Nir Soffer wrote: > On Wed, Mar 7, 2018 at 12:18 AM Richard W.M. Jones <rjones@redhat.com> > wrote: > > > Previous versions: > > v3: https://www.redhat.com/archives/libguestfs/2018-March/msg00000.html > > v2: > > https://www.redhat.com/archives/libguestfs/2018-February/msg00177.html > > v1: > >

http was missing in emulate_zero, so the code would fail with NameError. This can happen only when communicating with old imageio versions not supporting the "zero" feature. Testing with qemu-img 2.12 show that we never send emulated zero request because of the highestwrite mechanism, but it can break with older qemu-img-rhev used on RHEL. --- v2v/rhv-upload-plugin.py | 2 ++ 1 file

On Thu, Mar 22, 2018 at 5:25 PM Richard W.M. Jones <rjones@redhat.com> wrote: > PROBLEMS: > - -of qcow2 does not work, with multiple problems > * needs to set NBD size to something larger than virtual size > How is this related to the actual file size specified when you create a disk? In block storage, qcow2 image is always on a thin provisioned disk, which in oVirt is a

On Tue, Apr 10, 2018 at 1:44 PM Richard W.M. Jones <rjones@redhat.com> wrote: > We now have true zeroing support in oVirt imageio, thanks for that. > > However a problem is that ‘qemu-img convert’ issues zero requests for > the whole disk before starting the transfer. It does this using 32 MB > requests which take approx. 1 second each to execute on the oVirt side. >

Fix NameError introduced by inocomplete change to extract create_transfer() by extracting cancel_transfer() function. Finally extract finallize_transfer() function, dealing with the poorly documented and over complicated oVirt SDK. Tested with: libguestfs-1.40.2-4.fc30.x86_64 nbdkit-1.12.8-1.fc30.x86_64 Tested flows: - Successful import - Error in close() - by injecting long sleep in vdsm

We can use now ImageioClient to communicate with ovirt-imageio server on oVirt host. Using the client greatly simplifies the plugin, and enables new features like transparent proxy support. The client will use transfer_url if possible, or fall back to proxy_url. Since the client implements the buffer protocol, move to version 2 of the API for more efficient pread(). Another advantage the client

Remove some instances of "transfer" and "transfer_service" that are not used in current code. --- v2v/rhv-upload-plugin.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/v2v/rhv-upload-plugin.py b/v2v/rhv-upload-plugin.py index 10887c031..5c036c46a 100644 --- a/v2v/rhv-upload-plugin.py +++ b/v2v/rhv-upload-plugin.py @@ -263,7 +263,6 @@ def request_failed(h, r, msg):

Currently we handle only HTTP errors like getting unexpected response code from imageio server. However if we could not send the request, or some other error is raised, we failed to mark the handle as failed, and finalized the transfer in close(). This may fool virt-v2v to create a VM with an empty or partially uploaded disk. Decorate all the NBD hander functions with a @failing decorator,

On Wed, Mar 21, 2018 at 3:34 PM Richard W.M. Jones <rjones@redhat.com> wrote: > PROBLEMS: > - -of qcow2 does not work, with multiple problems > * needs to set NBD size to something larger than virtual size > > This adds a new output mode to virt-v2v. virt-v2v -o rhv-upload > streams images directly to an oVirt or RHV >= 4 Data Domain using the > oVirt SDK v4.

These two patches add support for using a Unix domain socket to directly access imageio in the case where imageio is running on the conversion host (usually that means virt-v2v is running on the RHV node and something else -- eg. CFME scripts -- arranges that the RHV node is the same one running imageio). Conversions in the normal case are not affected - they happen over TCP as usual. This was