Libguestfs - Mar 2018 - Re: [PATCH v7 6/6] v2v: Add -o rhv-upload output mode (RHBZ#1557273).

On Mon, Mar 26, 2018 at 11:41 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Sun, Mar 25, 2018 at 08:05:14PM +0000, Nir Soffer wrote:
> > When using the sdk, you can select both the image format and sparse,
so
> you
> > can create invalid combinations. oVirt selects the allocation for you.
> >
> > storage type  image format  sparse | allocation      creating
> >
>
-----------------------------------|--------------------------------------------------------
> > file          qcow2         true   | thin           sparse file of
image
> > size bytes
> > file          raw           false  | preallocated   preallocated file
of
> > provisioned_size bytes
> > file          raw           true   | thin           sparse file of
> > provisioned_size
> > bytes
> > file          qcow2         false  | -              unsupported
> > block         qcow2         true   | thin           logical volume of
> > initial_size bytes
> > block         raw           false  | preallocated   logical volume of
> > provisioned_size bytes
> > block         qcow2         false  | -              unsupported
> > block         raw           true   | -              unsupported
> >
> > The only case when selecting the sparse value is useful is raw file
> > on file based storage domain, allowing creation of sparse or
preallocated
> > disk.
>
> I don't think this is true.  Both LVs (lvmthin) and SAN LUNs can
> support sparse allocation meaningfully.

"sparse" is just another way for selecting "thin" or
"preallocated"
allocation
policy. In oVirt we emulate thin volume on block storage using regular
logical
volumes.

lvmthin does not support sharing a logical volume in a cluster. SAN LUN can
support
thin provisioning, but we don't support upload to LUN yet.

Nir

I found another problem which is sort of related to this thread.  I
hit a time-out waiting for the disk to be unlocked after creation:

https://github.com/libguestfs/libguestfs/commit/8081f54105bd990233f166170890192c1fd7d1f3#diff-5ca47c29ae13efa3959b8b28cf4dbd31R112
line 112

The disk size in this case is 40 GB and we currently wait 5 minutes.

On the ovirt-node there's an ‘fallocate’ process (still) running.
It's taken 23 minutes so far.

Because we didn't reach a satisfactory conclusion of the whole
sparse/format business, I am currently setting sparse = False
(format is always raw in my testing):

https://github.com/libguestfs/libguestfs/commit/8081f54105bd990233f166170890192c1fd7d1f3#diff-5ca47c29ae13efa3959b8b28cf4dbd31R92
line 92

I guess this is the cause of this problem in some sense.  Is there
something we can do?  I'm still unclear how to fix the sparse/format
mess, it seems unsolvable from the client side because we don't have
sufficient information to make a decision.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

And another problem ...

nbdkit: python[1]: error:
/home/rjones/d/libguestfs/tmp/rhvupload.riX9kG/rhv-upload-plugin.py: callback
failed: pwrite
Traceback (most recent call last):
  File
"/home/rjones/d/libguestfs/tmp/rhvupload.riX9kG/rhv-upload-plugin.py",
line 268, in pwrite
    http.send(buf)
  File "/usr/lib64/python3.6/http/client.py", line 986, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.6/ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "/usr/lib64/python3.6/ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "/usr/lib64/python3.6/ssl.py", line 636, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
qemu-img: error while writing sector 25343488: Input/output error

On the server side we see it's something to do with the ticket
expiring:

2018-03-29 12:59:24,215 INFO    (Thread-18701) [web] START [192.168.0.128] PUT
/images/4e8567f6-939d-471f-8ea8-e64d172c0a26
2018-03-29 12:59:24,216 INFO    (Thread-18701) [images] Writing 458752 bytes at
offset 12975407104 to
/rhev/data-center/mnt/ham0.home.annexia.org:_mnt_ovirt-data/7b4ae32d-bc79-4368-b683-3e0b58b64bf6/images/b4c31e0c-16c0-47d0-a162-9db4357a2dc5/89d70696-2075-4d30-b30e-86465060f677
for ticket 4e8567f6-939d-471f-8ea8-e64d172c0a26
2018-03-29 12:59:24,272 INFO    (Thread-18701) [directio] Operation stats:
<Clock(total=0.06, read=0.04, write=0.01, sync=0.00)>
2018-03-29 12:59:24,272 INFO    (Thread-18701) [web] FINISH [192.168.0.128] PUT
/images/4e8567f6-939d-471f-8ea8-e64d172c0a26: [200] 0 (0.05s)
2018-03-29 12:59:24,296 INFO    (Thread-18702) [web] START [192.168.0.128] PUT
/images/4e8567f6-939d-471f-8ea8-e64d172c0a26
2018-03-29 12:59:24,296 WARNING (Thread-18702) [web] ERROR [192.168.0.128] PUT
/images/4e8567f6-939d-471f-8ea8-e64d172c0a26: [403] Ticket
u'4e8567f6-939d-471f-8ea8-e64d172c0a26' expired (0.00s)

We had suspected this before but now I think I've proven that
it's a problem.

I believe the timeout was after 30 minutes (but we were writing
constantly the whole time).  Is this a bug in imageio?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On Thu, Mar 29, 2018 at 1:59 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> I found another problem which is sort of related to this thread.  I
> hit a time-out waiting for the disk to be unlocked after creation:
>
>
>
https://github.com/libguestfs/libguestfs/commit/8081f54105bd990233f166170890192c1fd7d1f3#diff-5ca47c29ae13efa3959b8b28cf4dbd31R112
> line 112
>
> The disk size in this case is 40 GB and we currently wait 5 minutes.
>
> On the ovirt-node there's an ‘fallocate’ process (still) running.
> It's taken 23 minutes so far.
>
This code is using posix_fallocate and it can be very slow on NFS < 4.2,
writing one byte to every 4k block.

> Because we didn't reach a satisfactory conclusion of the whole
> sparse/format business, I am currently setting sparse = False
> (format is always raw in my testing):
>
>
>
https://github.com/libguestfs/libguestfs/commit/8081f54105bd990233f166170890192c1fd7d1f3#diff-5ca47c29ae13efa3959b8b28cf4dbd31R92
> line 92
>
> I guess this is the cause of this problem in some sense.  Is there
> something we can do?  I'm still unclear how to fix the sparse/format
> mess, it seems unsolvable from the client side because we don't have
> sufficient information to make a decision.
>
I don't think always using sparse=False is a good solution. We have an issue
only with block storage, so there is no need to force sparse=False for every
kind of storage.

I think the behavior should be:

- if the user wants raw image (sparse=False) on file domain we can warn
that
creating preallocated disk on file storage can take lot of time, and we
should
wait until the disk is ready. Or we can always use thin for upload to file
based
storage, this is what we for uploading from the UI.

- if the user wants thin image on block storage, we must allocate the entire
image when we create a disk. In 4.2.4 we will support engine API to reduce
an image to the optimal size, and you will be able to call this when you
complete
the upload.

- if the user wants thin image on file storage, or raw image on block
storage,
we don't have any issue

In 4.3 we plan to improve engine extend volume api so it will update the
ticket
size when extending a disk. With this you will be able to create a thin
disk on
block storage and extend it when needed.

Nir

On Thu, Mar 29, 2018 at 3:07 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> And another problem ...
>
> nbdkit: python[1]: error:
> /home/rjones/d/libguestfs/tmp/rhvupload.riX9kG/rhv-upload-plugin.py:
> callback failed: pwrite
> Traceback (most recent call last):
>   File
>
"/home/rjones/d/libguestfs/tmp/rhvupload.riX9kG/rhv-upload-plugin.py",
line
> 268, in pwrite
>     http.send(buf)
>   File "/usr/lib64/python3.6/http/client.py", line 986, in send
>     self.sock.sendall(data)
>   File "/usr/lib64/python3.6/ssl.py", line 965, in sendall
>     v = self.send(data[count:])
>   File "/usr/lib64/python3.6/ssl.py", line 935, in send
>     return self._sslobj.write(data)
>   File "/usr/lib64/python3.6/ssl.py", line 636, in write
>     return self._sslobj.write(data)
> BrokenPipeError: [Errno 32] Broken pipe
> qemu-img: error while writing sector 25343488: Input/output error
>
> On the server side we see it's something to do with the ticket
> expiring:
>
> 2018-03-29 12:59:24,215 INFO    (Thread-18701) [web] START [192.168.0.128]
> PUT /images/4e8567f6-939d-471f-8ea8-e64d172c0a26
> 2018-03-29 12:59:24,216 INFO    (Thread-18701) [images] Writing 458752
> bytes at offset 12975407104 to
/rhev/data-center/mnt/ham0.home.annexia.org:_mnt_ovirt-data/7b4ae32d-bc79-4368-b683-3e0b58b64bf6/images/b4c31e0c-16c0-47d0-a162-9db4357a2dc5/89d70696-2075-4d30-b30e-86465060f677
> for ticket 4e8567f6-939d-471f-8ea8-e64d172c0a26
> 2018-03-29 12:59:24,272 INFO    (Thread-18701) [directio] Operation stats:
> <Clock(total=0.06, read=0.04, write=0.01, sync=0.00)>
> 2018-03-29 12:59:24,272 INFO    (Thread-18701) [web] FINISH
> [192.168.0.128] PUT /images/4e8567f6-939d-471f-8ea8-e64d172c0a26: [200] 0
> (0.05s)
> 2018-03-29 12:59:24,296 INFO    (Thread-18702) [web] START [192.168.0.128]
> PUT /images/4e8567f6-939d-471f-8ea8-e64d172c0a26
> 2018-03-29 12:59:24,296 WARNING (Thread-18702) [web] ERROR [192.168.0.128]
> PUT /images/4e8567f6-939d-471f-8ea8-e64d172c0a26: [403] Ticket
> u'4e8567f6-939d-471f-8ea8-e64d172c0a26' expired (0.00s)
>
> We had suspected this before but now I think I've proven that
> it's a problem.
>
> I believe the timeout was after 30 minutes (but we were writing
> constantly the whole time).  Is this a bug in imageio?
>
The ticket has expired. Engine is monitoring download/upload activity for
tickets
and extend the ticket when needed. We need complete daemon and engine logs
to check why the ticket was not extended.

Nir