Libguestfs - Jun 2018 - Re: [PATCH] v2v: -o rhv-upload: Optimize http request sending

On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer
wrote:
> On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer <nirsof@gmail.com> wrote:
> > +    headers = {"Content-Type":
"application/json",
> > +               "Content-Length", str(len(buf))}
There were a few Python syntax errors such as this one.  They
can be found by running:

  make -C v2v check TESTS=test-v2v-python-syntax.sh

However I have now fixed them.
> Do we have an easy way to test the plugin without running the entire
> virt-v2v pipeline?
Not really.  In fact we don't have any unit tests for -o rhv-upload
functionality because there's no way to simulate the imageio server.

However you can run virt-v2v locally against an oVirt instance without
needing VMware.  The command is rather lengthy, but here it is:

$ virt-builder fedora-27
$ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
    -o rhv-upload \
    -oc https://ovirt-engine.example.com/ovirt-engine/api \
    -os ovirt-data \
    -op /tmp/password \
    -of raw \
    -oo rhv-cafile=/tmp/ca.pem \
    -oo rhv-direct

/tmp/password should contain the oVirt admin password.
/tmp/ca.pem should contain the oVirt CA cert.

This will create a guest called ‘fedora-27’ which you'll need to
delete (on oVirt) afterwards.  You can add ‘-on name’ to name it
something else.

In any case I have fixed and verified this patch and will push it
soon, thanks.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer <nirsof@gmail.com>
wrote:
> > > +    headers = {"Content-Type":
"application/json",
> > > +               "Content-Length", str(len(buf))}
>
> There were a few Python syntax errors such as this one.  They
> can be found by running:
>
>   make -C v2v check TESTS=test-v2v-python-syntax.sh
>
Cool. Why not include this in "make check"?

>
> However I have now fixed them.
>
Thanks!

>
> > Do we have an easy way to test the plugin without running the entire
> > virt-v2v pipeline?
>
> Not really.  In fact we don't have any unit tests for -o rhv-upload
> functionality because there's no way to simulate the imageio server.
>
Actually it is easy to use real imageio server for testing.

1. install ovirt-imageio-daemon
2. configure it for testing

# /etc/ovirt-imageio-daemon/daemon.conf
[daemon]
# directory with this structure
#     certs/
#         vdsmcert.pem
#     keys
#         vdsmkey.pem


pki_dir = test/pki
poll_interval = 0.1

[images]
port = 9876

[tickets]
socket = /tmp/ovirt-imageio-daemon.sock

3. Now you can run it

/usr/bin/ovirt-imageio-daemon

To upload, without real engine, you can install your own ticket like this:

1. Create a ticket json:

    $ cat ticket.json
    {
        "uuid": "test",
        "size": 1073741824,
        "url": "file:///var/tmp/sd-uukd/vol-uuid",
        "timeout": 3000,
        "ops": ["read", "write"]
    }

2. Install the ticket

    $ curl --unix-socket /tmp/vdsm/ovirt-imageio-daemon.sock \
          -X PUT \
          --upload-file ticket.json \
          http://localhost/tickets/test

3. Create the image:

file:
touch /var/tmp/sd-uuuid/vol-uuid

(note that the file system must support direct I/O)

block:
lvcreate -n vol-uuid -L 1g sd-uuid


At this point you can upload or download using this transfer_url:
https://localhost:9876/images/test


We can make it even easier by supporting command line options so
you don't need to change /etc/ovirt-imageio-daemon, or even install
the package. For example:

    git clone https://github.com/oVirt/ovirt-imageio.git
    src/ovirt-image/daemon/ovirt-imageio-daemon -f my-test.conf

Another option is to start the server from your tests like this. This
is how we run our tests.

from ovirt_imageio_daemon import server
from ovirt_imageio_daemon import config

config.daemon.pki_dir = test/pki
config.daemon.poll_interval = 0.1
config.images.port = 9876
config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"

server.start(config)

# run your test here...

server.stop()

Note that we did not complete the port to python 3 yest, so upload tests
will work only with python 2.

The missing part is how to test without real engine. I think the best way
would
be to monkeypatch the ovirtsdk module so it does not send any requests,
and instead return fake responses. This add risk of having incorrect mocked
response, or not detecting wrong calls, but will make testing very easy,
which
make it easy for random contributors.


However you can run virt-v2v locally against an oVirt instance
without
> needing VMware.  The command is rather lengthy, but here it is:
>
> $ virt-builder fedora-27
> $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
>     -o rhv-upload \
>     -oc https://ovirt-engine.example.com/ovirt-engine/api \
>     -os ovirt-data \
>     -op /tmp/password \
>     -of raw \
>     -oo rhv-cafile=/tmp/ca.pem \
>     -oo rhv-direct
>
> /tmp/password should contain the oVirt admin password.
> /tmp/ca.pem should contain the oVirt CA cert.
>
> This will create a guest called ‘fedora-27’ which you'll need to
> delete (on oVirt) afterwards.  You can add ‘-on name’ to name it
> something else.
>
Should we document this?

>
> In any case I have fixed and verified this patch and will push it
> soon, thanks.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>

On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer
wrote:
> On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
<rjones@redhat.com>
> wrote:
> 
> > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer
<nirsof@gmail.com> wrote:
> > > > +    headers = {"Content-Type":
"application/json",
> > > > +               "Content-Length", str(len(buf))}
> >
> > There were a few Python syntax errors such as this one.  They
> > can be found by running:
> >
> >   make -C v2v check TESTS=test-v2v-python-syntax.sh
> >
> 
> Cool. Why not include this in "make check"?
It is!  That command is just if you want to run the single test.
> > Not really.  In fact we don't have any unit tests for -o
rhv-upload
> > functionality because there's no way to simulate the imageio
server.
> >
> 
> Actually it is easy to use real imageio server for testing.
> 
> 1. install ovirt-imageio-daemon
Yup, problem is this part.

However I'll look to see if we can make the tests run optionally _if_
someone has installed ovirt-imageio-daemon or has a local copy of the
sources, so the rest of the instructions are useful.  I think this
method looks most promising, but as you say I'd have to see about
mocking the other engine methods:
> Another option is to start the server from your tests like this. This
> is how we run our tests.
> 
> from ovirt_imageio_daemon import server
> from ovirt_imageio_daemon import config
> 
> config.daemon.pki_dir = test/pki
> config.daemon.poll_interval = 0.1
> config.images.port = 9876
> config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
> 
> server.start(config)
> 
> # run your test here...
> 
> server.stop()
[...]
> However you can run virt-v2v locally against an oVirt instance without
> > needing VMware.  The command is rather lengthy, but here it is:
> >
> > $ virt-builder fedora-27
> > $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> >     -o rhv-upload \
> >     -oc https://ovirt-engine.example.com/ovirt-engine/api \
> >     -os ovirt-data \
> >     -op /tmp/password \
> >     -of raw \
> >     -oo rhv-cafile=/tmp/ca.pem \
> >     -oo rhv-direct
> >
> > /tmp/password should contain the oVirt admin password.
> > /tmp/ca.pem should contain the oVirt CA cert.
> >
> > This will create a guest called ‘fedora-27’ which you'll need to
> > delete (on oVirt) afterwards.  You can add ‘-on name’ to name it
> > something else.
> >
> 
> Should we document this?
It's just the normal way to run virt-v2v, eg as documented here:

  http://libguestfs.org/virt-v2v.1.html#convert-from-vmware-to-rhv-ovirt

... except that I've changed the input side to use a local disk.

We did consider having an ‘-i builder’ input method (just for testing).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html