Libguestfs - Jun 2018 - Re: [PATCH] v2v: -o rhv-upload: Optimize http request sending

On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer
wrote:
> On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
<rjones@redhat.com>
> wrote:
> 
> > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer
<nirsof@gmail.com> wrote:
> > > > +    headers = {"Content-Type":
"application/json",
> > > > +               "Content-Length", str(len(buf))}
> >
> > There were a few Python syntax errors such as this one.  They
> > can be found by running:
> >
> >   make -C v2v check TESTS=test-v2v-python-syntax.sh
> >
> 
> Cool. Why not include this in "make check"?
It is!  That command is just if you want to run the single test.
> > Not really.  In fact we don't have any unit tests for -o
rhv-upload
> > functionality because there's no way to simulate the imageio
server.
> >
> 
> Actually it is easy to use real imageio server for testing.
> 
> 1. install ovirt-imageio-daemon
Yup, problem is this part.

However I'll look to see if we can make the tests run optionally _if_
someone has installed ovirt-imageio-daemon or has a local copy of the
sources, so the rest of the instructions are useful.  I think this
method looks most promising, but as you say I'd have to see about
mocking the other engine methods:
> Another option is to start the server from your tests like this. This
> is how we run our tests.
> 
> from ovirt_imageio_daemon import server
> from ovirt_imageio_daemon import config
> 
> config.daemon.pki_dir = test/pki
> config.daemon.poll_interval = 0.1
> config.images.port = 9876
> config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
> 
> server.start(config)
> 
> # run your test here...
> 
> server.stop()
[...]
> However you can run virt-v2v locally against an oVirt instance without
> > needing VMware.  The command is rather lengthy, but here it is:
> >
> > $ virt-builder fedora-27
> > $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> >     -o rhv-upload \
> >     -oc https://ovirt-engine.example.com/ovirt-engine/api \
> >     -os ovirt-data \
> >     -op /tmp/password \
> >     -of raw \
> >     -oo rhv-cafile=/tmp/ca.pem \
> >     -oo rhv-direct
> >
> > /tmp/password should contain the oVirt admin password.
> > /tmp/ca.pem should contain the oVirt CA cert.
> >
> > This will create a guest called ‘fedora-27’ which you'll need to
> > delete (on oVirt) afterwards.  You can add ‘-on name’ to name it
> > something else.
> >
> 
> Should we document this?
It's just the normal way to run virt-v2v, eg as documented here:

  http://libguestfs.org/virt-v2v.1.html#convert-from-vmware-to-rhv-ovirt

... except that I've changed the input side to use a local disk.

We did consider having an ‘-i builder’ input method (just for testing).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On Mon, Jun 18, 2018 at 9:23 PM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer wrote:
> > On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
<rjones@redhat.com>
> > wrote:
> >
> > > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer
<nirsof@gmail.com> wrote:
> > > > > +    headers = {"Content-Type":
"application/json",
> > > > > +               "Content-Length",
str(len(buf))}
> > >
> > > There were a few Python syntax errors such as this one.  They
> > > can be found by running:
> > >
> > >   make -C v2v check TESTS=test-v2v-python-syntax.sh
> > >
> >
> > Cool. Why not include this in "make check"?
>
> It is!  That command is just if you want to run the single test.
>
Strange, I did run "make check" but it did not fail on checking
the rhv plugin. It was failing because of unralted c-api test.

Maybe make aborted before running the v2v tests?

>
> > > Not really.  In fact we don't have any unit tests for -o
rhv-upload
> > > functionality because there's no way to simulate the imageio
server.
> > >
> >
> > Actually it is easy to use real imageio server for testing.
> >
> > 1. install ovirt-imageio-daemon
>
> Yup, problem is this part.
>
> However I'll look to see if we can make the tests run optionally _if_
> someone has installed ovirt-imageio-daemon or has a local copy of the
> sources, so the rest of the instructions are useful.  I think this
> method looks most promising, but as you say I'd have to see about
> mocking the other engine methods:
>
There is also:
https://github.com/kevin1024/vcrpy

I don't like these mocking libraries very much, but this may be useful.
After you run the tests once with a real system it can record all the
interactions, and the next time you run the tests it will use recorded data.

See https://il.pycon.org/2018/schedule/presentation/37/

>
> > Another option is to start the server from your tests like this. This
> > is how we run our tests.
> >
> > from ovirt_imageio_daemon import server
> > from ovirt_imageio_daemon import config
> >
> > config.daemon.pki_dir = test/pki
> > config.daemon.poll_interval = 0.1
> > config.images.port = 9876
> > config.tickets.socket = "/tmp/ovirt-imageio-daemon.sock"
> >
> > server.start(config)
> >
> > # run your test here...
> >
> > server.stop()
>
> [...]
> > However you can run virt-v2v locally against an oVirt instance without
> > > needing VMware.  The command is rather lengthy, but here it is:
> > >
> > > $ virt-builder fedora-27
> > > $ ./run virt-v2v -i disk /var/tmp/fedora-27.img \
> > >     -o rhv-upload \
> > >     -oc https://ovirt-engine.example.com/ovirt-engine/api \
> > >     -os ovirt-data \
> > >     -op /tmp/password \
> > >     -of raw \
> > >     -oo rhv-cafile=/tmp/ca.pem \
> > >     -oo rhv-direct
> > >
> > > /tmp/password should contain the oVirt admin password.
> > > /tmp/ca.pem should contain the oVirt CA cert.
> > >
> > > This will create a guest called ‘fedora-27’ which you'll need
to
> > > delete (on oVirt) afterwards.  You can add ‘-on name’ to name it
> > > something else.
> > >
> >
> > Should we document this?
>
> It's just the normal way to run virt-v2v, eg as documented here:
>
>   http://libguestfs.org/virt-v2v.1.html#convert-from-vmware-to-rhv-ovirt
>
> ... except that I've changed the input side to use a local disk.
>
> We did consider having an ‘-i builder’ input method (just for testing).
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
>

On Mon, Jun 18, 2018 at 09:58:23PM +0300, Nir Soffer
wrote:
> On Mon, Jun 18, 2018 at 9:23 PM Richard W.M. Jones
<rjones@redhat.com>
> wrote:
> 
> > On Mon, Jun 18, 2018 at 08:55:13PM +0300, Nir Soffer wrote:
> > > On Mon, Jun 18, 2018 at 1:37 PM Richard W.M. Jones
<rjones@redhat.com>
> > > wrote:
> > >
> > > > On Thu, Jun 14, 2018 at 09:24:48PM +0300, Nir Soffer wrote:
> > > > > On Thu, Jun 14, 2018 at 9:16 PM Nir Soffer
<nirsof@gmail.com> wrote:
> > > > > > +    headers = {"Content-Type":
"application/json",
> > > > > > +               "Content-Length",
str(len(buf))}
> > > >
> > > > There were a few Python syntax errors such as this one. 
They
> > > > can be found by running:
> > > >
> > > >   make -C v2v check TESTS=test-v2v-python-syntax.sh
> > > >
> > >
> > > Cool. Why not include this in "make check"?
> >
> > It is!  That command is just if you want to run the single test.
> >
> 
> Strange, I did run "make check" but it did not fail on checking
> the rhv plugin. It was failing because of unralted c-api test.
> 
> Maybe make aborted before running the v2v tests?
Right, it runs the tests in subdirectories in order and aborts at the
first subdirectory which fails (this is just how make / automake
works, it's nothing special to libguestfs):

  $ make print-subdirs | fold -72 -s
  common/mlstdutils generator tests/qemu test-data gnulib/lib 
  gnulib/tests common/errnostring common/protocol common/qemuopts 
  common/utils common/structs lib docs examples po common/mlutils 
  common/mlaugeas common/mlpcre daemon tests/daemon appliance tests/c-api 
  tests/tmpdirs tests/protocol tests/events tests/parallel tests/create 
  tests/disks tests/discard tests/mountable tests/network tests/lvm 
  tests/luks tests/md tests/selinux tests/relabel tests/ntfs tests/btrfs 
  tests/xfs tests/charsets tests/xml tests/mount-local tests/9p 
  tests/rsync tests/bigdirs tests/disk-labels tests/hotplug tests/nbd 
  tests/http tests/syslinux tests/journal tests/relative-paths 
  tests/gdisk tests/regressions tests/tsk tests/yara common/edit 
  common/options common/parallel common/progress common/visit 
  common/windows test-tool fish align cat diff df edit format inspector 
  make-fs rescue common/miniexpect p2v bash perl perl/examples ocaml 
  ocaml/examples python python/examples ruby ruby/examples java 
  java/examples php erlang erlang/examples lua lua/examples gobject 
  csharp common/mlgettext common/mlprogress common/mlvisit common/mlxml 
  common/mltools customize builder builder/templates get-kernel resize 
  sparsify sysprep v2v v2v/test-harness dib tools fuse 
  utils/boot-analysis utils/boot-benchmark utils/max-disks 
  utils/qemu-boot utils/qemu-speed-test po-docs

If you want it to continue after the first failure (this also applies
to every automake-using program) then do:

  make check -k
  less `find -name test-suite.log | xargs grep -l ^FAIL:`

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html