Libguestfs - Mar 2023 - [V2V PATCH v3 5/6] v2v, in-place: introduce --block-driver command line option

On 3/15/23 00:16, Richard W.M. Jones wrote:
> On Tue, Mar 14, 2023 at 04:06:18PM +0200, Andrey Drobyshev wrote:
>> Speaking of "make check": could you point out, for future
reference,
>> which particular sub-target you're referring to here?  I can see
these:
>> check-am, check-recursive, check-slow, check-TESTS, check-valgrind. 
And
>> none of them seems to refer to checking docs integrity.  Yet running
>> entire "make check" might be quite time consuming.
> 
> (FYI I'm on holiday at the moment, back 1st April)
Hi Richard,
Please enjoy your holiday, there's no urgency to answer this :)
> 
> 'make check' runs the test suite and as Laszlo said is reasonably
fast
> (on my machine anyway!).  Well, it should be around 5-15 mins.  You
> can add -j4 or -j`nproc` or similar to parallelise the tests.
> 
> 'make check-valgrind' runs the same tests but with valgrind.  This
is
> highly unlikely to affect this patch series which only touches OCaml
> code.
> 
> 'make check-slow' runs an extra set of tests that as you might
guess
> are quite slow.  I wouldn't bother with this for a simple patch.  I
> usually run it before major releases.
> 
> The other targets you mention are internally generated by automake.
> 
> Then you can run single tests, eg:
> 
> $ make check -C docs TESTS=" test-v2v-docs.sh "
Thanks for the detailed overview.  That is actually the answer to my
original question: I was looking for a sub-target which would check the
docs, and failed to see that instead there's a separate test for that
purpose.  And the reason for that is I tried running the suite as root
and without "--keep-going" option, thus causing the recursive
"check"
target to fail on tests/ before it gets to the docs/.

This raises another question.  If we run the "make check" suite
properly, i.e. as non-root, then:

1. libvirt is being chosen as the default input method;
2. Due to this patch libvirt_uri is set to "qemu:///session":
https://listman.redhat.com/archives/libguestfs/2013-December/msg00085.html.

Now if libvirtd is being run by root, qemu:///session won't work and
we'll get "could not connect to libvirt (URI = qemu:///session)".
That is exactly what I observe.

If I follow the docs (https://www.libguestfs.org/guestfs.3.html#backend)
and explicitly set LIBGUESTFS_BACKEND, it gets better.  I.e.

LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN


But then there's the test test-v2v-o-libvirt.sh which connects to
libvirtd not by the means of libguestfs, but rather invoking virsh
directly, which causes:

error: failed to connect to the hypervisor
error: Cannot recv data: Connection reset by peer

So the only way I'm able to successfully run the entire suite is this:

LIBVIRT_DEFAULT_URI=qemu:///system
LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN

My question is: is this how it's supposed to be?
> 
> Note that some individual tests depend on the test-data dir having
> been built first to build a bunch of phony guests:
> 
> $ make -C test-data check
> 
> (If you do 'make check' it will do the test-data dir first.)
> 
> Rich.
> 
Andrey

On 3/16/23 17:14, Andrey Drobyshev wrote:
> On 3/15/23 00:16, Richard W.M. Jones wrote:
>> On Tue, Mar 14, 2023 at 04:06:18PM +0200, Andrey Drobyshev wrote:
>>> Speaking of "make check": could you point out, for future
reference,
>>> which particular sub-target you're referring to here?  I can
see these:
>>> check-am, check-recursive, check-slow, check-TESTS, check-valgrind.
And
>>> none of them seems to refer to checking docs integrity.  Yet
running
>>> entire "make check" might be quite time consuming.
>>
>> (FYI I'm on holiday at the moment, back 1st April)
> 
> Hi Richard,
> Please enjoy your holiday, there's no urgency to answer this :)
> 
>>
>> 'make check' runs the test suite and as Laszlo said is
reasonably fast
>> (on my machine anyway!).  Well, it should be around 5-15 mins.  You
>> can add -j4 or -j`nproc` or similar to parallelise the tests.
>>
>> 'make check-valgrind' runs the same tests but with valgrind. 
This is
>> highly unlikely to affect this patch series which only touches OCaml
>> code.
>>
>> 'make check-slow' runs an extra set of tests that as you might
guess
>> are quite slow.  I wouldn't bother with this for a simple patch.  I
>> usually run it before major releases.
>>
>> The other targets you mention are internally generated by automake.
>>
>> Then you can run single tests, eg:
>>
>> $ make check -C docs TESTS=" test-v2v-docs.sh "
> 
> Thanks for the detailed overview.  That is actually the answer to my
> original question: I was looking for a sub-target which would check the
> docs, and failed to see that instead there's a separate test for that
> purpose.  And the reason for that is I tried running the suite as root
> and without "--keep-going" option, thus causing the recursive
"check"
> target to fail on tests/ before it gets to the docs/.
> 
> This raises another question.  If we run the "make check" suite
> properly, i.e. as non-root, then:
> 
> 1. libvirt is being chosen as the default input method;
I don't understand this. "Input method" is set with the
"-i" option.

Did you mean "default libguestfs backend"?

But even in that case, I don't understand. The default libguestfs
backend is supposed to be "direct".

If you have LIBGUESTFS_BACKEND permanently set to libvirt in your
environment, for various reasons, I'd suggest simply unsetting
LIBGUESTFS_BACKEND before running "make check".

Laszlo
> 2. Due to this patch libvirt_uri is set to "qemu:///session":
> https://listman.redhat.com/archives/libguestfs/2013-December/msg00085.html.
> 
> Now if libvirtd is being run by root, qemu:///session won't work and
> we'll get "could not connect to libvirt (URI =
qemu:///session)".
> That is exactly what I observe.
> 
> If I follow the docs (https://www.libguestfs.org/guestfs.3.html#backend)
> and explicitly set LIBGUESTFS_BACKEND, it gets better.  I.e.
> 
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> 
> But then there's the test test-v2v-o-libvirt.sh which connects to
> libvirtd not by the means of libguestfs, but rather invoking virsh
> directly, which causes:
> 
> error: failed to connect to the hypervisor
> error: Cannot recv data: Connection reset by peer
> 
> So the only way I'm able to successfully run the entire suite is this:
> 
> LIBVIRT_DEFAULT_URI=qemu:///system
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> My question is: is this how it's supposed to be?
> 
>>
>> Note that some individual tests depend on the test-data dir having
>> been built first to build a bunch of phony guests:
>>
>> $ make -C test-data check
>>
>> (If you do 'make check' it will do the test-data dir first.)
>>
>> Rich.
>>
> 
> Andrey
>

On Thu, Mar 16, 2023 at 06:14:05PM +0200, Andrey Drobyshev
wrote:
> Thanks for the detailed overview.  That is actually the answer to my
> original question: I was looking for a sub-target which would check the
> docs, and failed to see that instead there's a separate test for that
> purpose.  And the reason for that is I tried running the suite as root
> and without "--keep-going" option, thus causing the recursive
"check"
> target to fail on tests/ before it gets to the docs/.
> 
> This raises another question.  If we run the "make check" suite
> properly, i.e. as non-root, then:
> 
> 1. libvirt is being chosen as the default input method;
[libvirt backend]

As was already discussed in this thread, the upstream default is
direct (ie. run qemu directly without libvirt), but in Fedora and RHEL
we override that to always use the libvirt backend:

https://src.fedoraproject.org/rpms/libguestfs/blob/rawhide/f/libguestfs.spec#_732

Either can be overridden at runtime by setting $LIBGUESTFS_BACKEND.
> 2. Due to this patch libvirt_uri is set to "qemu:///session":
> https://listman.redhat.com/archives/libguestfs/2013-December/msg00085.html.
> 
> Now if libvirtd is being run by root, qemu:///session won't work and
> we'll get "could not connect to libvirt (URI =
qemu:///session)".
> That is exactly what I observe.
So this is the following bug (filed 11 years ago):

https://bugzilla.redhat.com/show_bug.cgi?id=890291

Tests should nevertheless work when run as non-root because they use a
session-mode libvirtd ("virtqemud") which is started in the background
when you run the tests (that's a feature of libvirt, not something
that we do explicitly).  The root libvirtd should never be used.

So I'm not sure about what's happening here exactly.  It does work for
me without taking any special action.  I never run the tests as root,
and do not recommend doing that.
> If I follow the docs (https://www.libguestfs.org/guestfs.3.html#backend)
> and explicitly set LIBGUESTFS_BACKEND, it gets better.  I.e.
> 
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> But then there's the test test-v2v-o-libvirt.sh which connects to
> libvirtd not by the means of libguestfs, but rather invoking virsh
> directly, which causes:
> 
> error: failed to connect to the hypervisor
> error: Cannot recv data: Connection reset by peer
> 
> So the only way I'm able to successfully run the entire suite is this:
> 
> LIBVIRT_DEFAULT_URI=qemu:///system
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> My question is: is this how it's supposed to be?
No I don't think so.

If you run these commands as non-root:

  $ virsh list --all

  $ ps -eo euser,ruser,suser,fuser,f,comm,label | grep virtqemud
  rjones   rjones   rjones   rjones   0 virtqemud      
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

there should be the session virtqemud owned by your user.  (There may
also be a root owned one, but ignore that.)

If you don't see that, then I suppose there's some problem with your
libvirt installation.

You can just try running the tests with LIBGUESTFS_BACKEND=direct
instead if libvirt is a problem.  Either way should work.

Rich.
> > 
> > Note that some individual tests depend on the test-data dir having
> > been built first to build a bunch of phony guests:
> > 
> > $ make -C test-data check
> > 
> > (If you do 'make check' it will do the test-data dir first.)
> > 
> > Rich.
> > 
> 
> Andrey
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html