Libguestfs - Mar 2023 - [V2V PATCH v3 5/6] v2v, in-place: introduce --block-driver command line option

On 3/16/23 17:14, Andrey Drobyshev wrote:
> On 3/15/23 00:16, Richard W.M. Jones wrote:
>> On Tue, Mar 14, 2023 at 04:06:18PM +0200, Andrey Drobyshev wrote:
>>> Speaking of "make check": could you point out, for future
reference,
>>> which particular sub-target you're referring to here?  I can
see these:
>>> check-am, check-recursive, check-slow, check-TESTS, check-valgrind.
And
>>> none of them seems to refer to checking docs integrity.  Yet
running
>>> entire "make check" might be quite time consuming.
>>
>> (FYI I'm on holiday at the moment, back 1st April)
> 
> Hi Richard,
> Please enjoy your holiday, there's no urgency to answer this :)
> 
>>
>> 'make check' runs the test suite and as Laszlo said is
reasonably fast
>> (on my machine anyway!).  Well, it should be around 5-15 mins.  You
>> can add -j4 or -j`nproc` or similar to parallelise the tests.
>>
>> 'make check-valgrind' runs the same tests but with valgrind. 
This is
>> highly unlikely to affect this patch series which only touches OCaml
>> code.
>>
>> 'make check-slow' runs an extra set of tests that as you might
guess
>> are quite slow.  I wouldn't bother with this for a simple patch.  I
>> usually run it before major releases.
>>
>> The other targets you mention are internally generated by automake.
>>
>> Then you can run single tests, eg:
>>
>> $ make check -C docs TESTS=" test-v2v-docs.sh "
> 
> Thanks for the detailed overview.  That is actually the answer to my
> original question: I was looking for a sub-target which would check the
> docs, and failed to see that instead there's a separate test for that
> purpose.  And the reason for that is I tried running the suite as root
> and without "--keep-going" option, thus causing the recursive
"check"
> target to fail on tests/ before it gets to the docs/.
> 
> This raises another question.  If we run the "make check" suite
> properly, i.e. as non-root, then:
> 
> 1. libvirt is being chosen as the default input method;
I don't understand this. "Input method" is set with the
"-i" option.

Did you mean "default libguestfs backend"?

But even in that case, I don't understand. The default libguestfs
backend is supposed to be "direct".

If you have LIBGUESTFS_BACKEND permanently set to libvirt in your
environment, for various reasons, I'd suggest simply unsetting
LIBGUESTFS_BACKEND before running "make check".

Laszlo
> 2. Due to this patch libvirt_uri is set to "qemu:///session":
> https://listman.redhat.com/archives/libguestfs/2013-December/msg00085.html.
> 
> Now if libvirtd is being run by root, qemu:///session won't work and
> we'll get "could not connect to libvirt (URI =
qemu:///session)".
> That is exactly what I observe.
> 
> If I follow the docs (https://www.libguestfs.org/guestfs.3.html#backend)
> and explicitly set LIBGUESTFS_BACKEND, it gets better.  I.e.
> 
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> 
> But then there's the test test-v2v-o-libvirt.sh which connects to
> libvirtd not by the means of libguestfs, but rather invoking virsh
> directly, which causes:
> 
> error: failed to connect to the hypervisor
> error: Cannot recv data: Connection reset by peer
> 
> So the only way I'm able to successfully run the entire suite is this:
> 
> LIBVIRT_DEFAULT_URI=qemu:///system
> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
> 
> My question is: is this how it's supposed to be?
> 
>>
>> Note that some individual tests depend on the test-data dir having
>> been built first to build a bunch of phony guests:
>>
>> $ make -C test-data check
>>
>> (If you do 'make check' it will do the test-data dir first.)
>>
>> Rich.
>>
> 
> Andrey
>

On 3/17/23 10:37, Laszlo Ersek wrote:
> On 3/16/23 17:14, Andrey Drobyshev wrote:
>> On 3/15/23 00:16, Richard W.M. Jones wrote:
>>> On Tue, Mar 14, 2023 at 04:06:18PM +0200, Andrey Drobyshev wrote:
>>>> Speaking of "make check": could you point out, for
future reference,
>>>> which particular sub-target you're referring to here?  I
can see these:
>>>> check-am, check-recursive, check-slow, check-TESTS,
check-valgrind.  And
>>>> none of them seems to refer to checking docs integrity.  Yet
running
>>>> entire "make check" might be quite time consuming.
>>>
>>> (FYI I'm on holiday at the moment, back 1st April)
>>
>> Hi Richard,
>> Please enjoy your holiday, there's no urgency to answer this :)
>>
>>>
>>> 'make check' runs the test suite and as Laszlo said is
reasonably fast
>>> (on my machine anyway!).  Well, it should be around 5-15 mins.  You
>>> can add -j4 or -j`nproc` or similar to parallelise the tests.
>>>
>>> 'make check-valgrind' runs the same tests but with
valgrind.  This is
>>> highly unlikely to affect this patch series which only touches
OCaml
>>> code.
>>>
>>> 'make check-slow' runs an extra set of tests that as you
might guess
>>> are quite slow.  I wouldn't bother with this for a simple
patch.  I
>>> usually run it before major releases.
>>>
>>> The other targets you mention are internally generated by automake.
>>>
>>> Then you can run single tests, eg:
>>>
>>> $ make check -C docs TESTS=" test-v2v-docs.sh "
>>
>> Thanks for the detailed overview.  That is actually the answer to my
>> original question: I was looking for a sub-target which would check the
>> docs, and failed to see that instead there's a separate test for
that
>> purpose.  And the reason for that is I tried running the suite as root
>> and without "--keep-going" option, thus causing the recursive
"check"
>> target to fail on tests/ before it gets to the docs/.
>>
>> This raises another question.  If we run the "make check"
suite
>> properly, i.e. as non-root, then:
>>
>> 1. libvirt is being chosen as the default input method;
> 
> I don't understand this. "Input method" is set with the
"-i" option.
> 
> Did you mean "default libguestfs backend"?
Sorry, you're right, I seem to have missed the terms here.  I meant the
libguestfs backend, which of course has nothing to do with v2v's input
method.
> 
> But even in that case, I don't understand. The default libguestfs
> backend is supposed to be "direct".
> 
> If you have LIBGUESTFS_BACKEND permanently set to libvirt in your
> environment, for various reasons, I'd suggest simply unsetting
> LIBGUESTFS_BACKEND before running "make check".
No, I don't have this set in my environment.  But here's the thing: in
RHEL, CentOS (and in other RHEL-like distros, I presume) libguestfs is
being built with the option "--with-default-backend=libvirt" set:

https://gitlab.com/redhat/centos-stream/rpms/libguestfs/-/blob/5089358fe5/libguestfs.spec#L744

And if you don't put extra effort in linking your freshly built v2v with
libguestfs also built from source, but rather link it against the
libguestfs present in the system -- then the issue still exists and the
question remains.

I'm wondering how do you happen to avoid this issue, supposing that
you're also doing your development on a RHEL-like OS?  Am I missing
something here?  And having all that said: wouldn't it be beneficial and
more robust to explicitly set libguestfs backend when running the test
suite?
> 
> Laszlo
> 
>> 2. Due to this patch libvirt_uri is set to "qemu:///session":
>>
https://listman.redhat.com/archives/libguestfs/2013-December/msg00085.html.
>>
>> Now if libvirtd is being run by root, qemu:///session won't work
and
>> we'll get "could not connect to libvirt (URI =
qemu:///session)".
>> That is exactly what I observe.
>>
>> If I follow the docs
(https://www.libguestfs.org/guestfs.3.html#backend)
>> and explicitly set LIBGUESTFS_BACKEND, it gets better.  I.e.
>>
>> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
>>
>>
>> But then there's the test test-v2v-o-libvirt.sh which connects to
>> libvirtd not by the means of libguestfs, but rather invoking virsh
>> directly, which causes:
>>
>> error: failed to connect to the hypervisor
>> error: Cannot recv data: Connection reset by peer
>>
>> So the only way I'm able to successfully run the entire suite is
this:
>>
>> LIBVIRT_DEFAULT_URI=qemu:///system
>> LIBGUESTFS_BACKEND=libvirt:qemu:///system make check -jN
>>
>> My question is: is this how it's supposed to be?
>>
>>>
>>> Note that some individual tests depend on the test-data dir having
>>> been built first to build a bunch of phony guests:
>>>
>>> $ make -C test-data check
>>>
>>> (If you do 'make check' it will do the test-data dir
first.)
>>>
>>> Rich.
>>>
>>
>> Andrey
>>
>

On 3/17/23 13:46, Andrey Drobyshev wrote:
> On 3/17/23 10:37, Laszlo Ersek wrote:
>> On 3/16/23 17:14, Andrey Drobyshev wrote:
>>> On 3/15/23 00:16, Richard W.M. Jones wrote:
>>>> On Tue, Mar 14, 2023 at 04:06:18PM +0200, Andrey Drobyshev
wrote:
>>>>> Speaking of "make check": could you point out,
for future reference,
>>>>> which particular sub-target you're referring to here? 
I can see these:
>>>>> check-am, check-recursive, check-slow, check-TESTS,
check-valgrind.  And
>>>>> none of them seems to refer to checking docs integrity. 
Yet running
>>>>> entire "make check" might be quite time
consuming.
>>>>
>>>> (FYI I'm on holiday at the moment, back 1st April)
>>>
>>> Hi Richard,
>>> Please enjoy your holiday, there's no urgency to answer this :)
>>>
>>>>
>>>> 'make check' runs the test suite and as Laszlo said is
reasonably fast
>>>> (on my machine anyway!).  Well, it should be around 5-15 mins. 
You
>>>> can add -j4 or -j`nproc` or similar to parallelise the tests.
>>>>
>>>> 'make check-valgrind' runs the same tests but with
valgrind.  This is
>>>> highly unlikely to affect this patch series which only touches
OCaml
>>>> code.
>>>>
>>>> 'make check-slow' runs an extra set of tests that as
you might guess
>>>> are quite slow.  I wouldn't bother with this for a simple
patch.  I
>>>> usually run it before major releases.
>>>>
>>>> The other targets you mention are internally generated by
automake.
>>>>
>>>> Then you can run single tests, eg:
>>>>
>>>> $ make check -C docs TESTS=" test-v2v-docs.sh "
>>>
>>> Thanks for the detailed overview.  That is actually the answer to
my
>>> original question: I was looking for a sub-target which would check
the
>>> docs, and failed to see that instead there's a separate test
for that
>>> purpose.  And the reason for that is I tried running the suite as
root
>>> and without "--keep-going" option, thus causing the
recursive "check"
>>> target to fail on tests/ before it gets to the docs/.
>>>
>>> This raises another question.  If we run the "make check"
suite
>>> properly, i.e. as non-root, then:
>>>
>>> 1. libvirt is being chosen as the default input method;
>>
>> I don't understand this. "Input method" is set with the
"-i" option.
>>
>> Did you mean "default libguestfs backend"?
> 
> Sorry, you're right, I seem to have missed the terms here.  I meant the
> libguestfs backend, which of course has nothing to do with v2v's input
> method.
> 
>>
>> But even in that case, I don't understand. The default libguestfs
>> backend is supposed to be "direct".
>>
>> If you have LIBGUESTFS_BACKEND permanently set to libvirt in your
>> environment, for various reasons, I'd suggest simply unsetting
>> LIBGUESTFS_BACKEND before running "make check".
> 
> No, I don't have this set in my environment.  But here's the thing:
in
> RHEL, CentOS (and in other RHEL-like distros, I presume) libguestfs is
> being built with the option "--with-default-backend=libvirt" set:
> 
>
https://gitlab.com/redhat/centos-stream/rpms/libguestfs/-/blob/5089358fe5/libguestfs.spec#L744
correct...
> 
> And if you don't put extra effort in linking your freshly built v2v
with
> libguestfs also built from source, but rather link it against the
> libguestfs present in the system -- then the issue still exists and the
> question remains.
... and that must be why I'm not seeing this issue on my RHEL-9.1
system, even with LIBGUESTFS_BACKEND unset: I had made it a "sticking
point" to build all libguestfs-related projects from source, and to
ensure they'd consume each other.

(... Except for augeas, because... well, let me not finish that sentence.)

It's not trivial at all; I've been referring to this dependency diagram
(constructed with Rich's help):

                       libvirt-ocaml ---------
                                              \
                       libnbd <--> nbdkit---   \
                                            \   \
  hivex    ----> libguestfs  --------------------> virt-v2v -> virt-p2v
              /              \                  /
   supermin --                -> guestfs-tools -

(And this diagram doesn't even show the libguestfs-common submodule,
which is shared by libguestfs, guestfs-tools and virt-v2v. If you want
to test common submodule changes, you need to play with superproject
patches like

diff --git a/.gitmodules b/.gitmodules
index 1343142128f6..7b1a84418c2b 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,3 +1,4 @@
 [submodule "common"]
        path = common
-       url = https://github.com/libguestfs/libguestfs-common
+       url = file:///home/lacos/src/v2v/libguestfs-common
+       branch = .

and commands like "git submodule sync". Very cumbersome.)

Building and running *all* the projects, from upstream checkouts, had
not even been possible ~1.5 years ago; that was why I started my
contributions to these projects with patches that would enable said
building/running, covering the *entire* project set.)

... So, I would propose to just set LIBGUESTFS_BACKEND=direct for
running the test suite :)
> 
> I'm wondering how do you happen to avoid this issue, supposing that
> you're also doing your development on a RHEL-like OS?
See above :)
> Am I missing
> something here?  And having all that said: wouldn't it be beneficial
and
> more robust to explicitly set libguestfs backend when running the test
> suite?
Yes, I think that LIBGUESTFS_BACKEND=direct makes sense as a built-in
default for the test suite, but we'll have to wait for Rich's opinion on
that.

Laszlo