thr3ads.net - Libguestfs - [Libguestfs] Figuring out some failing tests for libnbd [Jun 2021]

If this information is useful, please help other people find it:
Share via:

Martin Kletzander

2021-Jun-30 15:11 UTC

[Libguestfs] Figuring out some failing tests for libnbd

I am preparing more patches for CI to run check-valgrind and fix ongoing
errors but there are two issues I can not identify the reason why they
are failing.

- On debian-10 the info/info-can.sh started failing and the error
   message is just one of those I saw earlier in other places:

     libnbd: debug: nbd1: nbd_opt_abort: leave: error="nbd_opt_abort:
     invalid state: READY: the handle must be negotiating: Invalid
     argument"

- On Fedora rawhide I hit a random issue where a port in a URI was
   translated to its name and looking at the code I can not find how this
   could have happened.  Until this is fixed the test suite is unreliable
   and notification fatigue will cause everyone to start ignoring any
   future failures.

     /builds/nertpinx/libnbd/tests/.libs/aio-connect: actual URI
     nbd://127.0.0.1:altova-lm/ != expected URI nbd://127.0.0.1:35355/

- Both openSUSE builds are failing to run check-valgrind and it looks
   like it might be unrelated to libnbd, although it would be nice for
   someone else to confirm that.  For now I have disabled check-valgrind
   on those platforms in my branch.

- Similarly to openSUSE Ubuntu 20.04 fails in valgrind tests, but
   somewhere down the GnuTLS rabbit hole, which I presume is unrelated
   too, so I disabled check-valgrind on that one as well.

I will send the patches once they are cleaned up, but I wanted to let
everyone know what the current status is because eliminating all random
issues is essential to properly consuming CI results.

Thanks,
Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20210630/5e546638/attachment.sig>

Martin Kletzander

2021-Jun-30 15:49 UTC

head link

[Libguestfs] Figuring out some failing tests for libnbd

On Wed, Jun 30, 2021 at 05:11:45PM +0200, Martin Kletzander
wrote:>I am preparing more patches for CI to run check-valgrind and fix ongoing
>errors but there are two issues I can not identify the reason why they
>are failing.
>
>- On debian-10 the info/info-can.sh started failing and the error
>   message is just one of those I saw earlier in other places:
>
>     libnbd: debug: nbd1: nbd_opt_abort: leave: error="nbd_opt_abort:
>     invalid state: READY: the handle must be negotiating: Invalid
>     argument"
>
>- On Fedora rawhide I hit a random issue where a port in a URI was
>   translated to its name and looking at the code I can not find how this
>   could have happened.  Until this is fixed the test suite is unreliable
>   and notification fatigue will cause everyone to start ignoring any
>   future failures.
>
>     /builds/nertpinx/libnbd/tests/.libs/aio-connect: actual URI
>     nbd://127.0.0.1:altova-lm/ != expected URI nbd://127.0.0.1:35355/
>
Same happened on F33:

   /builds/nertpinx/libnbd/tests/.libs/lt-aio-connect: actual URI
   nbd://127.0.0.1:rt-helper/ != expected URI nbd://127.0.0.1:35006/
>- Both openSUSE builds are failing to run check-valgrind and it looks
>   like it might be unrelated to libnbd, although it would be nice for
>   someone else to confirm that.  For now I have disabled check-valgrind
>   on those platforms in my branch.
>
>- Similarly to openSUSE Ubuntu 20.04 fails in valgrind tests, but
>   somewhere down the GnuTLS rabbit hole, which I presume is unrelated
>   too, so I disabled check-valgrind on that one as well.
>
>I will send the patches once they are cleaned up, but I wanted to let
>everyone know what the current status is because eliminating all random
>issues is essential to properly consuming CI results.
>
I forgot to mention the pipeline with all the errors (before the
check-valgrind skips) is here:

   https://gitlab.com/nertpinx/libnbd/-/pipelines/329661257

And the latest (what I sent the patches from) is here:

   https://gitlab.com/nertpinx/libnbd/-/pipelines/329755114
>Thanks,
>Martin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20210630/9bbeee83/attachment.sig>

Richard W.M. Jones

2021-Jun-30 15:51 UTC

head link

[Libguestfs] Figuring out some failing tests for libnbd

On Wed, Jun 30, 2021 at 05:11:45PM +0200, Martin Kletzander
wrote:> I am preparing more patches for CI to run check-valgrind and fix ongoing
> errors but there are two issues I can not identify the reason why they
> are failing.
> 
> - On debian-10 the info/info-can.sh started failing and the error
>   message is just one of those I saw earlier in other places:
> 
>     libnbd: debug: nbd1: nbd_opt_abort: leave: error="nbd_opt_abort:
>     invalid state: READY: the handle must be negotiating: Invalid
>     argument"
I think it is probably not the error since it happens during normal
output.  The error must be something else.  Is this when using
valgrind or normally?  Do you have the complete log?
> - On Fedora rawhide I hit a random issue where a port in a URI was
>   translated to its name and looking at the code I can not find how this
>   could have happened.  Until this is fixed the test suite is unreliable
>   and notification fatigue will cause everyone to start ignoring any
>   future failures.
> 
>     /builds/nertpinx/libnbd/tests/.libs/aio-connect: actual URI
>     nbd://127.0.0.1:altova-lm/ != expected URI nbd://127.0.0.1:35355/
That is likely a real bug.  In libnbd 1.8 we added a new API which
creates the URI for connecting back to the server:

https://libguestfs.org/nbd_get_uri.3.html

When this particular test is run, the following code path is followed:

https://gitlab.com/nbdkit/libnbd/-/blob/c6109b5d37c08a430b3a55691dff72321eeac23e/lib/uri.c#L418

which uses getnameinfo.  I thought that by using NI_NUMERICHOST it
wouldn't translate the port number using /etc/services.  Apparently
that's not the case.

A simple reproducer would be the one attached, except that it
segfaults :-( which means now we've got 2 bugs ...

  $ nbdkit null --run './uri.py'
  nbdkit: external command was killed by signal 6

The stack trace is rather long so I attached that too.

Patches welcome for both issues.
> - Both openSUSE builds are failing to run check-valgrind and it looks
>   like it might be unrelated to libnbd, although it would be nice for
>   someone else to confirm that.  For now I have disabled check-valgrind
>   on those platforms in my branch.
What's the failure?
> - Similarly to openSUSE Ubuntu 20.04 fails in valgrind tests, but
>   somewhere down the GnuTLS rabbit hole, which I presume is unrelated
>   too, so I disabled check-valgrind on that one as well.
> 
> I will send the patches once they are cleaned up, but I wanted to let
> everyone know what the current status is because eliminating all random
> issues is essential to properly consuming CI results.
Thanks,

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
-------------- next part --------------
#!/usr/bin/python3

import nbd

h = nbd.NBD()
sockaddr = ("localhost", "10809")
h.aio_connect(sockaddr)
while h.aio_is_connecting():
    sys.sleep(1)
print("%s\n" % h.get_uri())
-------------- next part --------------
(gdb) bt
#0  __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007fcf164aa8a4 in __GI_abort () at abort.c:79
#2  0x00007fcf084bd5f5 in nbd_internal_py_aio_connect (self=<optimized
out>,
    args=<optimized out>)
    at /usr/src/debug/libnbd-1.8.0-2.fc35.x86_64/python/methods.c:2554
#3  0x00007fcf167681e8 in cfunction_call (func=0x7fcf089cdd50, 
    args=<optimized out>, kwargs=<optimized out>)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Objects/methodobject.c:548
#4  0x00007fcf16763dc4 in _PyObject_MakeTpCall (tstate=0x55958608b7e0, 
    callable=0x7fcf089cdd50, args=0x7fcf08b64ee0, nargs=2, keywords=0x0)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Objects/call.c:215
#5  0x00007fcf16760d03 in _PyObject_VectorcallTstate (kwnames=0x0, 
    nargsf=<optimized out>, args=0x7fcf08b64ee0, callable=<optimized
out>,
    tstate=0x55958608b7e0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/cpython/abstract.h:112
#6  _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, 
    args=0x7fcf08b64ee0, callable=<optimized out>, tstate=0x55958608b7e0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/cpython/abstract.h:99
#7  PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, 
    args=0x7fcf08b64ee0, callable=<optimized out>)
--Type <RET> for more, q to quit, c to continue without paging--
   3.10.0~b2-3.fc35.x86_64/Include/cpython/abstract.h:123
#8  call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>, trace_info=0x7ffc3a7f1b60, 
    tstate=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:5885
#9  _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized
out>,
    throwflag=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:4182
#10 0x00007fcf16759ec4 in _PyEval_EvalFrame (throwflag=0, f=0x7fcf08b64d60, 
    tstate=0x55958608b7e0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/internal/pycore_ceval.h:46
#11 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=2, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:5069
#12 0x00007fcf1675b28e in _PyObject_VectorcallTstate (kwnames=0x0, 
    nargsf=<optimized out>, args=0x7fcf08d09ba8, callable=0x7fcf089d2dd0, 
    tstate=0x55958608b7e0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/cpython/abstract.h:114
#13 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, 
    args=0x7fcf08d09ba8, callable=0x7fcf089d2dd0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/cpython/abstract.h:123
#14 call_function (kwnames=0x0, oparg=<optimized out>, 
    pp_stack=<synthetic pointer>, trace_info=0x7ffc3a7f1db0, 
    tstate=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:5885
#15 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized
out>,
    throwflag=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:4199
#16 0x00007fcf16759ec4 in _PyEval_EvalFrame (throwflag=0, f=0x7fcf08d09a40, 
    tstate=0x55958608b7e0)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Include/internal/pycore_ceval.h:46
#17 _PyEval_Vector (tstate=<optimized out>, con=<optimized out>, 
    locals=<optimized out>, args=<optimized out>, argcount=0, 
    kwnames=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:5069
#18 0x00007fcf167d42b4 in PyEval_EvalCode (co=0x7fcf08c1f7e0, 
    globals=0x7fcf08c21cc0, locals=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/ceval.c:1135
#19 0x00007fcf16803944 in run_eval_code_obj (tstate=0x55958608b7e0, 
    co=0x7fcf08c1f7e0, globals=0x7fcf08c21cc0, locals=0x7fcf08c21cc0)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/pythonrun.c:1289
#20 0x00007fcf167fefd6 in run_mod (mod=<optimized out>, 
    filename=<optimized out>, globals=0x7fcf08c21cc0,
locals=0x7fcf08c21cc0,
    flags=<optimized out>, arena=<optimized out>)
    at
/usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/pythonrun.c:1310--Type
<RET> for more, q to quit, c to continue without paging--

#21 0x00007fcf166d8c9f in pyrun_file (fp=<optimized out>, 
    filename=<optimized out>, start=<optimized out>,
globals=<optimized out>,
    locals=<optimized out>, closeit=<optimized out>,
flags=0x7ffc3a7f2158)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/pythonrun.c:1206
#22 0x00007fcf167f9cc2 in _PyRun_SimpleFileObject (fp=0x5595860903d0, 
    filename=0x7fcf08c0fa00, closeit=1, flags=0x7ffc3a7f2158)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/pythonrun.c:455
#23 0x00007fcf167f99c7 in _PyRun_AnyFileObject (fp=0x5595860903d0, 
    filename=0x7fcf08c0fa00, closeit=1, flags=0x7ffc3a7f2158)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Python/pythonrun.c:89
#24 0x00007fcf167f6c34 in pymain_run_file_obj (skip_source_first_line=0, 
    filename=0x7fcf08c0fa00, program_name=0x7fcf08cd29c0)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Modules/main.c:353
#25 pymain_run_file (config=0x55958606fd60)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Modules/main.c:372
#26 pymain_run_python (exitcode=0x7ffc3a7f2150)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Modules/main.c:587
#27 Py_RunMain ()
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Modules/main.c:666
#28 0x00007fcf167c54dd in Py_BytesMain (argc=<optimized out>, 
    argv=<optimized out>)
    at /usr/src/debug/python3.10-3.10.0~b2-3.fc35.x86_64/Modules/main.c:720
#29 0x00007fcf164abb75 in __libc_start_main (main=0x559585dfe160 <main>, 
    argc=2, argv=0x7ffc3a7f2368, init=<optimized out>, fini=<optimized
out>,
    rtld_fini=<optimized out>, stack_end=0x7ffc3a7f2358)
    at ../csu/libc-start.c:332
#30 0x0000559585dfe09e in _start ()

Libguestfs - Jun 2021 - Figuring out some failing tests for libnbd

[Libguestfs] Figuring out some failing tests for libnbd

[Libguestfs] Figuring out some failing tests for libnbd

[Libguestfs] Figuring out some failing tests for libnbd