search for: gnutls_bye

.../to/libnbd/run make check' will hang without this series. In short, this series is restoring the shutdown(SHUT_WR) call that got lost from plugins/nbd/nbd.c when nbdkit switched to libnbd in commit ab7760fc. Eric Blake (2): sockets: Add .shut_writes callback states: Add state for shutdown/gnutls_bye after NBD_CMD_DISC lib/internal.h | 4 +++- generator/state_machine.ml | 19 +++++++++++++++++-- generator/states-issue-command.c | 20 ++++++++++++++++++-- lib/crypto.c | 21 +++++++++++++++++---- lib/socket.c | 12 +++++++++++- 5...

...hat I also need to augment the loop condition in the reader thread to additionally break out of the loop when the pipe-to-self sees EOF even when nbd_aio_is_dead() has not yet been satisfied I'm also fairly confident that I'll need to patch libnbd to call shutdown(SHUT_WR) on plaintext and gnutls_bye(GNUTLS_SHUT_WR) on TLS connections after sending NBD_CMD_DISC, as that is something we lowt when the nbd plugin switched to libnbd, even after documenting in commit 430f8141 why it is important for preventing shutdown deadlocks. Eric Blake (2): nbd: Don't reference stale errno in reader loop...

...he reader thread is itself blocked on a poll() > that will never make additional progress. Tracing the race is > difficult: nbd_shutdown() sends NBD_CMD_DISC to the server, and the > NBD protocol does not require the server to send a response but does > not forbid the server from using gnutls_bye to at least gracefully end > the TLS session. So where a plaintext server just closes the socket > (and the resulting EOF exits our poll()), it appears that with TLS, we > can sometimes hit the situation where the server is waiting for a > response to gnutls_bye() (note that qemu does...

...ader thread, while the reader thread is itself blocked on a poll() that will never make additional progress. Tracing the race is difficult: nbd_shutdown() sends NBD_CMD_DISC to the server, and the NBD protocol does not require the server to send a response but does not forbid the server from using gnutls_bye to at least gracefully end the TLS session. So where a plaintext server just closes the socket (and the resulting EOF exits our poll()), it appears that with TLS, we can sometimes hit the situation where the server is waiting for a response to gnutls_bye() (note that qemu does not use this functio...

...on EOF - move state to CLOSED pthread_join stuck As can be seen, having two threads compete on poll() on the same fd is unwise. Worse, if the server uses a means that will cause us to see EOF even though the socket is still open (such as shutdown(SHUT_WR) on plaintext, or gnutls_bye() with TLS), but waits for us to close the socket first, we end up with two processes deadlocked on each other. (nbdkit as server has used gnutls_bye since commit bd9a52e0; qemu however does not use it.) Other commits such as 14ba4154, c70616f8, and 430f8141 show that getting the shutdown sequence...

I have c-client installed on a 64bit system running Gentoo. I am trying to run configure so I can test the IMAP voicemail functionality. But asterisk-1.4.22 # ./configure --with-imap=/usr/include/imap just gives me the following error. checking for gnutls_bye in -lgnutls... no checking for UW IMAP Toolkit c-client library... no checking for system c-client library...... no configure: *** configure: *** The UW IMAP Toolkit installation on this system appears to be broken. configure: *** Either correct the installation, or run configure configure: *** inc...

We're still seeing sporadic failures of 'nbdkit nbd tls=', and I'm still trying to come up with a root cause fix (it may involve smarter use of gnutls_bye() in libnbd). In the meantime, here's what we know: when the hang/failure happens, the 'nbdkit nbd tls=' client process is stuck in a poll() waiting to see EOF from the server, while the 'nbdkit example1' server process is stuck in a read waiting to see if the client will do a...

...ry=4294967295, ms=<optimized out>, ms@entry=0) at record.c:1307 #6 0x00007f30b333b555 in _gnutls_recv_int (session=session@entry=0x55c5cf561620, type=type@entry=GNUTLS_ALERT, data=data@entry=0x0, data_size=data_size@entry=0, seq=seq@entry=0x0, ms=0) at record.c:1773 #7 0x00007f30b333b703 in gnutls_bye (session=session@entry=0x55c5cf561620, how=how@entry=GNUTLS_SHUT_RDWR) at record.c:312 #8 0x000055c5c57af171 in crypto_close () at crypto.c:407 #9 0x000055c5c57aea58 in free_connection (conn=0x55c5cf560500) at connections.c:339 #10 handle_single_connection (sockin=<optimized out>, sockout=&...

...if (actual_size == -1) { fprintf (stderr, "%s\n", nbd_get_error ()); @@ -160,7 +174,7 @@ main (int argc, char *argv[]) /* XXX In future test more operations here. */ -#if !CERTS && !PSK +#if !TLS /* XXX qemu doesn't shut down the connection nicely (using * gnutls_bye) and because of this the following call will fail * with: -- 2.23.0

On Thu, Mar 26, 2020 at 02:34:41PM -0500, Eric Blake wrote: > We're still seeing sporadic failures of 'nbdkit nbd tls=', and I'm > still trying to come up with a root cause fix (it may involve smarter > use of gnutls_bye() in libnbd). In the meantime, here's what we know: > when the hang/failure happens, the 'nbdkit nbd tls=' client process is > stuck in a poll() waiting to see EOF from the server, while the > 'nbdkit example1' server process is stuck in a read waiting to see if > t...

I was a little surprised to find that LIBNBD_TLS_ALLOW worked out of the box, so I had to examine the logs whereupon I saw the magic message ... libnbd: debug: nbd1: nbd_connect_command: server refused TLS (policy), continuing with unencrypted connection I don't believe this path has ever been tested before. It's possible the tests could be improved if they actually checked for this

...@@ -9,12 +9,8 @@ Example code integrating with ppoll, pollfd, APR pollset (and others?). Example command line utils to copy in/out (like qemu-img convert). -NBD_OPT_INFO mode (like qemu-nbd -L). - NBD resize extension. -NBD_INFO_BLOCK_SIZE. - TLS should properly shut down the session (calling gnutls_bye). Performance: Chart it over various buffer sizes and threads, as that @@ -71,12 +67,3 @@ Suggested API improvements: maybe nbd_shutdown should wait for the subprocess or there should be another API to do this - capture error message when nbd_connect_command fails - - list exports -...

...nterop-nbdkit-tls-psk-allow-fallback /interop/interop-nbd-server /interop/interop-qemu-nbd /interop/interop-qemu-nbd-tls-certs diff --git a/TODO b/TODO index 21feb2f..642d39f 100644 --- a/TODO +++ b/TODO @@ -17,9 +17,6 @@ NBD_INFO_BLOCK_SIZE. TLS should properly shut down the session (calling gnutls_bye). -LIBNBD_TLS_ALLOW is not tested. Related to this, -nbd_get_tls_negotiated is not tested. - Implement nbd_connect + systemd socket activation. Improve function trace output so that: diff --git a/interop/Makefile.am b/interop/Makefile.am index 8a5b787..43350a8 100644 --- a/interop/Makefile.a...

...er/crypto.c b/server/crypto.c index 9cd1bb0..6f97f2c 100644 --- a/server/crypto.c +++ b/server/crypto.c @@ -47,6 +47,7 @@ #include <assert.h> #include "internal.h" +#include "utils.h" #ifdef HAVE_GNUTLS @@ -404,9 +405,9 @@ crypto_close (struct connection *conn) gnutls_bye (session, GNUTLS_SHUT_RDWR); if (sockin >= 0) - close (sockin); + close_or_nullify_fd (sockin); if (sockout >= 0 && sockin != sockout) - close (sockout); + close_or_nullify_fd (sockout); gnutls_deinit (session); conn->crypto_session = NULL; diff --git a/...

...bd_internal_run (h, cmd_issue) == -1) + debug (h, "option queued, ignoring state machine failure"); + return 0; +} diff --git a/TODO b/TODO index 4a0cd22..b6c676b 100644 --- a/TODO +++ b/TODO @@ -13,6 +13,17 @@ NBD resize extension. TLS should properly shut down the session (calling gnutls_bye). +Potential deadlock with nbd_add_meta_context: if a client sends enough +requests to the server that it blocks while writing, but the server +replies to requests as they come in rather than waiting for the end of +the client request, then the server can likewise block in writing +replies that li...

...psk +/interop/socket-activation-nbdkit +/interop/socket-activation-qemu-nbd /interop/structured-read /lib/api.c /lib/libnbd.pc diff --git a/TODO b/TODO index 642d39f..92e2c47 100644 --- a/TODO +++ b/TODO @@ -17,8 +17,6 @@ NBD_INFO_BLOCK_SIZE. TLS should properly shut down the session (calling gnutls_bye). -Implement nbd_connect + systemd socket activation. - Improve function trace output so that: - Long strings are truncated. - Strings with non-printable characters are escaped. diff --git a/generator/generator b/generator/generator index 3b63665..d0b4d46 100755 --- a/generator/generator +++...

...| 6 +++++ lib/internal.h | 3 +++ 5 files changed, 43 insertions(+), 4 deletions(-) diff --git a/TODO b/TODO index 642d39f..21feb2f 100644 --- a/TODO +++ b/TODO @@ -17,6 +17,9 @@ NBD_INFO_BLOCK_SIZE. TLS should properly shut down the session (calling gnutls_bye). +LIBNBD_TLS_ALLOW is not tested. Related to this, +nbd_get_tls_negotiated is not tested. + Implement nbd_connect + systemd socket activation. Improve function trace output so that: diff --git a/generator/generator b/generator/generator index 87a8cdf..28248ed 100755 --- a/generator/generato...

Slightly RFC, as I need more time to investigate why Unix sockets appeared to degrade with this patch. But as TCP sockets (over loopback to localhost) and TLS sessions (regardless of underlying Unix or TCP) both showed improvements, this looks like a worthwhile series. Eric Blake (2): server: Add support for corking server: Cork around grouped transmission send()s server/internal.h | 3

...psk +/interop/socket-activation-nbdkit +/interop/socket-activation-qemu-nbd /interop/structured-read /lib/api.c /lib/libnbd.pc diff --git a/TODO b/TODO index 2f23a34..71d678b 100644 --- a/TODO +++ b/TODO @@ -17,8 +17,6 @@ NBD_INFO_BLOCK_SIZE. TLS should properly shut down the session (calling gnutls_bye). -Implement nbd_connect + systemd socket activation. - Improve function trace output so that: - Long strings are truncated. - Strings with non-printable characters are escaped. diff --git a/configure.ac b/configure.ac index 35a4fed..2ebd47c 100644 --- a/configure.ac +++ b/configure.ac @@ -7...

I'm posting this now, as I'm at the end of a workday and I got things working for manual experimentation. Still to do: - write interop tests for qemu-nbd and nbdkit (including my proposed patch addition of qemu-nbd -A to show qemu:allocation-depth) - figure out if we can make 'nbdinfo --map' use the new API to automatically select all contexts advertised by the server Eric Blake