search for: nbd_aio_notify_read

...ab the underlying file descriptor using nbd_aio_get_fd, which is what you poll on. You also have to call nbd_aio_get_direction which returns READ, WRITE or BOTH (== READ|WRITE). You then set up some events mechanism (eg. poll, epoll, etc.), poll until the file descriptor is ready, and call one of nbd_aio_notify_read or nbd_aio_notify_write. The direction can change any time the handle state changes, which includes whenever you issue a command (eg. nbd_aio_pread), or whenever you call nbd_aio_notify_*. You therefore have to call nbd_aio_get_direction frequently. A typical loop using poll might look like:...

...d of control can hold the libnbd handle > lock (h->lock), and since there can only be one thread of control > running in each handle at any time, that thread can only be reading or > writing. Indeed, when the writer thread calls nbd_aio_pread(), it is contending on the same lock as the nbd_aio_notify_read() in the reader thread, so we'd have to split up to several finer-grained locks (maybe keep all locking APIs with a grab on h->lock at the beginning, but while holding that lock we then grab the h->rstate or h->wstate lock for the rest of the call, and drop the main h->lock at that...

..."aio_connect", { @@ -1838,7 +1846,7 @@ come from some other means such as C<nbd_aio_connect>. We are expected next to read from the server. If using L<poll(2)> you would set C<events = POLLIN>. If C<revents> returns C<POLLIN> -you would then call C<nbd_aio_notify_read>. +or C<POLLHUP> you would then call C<nbd_aio_notify_read>. Note that once libnbd reaches C<nbd_aio_is_ready>, this direction is returned even before a command is issued via C<nbd_aio_pwrite> and diff --git a/lib/poll.c b/lib/poll.c index 982b172..d356afe 100644 --- a...

...generator index a289741..1d4db23 100755 --- a/generator/generator +++ b/generator/generator @@ -1719,6 +1719,18 @@ We are expected next to read from the server. If using L<poll(2)> you would set C<events = POLLIN>. If C<revents> returns C<POLLIN> you would then call C<nbd_aio_notify_read>. +Note that once libnbd reaches C<nbd_aio_is_ready>, this direction is +returned even before a command is issued via C<nbd_aio_pwrite> and +friends. In a single-threaded use of libnbd, it is not worth polling +until after issuing a command, as otherwise the server will never wake +...

Rich recently patched things to generate one man page per function rather than libnbd-api.3 (nice), but in doing so got stumped by a problem with a fresh git clone (automake fails for any 'include' directive that does not already exist). I've figured out how to hack around it, but the hack requires GNU make. We already use GNU make constructs elsewhere (such as $(wildcard)), but

While working on a new test of what happens when the server goes away while commands are in flight, I managed to hit a race where I hit death from SIGPIPE instead of a clean transition to the DEAD state. I also found myself wanting to use nbd_poll from the test, but with a way to distinguish between the state machine progressing vs. hanging. Eric Blake (2): socket: Avoid SIGPIPE where possible

...t where the libnbd lock is held; as > > -such, it is unsafe for the callback to call any C<nbd_*> APIs on the > > -same nbd object, as it would cause deadlock. > > +The callbacks are invoked at a point where the libnbd lock is held, > > +typically during a call to C<nbd_aio_notify_read>, > > +C<nbd_aio_notify_write>, C<nbd_aio_poll>, or other call that can > > +advance libnbd's state machine. Depending on system load, it is even > > +possible for a callback to reached before completion of the > > > Shouldn't it be "to be r...

(Sorry for the late reply, was a bit involved in a qemu bug last week ...) On Fri, Jul 07, 2023 at 08:58:50AM +0000, Tage Johansson wrote: > On 7/6/2023 7:06 PM, Nir Soffer wrote: > > > - After calling for example aio_notify_read(3), can I know that the next > reading from the file descriptor would block? > > No, you have to call again aio_get_direction() and

...re invoked at a point where the libnbd lock is held; as > -such, it is unsafe for the callback to call any C<nbd_*> APIs on the > -same nbd object, as it would cause deadlock. > +The callbacks are invoked at a point where the libnbd lock is held, > +typically during a call to C<nbd_aio_notify_read>, > +C<nbd_aio_notify_write>, C<nbd_aio_poll>, or other call that can > +advance libnbd's state machine. Depending on system load, it is even > +possible for a callback to reached before completion of the Shouldn't it be "to be reached" instead of "...

...if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0) - fds[0].events |= POLLOUT; - - if (poll (fds, 1, -1) == -1) { - perror ("poll"); - goto error; - } - - if ((dir & LIBNBD_AIO_DIRECTION_READ) != 0 && - (fds[0].revents & POLLIN) != 0) - nbd_aio_notify_read (nbd); - else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0 && - (fds[0].revents & POLLOUT) != 0) - nbd_aio_notify_write (nbd); - /* If we can issue another request, do so. */ - if (want_to_send && (fds[0].revents & POLLOUT) != 0 && -...

...ion). When a command is submitted from an nbdkit-managed thread a byte is sent down this pipe. * The reader thread can either wake up because the socket is ready for read or write, or because of an indication on the pipe-to-self. * If the socket is ready for read or write then the normal nbd_aio_notify_read|write function is called which will move the state machine along, and also check for command completion. * If it's pipe-to-self indication then we will (after checking for command completion) check the direction flag again and reenter the poll. The reason for this is because when the ot...

...lts in: nbd .close nbd reader server nbd_shutdown poll - send NBD_CMD_DISC receive NBD_CMD_DISC shutdown(SHUT_WR) wake up on EOF nbd_aio_notify_read - move state to CLOSED nbd_aio_is_dead -> true break loop - poll pthread_join close fds nbd_close see EOF from client but sometimes things happen differently: nbd .close...

...ot;,\n"; > - pr " ev, nbd_internal_state_short_string (get_state (h)));\n"; > + pr " ev, nbd_internal_state_short_string (get_next_state (h)));\n"; I'm also wondering if we should treat external events more like interrupts, where calling nbd_aio_notify_read() in an incorrect state succeeds by setting a flag that is not cleared until we later reach a state that can actually care about the flag, rather than by rejecting the call out-right for being in the wrong state. (That is, states where we are NOT expecting a notify_read() are the same as setting a...

...ng -The callbacks are invoked at a point where the libnbd lock is held; as -such, it is unsafe for the callback to call any C<nbd_*> APIs on the -same nbd object, as it would cause deadlock. +The callbacks are invoked at a point where the libnbd lock is held, +typically during a call to C<nbd_aio_notify_read>, +C<nbd_aio_notify_write>, C<nbd_aio_poll>, or other call that can +advance libnbd's state machine. Depending on system load, it is even +possible for a callback to reached before completion of the +C<nbd_aio_*> call that specified the callback. As such, it is unsafe +fo...

...o_connect_unix \ - nbd_aio_connect_tcp \ - nbd_aio_connect_command \ - nbd_aio_pread \ - nbd_aio_pread_structured \ - nbd_aio_pwrite \ - nbd_aio_disconnect \ - nbd_aio_flush \ - nbd_aio_trim \ - nbd_aio_cache \ - nbd_aio_zero \ - nbd_aio_block_status \ - nbd_aio_get_fd \ - nbd_aio_get_direction \ - nbd_aio_notify_read \ - nbd_aio_notify_write \ - nbd_aio_is_created \ - nbd_aio_is_connecting \ - nbd_aio_is_ready \ - nbd_aio_is_processing \ - nbd_aio_is_dead \ - nbd_aio_is_closed \ - nbd_aio_command_completed \ - nbd_aio_peek_command_completed \ - nbd_aio_in_flight \ - nbd_connection_state \ - nbd_get_package_name...

This is working now, and incorporates all of the changes in Eric's review, *except* that it still doesn't retire commands (although this seems to make no obvious difference, except possibly a performance and memory impact). Rich.

...o_connect_unix \ + nbd_aio_connect_tcp \ + nbd_aio_connect_command \ + nbd_aio_pread \ + nbd_aio_pread_structured \ + nbd_aio_pwrite \ + nbd_aio_disconnect \ + nbd_aio_flush \ + nbd_aio_trim \ + nbd_aio_cache \ + nbd_aio_zero \ + nbd_aio_block_status \ + nbd_aio_get_fd \ + nbd_aio_get_direction \ + nbd_aio_notify_read \ + nbd_aio_notify_write \ + nbd_aio_is_created \ + nbd_aio_is_connecting \ + nbd_aio_is_ready \ + nbd_aio_is_processing \ + nbd_aio_is_dead \ + nbd_aio_is_closed \ + nbd_aio_command_completed \ + nbd_aio_peek_command_completed \ + nbd_aio_in_flight \ + nbd_connection_state \ + nbd_get_package_name...

...es libnbd need to support more complex thread models or can we rely on callers to use their own mutex on top? More comments below ... On Wed, Jun 05, 2019 at 10:23:57AM -0500, Eric Blake wrote: > I'm also wondering if we should treat external events more like > interrupts, where calling nbd_aio_notify_read() in an incorrect state > succeeds by setting a flag that is not cleared until we later reach a > state that can actually care about the flag, rather than by rejecting > the call out-right for being in the wrong state. (That is, states where > we are NOT expecting a notify_read() are t...

...if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0) + fds[0].events |= POLLOUT; + + if (poll (fds, 1, -1) == -1) { + perror ("poll"); + goto error; + } + + if ((dir & LIBNBD_AIO_DIRECTION_READ) != 0 && + (fds[0].revents & POLLIN) != 0) + nbd_aio_notify_read (conn); + else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0 && + (fds[0].revents & POLLOUT) != 0) + nbd_aio_notify_write (conn); + + /* If a command is ready to retire, retire it. */ + for (j = 0; j < in_flight; ++j) { + r = nbd_aio_command_completed...

...if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0) + fds[0].events |= POLLOUT; + + if (poll (fds, 1, -1) == -1) { + perror ("poll"); + goto error; + } + + if ((dir & LIBNBD_AIO_DIRECTION_READ) != 0 && + (fds[0].revents & POLLIN) != 0) + nbd_aio_notify_read (conn); + else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) != 0 && + (fds[0].revents & POLLOUT) != 0) + nbd_aio_notify_write (conn); + + /* If a command is ready to retire, retire it. */ + while ((done = nbd_aio_peek_command_completed (conn)) >= 0) { + f...