Libguestfs - Jun 2021 - nbd_aio_p{read,write} responsibilities

On Fri, Jun 18, 2021 at 03:38:16PM +0530, Abhay Raj Singh
wrote:
> I was going through the code and realized io_uring is similar to
> libev in the way that we listen to the io events ourselves but the
> difference is when we encounter the event we read it
> ourselves(io_uring), whereas in the case of libev we ask libnbd to
> read it for us. We notify that the connection is readable using
> aio_notify_read and the command is ran by the state machine the
> command was queued by an earlier call to nbd_aio_pread.
The current "nbd_aio_*" API isn't really designed around
completion
events.  You're thinking about the right thing: what would an nbd_*
API look like that could be implemented using io_uring?

I find it useful to write some example programs using a proposed new
API (obviously not programs that can be compiled) to explore what API
could work.
> My question is if we read from the socket using (let's say read[1])
> what do we need to do to inform the state machine we have done so,
> so that it can function properly as I am assuming nbd_aio_pread
> modifies the state in some way on enqueue and completion
The answer will be in the code :-)  I don't remember all the exact
details of how it works now.  The generated files are probably the
best place to start reading.
> [1]: io_uring has read but not recv in kernel 5.5
The main technical reason we currently use send/recv instead of
write/read is so we can pass the flags MSG_NOSIGNAL and MSG_MORE.
MSG_MORE is an optimization so we don't have to use it.  MSG_NOSIGNAL
is necessary, but at a first pass we could ignore it until we work out
what to do with it.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

Another way to proceed here could be to write a from-scratch version
of nbdcopy using io_uring, to get a feel for the "shape" of what that
would look like.  Then translate back from that into an NBD API
proposal.

For simplicity in implementation:

 - Only implement NBD oldstyle protocol, which is very simple.  You
   can find it documented here:
  
https://gitlab.com/nbdkit/nbdkit/-/blob/dd894ae009c16810605252ee6996d329c4a13327/common/protocol/nbd-protocol.h#L51
   nbdkit supports oldstyle if you add the -o option.

 - Only do NBD server -> NBD server copies, forget about local files,
   pipes, etc.

 - Don't implement any of the options, synchronous mode, etc.  Just
   hard-code all the settings.

I'm interested in seeing what this would look like.  Also it might
answer a few questions I have:

 - Do we need to use threads in nbdcopy if we have io_uring?

 - How many io_urings should we use?

 - Can we avoid kernel-userspace copies?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/