Libguestfs - Jun 2021 - nbdcpy: from scratch nbdcopy using io_uring

In a previous email, Mr. Jones suggested, writing a test implementation of
nbdcopy from scratch. First of all, thank you, Mr. Jones, as I got a better
perspective of the problem and the solution.

I have almost written the bare-bones implementation for handling
NBD-related stuff on which I will base the io_uring core piece by piece. It
is written in C++, as I am more comfortable with it. But, made it easily
translatable to C.

I have been working on a solution and have documented the current approach,
it would be great if you have a look at it and let me know what you think.

Approach document:
https://gitlab.com/rathod-sahaab/nbdcpy/-/blob/dev/docs/SOLUTION.md

Thanks and regards,
Abhay
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20210624/b54d6cca/attachment.htm>

On Thu, Jun 24, 2021 at 09:30:00PM +0530, Abhay Raj Singh
wrote:
> 
> In a previous email, Mr. Jones suggested, writing a test implementation of
> nbdcopy from scratch. First of all, thank you, Mr. Jones, as I got a better
> perspective of the problem and the solution.
> 
> I have almost written the bare-bones implementation for handling
NBD-related
> stuff on which I will base the io_uring core piece by piece. It is written
in
> C++, as I am more comfortable with it. But, made it easily translatable to
C.
> 
> I have been working on a solution and have documented the current approach,
it
> would be great if you have a look at it and let me know what you think.
> 
> Approach document:
> https://gitlab.com/rathod-sahaab/nbdcpy/-/blob/dev/docs/SOLUTION.md
So the document is fine.  Since this is throw-away code I wouldn't
worry about handling errors except to abort the program.

You can issue the read requests out of order, although perhaps it
isn't helpful.  And you're correct that you should try to limit the
number of requests in flight (otherwise you'll run out of memory when
you try to copy very large disks).  However 6 is probably too low - in
nbdcopy we use 64.

NBD supports multi-conn (well, technically speaking you should check a
flag on the server but that doesn't matter for testing).  So you can
open multiple TCP connections on each side, and issue multiple
commands in flight on each of those connections.  We found this
improves performance a lot.

The code looks very minimal at the moment.  I'm not very familiar with
the fmt:: class.  Does it successfully make a handshake to ?nbdkit -o?
yet?

I notice that although you create an io_uring you're not using it.
But I guess that's still to come.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW