Libguestfs - Mar 2019 - Re: nbdkit & qemu 2.12: qemu-img: Protocol error: simple reply when structured reply chunk was expected

On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake
wrote:
> On 3/23/19 6:42 AM, Richard W.M. Jones wrote:
> > (b) Try and get it fixed in RHEL.  I filed a BZ already but I guess it
> > won't be fixed any time soon: https://bugzilla.redhat.com/1692018
> 
> Several RHEL 7 bugs against NBD are pending - there's hope for RHEL
7.7,
> but you are also right that guessing timelines is hard.
Yes, my worry is that if we miss 7.7 then this bug will exist forever
since as far as I know we (Red Hat) are not planning any further qemu
improvements in 7.8 and above.
> > (c) Add a workaround in nbdkit.
> > 
> 
> And what workaround would that be? Looking at the qemu patch, the
> problem is that qemu asked for "base:allocation" and nbdkit
replied with
> 0 contexts. The workaround would either be to reply with
> NBD_REP_ERR_UNSUP (the behavior before NBD_OPT_SET_META_CONTEXT was
> added), or to implement "base:allocation" (even if we implement
the
> poor-man's version that always reports that the entire image is data,
> for all requests, without any feedback from the plugins).  The former
> feels odd, but the latter seems oddly appealing as progress towards our
> end goal (kind of how our initial implementation of NBD_CMD_WRITE_ZEROES
> was always advertised even before we finished wiring up plug support).
A good point here is what happens with the block-status branch[1].
Let's see:

$ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img convert $nbd
/var/tmp/out'
qemu-img: Payload too large
nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe

Oh dear.  I believe this is actually a bug in the block status code so
let's see if I can narrow this down first ...

Rich.

[1] https://github.com/rwmjones/nbdkit/tree/block-status

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On Sat, Mar 23, 2019 at 12:21:11PM +0000, Richard W.M. Jones
wrote:
> On Sat, Mar 23, 2019 at 06:57:14AM -0500, Eric Blake wrote:
> > And what workaround would that be? Looking at the qemu patch, the
> > problem is that qemu asked for "base:allocation" and nbdkit
replied with
> > 0 contexts. The workaround would either be to reply with
> > NBD_REP_ERR_UNSUP (the behavior before NBD_OPT_SET_META_CONTEXT was
> > added), or to implement "base:allocation" (even if we
implement the
> > poor-man's version that always reports that the entire image is
data,
> > for all requests, without any feedback from the plugins).  The former
> > feels odd, but the latter seems oddly appealing as progress towards
our
> > end goal (kind of how our initial implementation of
NBD_CMD_WRITE_ZEROES
> > was always advertised even before we finished wiring up plug support).
> 
> A good point here is what happens with the block-status branch[1].
> Let's see:
> 
> $ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img convert
$nbd /var/tmp/out'
> qemu-img: Payload too large
> nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe
> 
> Oh dear.  I believe this is actually a bug in the block status code so
> let's see if I can narrow this down first ...
OK I fixed this bug (in nbdkit).

The block-status branch *works* with qemu 2.12.0, so another option
here is to do nothing and wait until that work is finished and merged.

NB During this I discovered another (but minor) bug in qemu.  If you
feed qemu a long series of block status replies then it eventually
closes the connection.  However it does not print an error message.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On 3/23/19 7:58 AM, Richard W.M. Jones wrote:
>> A good point here is what happens with the block-status branch[1].
>> Let's see:
>>
>> $ ./nbdkit memory size=64M --run '/home/rjones/d/qemu/qemu-img
convert $nbd /var/tmp/out'
>> qemu-img: Payload too large
>> nbdkit: memory.1: error: write reply: NBD_CMD_BLOCK_STATUS: Broken pipe
>>
>> Oh dear.  I believe this is actually a bug in the block status code so
>> let's see if I can narrow this down first ...
> 
> OK I fixed this bug (in nbdkit).
Which is why the block-status branch is still in flux :)
> 
> The block-status branch *works* with qemu 2.12.0, so another option
> here is to do nothing and wait until that work is finished and merged.
Well, it works as long as you don't send a simple error reply (our mails
crossed, and I submitted that patch for that issue already).
> 
> NB During this I discovered another (but minor) bug in qemu.  If you
> feed qemu a long series of block status replies then it eventually
> closes the connection.  However it does not print an error message.
> 
Which version(s) of qemu? Was REQ_ONE in force or not? Was it a long
series of status replies that could have been coalesced, or were they
alternating status? Did the replies extend beyond the client's original
request?

I'd like to reproduce the failure, to determine if qemu needs more patches.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org