Libguestfs - Nov 2018 - Re: nbdkit: Could not read L1 table when reading exported qcow2

On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones
wrote:
> nbdkit: file.2: error: invalid request: offset and count are out of range:
offset=196608 count=512
Actually what happens even more precisely is that the underlying file
is not a multiple of 512 bytes (196624 ≡ 16 mod 512).

qemu-img (the client) issues a request for the final "sector" of the
file which goes beyond the end of the file.

You can "fix" this by using the truncate filter
(https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-truncate-filter.pod)
telling it to round up the size to the next multiple of 512 bytes:

$ nbdkit --filter=truncate file file=disk -f -v round-up=512

and this allows qemu-img to work:

$ qemu-img info nbd://localhost
image: nbd://localhost:10809
file format: qcow2
virtual size: 1.0G (1073741824 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

However the fundamental problem remains that you're exporting qcow2
bytes over the NBD connection which is not really what you wanted to
do.

You should use qemu-nbd instead since it understands qcow2 natively,
or use raw format disks if you want to use nbdkit for its other
features.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
> On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
>> nbdkit: file.2: error: invalid request: offset and count are out of
range: offset=196608 count=512
> 
> Actually what happens even more precisely is that the underlying file
> is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
> 
> qemu-img (the client) issues a request for the final "sector" of
the
> file which goes beyond the end of the file.
I'm still hoping to fix that bug in qemu before the 3.1 release, but 
that date is getting close :)
> 
> You can "fix" this by using the truncate filter
>
(https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-truncate-filter.pod)
> telling it to round up the size to the next multiple of 512 bytes:
> 
> $ nbdkit --filter=truncate file file=disk -f -v round-up=512
> 
> and this allows qemu-img to work:
Yes, this is the right workaround for now.
> 
> $ qemu-img info nbd://localhost
> image: nbd://localhost:10809
> file format: qcow2
Invoking qemu-img without '-f $format' means that qemu-img has to probe 
the image to guess its format. Here, the probe succeeded, but in general 
this is insecure (if the file was 'raw', but the probe returned anything
else, then a malicious guest could have tried to write metadata into the 
initial sector of their disk to try and get qemu to perform something on 
the guest's behalf due to the incorrect probe).  A more robust solution 
pairs either:

qemu-nbd -f raw file.qcow2   <=>   qemu-img info -f qcow2 nbd://localhost
nbdkit file file.qcow2   <=>   qemu-img info -f qcow2 nbd://localhost

where NBD is exposing the qcow2 metadata over the network (whether via 
qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or:

qemu-nbd -f qcow2 file.qcow2   <=>   qemu-img info -f raw nbd://localhost
qemu-nbd -f qcow2 file.qcow2   <=>   any other nbd client

where qemu-nbd is interpreting the qcow2 file, and only exposing the 
guest-visible data over the network.

Or, repeating the above in a different formulation - when you are 
interested in serving qcow2 data over the network, you HAVE to have 
exactly one of your two endpoints be aware that the qcow2 format is 
involved - either the server is agnostic and the client interprets qcow2 
(the former setup), or the server interprets qcow2 and the client is 
agnostic (the latter setup).

Then, as Rich pointed out, when you expose qcow2 over NBD (the former 
setup), the current NBD protocol does NOT allow resizes - so while 
reading the image works okay, writing to the image is likely to hit 
ENOSPC unless you pre-allocated the .qcow2 file being served.  That is 
likely to change upstream in the next six months (it's on my todo list 
to get NBD resize implemented), but there's still a percolation delay 
from upstream to using it in default OS installs.  On the other hand, 
when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw 
on the server end, then the server can resize things as needed and the 
client need not be aware that qcow2 is even involved.

As for why nbdkit doesn't have a plugin for exposing a local qcow2 file 
as raw - we don't see the point in duplicating what qemu-nbd can already 
do. Yes, someone could write such a plugin, but it will probably have 
worse performance than what you can already get out of using qemu-nbd as 
the server.
> However the fundamental problem remains that you're exporting qcow2
> bytes over the NBD connection which is not really what you wanted to
> do.
Maybe it is, if the qcow2 file is fully preallocated, but yeah, until 
NBD resize exists, most people don't want that combination.
> 
> You should use qemu-nbd instead since it understands qcow2 natively,
> or use raw format disks if you want to use nbdkit for its other
> features.
> 
> Rich.
> 
-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

On Wed, Nov 21, 2018 at 10:33 PM Eric Blake <eblake@redhat.com> wrote:
> On 11/21/18 4:39 AM, Richard W.M. Jones wrote:
> > On Wed, Nov 21, 2018 at 09:25:05AM +0000, Richard W.M. Jones wrote:
> >> nbdkit: file.2: error: invalid request: offset and count are out
of
> range: offset=196608 count=512
> >
> > Actually what happens even more precisely is that the underlying file
> > is not a multiple of 512 bytes (196624 ≡ 16 mod 512).
> >
> > qemu-img (the client) issues a request for the final
"sector" of the
> > file which goes beyond the end of the file.
>
> I'm still hoping to fix that bug in qemu before the 3.1 release, but
> that date is getting close :)
>
> >
> > You can "fix" this by using the truncate filter
> > (
>
https://github.com/libguestfs/nbdkit/blob/master/filters/truncate/nbdkit-truncate-filter.pod
> )
> > telling it to round up the size to the next multiple of 512 bytes:
> >
> > $ nbdkit --filter=truncate file file=disk -f -v round-up=512
> >
> > and this allows qemu-img to work:
>
> Yes, this is the right workaround for now.
>
> >
> > $ qemu-img info nbd://localhost
> > image: nbd://localhost:10809
> > file format: qcow2
>
> Invoking qemu-img without '-f $format' means that qemu-img has to
probe
> the image to guess its format. Here, the probe succeeded, but in general
> this is insecure (if the file was 'raw', but the probe returned
anything
> else, then a malicious guest could have tried to write metadata into the
> initial sector of their disk to try and get qemu to perform something on
> the guest's behalf due to the incorrect probe).  A more robust solution
> pairs either:
>
> qemu-nbd -f raw file.qcow2   <=>   qemu-img info -f qcow2
nbd://localhost
> nbdkit file file.qcow2   <=>   qemu-img info -f qcow2 nbd://localhost
>
> where NBD is exposing the qcow2 metadata over the network (whether via
> qemu-nbd or nbdkit), and the client HAS to be aware of qcow2, or:
>
> qemu-nbd -f qcow2 file.qcow2   <=>   qemu-img info -f raw
nbd://localhost
> qemu-nbd -f qcow2 file.qcow2   <=>   any other nbd client
>
> where qemu-nbd is interpreting the qcow2 file, and only exposing the
> guest-visible data over the network.
>
> Or, repeating the above in a different formulation - when you are
> interested in serving qcow2 data over the network, you HAVE to have
> exactly one of your two endpoints be aware that the qcow2 format is
> involved - either the server is agnostic and the client interprets qcow2
> (the former setup), or the server interprets qcow2 and the client is
> agnostic (the latter setup).
>
> Then, as Rich pointed out, when you expose qcow2 over NBD (the former
> setup), the current NBD protocol does NOT allow resizes - so while
> reading the image works okay, writing to the image is likely to hit
> ENOSPC unless you pre-allocated the .qcow2 file being served.  That is
> likely to change upstream in the next six months (it's on my todo list
> to get NBD resize implemented), but there's still a percolation delay
> from upstream to using it in default OS installs.  On the other hand,
> when you expose raw over NBD by using qemu-nbd to convert qcow2 to raw
> on the server end, then the server can resize things as needed and the
> client need not be aware that qcow2 is even involved.
>
> After nbd resize implemented, we can get actual disk size instread of
'unavailable', right?
Will it support disk extend, or just support disk shrinking?
> As for why nbdkit doesn't have a plugin for exposing a local qcow2 file
> as raw - we don't see the point in duplicating what qemu-nbd can
already
> do. Yes, someone could write such a plugin, but it will probably have
> worse performance than what you can already get out of using qemu-nbd as
> the server.
>
> > However the fundamental problem remains that you're exporting
qcow2
> > bytes over the NBD connection which is not really what you wanted to
> > do.
>
> Maybe it is, if the qcow2 file is fully preallocated, but yeah, until
> NBD resize exists, most people don't want that combination.
>
> >
> > You should use qemu-nbd instead since it understands qcow2 natively,
> > or use raw format disks if you want to use nbdkit for its other
> > features.
> >
> > Rich.
> >
>
> --
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
>

-- 
Best regards,
-----------------------------------
Han Han
Quality Engineer
Redhat.

Email: hhan@redhat.com
Phone: +861065339333