Libguestfs - Mar 2019 - Support for NBD block status in nbdkit & qemu NBD client

Hi Eric:

Here's the problem: We have a VMware VDDK file or snapshot.  The
virtual size of the snapshot is generally much larger than what's
allocated.  Copying the snapshot using:

  nbdkit vddk file=foo.vmdk --run 'qemu-img convert $nbd out'

is therefore much slower than it could be.  (In the real world version
of this we're actually copying the source over the network, but the
command line for that is too complicated to show in a simple example).

VDDK has an API for querying allocated blocks, see
‘VixDiskLib_QueryAllocatedBlocks’ in
https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf

It's awkward to use but Martin (CC'd) has played with it and it does
seem to work.

The question is if we add Structured Replies + Block Status support to
nbdkit & implement it for the VDDK plugin, will ‘qemu-img convert’
above be able to use that to optimize the copy?  My brief reading of
the qemu sources suggests that something has been implemented here so
it likely ought to work?

I guess the other question is if there are any limitations around
alignment that we should be aware of.  It seems as if VDDK uses a
larger cluster size for sparseness than its sector size (as does
qcow2).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

On 3/7/19 9:19 AM, Richard W.M. Jones wrote:
> Hi Eric:
> 
> Here's the problem: We have a VMware VDDK file or snapshot.  The
> virtual size of the snapshot is generally much larger than what's
> allocated.  Copying the snapshot using:
> 
>   nbdkit vddk file=foo.vmdk --run 'qemu-img convert $nbd out'
> 
> is therefore much slower than it could be.  (In the real world version
> of this we're actually copying the source over the network, but the
> command line for that is too complicated to show in a simple example).
> 
> VDDK has an API for querying allocated blocks, see
> ‘VixDiskLib_QueryAllocatedBlocks’ in
>
https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf
> 
> It's awkward to use but Martin (CC'd) has played with it and it
does
> seem to work.
> 
> The question is if we add Structured Replies + Block Status support to
> nbdkit & implement it for the VDDK plugin, will ‘qemu-img convert’
> above be able to use that to optimize the copy?  My brief reading of
> the qemu sources suggests that something has been implemented here so
> it likely ought to work?
Yes, my gut feel is that qemu-img IS able to take advantage of
NBD_CMD_BLOCK_STATUS to avoid reading sparse portions of the source file
during qemu-img convert, so it IS worth figuring out how to get
structured replies and block status wired into nbdkit. It's on my list
of fun projects to attempt (but trailing behind other stuff like libvirt
incremental snapshots at the moment).
> 
> I guess the other question is if there are any limitations around
> alignment that we should be aware of.  It seems as if VDDK uses a
> larger cluster size for sparseness than its sector size (as does
> qcow2).
In general, block status will naturally report holes aligned to
granularity, but it really shouldn't matter to correctness (reporting
holes is merely an optimization, after all).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org