search for: vixdisklib_queryallocatedblocks

This uses a new API VixDiskLib_QueryAllocatedBlocks provided in VDDK >= 6.7. Thanks: Martin Kletzander. --- plugins/vddk/vddk-structs.h | 15 +++- plugins/vddk/vddk.c | 138 ++++++++++++++++++++++++++++++++++++ 2 files changed, 152 insertions(+), 1 deletion(-) diff --git a/plugins/vddk/vddk-structs.h b/plugins/vddk/vddk-structs.h inde...

When using VIXDISKLIB_FLAG_OPEN_SINGLE_LINK, parent images are not taken into account. However, the data we get from VixDiskLib_QueryAllocatedBlocks() has such granularity that it might return incorrect information. For example when the top image has only one block allocated, we get information about the whole chunk being allocated. This results in the consequent read to return invalid data - the rest of the chunk is just zeros. To eliminate...

As I've spent really too long today investigating this, I want to document this in a public email, even though there's nothing really that interesting here. One thing you find from search for VDD 6.7 / VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be one of the very few users out there. And the other thing is that it's quite broken. All testing was done using two baremetal servers connected back to back through a gigabit ethernet switch. I used upstream qemu and nbdkit from git as of today. I used a...

Couple of fixes to return correct data and one nice-to-have clean-up which is not needed. I just find it nicer to read. Martin Kletzander (3): vddk: Use a separate handle for single-link=true vddk: Do not report hole extents to be zero with single-link=true vddk: Eliminate one needless goto plugins/vddk/vddk.c | 48 +++++++++++++++++++++++++++++++++------------ 1 file changed, 36

On Thu, Apr 11, 2019 at 06:55:55PM +0100, Richard W.M. Jones wrote: >As I've spent really too long today investigating this, I want to >document this in a public email, even though there's nothing really >that interesting here. One thing you find from search for VDD 6.7 / >VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be >one of the very few users out there. And the other thing is that it's >quite broken. > >All testing was done using two baremetal servers connected back to >back through a gigabit ethernet switch. I used upstream qemu and >nbdkit from gi...

I believe this roughly implements Nir's proposal here: https://www.redhat.com/archives/libguestfs/2020-August/msg00028.html Unfortunately it doesn't work for me. It actually slows things down quite a lot, for reasons I don't understand. Note the adjustment of the pool-max parameter and how it affects the total time. The results are quite reproducible. $ ./nbdkit -r -U - vddk

...gs) static int vddk_can_extents (void *handle) { - struct vddk_handle *h = handle; + struct handle *h = handle; + GET_VDDK_HANDLE_FOR_CURRENT_SCOPE (h, vddk_handle); VixError err; VixDiskLibBlockList *block_list; @@ -808,7 +920,7 @@ vddk_can_extents (void *handle) DEBUG_CALL ("VixDiskLib_QueryAllocatedBlocks", "handle, 0, %d sectors, %d sectors", VIXDISKLIB_MIN_CHUNK_SIZE, VIXDISKLIB_MIN_CHUNK_SIZE); - err = VixDiskLib_QueryAllocatedBlocks (h->handle, + err = VixDiskLib_QueryAllocatedBlocks (vddk_handle, 0, VIXD...

...ndle) > { > - struct vddk_handle *h = handle; > + struct handle *h = handle; > + GET_VDDK_HANDLE_FOR_CURRENT_SCOPE (h, vddk_handle); > VixError err; > VixDiskLibBlockList *block_list; > > @@ -808,7 +920,7 @@ vddk_can_extents (void *handle) > DEBUG_CALL ("VixDiskLib_QueryAllocatedBlocks", > "handle, 0, %d sectors, %d sectors", > VIXDISKLIB_MIN_CHUNK_SIZE, VIXDISKLIB_MIN_CHUNK_SIZE); > - err = VixDiskLib_QueryAllocatedBlocks (h->handle, > + err = VixDiskLib_QueryAllocatedBlocks (vddk_handle, >...

...;qemu-img convert $nbd out' is therefore much slower than it could be. (In the real world version of this we're actually copying the source over the network, but the command line for that is too complicated to show in a simple example). VDDK has an API for querying allocated blocks, see ‘VixDiskLib_QueryAllocatedBlocks’ in https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf It's awkward to use but Martin (CC'd) has played with it and it does seem to work. The question is if we add Stru...

In can_extents we test whether the VDDK function VixDiskLib_QueryAllocatedBlocks works before declaring that extents are supported. This VDDK function can fail in several ways but most notable if the server doesn't support querying extents. In this case VDDK raises an error message through the error_function callback, which we turn into a call to nbdkit_error. In most ca...

On 3/20/19 5:11 PM, Richard W.M. Jones wrote: > This uses a new API VixDiskLib_QueryAllocatedBlocks provided in > VDDK >= 6.7. > > Thanks: Martin Kletzander. > --- > plugins/vddk/vddk-structs.h | 15 +++- > plugins/vddk/vddk.c | 138 ++++++++++++++++++++++++++++++++++++ > 2 files changed, 152 insertions(+), 1 deletion(-) > > +static int > +vddk_extent...

Not sure what version we're up to, but this reimplements extents using the new simpler structure described in this thread: https://www.redhat.com/archives/libguestfs/2019-March/msg00077.html I also fixed most of the things that Eric pointed out in the previous review, although I need to go back over his replies and check I've got everything. This needs a bit more testing. However the

I'm not sure exactly which version we're up to, but let's say it's version 4. I'm a lot happier with this version: - all filters have been reviewed and changed where I think that's necessary - can_extents is properly defined and implemented now - NBD protocol is followed - I believe it addresses all previous review points where possible The "only" thing

I want to post this but mainly for discussion and early review. It's not safe for these patches to all go upstream yet (because not all filters have been checked/adjusted), but if any patches were to go upstream then probably 1 & 2 only are safe. File, VDDK, memory and data plugins all work, although I have only done minimal testing on them. The current tests, such as they are, all

...t;); > > + VixDiskLib_FreeBlockList (block_list); > > + return -1; > > + } > > + > > + position = offset + length; > > This inner loop might be long; you could add this: > > if (req_one) > break; The issue in practice is that VixDiskLib_QueryAllocatedBlocks is very expensive (because it's a remote procedure call), but we expect calling nbdkit_add_extent should be very cheap. However I'll experiment with breaking earlier from the inner loop. I need to do a lot more testing on this patch anyway. Rich. > > + } > > + DEBUG_CA...

...utput of NBD_OPT_{SET,LIST}_META_CONTEXT. For nbdkit-file-filter: - Fairly simple implementation using SEEK_HOLE/SEEK_DATA. - Not sure how we detect zeroes without reading the file. For nbdkit-memory-plugin: - Pretty simple implementation, which can even detect non-hole zeroes. For VDDK: - VixDiskLib_QueryAllocatedBlocks can return allocated blocks, but doesn't return holes separately (they are assumed from what is omitted from the list). No support for detecting zeroes that I can see. Some existing filters would have to be modified to correctly adjust ‘extents’ offsets: - nbdkit-offset-filter -...

This has already been pushed upstream. I am simply posting these here so we have a reference in the mailing list in case we find bugs later (as I'm sure we will - it's a complex patch series). Great thanks to Eric Blake for tireless review on this one. It also seems to have identified a few minor bugs in qemu along the way. Rich.

...n of allocated zeroes), but where a filter can be added that does read-based zero detection (slower, but potentially reports more zero blocks). > > For nbdkit-memory-plugin: > > - Pretty simple implementation, which can even detect non-hole zeroes. > > For VDDK: > > - VixDiskLib_QueryAllocatedBlocks can return allocated blocks, but > doesn't return holes separately (they are assumed from what is > omitted from the list). No support for detecting zeroes that I can > see. > > Some existing filters would have to be modified to correctly adjust > ‘extents’ offsets:...