similar to: [PATCH v2v] v2v: -it vddk: Don't use nbdkit readahead filter with VDDK (RHBZ#1832805).

I'm still testing this fix, so let's hold off the review for the moment. Also it may be better to specifically identify problematic servers rather than disabling this for every curl source. eg. I suspect that the problem is the Java server used by VCenter, so we might think about only disabling readahead for that single case. Rich.

This filter deliberately tries to coalesce reads into larger requests. Unfortunately VMware has low limits on the size of requests it can serve to a VDDK client and the larger requests would break with errors like this: nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes We already increase the maximum

Patch 1 was previously posted here: https://www.redhat.com/archives/libguestfs/2020-June/msg00086.html to handle this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1848862 I was able to observe this bug and for me at least disabling readahead seems to cure it. Patches 2 and 3 are simplifications, removing a now-undocumented feature of virt-v2v-copy-to-local and thus simplifying greatly the

As I've spent really too long today investigating this, I want to document this in a public email, even though there's nothing really that interesting here. One thing you find from search for VDD 6.7 / VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be one of the very few users out there. And the other thing is that it's quite broken. All testing was done using

On 6/19/20 7:47 AM, Richard W.M. Jones wrote: > This appears to be the cause of timeouts during the conversion step > where VMware VCenter server's Tomcat HTTPS server stops responding to > requests (or rather, responds only with 503 errors). The server later > recovers and in fact because of the retry filter the conversion > usually succeeds, but I found that we can avoid the

This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++----

Part 2 fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1842440 Actually this fix on its own should be sufficient, but probably we want the nbdkit fixes too. Note this uses actual OCaml 4.05 features! ("let open" and the Unix.tcgetattr functions). I checked that both features are available on RHEL 7's OCaml. Rich.

On Thu, Apr 11, 2019 at 06:55:55PM +0100, Richard W.M. Jones wrote: >As I've spent really too long today investigating this, I want to >document this in a public email, even though there's nothing really >that interesting here. One thing you find from search for VDD 6.7 / >VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be >one of the very few users out

Note this fix is incomplete / not ready for upstream: (1) We haven't definitively proven this is the issue. Mor is still investigating and producing the proof. (2) We will need to make a similar change to virt-v2v documentation. Rich.

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu

v3 posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00200.html v4: - The first patch in the v3 series was just a trivial doc whitespace fix so I pushed it. - There's a new patch using the nbdkit-retry-filter. This is not actually upstream in nbdkit but we know enough about how it will work. - Rebased against master and reran the tests. Rich.

On Fri, Apr 12, 2019 at 03:52:58PM +0200, Martin Kletzander wrote: > On Thu, Apr 11, 2019 at 06:55:55PM +0100, Richard W.M. Jones wrote: > >This seems to be a bug in VDDK, possibly connected with the fact that > >we fork after initializing VDDK but before doing the > >VixDiskLib_ConnectEx. I suspect it's something to do with the PID > >changing. > > >

v2 was posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00115.html This also has links to earlier versions. v3: - The 01/11 patch in v2 included a bunch of unnecessary plus one necessary change to how input_password is passed around. I moved the necessary change into the final patch (implementing SSH password authentication) and dropped the rest. - The 01/11

On Mon, Apr 15, 2019 at 04:30:47PM +0200, Martin Kletzander wrote: > On Fri, Apr 12, 2019 at 04:30:02PM +0100, Richard W.M. Jones wrote: > > +-- nbdkit monitoring process > > | > > +-- first child = nbdkit > > | > > +-- second child = ‘--run’ command > > > >so when the second child exits, the monitoring process (which is doing > >nothing

Originally posted here: https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00054 https://www.redhat.com/archives/libguestfs/2019-April/msg00076.html https://www.redhat.com/archives/libguestfs/2019-April/msg00126.html This is a rebase on top of current master branch with no other changes. The first patch in the old series was pushed a while back, and the last "TEMPORARY"

I'm a lot happier with this version: no mucking with dlmopen(). It does add a bit more coupling between nbdkit proper and the vddk plugin (namely, nbdkit now exports a new function that the vddk plugin relies on), but patch 2 adds testsuite coverage of the new function so we shouldn't regress. Patch 1 and 2 are new, patch 3 is unchanged from when Rich posted it in v2, and patch 4 is

It does not need configuring and speeds up the process of requesting extents if the client asks for them one by one (like qemu-img). Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- v2v/input_libvirt_vddk.ml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/v2v/input_libvirt_vddk.ml b/v2v/input_libvirt_vddk.ml index a8e4fd5ac78e..cc63f77ce825 100644 ---

On Fri, Sep 20, 2019 at 10:28:18AM +0100, Richard W.M. Jones wrote: >The readahead filter is a self-configuring filter that makes >sequential reads faster when the plugin is slow (and all of the >plugins we use here are always slow). > >I observed the behaviour of the readahead filter with our qcow2 >overlay when converting a guest from a vCenter source. Even when >doing

On Fri, Apr 12, 2019 at 04:30:02PM +0100, Richard W.M. Jones wrote: >On Fri, Apr 12, 2019 at 03:52:58PM +0200, Martin Kletzander wrote: >> On Thu, Apr 11, 2019 at 06:55:55PM +0100, Richard W.M. Jones wrote: >> >This seems to be a bug in VDDK, possibly connected with the fact that >> >we fork after initializing VDDK but before doing the >> >VixDiskLib_ConnectEx.

Differences from v4: Patch 1 is simplified: I realized that since we already use -rdynamic for nbdkit (after all, we WANT our dlopen()d plugins to be able to call our nbdkit_* exports), it is only a matter of adding dlopen to the set of symbols that we export. With that done, there is no separate shared library needed; our dlopen shim is now part of nbdkit proper, and we don't have to tweak