similar to: [PATCH v2v 0/2] v2v: nbdkit: Don't use password=- parameter.

v3 posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00200.html v4: - The first patch in the v3 series was just a trivial doc whitespace fix so I pushed it. - There's a new patch using the nbdkit-retry-filter. This is not actually upstream in nbdkit but we know enough about how it will work. - Rebased against master and reran the tests. Rich.

Patch 1 was previously posted here: https://www.redhat.com/archives/libguestfs/2020-June/msg00086.html to handle this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1848862 I was able to observe this bug and for me at least disabling readahead seems to cure it. Patches 2 and 3 are simplifications, removing a now-undocumented feature of virt-v2v-copy-to-local and thus simplifying greatly the

v2 was posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00115.html This also has links to earlier versions. v3: - The 01/11 patch in v2 included a bunch of unnecessary plus one necessary change to how input_password is passed around. I moved the necessary change into the final patch (implementing SSH password authentication) and dropped the rest. - The 01/11

Originally posted here: https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00054 https://www.redhat.com/archives/libguestfs/2019-April/msg00076.html https://www.redhat.com/archives/libguestfs/2019-April/msg00126.html This is a rebase on top of current master branch with no other changes. The first patch in the old series was pushed a while back, and the last "TEMPORARY"

This is built on top of the following patch series: https://www.redhat.com/archives/libguestfs/2019-April/msg00054.html Rich.

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu

Trivial fix. We really need a regression test for all v2v inpus related to nbdkit. There is actually nothing at all at the moment. Of course if it was easy to test inputs over the network from a ‘make check’ rule then we'd be doing it already. I thought about adding something like a ‘-it file’ option which would use nbdkit-file-plugin for test only. However I'm rather cautious about

This is the simplest solution to this problem. There are two other possible fixes I considered: Increase the documented limit (see http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-memory-limits). However at the moment we know the current limit works through extensive testing (without readahead), plus I have no idea nor any way to test if larger limits are supported by

I'm still testing this fix, so let's hold off the review for the moment. Also it may be better to specifically identify problematic servers rather than disabling this for every curl source. eg. I suspect that the problem is the Java server used by VCenter, so we might think about only disabling readahead for that single case. Rich.

On Fri, Sep 20, 2019 at 10:28:18AM +0100, Richard W.M. Jones wrote: >The readahead filter is a self-configuring filter that makes >sequential reads faster when the plugin is slow (and all of the >plugins we use here are always slow). > >I observed the behaviour of the readahead filter with our qcow2 >overlay when converting a guest from a vCenter source. Even when >doing

It does not need configuring and speeds up the process of requesting extents if the client asks for them one by one (like qemu-img). Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- Essentially a v2 of the following patch since now there is a probing function: https://www.redhat.com/archives/libguestfs/2019-September/msg00249.html v2v/nbdkit.ml | 7 +++++++ 1 file changed, 7

Commit c3a54d6aed6dfc65f9ffa59976bb8d20044c03a8 ("v2v: Add standalone nbdkit module.") was supposed to be a simple refactoring but it broke the --bandwidth and --bandwidth-file options (amongst other things). Because of an extra '=' character which was accidentally left over, it would add an extra character in the nbdkit-rate-filter command line. For example: virt-v2v ..

This filter deliberately tries to coalesce reads into larger requests. Unfortunately VMware has low limits on the size of requests it can serve to a VDDK client and the larger requests would break with errors like this: nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes We already increase the maximum

The tests assume probe_filter returns true if the filter is available (and not the other way around). Signed-off-by: Tomáš Golembiovský <tgolembi@redhat.com> --- v2v/nbdkit.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/v2v/nbdkit.ml b/v2v/nbdkit.ml index 77d2a506..00122bec 100644 --- a/v2v/nbdkit.ml +++ b/v2v/nbdkit.ml @@ -142,7 +142,7 @@ let common_create

This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++----

This refactoring takes the nbdkit-specific code from the ‘-it vddk’ mode and puts it into a separate module. The idea is to reuse this module (in future commits) to support ssh, curl and rate limiting. This refactoring is not quite neutral because it moves some of the prechecking into the Nbdkit.create function. This means it will happen a little bit later in the cycle (in input#source instead

The readahead filter is a self-configuring filter that makes sequential reads faster when the plugin is slow (and all of the plugins we use here are always slow). I observed the behaviour of the readahead filter with our qcow2 overlay when converting a guest from a vCenter source. Even when doing random reads, qemu issues 64K reads which happen to also be the minimum request size of the

Use -ip instead of --password-file, and various refactorings. It strikes me that we should probably deprecate and eventually remove virt-v2v-copy-to-local. With the introduction of the new SSH and VDDK transports, and with RHEL 5 Xen becoming more irrelevant, it's no longer needed. Rich.

This isn't quite the full thing. I think that Pino is also working on replacing the ssh and scp commands in the v2v/input_vmx.ml file with libssh. Without those changes, -i vmx will still issue raw ssh and scp commands, which will use ssh-agent (or keyboard-interactive). The Xen input method doesn't use raw ssh and scp commands, so that one is OK. Rich.

This way errors that occur while reading the password file are properly propagated, instead of being reported as errors of the libvirt authentication callback. Reported by: Ming Xie. --- v2v/libvirt_utils.ml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/v2v/libvirt_utils.ml b/v2v/libvirt_utils.ml index 4d0b8639..1a24b049 100644 --- a/v2v/libvirt_utils.ml +++