similar to: Thoughts on nbdkit automatic reconnection

On 9/19/19 4:50 AM, Richard W.M. Jones wrote: > On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones wrote: >> We have a running problem with the nbdkit VDDK plugin where the VDDK >> side apparently disconnects or the network connection is interrupted. >> During a virt-v2v conversion this causes the entire operation to fail, >> and since v2v conversions take many

I have an update on the networking issue: - After the deep dive into the logs of the firewall by customer's security team, it turns out that even though there were some disconnections, the time-stamps do not match. This means that we got the disconnected by something else (ESXi or conversion host perhaps) - As we mentioned in the chat briefly, there could be general keep-alive issues on both

On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones wrote: > We have a running problem with the nbdkit VDDK plugin where the VDDK > side apparently disconnects or the network connection is interrupted. > During a virt-v2v conversion this causes the entire operation to fail, > and since v2v conversions take many hours that's not a happy outcome. > > (Aside: I should

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00199.html v2: - Adds a fairly simple yet comprehensive test using sh plugin. - Rebase and retest. Patch 1 is a misc patch not really related to the series. Rich.

This is a retry filter implementation as outlined here: https://www.redhat.com/archives/libguestfs/2019-September/msg00167.html It is only lightly tested. One way to test it is to try an SSH copy (see the commit message for patch 2/2), and in the middle of the copy kill the per-connection sshd on the remote machine. You will see that the copy recovers after a few seconds. Add the nbdkit -v

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

On 9/19/19 10:26 AM, Richard W.M. Jones wrote: > This filter can be used to transparently reopen/retry when a plugin > fails. The connection is closed and reopened which for most plugins > causes them to attempt to reconnect to their source. > > For example if doing a long or slow SSH copy: > > nbdkit -U - ssh host=remote /var/tmp/test.iso \ > --run 'qemu-img

v3 posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00200.html v4: - The first patch in the v3 series was just a trivial doc whitespace fix so I pushed it. - There's a new patch using the nbdkit-retry-filter. This is not actually upstream in nbdkit but we know enough about how it will work. - Rebased against master and reran the tests. Rich.

On Tue, Nov 19, 2019 at 04:14:45PM +0200, Nir Soffer wrote: >On Tue, Nov 19, 2019 at 3:36 PM Martin Kletzander <mkletzan@redhat.com> >wrote: > >> On Tue, Nov 19, 2019 at 01:48:31PM +0100, Martin Kletzander wrote: >> >On Tue, Nov 19, 2019 at 02:24:11PM +0200, Nir Soffer wrote: >> >>On Tue, Nov 19, 2019 at 2:16 PM Martin Kletzander

This is the simplest solution to this problem. There are two other possible fixes I considered: Increase the documented limit (see http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-memory-limits). However at the moment we know the current limit works through extensive testing (without readahead), plus I have no idea nor any way to test if larger limits are supported by

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu

v2 was posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00115.html This also has links to earlier versions. v3: - The 01/11 patch in v2 included a bunch of unnecessary plus one necessary change to how input_password is passed around. I moved the necessary change into the final patch (implementing SSH password authentication) and dropped the rest. - The 01/11

Originally posted here: https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00054 https://www.redhat.com/archives/libguestfs/2019-April/msg00076.html https://www.redhat.com/archives/libguestfs/2019-April/msg00126.html This is a rebase on top of current master branch with no other changes. The first patch in the old series was pushed a while back, and the last "TEMPORARY"

Nir, BTW what are you using for performance testing? As far as I can tell it's not possible to make qemu-img convert use multi-conn when connecting to the source (which is going to be a problem if we want to use this stuff in virt-v2v). Instead I've hacked up a copy of this program from libnbd: https://github.com/libguestfs/libnbd/blob/master/examples/threaded-reads-and-writes.c so

I'm still testing this fix, so let's hold off the review for the moment. Also it may be better to specifically identify problematic servers rather than disabling this for every curl source. eg. I suspect that the problem is the Java server used by VCenter, so we might think about only disabling readahead for that single case. Rich.

v6 was here: https://www.redhat.com/archives/libguestfs/2018-March/msg00126.html This makes a number of significant changes: - Input and output options now use a uniform set of -io and -oo parameters. - For -o rhv-upload, we use ‘-oo rhv-cafile=/tmp/ca.pem’ etc. The ‘--rhv*’ options have been dropped. - Rearranges the documentation. - As before includes (untested) support for zero,

From: Martin Kletzander <mkletzan@redhat.com> Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- plugins/vddk/nbdkit-vddk-plugin.pod | 7 ++++++- plugins/vddk/vddk-structs.h | 1 + plugins/vddk/vddk.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod

Proposed small change to the command line of virt-v2v when specifying that you want VDDK mode. Rich.

The first patch was previously posted here: https://www.redhat.com/archives/libguestfs/2017-December/msg00018.html That patch hasn't changed except that I made the ‘input_transport’ variable type-safe. The second patch adds a significant new mode for liberating data from VMware: the ability to copy VMs over SSH directly from ESXi hypervisors. Although this requires enabling SSH access (a

This patch isn't quite complete (see ‘assert false’). Currently we have a bunch of ad hoc options like --vddk* and --vdsm* (and proposed to add --rhv*) to handle extra parameters for input and output modes/transports. This complicates the command line parsing and also the clarity of the command line (becauseit's not very obvious which options apply to which side of the conversion).