Libguestfs - Sep 2019 - Thoughts on nbdkit automatic reconnection

We have a running problem with the nbdkit VDDK plugin where the VDDK
side apparently disconnects or the network connection is interrupted.
During a virt-v2v conversion this causes the entire operation to fail,
and since v2v conversions take many hours that's not a happy outcome.

(Aside: I should say that we see many cases where it's claimed that
the connection was dropped, but often when we examine them in detail
the cause is something else.  But it seems like this disconnection
thing does happen sometimes.)

To put this isn't concrete terms which don't involve v2v, let's say
you were doing something like:

  nbdkit ssh host=remote /var/tmp/test.iso \
    --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2'

which copies a file over ssh to local.  If /var/tmp/test.iso is very
large and/or the connection is very slow, and the network connection
is interrupted then the whole operation fails.  If nbdkit could
retry/reconnect on failure then the operation might succeed.

There are lots of parameters associated with retrying, eg:

 - how often should you retry before giving up?

 - how long should you wait between retries?

 - which errors should cause a retry, which are a hard failure?

So I had an idea we could implement this as a generic "retry" filter,
like:

  nbdkit ssh ... --filter=retry retries=5 retry-delay=5 retry-exponential=yes

This cannot be implemented with the current design of filters because
a filter would have to call the plugin .close and .open methods, but
filters don't have access to those from regular data functions, and in
any case this would cause a new plugin handle to be allocated.

We could probably do it if we added a special .reopen method to
plugins.  We could either require plugins which support the concept of
retrying to implement this, or we could have a generic implementation
in server/backend.c which would call .close, .open and cope with the
new handle.

Another way to do this would be to modify each plugin to add the
feature.  nbdkit-nbd-plugin has this for a very limited case, but no
others do, and it's quite complex to implement in plugins.  As far as
I can see it involves checking the return value of any data call that
the plugin makes and performing the reconnection logic, while not
changing the handle (so just calling self->close, self->open isn't
going to work).

If anyone has any thoughts about this I'd be happy to hear them.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

On 9/18/19 7:59 AM, Richard W.M. Jones wrote:
> We have a running problem with the nbdkit VDDK plugin where the VDDK
> side apparently disconnects or the network connection is interrupted.
> During a virt-v2v conversion this causes the entire operation to fail,
> and since v2v conversions take many hours that's not a happy outcome.
> 
> (Aside: I should say that we see many cases where it's claimed that
> the connection was dropped, but often when we examine them in detail
> the cause is something else.  But it seems like this disconnection
> thing does happen sometimes.)
nbdkit is not alone - qemu is currently trying to add patches to add nbd
reconnect:

https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg03621.html
> 
> To put this isn't concrete terms which don't involve v2v, let's
say
> you were doing something like:
> 
>   nbdkit ssh host=remote /var/tmp/test.iso \
>     --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2'
> 
> which copies a file over ssh to local.  If /var/tmp/test.iso is very
> large and/or the connection is very slow, and the network connection
> is interrupted then the whole operation fails.  If nbdkit could
> retry/reconnect on failure then the operation might succeed.
> 
> There are lots of parameters associated with retrying, eg:
> 
>  - how often should you retry before giving up?
> 
>  - how long should you wait between retries?
> 
>  - which errors should cause a retry, which are a hard failure?
- do you want TCP keepalive active during the session
> 
> So I had an idea we could implement this as a generic "retry"
filter,
> like:
> 
>   nbdkit ssh ... --filter=retry retries=5 retry-delay=5
retry-exponential=yes
Interesting idea.
> 
> This cannot be implemented with the current design of filters because
> a filter would have to call the plugin .close and .open methods, but
> filters don't have access to those from regular data functions, and in
> any case this would cause a new plugin handle to be allocated.
Our .open handling is already odd: we document but do not enforce that a
filter must call next_open on success, but does not necessarily do so on
failure.  Depending on where things fail, it may be possible that we
have a memory leak and/or end up calling .close without a matching
.open; I'm trying to come up with a definitive test case demonstrating
if that is a problem.  I noticed that while trying to make nbdkit return
NBD_REP_ERR_XXX when .open fails, rather than dropping the connection
altogether (since that's a case where a single TCP connection would need
to result in multiple .open/.close pairings).
> We could probably do it if we added a special .reopen method to
> plugins.  We could either require plugins which support the concept of
> retrying to implement this, or we could have a generic implementation
> in server/backend.c which would call .close, .open and cope with the
> new handle.
It sounds like something that only needs to be exposed for filters to
use; I'm having a hard time seeing how a plugin would do it, so keeping
the magic in server/backend.c sounds reasonable.
> Another way to do this would be to modify each plugin to add the
> feature.  nbdkit-nbd-plugin has this for a very limited case, but no
> others do, and it's quite complex to implement in plugins.  As far as
> I can see it involves checking the return value of any data call that
> the plugin makes and performing the reconnection logic, while not
> changing the handle (so just calling self->close, self->open
isn't
> going to work).
> 
> If anyone has any thoughts about this I'd be happy to hear them.
> 
> Rich.
> 
-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones
wrote:
>We have a running problem with the nbdkit VDDK plugin where the VDDK
>side apparently disconnects or the network connection is interrupted.
>During a virt-v2v conversion this causes the entire operation to fail,
>and since v2v conversions take many hours that's not a happy outcome.
>
>(Aside: I should say that we see many cases where it's claimed that
>the connection was dropped, but often when we examine them in detail
>the cause is something else.  But it seems like this disconnection
>thing does happen sometimes.)
>
>To put this isn't concrete terms which don't involve v2v, let's
say
>you were doing something like:
>
>  nbdkit ssh host=remote /var/tmp/test.iso \
>    --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2'
>
>which copies a file over ssh to local.  If /var/tmp/test.iso is very
>large and/or the connection is very slow, and the network connection
>is interrupted then the whole operation fails.  If nbdkit could
>retry/reconnect on failure then the operation might succeed.
>
>There are lots of parameters associated with retrying, eg:
>
> - how often should you retry before giving up?
>
> - how long should you wait between retries?
>
> - which errors should cause a retry, which are a hard failure?
>
>So I had an idea we could implement this as a generic "retry"
filter,
>like:
>
>  nbdkit ssh ... --filter=retry retries=5 retry-delay=5
retry-exponential=yes
>
>This cannot be implemented with the current design of filters because
>a filter would have to call the plugin .close and .open methods, but
>filters don't have access to those from regular data functions, and in
>any case this would cause a new plugin handle to be allocated.
>
>We could probably do it if we added a special .reopen method to
>plugins.  We could either require plugins which support the concept of
>retrying to implement this, or we could have a generic implementation
>in server/backend.c which would call .close, .open and cope with the
>new handle.
>
>Another way to do this would be to modify each plugin to add the
>feature.  nbdkit-nbd-plugin has this for a very limited case, but no
>others do, and it's quite complex to implement in plugins.  As far as
>I can see it involves checking the return value of any data call that
>the plugin makes and performing the reconnection logic, while not
>changing the handle (so just calling self->close, self->open isn't
>going to work).
>
>If anyone has any thoughts about this I'd be happy to hear them.
>
Maybe this is a special case for which it would be easier to implement this in
nbdkit itself rather than a filter or plugin.  But to be honest when I read the
first paragraph my first two immediate thoughts were:

 1) this needs to be configurable and

 2) having this as a filter would be really cool.

I don't know about how much the implementation needs to change etc.  But if
plugins expose proper error outputs then the "to retry or not to
retry" question
can be also configurable based on the errno?  Just food for thoughts.

Have a nice day,
Martin
>Rich.
>
>-- 
>Richard Jones, Virtualization Group, Red Hat
http://people.redhat.com/~rjones
>Read my programming and virtualization blog: http://rwmj.wordpress.com
>virt-builder quickly builds VMs from scratch
>http://libguestfs.org/virt-builder.1.html
>
>_______________________________________________
>Libguestfs mailing list
>Libguestfs@redhat.com
>https://www.redhat.com/mailman/listinfo/libguestfs

On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones
wrote:
> We have a running problem with the nbdkit VDDK plugin where the VDDK
> side apparently disconnects or the network connection is interrupted.
> During a virt-v2v conversion this causes the entire operation to fail,
> and since v2v conversions take many hours that's not a happy outcome.
> 
> (Aside: I should say that we see many cases where it's claimed that
> the connection was dropped, but often when we examine them in detail
> the cause is something else.  But it seems like this disconnection
> thing does happen sometimes.)
It turns out in the customer case that led us to talk about this, a
Checkpoint firewall was forcing the VDDK control connection to be
closed after an idle period.  (The VDDK connection as a whole was not
actually idle because data was being copied over the separate data
port, but the firewall did not associate the two ports).  I believe
nbdkit-retry-filter would have helped in this case because reopening
the VDDK connection will reestablish the control/metadata connection,
and therefore I am looking at an implementation now.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

I have an update on the networking issue:
- After the deep dive into the logs of the firewall by customer's security
team, it turns out that even though there were some disconnections, the
time-stamps do not match.
 This means that we got the disconnected by something else (ESXi or
conversion host perhaps)
- As we mentioned in the chat briefly, there could be general keep-alive
issues on both RHEL (conversion host) and ESXi side.
 We changed the keep-alive settings in RHEL, but could not find the
equvalent in VMware as of yet.
- I found on a few spots that there are some vddk (vixDiskLib.nfc*)
settings which can configure NFC keep-alives and timeouts, but I do not
understand it deeply enough to see if anything would help.

Whatever may be the cause, a retry filter would most likely solve the
problem.

Since we are fairly certain that we would encounter another failure with
VDDK how the situation stands now, we are trying SSH transport to see how
that will go.

Cheers,

*Nenad Perić*

PRINCIPAL SOFTWARE ENGINEER

Red Hat - Migration Engineering

nenad@redhat.com

<http://redhat.com>


On Thu, Sep 19, 2019 at 11:50 AM Richard W.M. Jones <rjones@redhat.com>
wrote:
> On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones wrote:
> > We have a running problem with the nbdkit VDDK plugin where the VDDK
> > side apparently disconnects or the network connection is interrupted.
> > During a virt-v2v conversion this causes the entire operation to fail,
> > and since v2v conversions take many hours that's not a happy
outcome.
> >
> > (Aside: I should say that we see many cases where it's claimed
that
> > the connection was dropped, but often when we examine them in detail
> > the cause is something else.  But it seems like this disconnection
> > thing does happen sometimes.)
>
> It turns out in the customer case that led us to talk about this, a
> Checkpoint firewall was forcing the VDDK control connection to be
> closed after an idle period.  (The VDDK connection as a whole was not
> actually idle because data was being copied over the separate data
> port, but the firewall did not associate the two ports).  I believe
> nbdkit-retry-filter would have helped in this case because reopening
> the VDDK connection will reestablish the control/metadata connection,
> and therefore I am looking at an implementation now.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-top is 'top' for virtual machines.  Tiny program with many
> powerful monitoring features, net stats, disk stats, logging, etc.
> http://people.redhat.com/~rjones/virt-top
>

On 9/19/19 4:50 AM, Richard W.M. Jones wrote:
> On Wed, Sep 18, 2019 at 01:59:01PM +0100, Richard W.M. Jones wrote:
>> We have a running problem with the nbdkit VDDK plugin where the VDDK
>> side apparently disconnects or the network connection is interrupted.
>> During a virt-v2v conversion this causes the entire operation to fail,
>> and since v2v conversions take many hours that's not a happy
outcome.
>>
>> (Aside: I should say that we see many cases where it's claimed that
>> the connection was dropped, but often when we examine them in detail
>> the cause is something else.  But it seems like this disconnection
>> thing does happen sometimes.)
> 
> It turns out in the customer case that led us to talk about this, a
> Checkpoint firewall was forcing the VDDK control connection to be
> closed after an idle period.
Would TCP keepalive help here?
>  (The VDDK connection as a whole was not
> actually idle because data was being copied over the separate data
> port, but the firewall did not associate the two ports).  I believe
> nbdkit-retry-filter would have helped in this case because reopening
> the VDDK connection will reestablish the control/metadata connection,
> and therefore I am looking at an implementation now.
Compared to re-establishing connection, keepalive is a simpler tweak
(although having the option for both is even better...)

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org