search for: do_retri

After a retry, if the second connection has fewer permissions than the first, but we blindly call next_ops->FOO, we end up triggering an assertion failure in backend.c. This is particularly noticeable when the force_readonly flag is in effect, as that makes it much easier for there to be fewer permissions than before. Add testsuite coverage of pwrite to demonstrate. Signed-off-by: Eric Blake

Now that a filter can open a backend as many times as it wants, there's no longer a technical reason we can't retry .open. However, adding retry logic here does mean we have to weaken an assert in the server backend code, since prepare can now be reached more than once. Test coverage will be added in a separate patch, so that it becomes easy to swap patch order and see that the test

This filter can be used to transparently reopen/retry when a plugin fails. The connection is closed and reopened which for most plugins causes them to attempt to reconnect to their source. For example if doing a long or slow SSH copy: nbdkit -U - ssh host=remote /var/tmp/test.iso \ --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2' if the SSH connection or network goes

On Thu, Oct 03, 2019 at 09:54:37PM -0500, Eric Blake wrote: > Although it is unusual for a plugin to shrink size on reopen, it is > not impossible. Failure to check our bounds could result in violating > assumptions in the plugin that all requests are in-bounds. Note that > if the plugin gains a larger size, we merely never access the new tail > of the file (when the NBD protocol

This filter can be used to transparently reopen/retry when a plugin fails. The connection is closed and reopened which for most plugins causes them to attempt to reconnect to their source. For example if doing a long or slow SSH copy: nbdkit -U - ssh host=remote /var/tmp/test.iso \ --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2' if the SSH connection or network goes

Includes a rework of the previously posted patch for --run improvements (mostly with improved comments and commit message; I decided that waiting for the captive nbdkit to exit was overkill), and four new patches. The tests are intentionally separate, to allow rearranging the order of the series to see the failures being fixed. Eric Blake (6): server: Propagate unexpected nbdkit failure with

On Fri, Jan 27, 2023 at 02:41:22PM -0600, Eric Blake wrote: > Now that a filter can open a backend as many times as it wants, > there's no longer a technical reason we can't retry .open. However, > adding retry logic here does mean we have to weaken an assert in the > server backend code, since prepare can now be reached more than once. > > Test coverage will be added

Although it is unusual for a plugin to shrink size on reopen, it is not impossible. Failure to check our bounds could result in violating assumptions in the plugin that all requests are in-bounds. Note that if the plugin gains a larger size, we merely never access the new tail of the file (when the NBD protocol ever gains dynamic resize support, we'll get to revisit how the retry filter fits

In https://bugzilla.redhat.com/show_bug.cgi?id=1841820, it was pointed out that the retry filter not retrying .open means that an ssh connection (such as in a vmx+ssh v2v conversion) fails when the ssh connection itself cannot be retried. A year ago, this was an inherent limitation of our retry implementation; but in the meantime, my work to allow filters to open independent backends has made it

This is a retry filter implementation as outlined here: https://www.redhat.com/archives/libguestfs/2019-September/msg00167.html It is only lightly tested. One way to test it is to try an SSH copy (see the commit message for patch 2/2), and in the middle of the copy kill the per-connection sshd on the remote machine. You will see that the copy recovers after a few seconds. Add the nbdkit -v

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00199.html v2: - Adds a fairly simple yet comprehensive test using sh plugin. - Rebase and retest. Patch 1 is a misc patch not really related to the series. Rich.

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

The retry filter defaults to 5 retries, but when run with verbose details produces some confusing output: $ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)& ./nbdkit -f -v -U - \ null 1G --filter=retry --filter=noextents --filter=error error-rate=100% \ error-file=/tmp/inject --filter=delay rdelay=1 \ --run 'qemu-img convert $nbd out.img' ... nbdkit: null[1]: debug: noextents:

On 4/1/20 5:12 AM, Richard W.M. Jones wrote: > On Tue, Mar 31, 2020 at 04:00:22PM -0500, Eric Blake wrote: >> The retry filter defaults to 5 retries, but when run with verbose >> details produces some confusing output: >> >> $ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)& ./nbdkit -f -v -U - \ >> null 1G --filter=retry --filter=noextents --filter=error

I'm still working on another set of patches to have reopen call .finalize/.prepare (so that another filter can safely appear between retry and the plugin), but for tonight, these are the patches I think are ready to go. Eric Blake (4): retry: Handle can_fua and can_fast_zero changes tests: Test retry with different fua/fast-zero flags server: Close backends if a filter's .open fails

On Tue, Mar 31, 2020 at 04:00:22PM -0500, Eric Blake wrote: > The retry filter defaults to 5 retries, but when run with verbose > details produces some confusing output: > > $ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)& ./nbdkit -f -v -U - \ > null 1G --filter=retry --filter=noextents --filter=error error-rate=100% \ > error-file=/tmp/inject --filter=delay rdelay=1

On Wed, Apr 01, 2020 at 08:46:53AM -0500, Eric Blake wrote: > On 4/1/20 5:12 AM, Richard W.M. Jones wrote: > >On Tue, Mar 31, 2020 at 04:00:22PM -0500, Eric Blake wrote: > >>The retry filter defaults to 5 retries, but when run with verbose > >>details produces some confusing output: > >> > >>$ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)&

I still don't have .prepare/.finalize working cleanly across reopen, but did find a nasty bug where a botched assertion means we failed to notice reads beyond EOF in both the xz and retry filter. Refactoring backend.c will make .finalize work easier. Eric Blake (5): xz: Avoid reading beyond EOF retry: Check size before transactions tests: Test retry when get_size values change

I think this is my last round of patches for issues I identified with the retry filter. With this in place, it should be safe to interject another filter in between retry and the plugin. Eric Blake (5): retry: Don't call into closed plugin tests: Refactor test-retry-reopen-fail.sh tests: Enhance retry test to cover failed reopen server: Move prepare/finalize/close recursion to

To allow filters to modify the export name as it passes through the layers this commit makes several changes: The filter .open callback now takes an extra parameter, the export name. This is always non-NULL (for oldstyle it is ""). This string has a short lifetime and filters that need to hang on to it must take a copy. The filter must pass the exportname parameter down to the next