similar to: Readahead in the nbdkit curl plugin

On 6/19/20 7:47 AM, Richard W.M. Jones wrote: > This appears to be the cause of timeouts during the conversion step > where VMware VCenter server's Tomcat HTTPS server stops responding to > requests (or rather, responds only with 503 errors). The server later > recovers and in fact because of the retry filter the conversion > usually succeeds, but I found that we can avoid the

I'm still testing this fix, so let's hold off the review for the moment. Also it may be better to specifically identify problematic servers rather than disabling this for every curl source. eg. I suspect that the problem is the Java server used by VCenter, so we might think about only disabling readahead for that single case. Rich.

A suggested readahead filter. I've only lightly tested this, but it seems to work fine with qemu-img convert. The commit needs proper tests. Rich.

On Fri, Sep 20, 2019 at 10:28:18AM +0100, Richard W.M. Jones wrote: >The readahead filter is a self-configuring filter that makes >sequential reads faster when the plugin is slow (and all of the >plugins we use here are always slow). > >I observed the behaviour of the readahead filter with our qcow2 >overlay when converting a guest from a vCenter source. Even when >doing

Simpler, and including tests. Rich.

I do have a question about whether patch 2 is right, or whether I've exposed a bigger problem in the truncate (and possibly other) filter, but the rest seem fairly straightforward. Eric Blake (4): server: Check for pthread lock failures truncate: Factor out reading real_size under mutex plugins: Check for mutex failures filters: Check for mutex failures filters/cache/cache.c

This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++----

The readahead filter is a self-configuring filter that makes sequential reads faster when the plugin is slow (and all of the plugins we use here are always slow). I observed the behaviour of the readahead filter with our qcow2 overlay when converting a guest from a vCenter source. Even when doing random reads, qemu issues 64K reads which happen to also be the minimum request size of the

Issue: Vista reads slowly from a samba server. This appears to pop up periodically here and elsewhere. My samba.conf file has: [homes] ... vfs objects = readahead As suggested elsewhere. Writes are approximately 17-18MB/s which is acceptable. Reads are in the 8MB/s range which is appalingly slow. Using linux smbclient and windows XP clients I can read at 25+MB/s. I've enabled vfs

Quick question... The docs for vfs readahead mention Windows Vista, but what about modern Windows versions? Windows Server 2016 or Windows Server 2019? Thank you, -- BOB BUCK SENIOR PLATFORM SOFTWARE ENGINEER SKIDMORE, OWINGS & MERRILL 7 WORLD TRADE CENTER 250 GREENWICH STREET NEW YORK, NY 10007 T (212) 298-9624 ROBERT.BUCK at SOM.COM

This is a retry filter implementation as outlined here: https://www.redhat.com/archives/libguestfs/2019-September/msg00167.html It is only lightly tested. One way to test it is to try an SSH copy (see the commit message for patch 2/2), and in the middle of the copy kill the per-connection sshd on the remote machine. You will see that the copy recovers after a few seconds. Add the nbdkit -v

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order

Patch 1 was previously posted here: https://www.redhat.com/archives/libguestfs/2020-June/msg00086.html to handle this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1848862 I was able to observe this bug and for me at least disabling readahead seems to cure it. Patches 2 and 3 are simplifications, removing a now-undocumented feature of virt-v2v-copy-to-local and thus simplifying greatly the

Since v1: - rework .can_cache to be tri-state, with default of no advertisement (ripple effect through other patches) - add a lot more patches in order to round out filter support And in the meantime, Rich pushed NBD_CMD_CACHE support into libnbd, so in theory we now have a way to test cache commands through the entire stack. Eric Blake (24): server: Internal hooks for implementing

This is the simplest solution to this problem. There are two other possible fixes I considered: Increase the documented limit (see http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-memory-limits). However at the moment we know the current limit works through extensive testing (without readahead), plus I have no idea nor any way to test if larger limits are supported by

As I've spent really too long today investigating this, I want to document this in a public email, even though there's nothing really that interesting here. One thing you find from search for VDD 6.7 / VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be one of the very few users out there. And the other thing is that it's quite broken. All testing was done using

v2 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00221.html I think this is more like "the one". It handles reopen failing correctly, and there is a second test for that. I also ran my sshd tests locally and it worked in all scenarios I could think up (except of course sshd not being available at the start, but we want that to fail). Rich.

While iflag=count_bytes combined with bs > 1 allows for more efficient operation, it is a feature of GNU dd, and not present on other implementations such as BSD. Sticking to just POSIX features makes things more portable. Signed-off-by: Eric Blake <eblake@redhat.com> --- docs/nbdkit-loop.pod | 2 +- docs/nbdkit.pod | 4 ++--

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

Here's what I'm currently playing with; I'm not ready to commit anything until I rebase my FUA work on top of this, as I only want to break filter ABI once between releases. Eric Blake (2): backend: Rework internal/filter error return semantics filters: Add log filter TODO | 2 - docs/nbdkit-filter.pod | 84 +++++++-- docs/nbdkit.pod