similar to: [PATCH nbdkit] v2v: Disable readahead for VMware curl sources too (RHBZ#1848862).

This is the simplest solution to this problem. There are two other possible fixes I considered: Increase the documented limit (see http://libguestfs.org/virt-v2v-input-vmware.1.html#vddk:-esxi-nfc-service-memory-limits). However at the moment we know the current limit works through extensive testing (without readahead), plus I have no idea nor any way to test if larger limits are supported by

Patch 1 was previously posted here: https://www.redhat.com/archives/libguestfs/2020-June/msg00086.html to handle this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1848862 I was able to observe this bug and for me at least disabling readahead seems to cure it. Patches 2 and 3 are simplifications, removing a now-undocumented feature of virt-v2v-copy-to-local and thus simplifying greatly the

This appears to be the cause of timeouts during the conversion step where VMware VCenter server's Tomcat HTTPS server stops responding to requests (or rather, responds only with 503 errors). The server later recovers and in fact because of the retry filter the conversion usually succeeds, but I found that we can avoid the problem by disabling readahead. --- v2v/nbdkit_sources.ml | 8 ++++----

Part 2 fix for: https://bugzilla.redhat.com/show_bug.cgi?id=1842440 Actually this fix on its own should be sufficient, but probably we want the nbdkit fixes too. Note this uses actual OCaml 4.05 features! ("let open" and the Unix.tcgetattr functions). I checked that both features are available on RHEL 7's OCaml. Rich.

On 6/19/20 7:47 AM, Richard W.M. Jones wrote: > This appears to be the cause of timeouts during the conversion step > where VMware VCenter server's Tomcat HTTPS server stops responding to > requests (or rather, responds only with 503 errors). The server later > recovers and in fact because of the retry filter the conversion > usually succeeds, but I found that we can avoid the

This filter deliberately tries to coalesce reads into larger requests. Unfortunately VMware has low limits on the size of requests it can serve to a VDDK client and the larger requests would break with errors like this: nbdkit: vddk[3]: error: [NFC ERROR] NfcFssrvrProcessErrorMsg: received NFC error 5 from server: Failed to allocate the requested 33554456 bytes We already increase the maximum

Trivial fix. We really need a regression test for all v2v inpus related to nbdkit. There is actually nothing at all at the moment. Of course if it was easy to test inputs over the network from a ‘make check’ rule then we'd be doing it already. I thought about adding something like a ‘-it file’ option which would use nbdkit-file-plugin for test only. However I'm rather cautious about

I'm trying to replicate the features of the qemu curl plugin in nbdkit's curl plugin, in order that we can use nbdkit in virt-v2v to access VMware servers. I've implemented everything else so far [not posted yet] except for readahead. To my surprise actually, qemu's curl driver implements readahead itself. I thought it was a curl feature. I'm not completely clear _how_ it

On Fri, Sep 20, 2019 at 10:28:18AM +0100, Richard W.M. Jones wrote: >The readahead filter is a self-configuring filter that makes >sequential reads faster when the plugin is slow (and all of the >plugins we use here are always slow). > >I observed the behaviour of the readahead filter with our qcow2 >overlay when converting a guest from a vCenter source. Even when >doing

Hi all, I think I've spotted a few stupidities (bugs) in the current version of kernel-utils (kernel-utils-2.4-13.1.80). I'm sure these are all propagated from upstream, but I hope someone could have a quick look to verify this and see if we either can push complaints upwards, or provide local fixes. The kernel-utils package provides several 'kernel-type' functions -

A suggested readahead filter. I've only lightly tested this, but it seems to work fine with qemu-img convert. The commit needs proper tests. Rich.

Quick question... The docs for vfs readahead mention Windows Vista, but what about modern Windows versions? Windows Server 2016 or Windows Server 2019? Thank you, -- BOB BUCK SENIOR PLATFORM SOFTWARE ENGINEER SKIDMORE, OWINGS & MERRILL 7 WORLD TRADE CENTER 250 GREENWICH STREET NEW YORK, NY 10007 T (212) 298-9624 ROBERT.BUCK at SOM.COM

Simpler, and including tests. Rich.

As I've spent really too long today investigating this, I want to document this in a public email, even though there's nothing really that interesting here. One thing you find from search for VDD 6.7 / VixDiskLib_QueryAllocatedBlocks issues with Google is that we must be one of the very few users out there. And the other thing is that it's quite broken. All testing was done using

This series (except the last one) changes virt-v2v to use nbdkit for several input modes: -i vmx -it vddk: No change in functionality, as this already uses nbdkit-vddk-plugin, but the code is refactored for the other modes to use. -i libvirtxml: Use nbdkit-curl-plugin instead of qemu curl. vCenter: Use nbdkit-curl-plugin instead of qemu curl. xen: Use nbdkit-ssh-plugin instead of qemu

There's more that can be done (in particular, use of CLEANUP_FREE), but this is enough to at least see if I'm on the right track. I couldn't figure out an obvious difference between common/include and common/utils, but it looks like the former is for things that are inlineable via .h only, while the latter is when you need to link in a convenience library, so this landed in the

v3 posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00200.html v4: - The first patch in the v3 series was just a trivial doc whitespace fix so I pushed it. - There's a new patch using the nbdkit-retry-filter. This is not actually upstream in nbdkit but we know enough about how it will work. - Rebased against master and reran the tests. Rich.

v2 was posted here: https://www.redhat.com/archives/libguestfs/2019-July/msg00115.html This also has links to earlier versions. v3: - The 01/11 patch in v2 included a bunch of unnecessary plus one necessary change to how input_password is passed around. I moved the necessary change into the final patch (implementing SSH password authentication) and dropped the rest. - The 01/11

Originally posted here: https://www.redhat.com/archives/libguestfs/2019-April/thread.html#00054 https://www.redhat.com/archives/libguestfs/2019-April/msg00076.html https://www.redhat.com/archives/libguestfs/2019-April/msg00126.html This is a rebase on top of current master branch with no other changes. The first patch in the old series was pushed a while back, and the last "TEMPORARY"

Hi, We have file readahead to do asyn file read, but has no metadata readahead. For a list of files, their metadata is stored in fragmented disk space and metadata read is a sync operation, which impacts the efficiency of readahead much. The patches try to add meatadata readahead for btrfs. In btrfs, metadata is stored in btree_inode. Ideally, if we could hook the inode to a fd so we could use