similar to: nbdkit design: varying the rate limit parameter

On 3/26/19 11:13 AM, Richard W.M. Jones wrote: > Hi Eric, > > We may have our first user (Tomas, CC'd) of the rate filter to limit > NBD connections. As you recall it lets you do commands like: > > nbdkit --filter=rate memory size=64G rate=10M > > to limit network bandwidth to 10 Mbps. However a twist is that he > needs to vary the parameter while nbdkit is

v2 turned out to be much more involved, as I ended up fixing several things along the way that I noticed while debugging a feature addition. Eric Blake (5): maint: Improve ./nbdkit option parsing main: Saner newline printing during --help utils: Add nbdkit_parse_bool main: Use new bool parser for --tls log: Allow user option of appending to log docs/nbdkit-plugin.pod | 11

In v7: everything should work now! The re-exec code is slightly simplified, with Rich's suggestion to pass the original LD_LIBRARY_PATH rather than just the prefix being added, and I've now finished wiring up the initial dlopen() check into code that correctly computes the right prefix dir to add to LD_LIBRARY_PATH. Eric Blake (1): vddk: Add re-exec with altered environment Richard

I suppose we may as well implement the only other flag too ... It's not clear what this does, something like O_DIRECT I imagine. Rich.

Should we use attribute((nonnull)) at all? There's a very interesting history of this in libvirt -- try looking at commit eefb881 plus the commits referencing eefb881 -- but it does seem to work for me using recent GCC and Clang. I only did a few functions because annotating them gets old quickly... Rich.

On 3/5/19 4:38 AM, Richard W.M. Jones wrote: > --- > filters/delay/nbdkit-delay-filter.pod | 4 +- > filters/rate/nbdkit-rate-filter.pod | 84 +++++++++ > configure.ac | 2 + > filters/rate/bucket.h | 62 +++++++ > filters/rate/bucket.c | 173 +++++++++++++++++++ > filters/rate/rate.c | 235

The 0.9.8 release breaks API, requiring a number of changes: - Use symbolic constants instead of magic numbers/open-coded strings (well, the string for "base:allocation" was present before this libnbd bump) - Change callbacks to drop the valid_flag parameter - Add _is to nbd_read_only call - Drop the _callback suffix on nbd_aio_FOO calls - Add a struct for managing callback/user_data

On 4/1/20 5:12 AM, Richard W.M. Jones wrote: > On Tue, Mar 31, 2020 at 04:00:22PM -0500, Eric Blake wrote: >> The retry filter defaults to 5 retries, but when run with verbose >> details produces some confusing output: >> >> $ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)& ./nbdkit -f -v -U - \ >> null 1G --filter=retry --filter=noextents --filter=error

From: Martin Kletzander <mkletzan@redhat.com> Signed-off-by: Martin Kletzander <mkletzan@redhat.com> --- plugins/vddk/nbdkit-vddk-plugin.pod | 7 ++++++- plugins/vddk/vddk-structs.h | 1 + plugins/vddk/vddk.c | 10 ++++++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod

We noticed while writing various libnbd tests that when the delay filter is in use, there are scenarios where we had to resort to SIGKILL to get rid of nbdkit, because it was non-responsive to SIGINT. I'm still trying to figure out the best way to add testsuite coverage of this, but already proved to myself that it works from the command line, under two scenarios that both used to cause long

Here's changes to the file plugin (which I'm happy with) and a new exportname filter (which is still at RFC stage; I need to finish implementing strict mode in .open, and add tests). I also discovered that we really want .list_exports and .open to know when they are used on plaintext vs. tls clients for --tls=on, and we may want to split out a new .default_export callback rather than

This is a second version of the limit filter. v1 was posted here: https://www.redhat.com/archives/libguestfs/2020-March/msg00015.html I didn't bother to repost the other three patches because they are the same. The difference is this version of the filter takes security more seriously. It shouldn't be possible for malicious clients to connect more than limit=N times to the plugin now,

The retry filter defaults to 5 retries, but when run with verbose details produces some confusing output: $ rm -f /tmp/inject; (sleep 5s; touch /tmp/inject)& ./nbdkit -f -v -U - \ null 1G --filter=retry --filter=noextents --filter=error error-rate=100% \ error-file=/tmp/inject --filter=delay rdelay=1 \ --run 'qemu-img convert $nbd out.img' ... nbdkit: null[1]: debug: noextents:

I pushed a few of the msys2 patches upstream. I changed the way that $(SHARED_LDFLAGS) works so it's more to my liking, and the others were pushed unchanged. Three patches remain which I'm posting on the mailing list for proper review. Rich.

libnbd's copy/copy-nbd-error.sh was triggering an assertion failure in nbdkit: $ nbdcopy -- [ nbdkit --exit-with-parent -v --filter=error pattern 5M error-pread-rate=0.5 ] null: ... nbdkit: pattern.2: debug: error-inject: pread count=262144 offset=4718592 nbdkit: pattern.2: debug: pattern: pread count=262144 offset=4718592 nbdkit: pattern.1: debug: error-inject: pread count=262144

I got annoyed by qemu-nbd's default of only allowing a single connection; combine that with nbdkit's nbd plugin, and even 'qemu-nbd --list' of nbdkit counts as the single connection and immediately hangs up. If we introduce a shared mode, then 'qemu-nbd --list' can connect as many times as it wants without killing the original qemu-nbd wrapped by nbdkit. But this in turn

This is a retry filter implementation as outlined here: https://www.redhat.com/archives/libguestfs/2019-September/msg00167.html It is only lightly tested. One way to test it is to try an SSH copy (see the commit message for patch 2/2), and in the middle of the copy kill the per-connection sshd on the remote machine. You will see that the copy recovers after a few seconds. Add the nbdkit -v

I'm a lot happier with this version: no mucking with dlmopen(). It does add a bit more coupling between nbdkit proper and the vddk plugin (namely, nbdkit now exports a new function that the vddk plugin relies on), but patch 2 adds testsuite coverage of the new function so we shouldn't regress. Patch 1 and 2 are new, patch 3 is unchanged from when Rich posted it in v2, and patch 4 is

For virt-v2v we have been discussing how to limit network bandwidth. The initial discussion has been around how to use cgroups to do this limiting, and that is still probably what we will go with in the end. However this patch gives us another possibility for certain virt-v2v inputs, especially VDDK. We could apply a filter on top of the nbdkit plugin which limits the rate at which it copies

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00199.html v2: - Adds a fairly simple yet comprehensive test using sh plugin. - Rebase and retest. Patch 1 is a misc patch not really related to the series. Rich.