similar to: [nbdkit PATCH 0/2] More on .thread_model

I'm still working my way through the filters before this series will be complete, but this is enough of a start to at least get some feedback on the idea of implementing another NBD protocol extension. Eric Blake (9): server: Internal hooks for implementing NBD_CMD_CACHE plugins: Add .cache callback file, split: Implement .cache with posix_fadvise nbd: Implement NBD_CMD_CACHE

Being able to programmatically force nbdkit to be less parallel can be useful during testing. I was less sure about patch 3, but if you like it, I'm inclined to instead squash it into patch 1. This patch is written to apply after my NBD_CMD_CACHE work (since I touched the nocache filter); but can be rearranged if we think this series should go in first while that one undergoes any adjustments

When I added .can_FOO caching in 1.16, I missed the case that the sh plugin itself was calling .can_flush twice in some situations (in order to default .can_fua). Then right after, I regressed it to call .can_zero twice (in order to default .can_fast_zero). I also missed that .thread_model could use better caching, because at the time, I did not add testsuite coverage. Fix that now. Eric Blake

This is a cover letter to a series of patches being proposed in tandem to four different projects: - nbd: Document a new NBD_CMD_FLAG_FAST_ZERO command flag - qemu: Implement the flag for both clients and server - libnbd: Implement the flag for clients - nbdkit: Implement the flag for servers, including the nbd passthrough client If you want to test the patches together, I've pushed a

v1 was here: https://www.redhat.com/archives/libguestfs/2018-December/thread.html#00123 v2: - Document "-" instead of "script=-" and use it in the test; and verify this also works on FreeBSD; and verify that it doesn't depend on the particular behaviour of our wrapper script and should work with installed nbdkit too. - Fix handling of zero flags parameter. -

Some feature additions to the shell script plugin allow us to test the export flags field reasonably easily. Rich.

Since v1: - rework .can_cache to be tri-state, with default of no advertisement (ripple effect through other patches) - add a lot more patches in order to round out filter support And in the meantime, Rich pushed NBD_CMD_CACHE support into libnbd, so in theory we now have a way to test cache commands through the entire stack. Eric Blake (24): server: Internal hooks for implementing

Allow plugins to affect the handling of the new NBD_CMD_FLAG_FAST_ZERO flag, then update affected plugins. In particular, in-memory plugins are always fast; the full plugin is better served by omitting .zero and relying on .pwrite fallback; nbd takes advantage of libnbd extensions proposed in parallel to pass through support; and v2 language bindings expose the choice to their scripts. The

This is a major rewrite compared to my v1 series, where I've tried a lot harder to ensure that we still accommodate building on Haiku (although I have not actually yet fired up a Haiku VM to try it for myself). I also managed to make the sh plugin fully parallel, on capable platforms. See also my question on patch 10 on whether I've picked the best naming convention. Eric Blake (17):

Until the next few patches expose and implement new callbacks for filters and plugins, our initial implementation for NBD_CMD_CACHE is to just blindly advertise the feature, and call into .pread with an ignored buffer. Note that for bisection reasons, this patch treats any use of a filter as a forced .can_cache of false to bypass the filter's caching; once all affected filters are patched to

In commit c306fa93ab and neighbors (v1.15.1), a concerted effort went into caching the results of .can_FOO callbacks, with commit messages demonstrating that a plugin with a slow callback should not have that delay magnified multiple times. But nothing was added to the testsuite at the time, and with the sh and eval plugins, we still have two scenarios where we did not take advantage of the

When we added support for .extents, we had nbdkit unconditionally support structured replies if the client requests them, and the plugin's .can_extents has no impact on what the server advertises. However, while the plugin API doesn't care whether the client requested SR, there is still a case to be made for allowing a filter to prevent SR, at least for testing purposese (such as

On Fri, Aug 02, 2019 at 02:26:16PM -0500, Eric Blake wrote: > When we first created the sh plugin, we copied the thread model used > in most other language bindings of SERIALIZE_ALL_REQUESTS, because it > was easier to reason about and there was no way for a script to > override our choice. However, we've since added support for scripts > to request a tighter model when needed,

First thing to say is that I need to do a *lot* more testing on this, so this is just an early peek. In particular, although it passed ‘make check && make check-valgrind’ I have *not* tested it against a multi-conn-aware client such as the Linux kernel >= 4.9. This implements NBD_FLAG_CAN_MULTI_CONN, described in the protocol doc as: "NBD_FLAG_CAN_MULTI_CONN: Indicates that

This is a revision of my .default_export work, plus new work on .export_descriptions and a new exportname filter. I think it is now ready to check in. Things I'd still like in 1.22: - the file plugin should implement .list_exports (patch already posted, but it needs rebasing on this series) - the ext2 filter should override .list_exports when in exportname mode - the nbd plugin should be

Similar to the existing fua, nozero and noextents filters, add a filter to make it easy to override the basic caching functionality, in part to facilitate timing tests of whether a plugin's cache implementation is worthwhile. A worthwhile test to add to the testsuite would connect the log filter both before and after the nocache filter, to prove how caching requests are altered. However,

It's been several weeks since I posted v2 (I got distracted by improving libnbd to better test things, which in turn surfaced some major memory leak problems in nbdsh that are now fixed). Many of the patches are minor rebases from v2, with the biggest changes being fallout from: - patch 2: rename nbdkit_add_default_export to nbdkit_use_default_export - overall: this missed 1.22, so update

If you're following nbdkit development upstream you'll have seen that we are still making changes to the .list_exports and related APIs. The current .list_exports API upstream is not how it will look finally. The latest set of proposals was here: https://www.redhat.com/archives/libguestfs/2020-August/thread.html#00330 At the same time I'd like to do an nbdkit 1.22 (stable) release.

Either patch 1 or patch 2 in isolation is sufficient to fix the problem that Rich forwarded on from an archlinux tester (name so I can credit them?). But both patches should be applied, as well as backported to appropriate stable branches, to maximize cross-version interoperability of nbdkit vs. plugins. Patch 3 will let us detect future similar bugs much faster. I want patch 4 to ensure that

On Fri, Aug 07, 2020 at 05:00:52PM -0500, Eric Blake wrote: > The next patch wants to add a filter that will prevent DoS attacks > from a plaintext client; to be successful, the filter must guarantee > that nbdkit did not settle on SERIALIZE_CONNECTIONS. The easiest way > to solve this is to expose the final thread model to .get_ready, which > is after the point where