similar to: [nbdkit PATCH] cache: Reduce use of bounce-buffer

Make the implicit lock explicit, and hold it around blk_* operations. This allows us to relax the thread model for the filter to NBDKIT_THREAD_MODEL_PARALLEL. --- filters/cache/blk.h | 7 ++++++ filters/cache/cache.c | 57 +++++++++++++++++++++++++++++++------------ 2 files changed, 49 insertions(+), 15 deletions(-) diff --git a/filters/cache/blk.h b/filters/cache/blk.h index 24bf6a1..ab9134e

Although the time spent in memcpy/memset probably pales in comparison to time spent in socket I/O, it's still worth worth reducing the number of times we have to utilize a bounce buffer when we already have aligned data. Note that blocksize, cache, and cow all do block fragmentation and bounce-buffer alignment; this brings the logic in cache and cow (which were copied from one another) more

On Sat, May 11, 2019 at 03:30:04PM -0500, Eric Blake wrote: > Although the time spent in memcpy/memset probably pales in comparison > to time spent in socket I/O, it's still worth worth reducing the > number of times we have to utilize a bounce buffer when we already > have aligned data. > > Signed-off-by: Eric Blake <eblake@redhat.com> > --- >

Here's the patch (on top of the preceeding one) which uses a bitmap instead of SEEK_DATA. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org --4VrXvz3cwkc87Wze

Previously, we let a plugin set an error in either thread-local storage (nbdkit_set_error()) or errno, then connections.c would decode which error to use. But with filters in the mix, it is very difficult for a filter to know what error was set by the plugin (particularly since nbdkit_set_error() has no public counterpart for reading the thread-local storage). What's more, if a filter does

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache

Commit 975dab14 argued that for simple lock/unlock sequences, it was easier to avoid the cleanup.h macros. But since that time, we added additional sanity checking to the macros, at which point the boilerplate of inlining that sanity checking is outweighed compared to just using the macros in more places. Signed-off-by: Eric Blake <eblake@redhat.com> --- filters/cache/cache.c | 23

--- configure.ac | 5 + filters/Makefile.am | 4 + filters/cow/Makefile.am | 65 +++++++ filters/cow/cow.c | 392 ++++++++++++++++++++++++++++++++++++++ filters/cow/nbdkit-cow-filter.pod | 162 ++++++++++++++++ tests/Makefile.am | 6 + tests/test-cow.sh | 98 ++++++++++ 7 files changed, 732

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order

This adds a cache filter, which works like the COW filter in reverse. For realistic use it needs a bit more work, especially to add limits on the size of the cache, a more sensible cache replacement policy, and perhaps some kind of background worker to write dirty blocks out. Rich.

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

--- filters/cache/lru.h | 54 ++++++++++++++ filters/cache/blk.c | 12 +++ filters/cache/lru.c | 150 ++++++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 2 + 4 files changed, 218 insertions(+) diff --git a/filters/cache/lru.h b/filters/cache/lru.h new file mode 100644 index 0000000..4faefcd --- /dev/null +++ b/filters/cache/lru.h @@ -0,0 +1,54 @@ +/* nbdkit

--- filters/cache/lru.h | 54 ++++++++++++++ filters/cache/blk.c | 12 +++ filters/cache/lru.c | 151 ++++++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 2 + 4 files changed, 219 insertions(+) diff --git a/filters/cache/lru.h b/filters/cache/lru.h new file mode 100644 index 0000000..4faefcd --- /dev/null +++ b/filters/cache/lru.h @@ -0,0 +1,54 @@ +/* nbdkit

Annotate some function parameters with attribute((nonnull)). Only do this for internal headers where we are sure that we will be using sufficiently recent GCC or Clang. For the public header files (ie. include/nbdkit-*.h) it may be that people building out of tree plugins are using old GCC which had problems, or even other compilers that don't support this extension at all. Libvirt has an

v4: https://www.redhat.com/archives/libguestfs/2019-January/msg00032.html v5: - Now we set the block size at run time. I'd like to say that I was able to test this change, but unfortunately I couldn't find any easy way to create a filesystem on x86-64 with a block size > 4K. Ext4 doesn't support it at all, and XFS doesn't support block size > page size (and I

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

This patch series enhances the cache filter in a few ways, primarily adding a "cache-on-read" feature (similar to qemu's copyonread); and adding the ability to limit the cache size and the antecedent of that which is having a method to reclaim cache blocks. As the cache is stored as a sparse temporary file, reclaiming cache blocks simply means punching holes in the temporary file.

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache