similar to: [PATCH nbdkit] filters: cache, cow: Handle bitmap overflow on 32 bit architectures.

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

Here's the patch (on top of the preceeding one) which uses a bitmap instead of SEEK_DATA. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org --4VrXvz3cwkc87Wze

I have some patches I'm working on to fix the cache filter. However this is a prelude. It should be simply pure refactoring. All tests pass still. Rich.

This is exactly the same as v1: https://www.redhat.com/archives/libguestfs/2018-December/msg00004.html except that it now frees the bitmap on unload (which the old code did not - there was always a memory leak). Rich.

v2: https://www.redhat.com/archives/libguestfs/2018-December/msg00039.html v2 -> v3: - Fix all the issues raised in Eric's review. - Precompute some numbers to make the calculations easier. - Calculations now use bitshifts and masks in preference to division and modulo. - Clear existing bits before setting (which fixes a bug in the cache filter). Rich.

On 12/2/18 10:33 AM, Richard W.M. Jones wrote: > The cow and cache filters both use a bitmap mapping virtual disk > blocks to status stored in the bitmap. The implementation of the > bitmaps is very similar because one was derived from the other when > the filters were implemented. > > The main difference is the cow filter uses a simple bitmap (one bit > per block), whereas

The following commands: nbdkit -fv --filter=cow memory size=512 --run 'qemu-img info $nbd' nbdkit -fv --filter=cache memory size=512 --run 'qemu-img info $nbd' both fail with: nbdkit: memory[1]: error: realloc: Success Initial git bisect pointed to commit 3166d2bcbfd2 (but I don't believe that commit is the real cause, it merely exposes the bug). The reason this

The following commands: nbdkit -fv --filter=cow memory size=512 --run 'qemu-img info $nbd' nbdkit -fv --filter=cache memory size=512 --run 'qemu-img info $nbd' nbdkit -fv --filter=cow null --run 'qemu-img info $nbd' all fail with: nbdkit: memory[1]: error: realloc: Success Initial git bisect pointed to commit 3166d2bcbfd2 (this is not real cause, it merely

This adds a cache filter, which works like the COW filter in reverse. For realistic use it needs a bit more work, especially to add limits on the size of the cache, a more sensible cache replacement policy, and perhaps some kind of background worker to write dirty blocks out. Rich.

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

Make the implicit lock explicit, and hold it around blk_* operations. This allows us to relax the thread model for the filter to NBDKIT_THREAD_MODEL_PARALLEL. --- filters/cache/blk.h | 7 ++++++ filters/cache/cache.c | 57 +++++++++++++++++++++++++++++++------------ 2 files changed, 49 insertions(+), 15 deletions(-) diff --git a/filters/cache/blk.h b/filters/cache/blk.h index 24bf6a1..ab9134e

--- filters/cache/lru.h | 54 ++++++++++++++ filters/cache/blk.c | 12 +++ filters/cache/lru.c | 150 ++++++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 2 + 4 files changed, 218 insertions(+) diff --git a/filters/cache/lru.h b/filters/cache/lru.h new file mode 100644 index 0000000..4faefcd --- /dev/null +++ b/filters/cache/lru.h @@ -0,0 +1,54 @@ +/* nbdkit

--- filters/cache/lru.h | 54 ++++++++++++++ filters/cache/blk.c | 12 +++ filters/cache/lru.c | 151 ++++++++++++++++++++++++++++++++++++++ filters/cache/Makefile.am | 2 + 4 files changed, 219 insertions(+) diff --git a/filters/cache/lru.h b/filters/cache/lru.h new file mode 100644 index 0000000..4faefcd --- /dev/null +++ b/filters/cache/lru.h @@ -0,0 +1,54 @@ +/* nbdkit

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00100.html In v2 I've changed the patch so it avoids calling realloc at all in this case. The patch is a bit longer this way. But I don't see any other alternative if we are to avoid having a "realloc wrapper" of some kind that we use everywhere, which I guess we should avoid because it makes plugins

On 01/26/2018 07:17 AM, Richard W.M. Jones wrote: > When compiling on armv7 it becomes clear from the compiler warnings > that the current code is wrong. > > The bitmap has to be allocated in virtual memory, so use size_t to > describe the length of the bitmap. When changing the length of the > bitmap, compute the new size as an unsigned 64 bit int, and then check > whether

Annotate some function parameters with attribute((nonnull)). Only do this for internal headers where we are sure that we will be using sufficiently recent GCC or Clang. For the public header files (ie. include/nbdkit-*.h) it may be that people building out of tree plugins are using old GCC which had problems, or even other compilers that don't support this extension at all. Libvirt has an

The blocksize filter already rounds sizes down, to avoid having to special-case a partial action at the end of an unaligned file. But our current behavior for cache and cow is not similarly careful, and can end up causing errors by passing a too-large count to the plugin. The bulk of our plugins are somewhat safe, in that they either fail the pread (for example, the file plugin gets a short read

Right now, the cache and cow filters always round up requests to blksize boundaries (blksize for cache is dynamically determined at connection start, for cow is fixed as BLKSIZE). Which is fine for the bulk of the underlying file, but can cause problems when reading past EOF for a partial tail of an underlying plugin. We aren't validating that filter calls to next_ops are within bounds; and