similar to: [PATCH nbdkit] common: Move shared bitmap code to a common library.

This is exactly the same as v1: https://www.redhat.com/archives/libguestfs/2018-December/msg00004.html except that it now frees the bitmap on unload (which the old code did not - there was always a memory leak). Rich.

v2: https://www.redhat.com/archives/libguestfs/2018-December/msg00039.html v2 -> v3: - Fix all the issues raised in Eric's review. - Precompute some numbers to make the calculations easier. - Calculations now use bitshifts and masks in preference to division and modulo. - Clear existing bits before setting (which fixes a bug in the cache filter). Rich.

This adds a cache filter, which works like the COW filter in reverse. For realistic use it needs a bit more work, especially to add limits on the size of the cache, a more sensible cache replacement policy, and perhaps some kind of background worker to write dirty blocks out. Rich.

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

The cow and cache filters both use a bitmap mapping virtual disk blocks to status stored in the bitmap. The implementation of the bitmaps is very similar because one was derived from the other when the filters were implemented. The main difference is the cow filter uses a simple bitmap (one bit per block), whereas the cache filter uses two bits per block. This commit abstracts the bitmap

Here's the patch (on top of the preceeding one) which uses a bitmap instead of SEEK_DATA. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org --4VrXvz3cwkc87Wze

When compiling on armv7 it becomes clear from the compiler warnings that the current code is wrong. The bitmap has to be allocated in virtual memory, so use size_t to describe the length of the bitmap. When changing the length of the bitmap, compute the new size as an unsigned 64 bit int, and then check whether or not it is too large to fit into size_t before casting it. ---

The following commands: nbdkit -fv --filter=cow memory size=512 --run 'qemu-img info $nbd' nbdkit -fv --filter=cache memory size=512 --run 'qemu-img info $nbd' both fail with: nbdkit: memory[1]: error: realloc: Success Initial git bisect pointed to commit 3166d2bcbfd2 (but I don't believe that commit is the real cause, it merely exposes the bug). The reason this

v1 was here: https://www.redhat.com/archives/libguestfs/2019-September/msg00100.html In v2 I've changed the patch so it avoids calling realloc at all in this case. The patch is a bit longer this way. But I don't see any other alternative if we are to avoid having a "realloc wrapper" of some kind that we use everywhere, which I guess we should avoid because it makes plugins

These are essentially identical to what was previously posted as patches 6/9 through 9/9 here: https://www.redhat.com/archives/libguestfs/2018-December/msg00145.html except that it has been rebased onto the current git master and retested thoroughly. Rich.

On 12/2/18 10:33 AM, Richard W.M. Jones wrote: > The cow and cache filters both use a bitmap mapping virtual disk > blocks to status stored in the bitmap. The implementation of the > bitmaps is very similar because one was derived from the other when > the filters were implemented. > > The main difference is the cow filter uses a simple bitmap (one bit > per block), whereas

This patch series enhances the cache filter in a few ways, primarily adding a "cache-on-read" feature (similar to qemu's copyonread); and adding the ability to limit the cache size and the antecedent of that which is having a method to reclaim cache blocks. As the cache is stored as a sparse temporary file, reclaiming cache blocks simply means punching holes in the temporary file.

v1 was here: https://www.redhat.com/archives/libguestfs/2019-January/msg00008.html In v2 I have provided two patches: The first patch extends attribute((nonnull)) to most internal functions, but not to the external API. The second patch uses a macro so that attribute((format)) is only used in the public API on GCC or Clang. At least in theory these headers could be used by a C compiler which

v4: https://www.redhat.com/archives/libguestfs/2019-January/msg00032.html v5: - Now we set the block size at run time. I'd like to say that I was able to test this change, but unfortunately I couldn't find any easy way to create a filesystem on x86-64 with a block size > 4K. Ext4 doesn't support it at all, and XFS doesn't support block size > page size (and I

Patch 1 is the same as last time, except for a minor comment fix. Patch 2 should address everything that Eric mentioned in his review, and has been retested. Rich.

v3 was broken by a bad rebase, so let's forget about that one. Compared to v2: - Patch 1 is the same except for a minor comment change. - Patch 2 splits the reclaim code into a separate file (filters/cache/reclaim.c) - Addressed Eric's comments from his review of v2. - Retested on Linux and FreeBSD.

While working on can_FOO caching, at one point I got confused by whether 'readonly' meant the global set by -r or a local passed to .open(). A quick attempt to compile with -Wshadow found several other confusing points; this series gets us to the point that we can add -Wshadow to builds with --enable-gcc-warnings. Eric Blake (10): server: Avoid -Wshadow warnings guestfs: Avoid

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order