search for: blk_

...plan was to have a background thread doing the reclaim. > However that cannot work given the design of filters, because a > background thread cannot access the next_ops struct which is only > available during requests. > > Therefore we spread the work over the request threads. Each blk_* > function checks whether there is work to do, and if there is will > reclaim up to two blocks from the cache (thus ensuring that we always > make forward progress, since each blk_* function can only add at most > one block to the cache). > --- > +=head1 CACHE MAXIMUM SIZE >...

On 12/28/18 12:45 PM, Richard W.M. Jones wrote: > Make the implicit lock explicit, and hold it around blk_* operations. > This allows us to relax the thread model for the filter to > NBDKIT_THREAD_MODEL_PARALLEL. > --- > filters/cache/blk.h | 7 ++++++ > filters/cache/cache.c | 57 +++++++++++++++++++++++++++++++------------ > 2 files changed, 49 insertions(+), 15 deletions(-) Nic...

...4457031 2% fsstat fsinfo pathconf commit 48529 0% 31463 0% 0 0% 11339977 0% Output of iostat: Linux 2.4.27-3-686-smp (data.clm.net4all.ch) 21. 07. 06 cpu-moy: %user %nice %sys %iowait %idle 0,16 0,00 1,65 0,00 98,19 Device: tps Blk_lus/s Blk_?crits/s Blk_lus Blk_?crits scsi/host0/bus0/target0/lun0/disc 165,50 310,46 617,07 1800983948 3579692632 scsi/host0/bus0/target0/lun0/part1 165,48 309,32 617,06 1794404866 3579616544 scsi/host0/bus0/target0/lun0/part5...

Make the implicit lock explicit, and hold it around blk_* operations. This allows us to relax the thread model for the filter to NBDKIT_THREAD_MODEL_PARALLEL. --- filters/cache/blk.h | 7 ++++++ filters/cache/cache.c | 57 +++++++++++++++++++++++++++++++------------ 2 files changed, 49 insertions(+), 15 deletions(-) diff --git a/filters/cache/blk.h...

...w-filter.pod mv $@.t $@ endif -endif diff --git a/filters/cow/cow.c b/filters/cow/cow.c index 287c94e..2b023af 100644 --- a/filters/cow/cow.c +++ b/filters/cow/cow.c @@ -38,20 +38,22 @@ * takes up no space. * * We confine all pread/pwrite operations to the filesystem block - * size. The blk_read and blk_write functions below always happen on - * whole filesystem block boundaries. A smaller-than-block-size - * pwrite will turn into a read-modify-write of a whole block. We - * also assume that the plugin returns the same immutable data for - * each pread call we make, and optimize on t...

v3 was broken by a bad rebase, so let's forget about that one. Compared to v2: - Patch 1 is the same except for a minor comment change. - Patch 2 splits the reclaim code into a separate file (filters/cache/reclaim.c) - Addressed Eric's comments from his review of v2. - Retested on Linux and FreeBSD.

...ut perhaps you can have both styles, and provide a config knob for users to tune which of the two styles they want? > +++ b/filters/cow/cow.c > @@ -38,20 +38,22 @@ > * takes up no space. > * > * We confine all pread/pwrite operations to the filesystem block > - * size. The blk_read and blk_write functions below always happen on > - * whole filesystem block boundaries. A smaller-than-block-size > - * pwrite will turn into a read-modify-write of a whole block. We > - * also assume that the plugin returns the same immutable data for > - * each pread call we mak...

...plan was to have a background thread doing the reclaim. > However that cannot work given the design of filters, because a > background thread cannot access the next_ops struct which is only > available during requests. > > Therefore we spread the work over the request threads. Each blk_* > function checks whether there is work to do, and if there is will > reclaim up to two blocks from the cache (thus ensuring that we always > make forward progress, since each blk_* function can only add at most > one block to the cache). > --- > filters/cache/nbdkit-cache-filte...

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache (thus ensuring that we always make forward progress, since each blk_* function can only add at most one block to the cache). --- filters/cache/nbdkit-cache-filter.pod | 70 +++++++++++ filte...

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache (thus ensuring that we always make forward progress, since each blk_* function can only add at most one block to the cache). --- filters/cache/nbdkit-cache-filter.pod | 71 +++++++++++ filte...

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache (thus ensuring that we always make forward progress, since each blk_* function can only add at most one block to the cache). --- filters/cache/nbdkit-cache-filter.pod | 50 ++++++++ filters/...

v4: https://www.redhat.com/archives/libguestfs/2019-January/msg00032.html v5: - Now we set the block size at run time. I'd like to say that I was able to test this change, but unfortunately I couldn't find any easy way to create a filesystem on x86-64 with a block size > 4K. Ext4 doesn't support it at all, and XFS doesn't support block size > page size (and I

...the cache is stored as a sparse temporary file, reclaiming cache blocks simply means punching holes in the temporary file. The tricky bit is working out where to punch the holes to avoid reclaiming recently/frequently used blocks. I believe the status of the patch series is: [1/9] cache: Rename blk_writeback function. [2/9] cache: Add cache-on-read mode. [3/9] common/bitmap: Include <nbdkit-plugin.h>. [4/9] cache: Split out the block/bitmap code into - All of the above patches are pretty uncontroversial and ready for review and upstreaming. [5/9] cache: Allow this filter to serve r...

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache (thus ensuring that we always make forward progress, since each blk_* function can only add at most one block to the cache). --- filters/cache/nbdkit-cache-filter.pod | 50 ++++++ filters/ca...

These are essentially identical to what was previously posted as patches 6/9 through 9/9 here: https://www.redhat.com/archives/libguestfs/2018-December/msg00145.html except that it has been rebased onto the current git master and retested thoroughly. Rich.

Patch 1 is the same as last time, except for a minor comment fix. Patch 2 should address everything that Eric mentioned in his review, and has been retested. Rich.

The original plan was to have a background thread doing the reclaim. However that cannot work given the design of filters, because a background thread cannot access the next_ops struct which is only available during requests. Therefore we spread the work over the request threads. Each blk_* function checks whether there is work to do, and if there is will reclaim up to two blocks from the cache (thus ensuring that we always make forward progress, since each blk_* function can only add at most one block to the cache). Another large change is that the cache block size can no longer be...

...06007e..ae4a3af 100644 --- a/filters/cow/cow.c +++ b/filters/cow/cow.c @@ -51,8 +51,6 @@ #include "minmax.h" #include "rounding.h" -#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL - /* In order to handle parallel requests safely, this lock must be held * when calling any blk_* functions. */ diff --git a/filters/delay/delay.c b/filters/delay/delay.c index 486a24e..11681c8 100644 --- a/filters/delay/delay.c +++ b/filters/delay/delay.c @@ -40,8 +40,6 @@ #include <nbdkit-filter.h> -#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL - static int delay_read_ms = 0;...

...n the design and implementation of this filter: + * + * The filter works by creating a large, sparse temporary file, the + * same size as the underlying device. Being sparse, initially this + * takes up no space. + * + * We confine all pread/pwrite operations to the filesystem block + * size. The blk_read and blk_write functions below always happen on + * whole filesystem block boundaries. A smaller-than-block-size + * pwrite will turn into a read-modify-write of a whole block. We + * also assume that the plugin returns the same immutable data for + * each pread call we make, and optimize on t...