search for: blk_write

...lock, err); if (r != -1) { @@ -319,6 +349,33 @@ cache_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, buf += n; count -= n; offset += n; + blknum++; + } + + /* Aligned body */ + while (count >= blksize) { + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + r = blk_write (next_ops, nxdata, blknum, buf, flags, err); + if (r == -1) + return -1; + + buf += blksize; + count -= blksize; + offset += blksize; + blknum++; + } + + /* Unaligned tail */ + if (count) { + assert (block); + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); + r = blk_rea...

...turn -1; + /* Do a read-modify-write operation on the current block. + * Hold the lock over the whole operation. + */ + pthread_mutex_lock (&lock); + r = blk_read (next_ops, nxdata, blknum, block, err); + if (r != -1) { + memcpy (&block[blkoffs], buf, n); + r = blk_write (next_ops, nxdata, blknum, block, flags, err); } - memcpy (&block[blkoffs], buf, n); - if (blk_write (next_ops, nxdata, blknum, block, flags, err) == -1) { + pthread_mutex_unlock (&lock); + if (r == -1) { free (block); return -1; } @@ -278,6 +297,7 @@ cac...

Based on Rich's review of my v1 that touched only cache.c, I have now tried to bring all three filters with alignment rounding in line with one another. There is definitely room for future improvements once we teach nbdkit to let filters and plugins advertise block sizes, but I'm hoping to get NBD_CMD_CACHE implemented first. Eric Blake (2): blocksize: Process requests in linear order

...on the current block. * Hold the lock over the whole operation. */ - pthread_mutex_lock (&lock); + ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); r = blk_read (next_ops, nxdata, blknum, block, err); if (r != -1) { memcpy (&block[blkoffs], buf, n); r = blk_write (next_ops, nxdata, blknum, block, flags, err); } - pthread_mutex_unlock (&lock); if (r == -1) return -1; @@ -371,13 +370,12 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata, /* Do a read-modify-write operation on the current block. * Hold the lock ov...

...che_pwrite (struct nbdkit_next_ops *next_ops, void *nxdata, * Hold the lock over the whole operation. */ ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock); - r = blk_read (next_ops, nxdata, blknum, block, err); - if (r != -1) { - memcpy (&block[blkoffs], buf, n); - r = blk_write (next_ops, nxdata, blknum, block, flags, err); + if (blkoffs || n < blksize) { + r = blk_read (next_ops, nxdata, blknum, block, err); + if (r != -1) { + memcpy (&block[blkoffs], buf, n); + r = blk_write (next_ops, nxdata, blknum, block, flags, err); + } }...

...mv $@.t $@ endif -endif diff --git a/filters/cow/cow.c b/filters/cow/cow.c index 287c94e..2b023af 100644 --- a/filters/cow/cow.c +++ b/filters/cow/cow.c @@ -38,20 +38,22 @@ * takes up no space. * * We confine all pread/pwrite operations to the filesystem block - * size. The blk_read and blk_write functions below always happen on - * whole filesystem block boundaries. A smaller-than-block-size - * pwrite will turn into a read-modify-write of a whole block. We - * also assume that the plugin returns the same immutable data for - * each pread call we make, and optimize on this basis. + * siz...

Eric, you'll probably find the design "interesting" ... It does work, for me at least. Rich.

...and implementation of this filter: + * + * The filter works by creating a large, sparse temporary file, the + * same size as the underlying device. Being sparse, initially this + * takes up no space. + * + * We confine all pread/pwrite operations to the filesystem block + * size. The blk_read and blk_write functions below always happen on + * whole filesystem block boundaries. A smaller-than-block-size + * pwrite will turn into a read-modify-write of a whole block. We + * also assume that the plugin returns the same immutable data for + * each pread call we make, and optimize on this basis. + * + *...

I do have a question about whether patch 2 is right, or whether I've exposed a bigger problem in the truncate (and possibly other) filter, but the rest seem fairly straightforward. Eric Blake (4): server: Check for pthread lock failures truncate: Factor out reading real_size under mutex plugins: Check for mutex failures filters: Check for mutex failures filters/cache/cache.c

...ops, void *nxdata, uint64_t blknum, uint8_t *block, int *err); +extern int blk_read (struct nbdkit_next_ops *next_ops, void *nxdata, + uint64_t blknum, uint8_t *block, int *err) + __attribute__((__nonnull__ (1, 2, 4, 5))); /* Write to the cache and the plugin. */ -extern int blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, uint64_t blknum, const uint8_t *block, uint32_t flags, int *err); +extern int blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, + uint64_t blknum, const uint8_t *block, +...

...the cache is stored as a sparse temporary file, reclaiming cache blocks simply means punching holes in the temporary file. The tricky bit is working out where to punch the holes to avoid reclaiming recently/frequently used blocks. I believe the status of the patch series is: [1/9] cache: Rename blk_writeback function. [2/9] cache: Add cache-on-read mode. [3/9] common/bitmap: Include <nbdkit-plugin.h>. [4/9] cache: Split out the block/bitmap code into - All of the above patches are pretty uncontroversial and ready for review and upstreaming. [5/9] cache: Allow this filter to serve reques...

v1 was here: https://www.redhat.com/archives/libguestfs/2019-January/msg00008.html In v2 I have provided two patches: The first patch extends attribute((nonnull)) to most internal functions, but not to the external API. The second patch uses a macro so that attribute((format)) is only used in the public API on GCC or Clang. At least in theory these headers could be used by a C compiler which

....errno_is_preserved = plugin_errno_is_preserved, .open = plugin_open, .prepare = plugin_prepare, .finalize = plugin_finalize, diff --git a/filters/cache/cache.c b/filters/cache/cache.c index 9473f2c..2ae6f01 100644 --- a/filters/cache/cache.c +++ b/filters/cache/cache.c @@ -292,6 +292,7 @@ blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, uint64_t blknum, const uint8_t *block) { off_t offset = blknum * BLKSIZE; + int r; nbdkit_debug ("cache: blk_writethrough block %" PRIu64 " (offset %" PRIu64 ")", @@ -3...

...u can have both styles, and provide a config knob for users to tune which of the two styles they want? > +++ b/filters/cow/cow.c > @@ -38,20 +38,22 @@ > * takes up no space. > * > * We confine all pread/pwrite operations to the filesystem block > - * size. The blk_read and blk_write functions below always happen on > - * whole filesystem block boundaries. A smaller-than-block-size > - * pwrite will turn into a read-modify-write of a whole block. We > - * also assume that the plugin returns the same immutable data for > - * each pread call we make, and optimize on...

Here's what I'm currently playing with; I'm not ready to commit anything until I rebase my FUA work on top of this, as I only want to break filter ABI once between releases. Eric Blake (2): backend: Rework internal/filter error return semantics filters: Add log filter TODO | 2 - docs/nbdkit-filter.pod | 84 +++++++-- docs/nbdkit.pod

...lru_set_recently_accessed (blknum); } return 0; } @@ -172,6 +181,7 @@ blk_read (struct nbdkit_next_ops *next_ops, void *nxdata, nbdkit_error ("pread: %m"); return -1; } + lru_set_recently_accessed (blknum); return 0; } } @@ -196,6 +206,7 @@ blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, return -1; bitmap_set_blk (&bm, blknum, BLOCK_CLEAN); + lru_set_recently_accessed (blknum); return 0; } @@ -222,6 +233,7 @@ blk_write (struct nbdkit_next_ops *next_ops, void *nxdata, return -1; } bitmap_set_blk...

...lru_set_recently_accessed (blknum); } return 0; } @@ -172,6 +181,7 @@ blk_read (struct nbdkit_next_ops *next_ops, void *nxdata, nbdkit_error ("pread: %m"); return -1; } + lru_set_recently_accessed (blknum); return 0; } } @@ -196,6 +206,7 @@ blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, return -1; bitmap_set_blk (&bm, blknum, BLOCK_CLEAN); + lru_set_recently_accessed (blknum); return 0; } @@ -222,6 +233,7 @@ blk_write (struct nbdkit_next_ops *next_ops, void *nxdata, return -1; } bitmap_set_blk...

Patch 1 is the same as last time, except for a minor comment fix. Patch 2 should address everything that Eric mentioned in his review, and has been retested. Rich.

v3 was broken by a bad rebase, so let's forget about that one. Compared to v2: - Patch 1 is the same except for a minor comment change. - Patch 2 splits the reclaim code into a separate file (filters/cache/reclaim.c) - Addressed Eric's comments from his review of v2. - Retested on Linux and FreeBSD.

...&bm, blknum, BLOCK_NOT_CACHED); + reclaim (); + nbdkit_debug ("cache: blk_read block %" PRIu64 " (offset %" PRIu64 ") is %s", blknum, (uint64_t) offset, state == BLOCK_NOT_CACHED ? "not cached" : @@ -193,6 +222,8 @@ blk_writethrough (struct nbdkit_next_ops *next_ops, void *nxdata, { off_t offset = blknum * BLKSIZE; + reclaim (); + nbdkit_debug ("cache: writethrough block %" PRIu64 " (offset %" PRIu64 ")", blknum, (uint64_t) offset); @@ -224,6 +255,8 @@ blk_write...