search for: bitmap_get_blk

...64_t new_size) return 0; } + +int64_t +bitmap_next (const struct bitmap *bm, uint64_t blk) +{ + uint64_t limit = bm->size * bm->ibpb; + const uint8_t *p; + + /* Align to the next byte boundary. */ + for (; blk < limit && (blk & (bm->ibpb-1)) != 0; ++blk) { + if (bitmap_get_blk (bm, blk, 0) != 0) + return blk; + } + if (blk == limit) + return -1; + + /* Now we're at a byte boundary we can use a fast string function to + * find the next non-zero byte. + */ + p = &bm->bitmap[blk >> (3 - bm->bitshift)]; + p = (const uint8_t *) next_non_ze...

These are essentially identical to what was previously posted as patches 6/9 through 9/9 here: https://www.redhat.com/archives/libguestfs/2018-December/msg00145.html except that it has been rebased onto the current git master and retested thoroughly. Rich.

...]; > +static unsigned c0 = 0, c1 = 0; Static variables already start life zero-initialized. > + > +void > +lru_set_recently_accessed (uint64_t blknum) > +{ > + /* If the block is already set in the first bitmap, don't need to do > + * anything. > + */ > + if (bitmap_get_blk (&bm[0], blknum, false)) > + return; And this implements the latter (the most recent N blocks accessed, not the most recent N accesses). > + > + bitmap_set_blk (&bm[0], blknum, true); > + c0++; > + > + /* If we've reached N/2 then we need to swap over the bitma...

...mp;bm[1], new_size) == -1) + return -1; + + /* XXX Choose this better. */ + N = MAX (new_size / BLKSIZE / 4, 100); + + return 0; +} + +void +lru_set_recently_accessed (uint64_t blknum) +{ + /* If the block is already set in the first bitmap, don't need to do + * anything. + */ + if (bitmap_get_blk (&bm[0], blknum, false)) + return; + + bitmap_set_blk (&bm[0], blknum, true); + c0++; + + /* If we've reached N/2 then we need to swap over the bitmaps. */ + if (c0 >= N/2) { + struct bitmap tmp; + + tmp = bm[0]; + bm[0] = bm[1]; + bm[1] = tmp; + c1 = c0; + +...

...mp;bm[1], new_size) == -1) + return -1; + + /* XXX Choose this better. */ + N = MAX (new_size / BLKSIZE / 4, 100); + + return 0; +} + +void +lru_set_recently_accessed (uint64_t blknum) +{ + /* If the block is already set in the first bitmap, don't need to do + * anything. + */ + if (bitmap_get_blk (&bm[0], blknum, false)) + return; + + bitmap_set_blk (&bm[0], blknum, true); + c0++; + + /* If we've reached N/2 then we need to swap over the bitmaps. */ + if (c0 >= N/2) { + struct bitmap tmp; + + tmp = bm[0]; + bm[0] = bm[1]; + bm[1] = tmp; + c1 = c0; + +...

...the virtual disk size in bytes. + * Returns -1 on error, setting nbdkit_error. + */ +extern int bitmap_resize (struct bitmap *bm, uint64_t new_size); + +/* Return the bit(s) associated with the given block. + * If the request is out of range, returns the default value. + */ +static inline unsigned +bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) +{ + uint64_t blk_offset = blk / (8 / bm->bpb); + unsigned blk_bit = bm->bpb * (blk % (8 / bm->bpb)); + unsigned mask = (1 << bm->bpb) - 1; + + if (blk_offset >= bm->size) { + nbdkit_debug ("bitmap_get: b...

This is exactly the same as v1: https://www.redhat.com/archives/libguestfs/2018-December/msg00004.html except that it now frees the bitmap on unload (which the old code did not - there was always a memory leak). Rich.

...the virtual disk size in bytes. + * Returns -1 on error, setting nbdkit_error. + */ +extern int bitmap_resize (struct bitmap *bm, uint64_t new_size); + +/* Return the bit(s) associated with the given block. + * If the request is out of range, returns the default value. + */ +static inline unsigned +bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) +{ + uint64_t blk_offset = blk / (8 / bm->bpb); + unsigned blk_bit = bm->bpb * (blk % (8 / bm->bpb)); + unsigned mask = (1 << bm->bpb) - 1; + + if (blk_offset >= bm->size) { + nbdkit_debug ("bitmap_get: b...

...\ + unsigned blk_bit = (bm)->bpb * ((blk) & ((bm)->ibpb - 1)); \ + unsigned mask = ((1 << (bm)->bpb) - 1) << blk_bit + +/* Return the bit(s) associated with the given block. + * If the request is out of range, returns the default value. + */ +static inline unsigned +bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) +{ + BITMAP_OFFSET_BIT_MASK (bm, blk); + + if (blk_offset >= bm->size) { + nbdkit_debug ("bitmap_get: block number is out of range"); + return default_; + } + + return (bm->bitmap[blk_offset] & mask) >&gt...

I have some patches I'm working on to fix the cache filter. However this is a prelude. It should be simply pure refactoring. All tests pass still. Rich.

v2: https://www.redhat.com/archives/libguestfs/2018-December/msg00039.html v2 -> v3: - Fix all the issues raised in Eric's review. - Precompute some numbers to make the calculations easier. - Calculations now use bitshifts and masks in preference to division and modulo. - Clear existing bits before setting (which fixes a bug in the cache filter). Rich.

This patch series enhances the cache filter in a few ways, primarily adding a "cache-on-read" feature (similar to qemu's copyonread); and adding the ability to limit the cache size and the antecedent of that which is having a method to reclaim cache blocks. As the cache is stored as a sparse temporary file, reclaiming cache blocks simply means punching holes in the temporary file.

Should we use attribute((nonnull)) at all? There's a very interesting history of this in libvirt -- try looking at commit eefb881 plus the commits referencing eefb881 -- but it does seem to work for me using recent GCC and Clang. I only did a few functions because annotating them gets old quickly... Rich.

Patch 1 is the same as last time, except for a minor comment fix. Patch 2 should address everything that Eric mentioned in his review, and has been retested. Rich.

...memset (bm->bitmap, 0, bm->size); @@ -125,7 +126,7 @@ bitmap_clear (struct bitmap *bm) /* Return the bit(s) associated with the given block. * If the request is out of range, returns the default value. */ -static inline unsigned +static inline unsigned __attribute__((__nonnull__ (1))) bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) { BITMAP_OFFSET_BIT_MASK (bm, blk); @@ -139,7 +140,7 @@ bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) } /* As above but works with virtual disk offset in bytes. */ -static inline unsigned +static inline u...

v1 was here: https://www.redhat.com/archives/libguestfs/2019-January/msg00008.html In v2 I have provided two patches: The first patch extends attribute((nonnull)) to most internal functions, but not to the external API. The second patch uses a macro so that attribute((format)) is only used in the public API on GCC or Clang. At least in theory these headers could be used by a C compiler which

v3 was broken by a bad rebase, so let's forget about that one. Compared to v2: - Patch 1 is the same except for a minor comment change. - Patch 2 splits the reclaim code into a separate file (filters/cache/reclaim.c) - Addressed Eric's comments from his review of v2. - Retested on Linux and FreeBSD.

...rns -1 on error, setting nbdkit_error. > + */ > +extern int bitmap_resize (struct bitmap *bm, uint64_t new_size); > + > +/* Return the bit(s) associated with the given block. > + * If the request is out of range, returns the default value. > + */ > +static inline unsigned > +bitmap_get_blk (const struct bitmap *bm, uint64_t blk, unsigned default_) > +{ > + uint64_t blk_offset = blk / (8 / bm->bpb); > + unsigned blk_bit = bm->bpb * (blk % (8 / bm->bpb)); Would using << and >> instead of / and % aid the compiler, since we know we have powers of 2 based...

v4: https://www.redhat.com/archives/libguestfs/2019-January/msg00032.html v5: - Now we set the block size at run time. I'd like to say that I was able to test this change, but unfortunately I couldn't find any easy way to create a filesystem on x86-64 with a block size > 4K. Ext4 doesn't support it at all, and XFS doesn't support block size > page size (and I

...2, +}; + +static enum reclaim_state reclaiming = NOT_RECLAIMING; +static uint64_t reclaim_blk; +#endif + +static void reclaim (void); + int blk_init (void) { @@ -146,6 +173,8 @@ blk_read (struct nbdkit_next_ops *next_ops, void *nxdata, off_t offset = blknum * BLKSIZE; enum bm_entry state = bitmap_get_blk (&bm, blknum, BLOCK_NOT_CACHED); + reclaim (); + nbdkit_debug ("cache: blk_read block %" PRIu64 " (offset %" PRIu64 ") is %s", blknum, (uint64_t) offset, state == BLOCK_NOT_CACHED ? "not cached" : @@ -193,6 +222,8 @...