similar to: [2.2-UNSTABLE] compilation error: 'POSIX_FADV_WILLNEED' undeclared

Add wrappers around posix_fadvise and use them in places we were calling posix_fadvise directly before. Also in virt-builder we were doing this (and ignoring the result): posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); However the POSIX_FADV_* flags are _not_ bitmasks! In fact POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED == POSIX_FADV_NOREUSE so we were giving a completely

Writing "file-like" plugins is hard because you have to implement your own .zero, .trim, .extents, etc, and that is very complicated. However implementations of these functions already exist in the file plugin. By factoring out the file plugin into a separate "fileops" mini-library we can reuse these implementations in other plugins. This refactoring commit creates a new

You can use these flags as described in the manual page to optimize access patterns, and to get better behaviour with the page cache in some scenarios. For my testing I used the cachedel and cachestats utilities written by Julius Plenz (https://github.com/Feh/nocache). I started with a 32 GB file of random data on a machine with about 32 GB of RAM. At the beginning of the test I evicted the

The way we used posix_fadvise was wrong, and yet right! Rich.

On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential > cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd >

--- builder/pxzcat-c.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builder/pxzcat-c.c b/builder/pxzcat-c.c index 0bbd296..dec9cc2 100644 --- a/builder/pxzcat-c.c +++ b/builder/pxzcat-c.c @@ -214,8 +214,10 @@ pxzcat (value filenamev, value outputfilev, unsigned nr_threads) unix_error (err, (char *) "ftruncate", outputfilev); } +#if defined HAVE_POSIX_FADVISE /*

On Sat, Aug 08, 2020 at 01:24:02AM +0300, Nir Soffer wrote: > The new cache=none mode is misleading since it does not avoid usage of > the page cache. When using shared storage, we may get stale data from > the page cache. When writing, we flush after every write which is > inefficient and unneeded. > Hmm. This isn't actually what the cache=none parameter is currently doing.

On 8/7/20 6:31 AM, Richard W.M. Jones wrote: > You can use these flags as described in the manual page to optimize > access patterns, and to get better behaviour with the page cache in > some scenarios. And if you guess wrong, it is only a performance penalty, not a correctness issue. > > For my testing I used the cachedel and cachestats utilities written by > Julius Plenz

On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd "json:{\"file.driver\":\"null-co\",\"file.size\":\"1E\"}"' ; free -m ; cachestats /var/tmp/random > > Hmm - the -W actually says that qemu-img is

The new cache=none mode is misleading since it does not avoid usage of the page cache. When using shared storage, we may get stale data from the page cache. When writing, we flush after every write which is inefficient and unneeded. Rename the cache modes to: - writeback - write complete when the system call returned, and the data was copied to the page cache. - writethrough - write completes

Hello! I am writing a Linux server, and I want to pre-fetch a number of CIFS files and cache locally. The files are on a CIFS share mounted as read only. I was planning on opening the file, calling posix_fadvise(........,POSIX_FADV_WILLNEED) on the file descriptor, and then closing the file. Will this work? Is there a better way of doing this? Thanks!! Aaron

I added a new mail_prefetch_count setting to Dovecot v2.1 (http://hg.dovecot.org/dovecot-2.1/). Its main purpose is to avoid unnecessary latency while waiting for replies from remote server with imapc, i.e. instead of doing (FETCH, wait, FETCH, wait, ...) it can do (FETCH, FETCH, .., wait). For the same reason it's also going to be very useful once I add support for key-value databases / other

On Sun, Aug 9, 2020 at 12:28 AM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Sat, Aug 08, 2020 at 01:24:02AM +0300, Nir Soffer wrote: > > The new cache=none mode is misleading since it does not avoid usage of > > the page cache. When using shared storage, we may get stale data from > > the page cache. When writing, we flush after every write which is > >

More posix_fadvise stuff, and document what Linux really does with these calls. Also fixes a nasty bug in virt-builder. Rich.

There's a lot of code in nbdkit-file-plugin which it would be nice to reuse elsewhere. One possible approach (as outlined here) is simply to move the file callbacks (like file.pread, file.pwrite, file.zero etc) to a new mini-library. They can then be consumed by other plugins fairly easily by doing: static void * foo_open (int readonly) { struct fileops *fops; int fd, flags; /*

I'm still working my way through the filters before this series will be complete, but this is enough of a start to at least get some feedback on the idea of implementing another NBD protocol extension. Eric Blake (9): server: Internal hooks for implementing NBD_CMD_CACHE plugins: Add .cache callback file, split: Implement .cache with posix_fadvise nbd: Implement NBD_CMD_CACHE

https://bugzilla.samba.org/show_bug.cgi?id=7004 Summary: Use posix_fadvise to free cached file contents when done Product: rsync Version: 3.0.6 Platform: All OS/Version: Linux Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: ted at

On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Fri, Aug 07, 2020 at 04:43:12PM +0300, Nir Soffer wrote: > > On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > > > I'm not sure if or even how we could ever do a robust O_DIRECT > > > > > > > We can let the plugin an filter deal with

Needs some work still, see in particular the commit message for patch 3. Rich.

Since v1: - rework .can_cache to be tri-state, with default of no advertisement (ripple effect through other patches) - add a lot more patches in order to round out filter support And in the meantime, Rich pushed NBD_CMD_CACHE support into libnbd, so in theory we now have a way to test cache commands through the entire stack. Eric Blake (24): server: Internal hooks for implementing