similar to: builder: posix_fadvise fixes.

More posix_fadvise stuff, and document what Linux really does with these calls. Also fixes a nasty bug in virt-builder. Rich.

Add wrappers around posix_fadvise and use them in places we were calling posix_fadvise directly before. Also in virt-builder we were doing this (and ignoring the result): posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); However the POSIX_FADV_* flags are _not_ bitmasks! In fact POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED == POSIX_FADV_NOREUSE so we were giving a completely

--- src/inspect-apps.c | 13 ++++++++++++- src/inspect-fs-windows.c | 6 ++++++ src/journal.c | 5 +++++ 3 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/inspect-apps.c b/src/inspect-apps.c index 20cf00a..8fbae9c 100644 --- a/src/inspect-apps.c +++ b/src/inspect-apps.c @@ -35,11 +35,22 @@ #include <sys/endian.h> #endif -/* be32toh is usually a macro

--- pxzcat.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pxzcat.c b/pxzcat.c index 4ab8689..9bcdc36 100644 --- a/pxzcat.c +++ b/pxzcat.c @@ -29,10 +29,11 @@ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ +#define _GNU_SOURCE #include <config.h>

On Wed, Oct 23, 2013 at 03:38:30AM +0100, Pádraig Brady wrote: [...] By the way, Eric Sandeen solved the problem. It's a genuine misfeature in ext4 called auto_da_alloc which causes a flush on close if the file has been truncated (ftruncate or O_TRUNC) and the file size is zero bytes. I added these patches which work around the issue:

--- pxzcat.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/pxzcat.c b/pxzcat.c index 9bcdc36..55ccfc0 100644 --- a/pxzcat.c +++ b/pxzcat.c @@ -44,10 +44,11 @@ #include <sys/types.h> #include <error.h> #include <errno.h> #include <getopt.h> #include <pthread.h> +#include <linux/falloc.h> #include <lzma.h> #define DEBUG

You can use these flags as described in the manual page to optimize access patterns, and to get better behaviour with the page cache in some scenarios. For my testing I used the cachedel and cachestats utilities written by Julius Plenz (https://github.com/Feh/nocache). I started with a 32 GB file of random data on a machine with about 32 GB of RAM. At the beginning of the test I evicted the

See http://rwmj.wordpress.com/2014/11/23/mapping-files-to-disk/

Does *not* incorporate changes suggested by Pino yet. Rich.

On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd "json:{\"file.driver\":\"null-co\",\"file.size\":\"1E\"}"' ; free -m ; cachestats /var/tmp/random > > Hmm - the -W actually says that qemu-img is

The new cache=none mode is misleading since it does not avoid usage of the page cache. When using shared storage, we may get stale data from the page cache. When writing, we flush after every write which is inefficient and unneeded. Rename the cache modes to: - writeback - write complete when the system call returned, and the data was copied to the page cache. - writethrough - write completes

https://bugzilla.samba.org/show_bug.cgi?id=7004 Summary: Use posix_fadvise to free cached file contents when done Product: rsync Version: 3.0.6 Platform: All OS/Version: Linux Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: ted at

--- builder/pxzcat-c.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builder/pxzcat-c.c b/builder/pxzcat-c.c index 0bbd296..dec9cc2 100644 --- a/builder/pxzcat-c.c +++ b/builder/pxzcat-c.c @@ -214,8 +214,10 @@ pxzcat (value filenamev, value outputfilev, unsigned nr_threads) unix_error (err, (char *) "ftruncate", outputfilev); } +#if defined HAVE_POSIX_FADVISE /*

As discussed in the thread on validating $TERM, it would be good to have a common function to do this. This is such a function. The main advantage is it includes unit tests which the previous functions did not. Rich.

While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed

On Fri, Aug 07, 2020 at 04:43:12PM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > > I'm not sure if or even how we could ever do a robust O_DIRECT > > > > We can let the plugin an filter deal with that. The simplest solution is to > drop it on the user and require aligned requests. I mean this is very error

There's a lot of code in nbdkit-file-plugin which it would be nice to reuse elsewhere. One possible approach (as outlined here) is simply to move the file callbacks (like file.pread, file.pwrite, file.zero etc) to a new mini-library. They can then be consumed by other plugins fairly easily by doing: static void * foo_open (int readonly) { struct fileops *fops; int fd, flags; /*

This is a quick hack to experiment with parallel threading model in the python plugin. Changes: - Use aligned buffers to make it possible to use O_DIRECT. Using parallel I/O does not buy us much when using buffered I/O. pwrite() copies data to the page cache, and pread() reads data from the page cache. - Disable extents in the file plugin. This way we can compare it with the python

Needs some work still, see in particular the commit message for patch 3. Rich.

Since v1: - rework .can_cache to be tri-state, with default of no advertisement (ripple effect through other patches) - add a lot more patches in order to round out filter support And in the meantime, Rich pushed NBD_CMD_CACHE support into libnbd, so in theory we now have a way to test cache commands through the entire stack. Eric Blake (24): server: Internal hooks for implementing