search for: posix_fadvise

The way we used posix_fadvise was wrong, and yet right! Rich.

Add wrappers around posix_fadvise and use them in places we were calling posix_fadvise directly before. Also in virt-builder we were doing this (and ignoring the result): posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); However the POSIX_FADV_* flags are _not_ bitmasks! In fact POSIX_FADV_RANDOM|POSIX_FADV_DON...

https://bugzilla.samba.org/show_bug.cgi?id=7004 Summary: Use posix_fadvise to free cached file contents when done Product: rsync Version: 3.0.6 Platform: All OS/Version: Linux Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: wayned at samba.org Report...

More posix_fadvise stuff, and document what Linux really does with these calls. Also fixes a nasty bug in virt-builder. Rich.

...ad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2 \ -I/usr/local/include -MT fs-posix.lo -MD -MP -MF .deps/fs-posix.Tpo \ -c fs-posix.c -fPIC -DPIC -o .libs/fs-posix.o | fs-posix.c: In function 'fs_posix_prefetch': | fs-posix.c:298: warning: implicit declaration of function 'posix_fadvise' | fs-posix.c:298: error: 'POSIX_FADV_WILLNEED' undeclared (first use in this function) | fs-posix.c:298: error: (Each undeclared identifier is reported only once | fs-posix.c:298: error: for each function it appears in.) | gmake[3]: *** [fs-posix.lo] Error 1 | gmake[3]: Leaving directo...

...ache=dontneed? I can't think of a good name! > > > > Yes, don't call it none if you use the cache. > > How about advise=? > > I would keep cache semantics similar to qemu. qemu uses cache=none as a synonym for O_DIRECT, but AFAIK it has nothing that tries to use posix_fadvise(DONTNEED) with or without Linus's double buffering technique. qemu does use posix_fadvise(DONTNEED) in one place but AFAICT it is only used for live migration. ... > We already tried this with dd and the results were not good. These ones? https://www.redhat.com/archives/libguestfs/2020-Au...

...hich work around the issue: http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=68640d56b2ea96401a1355ab56603b0837058d21 http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=05f0de58de6cbcbdc40f5a661d406b3fbe5a9060 > > /* Tell the kernel we won't read the output file. */ > > posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); > > I know this isn't related to your patch, > but I don't think you can set a flag like this. > These are rather operations that need to be done > while writing I think. > > For ref I made the above call after writin...

...pends on your operating system, but for Linux using C<normal> causes > >+the kernel to read-ahead, C<sequential> causes the kernel to > >+read-ahead twice as much as C<normal>, and C<random> turns off > >+read-ahead. > > Is it worth a mention of L<posix_fadvise(3)> here, to let the user > get some idea of what their operating system supports? Yes I had this at one point but I seem to have dropped it. Will add it back, thanks. > >+=head2 Reducing evictions from the page cache > >+ > >+If the file is very large and you known the c...

...utputfile); - if (ftruncate (ofd, size) == -1) - error (EXIT_FAILURE, errno, "ftruncate: %s", outputfile); + if (fallocate (ofd, 0, 0, size) == -1) + error (EXIT_FAILURE, errno, "fallocate: %s", outputfile); /* Tell the kernel we won't read the output file. */ posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); /* Iterate over blocks. */ -- 1.8.4.1.563.g8e6fc32

...e=none + =head2 Files on tmpfs If you want to expose a file that resides on a file system known to diff --git a/plugins/file/file.c b/plugins/file/file.c index 076e7531..a8e37cd9 100644 --- a/plugins/file/file.c +++ b/plugins/file/file.c @@ -66,6 +66,18 @@ static char *filename = NULL; +/* posix_fadvise mode: -1 = don't set it, or POSIX_FADV_*. */ +static int fadvise_mode = +#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_NORMAL) + POSIX_FADV_NORMAL +#else + -1 +#endif + ; + +/* cache mode */ +static enum { cache_default, cache_none } cache_mode = cache_default; + /* Any ca...

...me! > > > > > > > Yes, don't call it none if you use the cache. > > > > How about advise=? > > > > I would keep cache semantics similar to qemu. > > qemu uses cache=none as a synonym for O_DIRECT, but AFAIK it has > nothing that tries to use posix_fadvise(DONTNEED) with or without > Linus's double buffering technique. Yes, this is the right way. posix_fadvise is not a replacement for O_DIRECT. > qemu does use > posix_fadvise(DONTNEED) in one place but AFAICT it is only used for > live migration. > > ... > > We already t...

...redhat.com> >> --- > >> diff --git a/configure.ac b/configure.ac >> index b6d60c3df6a1..62fe470b6cd5 100644 >> --- a/configure.ac >> +++ b/configure.ac >> @@ -132,11 +132,16 @@ dnl Check for various libc functions, all optional. >> dnl >> dnl posix_fadvise helps to optimise linear reads and writes. >> dnl >> +dnl When /proc/sys/kernel/core_pattern starts with a pipe (|) symbol, Linux >> +dnl ignores "ulimit -c" and (equivalent) setrlimit(RLIMIT_CORE) actions, for >> +dnl disabling core dumping. Only prctl() can be u...

...else if (strcmp (value, "writethrough") == 0) + cache_mode = cache_writethrough; else { nbdkit_error ("unknown cache mode: %s", value); return -1; @@ -423,7 +423,7 @@ file_pread (void *handle, void *buf, uint32_t count, uint64_t offset, #ifdef HAVE_POSIX_FADVISE /* On Linux this will evict the pages we just read from the page cache. */ - if (cache_mode == cache_none) + if (cache_mode == cache_writethrough) posix_fadvise (h->fd, orig_offset, orig_count, POSIX_FADV_DONTNEED); #endif @@ -441,11 +441,11 @@ file_pwrite (void *handle, const void...

...eloper to get their name in the Linux kernel :-) Feature additions like this have been done before by us, and we should continue where we can. It is a relatively easy entry for contributing to the Linux kernel. > [1] https://linux.die.net/man/2/fadvise As well as local man-pages for fadvise64/posix_fadvise. Showing that we have support for this, suggests that the filesystem becomes more mature and gains advanced features. This should impress users and might open up more interest for certain (HPC?) use-cases. Thanks, Niels > > regards, > Raghavendra > _________________________________...

I am using dovecot 2.2.13 on a VPS. One (only one!) of the users is experiencing IMAP disconnections. The maillog file is full of errors about corrupted transaction log and index files. The configurations follows: # 2.2.13: /etc/dovecot/dovecot.conf # OS: Linux 2.6.18-028stab107.1 x86_64 CentOS release 6.4 (Final) auth_cache_size = 1 k auth_mechanisms = plain login auth_username_chars =

With recent discussion on the LKML[1], it seems likely that Linux will finally support posix_fadvise in a useful way with the FADV_DONTNEED flag. This should allow us to minimize the effect of rsync on the system's working set. Add the necessary wrapper to syscall.c. [1] http://lkml.org/lkml/2010/11/21/59 --- syscall.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) di...

...system, but for Linux using C<normal> causes > > >+the kernel to read-ahead, C<sequential> causes the kernel to > > >+read-ahead twice as much as C<normal>, and C<random> turns off > > >+read-ahead. > > > > Is it worth a mention of L<posix_fadvise(3)> here, to let the user > > get some idea of what their operating system supports? > > Yes I had this at one point but I seem to have dropped it. Will > add it back, thanks. > > > >+=head2 Reducing evictions from the page cache > > >+ > > >+If the...

...char *outputfile, unsigned nr_threads) { int fd, ofd; + off_t hole_start, data_start; uint64_t size; lzma_index *idx; /* Open the file. */ fd = open (filename, O_RDONLY); @@ -176,10 +178,29 @@ xzfile_uncompress (const char *filename, const char *outputfile, posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); /* Iterate over blocks. */ iter_blocks (idx, nr_threads, filename, fd, outputfile, ofd); + /* discard ranges that were allocated but not written */ + data_start = 0; + while (data_start < size) { + hole_start = lseek (ofd, data_sta...

...adding a new fop to struct file_operations that is common across the entire VFS and likely won't fly with the kernel folks. I could be wrong in understanding all of this. :-) Regards, Ravi > >> [1] https://linux.die.net/man/2/fadvise > As well as local man-pages for fadvise64/posix_fadvise. > > Showing that we have support for this, suggests that the filesystem > becomes more mature and gains advanced features. This should impress > users and might open up more interest for certain (HPC?) use-cases. > > Thanks, > Niels > > >> regards, >> Raghave...

While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed