search for: posix_fadv_dontneed

The way we used posix_fadvise was wrong, and yet right! Rich.

Add wrappers around posix_fadvise and use them in places we were calling posix_fadvise directly before. Also in virt-builder we were doing this (and ignoring the result): posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); However the POSIX_FADV_* flags are _not_ bitmasks! In fact POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED == POSIX_FADV_NOREUSE so we were giving a completely different hint from what we thought we were giving. --- builder/pxzcat-c.c | 5 +-- src/guestfs-internal-frontend.h | 5 +++ src/...

...ice = S_ISBLK (statbuf.st_mode); h->sector_size = 4096; /* Start with safe guess */ @@ -329,6 +392,10 @@ file_pread (void *handle, void *buf, uint32_t count, uint64_t offset, uint32_t flags) { struct handle *h = handle; +#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_DONTNEED) + uint32_t orig_count = count; + uint64_t orig_offset = offset; +#endif while (count > 0) { ssize_t r = pread (h->fd, buf, count, offset); @@ -345,6 +412,12 @@ file_pread (void *handle, void *buf, uint32_t count, uint64_t offset, offset += r; } +#ifdef HAVE_POSIX_FADVIS...

...che=none. cache=advise? cache=dontneed? I can't think of a good name! > >@@ -355,6 +428,17 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, > > { > > struct handle *h = handle; > >+#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_DONTNEED) > >+ uint32_t orig_count = count; > >+ uint64_t orig_offset = offset; > >+ > >+ /* If cache=none we want to force pages we have just written to the > >+ * file to be flushed to disk so we can immediately evict them from > >+ * the page cache. > >+...

...? I would keep cache semantics similar to qemu. > > >@@ -355,6 +428,17 @@ file_pwrite (void *handle, const void *buf, > uint32_t count, uint64_t offset, > > > { > > > struct handle *h = handle; > > >+#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_DONTNEED) > > >+ uint32_t orig_count = count; > > >+ uint64_t orig_offset = offset; > > >+ > > >+ /* If cache=none we want to force pages we have just written to the > > >+ * file to be flushed to disk so we can immediately evict them from > > >+...

...hile there was at least one[1] attempt at making kernels' limited fadvise implementations work for rsync, it was quite awkward and it was understandably deemed inappropriate for merge. As a result of discussions with Minchan Kim and KOSAKI Motohiro, Linux has had reasonable support for handling POSIX_FADV_DONTNEED[3]. In particular, issuing this hint will compel the kernel to try reclaiming the affected pages more quickly, which should reduce memory pressure by dirty pages. The accompanying patches adds support for this hint when available. Currently DONTNEED is issued unconditionally although it could be c...

...-1) - error (EXIT_FAILURE, errno, "ftruncate: %s", outputfile); + if (fallocate (ofd, 0, 0, size) == -1) + error (EXIT_FAILURE, errno, "fallocate: %s", outputfile); /* Tell the kernel we won't read the output file. */ posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); /* Iterate over blocks. */ -- 1.8.4.1.563.g8e6fc32

...hes. > > We tried to use fadvice but it did not help. The only way to avoid such issues > is with O_SYNC or O_DIRECT. O_SYNC is much slower but this is the path > we took for now in this flow. I'm interested in more background about this, because while it is true that O_DIRECT and POSIX_FADV_DONTNEED are not exactly equivalent, I think I've shown here that DONTNEED can be used to avoid polluting the page cache. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you e...

...on. But in the meantime, cache=none is fairly nice while still avoiding O_DIRECT. > @@ -355,6 +428,17 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, > { > struct handle *h = handle; > > +#if defined (HAVE_POSIX_FADVISE) && defined (POSIX_FADV_DONTNEED) > + uint32_t orig_count = count; > + uint64_t orig_offset = offset; > + > + /* If cache=none we want to force pages we have just written to the > + * file to be flushed to disk so we can immediately evict them from > + * the page cache. > + */ > + if (cache_mode =...

...read (void *handle, void *buf, uint32_t count, uint64_t offset, #ifdef HAVE_POSIX_FADVISE /* On Linux this will evict the pages we just read from the page cache. */ - if (cache_mode == cache_none) + if (cache_mode == cache_writethrough) posix_fadvise (h->fd, orig_offset, orig_count, POSIX_FADV_DONTNEED); #endif @@ -441,11 +441,11 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, uint32_t orig_count = count; uint64_t orig_offset = offset; - /* If cache=none we want to force pages we have just written to the - * file to be flushed to disk so we can immedi...

...gned nr_threads) { int fd, ofd; + off_t hole_start, data_start; uint64_t size; lzma_index *idx; /* Open the file. */ fd = open (filename, O_RDONLY); @@ -176,10 +178,29 @@ xzfile_uncompress (const char *filename, const char *outputfile, posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); /* Iterate over blocks. */ iter_blocks (idx, nr_threads, filename, fd, outputfile, ofd); + /* discard ranges that were allocated but not written */ + data_start = 0; + while (data_start < size) { + hole_start = lseek (ofd, data_start, SEEK_HOLE); + if (hole_start == (off_t) -1...

...OS/Version: Linux Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: ted at midg3t.net QAContact: rsync-qa at samba.org It would be good if rsync used posix_fadvise with POSIX_FADV_DONTNEED to notify the kernel when it no longer needs file contents that have been synced to prevent useful cache contents being dropped in favor of never-reused rsynced files. Tobias Oetiker wrote a patch & documentation about this: http://insights.oetiker.ch/linux/fadvise.html -- Configure bugma...

https://bugzilla.samba.org/show_bug.cgi?id=7876 Summary: please implement o_direct Product: rsync Version: 3.1.0 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: costinel at gmail.com QAContact:

...exia.org/?p=pxzcat.git;a=commitdiff;h=68640d56b2ea96401a1355ab56603b0837058d21 http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=05f0de58de6cbcbdc40f5a661d406b3fbe5a9060 > > /* Tell the kernel we won't read the output file. */ > > posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); > > I know this isn't related to your patch, > but I don't think you can set a flag like this. > These are rather operations that need to be done > while writing I think. > > For ref I made the above call after writing various chunks to disk in: > https://github....

--- src/inspect-apps.c | 13 ++++++++++++- src/inspect-fs-windows.c | 6 ++++++ src/journal.c | 5 +++++ 3 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/inspect-apps.c b/src/inspect-apps.c index 20cf00a..8fbae9c 100644 --- a/src/inspect-apps.c +++ b/src/inspect-apps.c @@ -35,11 +35,22 @@ #include <sys/endian.h> #endif -/* be32toh is usually a macro

On Monday, February 10, 2020 11:57 AM, Tetsuo Handa wrote: > Then, "node-A's NR_FILE_PAGES is already 0 and node-B's NR_FILE_PAGES is > not 0, but allocation request which triggered this shrinker wants to allocate > from only node-A" > would be confused by this change, for the pagecache pages for allocating > thread's interested node are already depleted but

On Monday, February 10, 2020 11:57 AM, Tetsuo Handa wrote: > Then, "node-A's NR_FILE_PAGES is already 0 and node-B's NR_FILE_PAGES is > not 0, but allocation request which triggered this shrinker wants to allocate > from only node-A" > would be confused by this change, for the pagecache pages for allocating > thread's interested node are already depleted but

...uint32_t count, uint64_t offset, > > #ifdef HAVE_POSIX_FADVISE > /* On Linux this will evict the pages we just read from the page cache. */ > - if (cache_mode == cache_none) > + if (cache_mode == cache_writethrough) > posix_fadvise (h->fd, orig_offset, orig_count, POSIX_FADV_DONTNEED); > #endif > > @@ -441,11 +441,11 @@ file_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset, > uint32_t orig_count = count; > uint64_t orig_offset = offset; > > - /* If cache=none we want to force pages we have just written to the > - * fil...

...d not help. The only way to avoid such issues > > > is with O_SYNC or O_DIRECT. O_SYNC is much slower but this is the path > > > we took for now in this flow. > > > > I'm interested in more background about this, because while it is true > > that O_DIRECT and POSIX_FADV_DONTNEED are not exactly equivalent, I > > think I've shown here that DONTNEED can be used to avoid polluting the > > page cache. > > This fixes the minor issue of polluting the page cache, but it does not help > to avoid stale data in the cache, or unconfrolled flushes. > >...

...ormal I/O requests." ;-) > > Didn't this one. The discussion was about guest pagecache pages v.s. guest balloon pages. > Why is host's pagecache here? I'm expecting a mode: "Guests should try to minimize pagecache pages (and teach host to treat reclaimed pages as if POSIX_FADV_DONTNEED) instead of managing guest balloon pages". In other words, as if while :; sleep 5; echo 1 > /proc/sys/vm/drop_caches; done is running in the guest's kernel. And as if echo 2 > /proc/sys/vm/drop_caches is triggered in the guest's kernel when host requested guests to reclai...