similar to: [RFC PATCH] fadvise support in rsync

I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern. It looks like a

While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed

Hi List, I have updated my rsync fadvise patch which stops rsync from ousting all your other data from cache when running large jobs. I have also written an article about the whole issue. http://insights.oetiker.ch/linux/fadvise.html cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten http://it.oetiker.ch tobi@oetiker.ch ++41 62 213 9902

Hi all, In a conversation between me, Milind and Csaba, Milind pointed out fadvise(2) [1] and its potential benefits to Glusterfs' caching translators like read-ahead etc. After discussing about it, we agreed that our performance translators can leverage the hints to provide better performance. Now the question is how commonly applications actually provide hints? Is it something that is used

On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd "json:{\"file.driver\":\"null-co\",\"file.size\":\"1E\"}"' ; free -m ; cachestats /var/tmp/random > > Hmm - the -W actually says that qemu-img is

You can use these flags as described in the manual page to optimize access patterns, and to get better behaviour with the page cache in some scenarios. For my testing I used the cachedel and cachestats utilities written by Julius Plenz (https://github.com/Feh/nocache). I started with a 32 GB file of random data on a machine with about 32 GB of RAM. At the beginning of the test I evicted the

On Sat, Aug 08, 2020 at 02:14:22AM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020 at 5:36 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > On Fri, Aug 07, 2020 at 05:29:24PM +0300, Nir Soffer wrote: > > > On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > These ones? > > > >

Use the FADV_DONTNEED fadvise hint after finishing writing to a destinataion fd in the receiver. --- receiver.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/receiver.c b/receiver.c index 39c5e49..33b21fb 100644 --- a/receiver.c +++ b/receiver.c @@ -721,6 +721,12 @@ int recv_files(int f_in, char *local_name) recv_ok = receive_data(f_in, fnamecmp, fd1,

On Fri, Aug 11, 2017 at 05:50:47PM +0530, Ravishankar N wrote: > > > On 08/11/2017 04:51 PM, Niels de Vos wrote: > > On Fri, Aug 11, 2017 at 12:47:47AM -0400, Raghavendra Gowdappa wrote: > > > Hi all, > > > > > > In a conversation between me, Milind and Csaba, Milind pointed out > > > fadvise(2) [1] and its potential benefits to Glusterfs'

Hi, We have file readahead to do asyn file read, but has no metadata readahead. For a list of files, their metadata is stored in fragmented disk space and metadata read is a sync operation, which impacts the efficiency of readahead much. The patches try to add meatadata readahead for btrfs. In btrfs, metadata is stored in btree_inode. Ideally, if we could hook the inode to a fd so we could use

On 08/11/2017 04:51 PM, Niels de Vos wrote: > On Fri, Aug 11, 2017 at 12:47:47AM -0400, Raghavendra Gowdappa wrote: >> Hi all, >> >> In a conversation between me, Milind and Csaba, Milind pointed out >> fadvise(2) [1] and its potential benefits to Glusterfs' caching >> translators like read-ahead etc. After discussing about it, we agreed >> that our

On Fri, Aug 11, 2017 at 12:47:47AM -0400, Raghavendra Gowdappa wrote: > Hi all, > > In a conversation between me, Milind and Csaba, Milind pointed out > fadvise(2) [1] and its potential benefits to Glusterfs' caching > translators like read-ahead etc. After discussing about it, we agreed > that our performance translators can leverage the hints to provide > better

On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential > cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd >

On Fri, Aug 07, 2020 at 05:29:24PM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > These ones? > > https://www.redhat.com/archives/libguestfs/2020-August/msg00078.html > > No, we had a bug when copying image from glance caused sanlock timeouts > because of the unpredictable page cache flushes. > > We

The new cache=none mode is misleading since it does not avoid usage of the page cache. When using shared storage, we may get stale data from the page cache. When writing, we flush after every write which is inefficient and unneeded. Rename the cache modes to: - writeback - write complete when the system call returned, and the data was copied to the page cache. - writethrough - write completes

Use the FADV_DONTNEED fadvise hint after finishing reading an origin fd in the sender. --- sender.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/sender.c b/sender.c index 59dae7d..a934bfe 100644 --- a/sender.c +++ b/sender.c @@ -338,6 +338,12 @@ void send_files(int f_in, int f_out) if (do_progress) end_progress(st.st_size); + if

https://bugzilla.samba.org/show_bug.cgi?id=7876 Summary: please implement o_direct Product: rsync Version: 3.1.0 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: costinel at gmail.com QAContact:

With recent discussion on the LKML[1], it seems likely that Linux will finally support posix_fadvise in a useful way with the FADV_DONTNEED flag. This should allow us to minimize the effect of rsync on the system's working set. Add the necessary wrapper to syscall.c. [1] http://lkml.org/lkml/2010/11/21/59 --- syscall.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)

Take 2... this version only adds code to discard the cache on Linux and uses the right name for the fadvise call. It''s a bit ugly - I basically just put #ifdef __linux__ around the fadvise calls - but I don''t have the ability to develop something suitable for Solaris... > Attached is a patch to unstable that stops save/restore/dump from > hosing > Dom0 when dealing

On Sat, Aug 19, 2017 at 4:27 PM, Csaba Henk <chenk at redhat.com> wrote: > Hi Niels, > > On Fri, Aug 11, 2017 at 2:33 PM, Niels de Vos <ndevos at redhat.com> wrote: > > On Fri, Aug 11, 2017 at 05:50:47PM +0530, Ravishankar N wrote: > [...] > >> To me it looks like fadvise (mm/fadvise.c) affects only the linux page > cache > >> behavior and is