similar to: DO NOT REPLY [Bug 7876] New: please implement o_direct

Hi List, I have updated my rsync fadvise patch which stops rsync from ousting all your other data from cache when running large jobs. I have also written an article about the whole issue. http://insights.oetiker.ch/linux/fadvise.html cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten http://it.oetiker.ch tobi@oetiker.ch ++41 62 213 9902

Warning for kernel folks: I'm not much of an mm person; let me know if I got anything horribly wrong. Many folks use rsync in their nightly backup jobs. In these applications, speed is of minimal concern and should be sacrificed in order to minimize the effect of rsync on the rest of the machine. When rsync is working on a large directory it can quickly fill the page cache with written data,

https://bugzilla.samba.org/show_bug.cgi?id=7004 Summary: Use posix_fadvise to free cached file contents when done Product: rsync Version: 3.0.6 Platform: All OS/Version: Linux Status: NEW Severity: enhancement Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: ted at

On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd "json:{\"file.driver\":\"null-co\",\"file.size\":\"1E\"}"' ; free -m ; cachestats /var/tmp/random > > Hmm - the -W actually says that qemu-img is

You can use these flags as described in the manual page to optimize access patterns, and to get better behaviour with the page cache in some scenarios. For my testing I used the cachedel and cachestats utilities written by Julius Plenz (https://github.com/Feh/nocache). I started with a 32 GB file of random data on a machine with about 32 GB of RAM. At the beginning of the test I evicted the

While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed

The new cache=none mode is misleading since it does not avoid usage of the page cache. When using shared storage, we may get stale data from the page cache. When writing, we flush after every write which is inefficient and unneeded. Rename the cache modes to: - writeback - write complete when the system call returned, and the data was copied to the page cache. - writethrough - write completes

I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern. It looks like a

Hi, We have file readahead to do asyn file read, but has no metadata readahead. For a list of files, their metadata is stored in fragmented disk space and metadata read is a sync operation, which impacts the efficiency of readahead much. The patches try to add meatadata readahead for btrfs. In btrfs, metadata is stored in btree_inode. Ideally, if we could hook the inode to a fd so we could use

I would appreciate the addition of this option, which would cause files opened by rsync to be opened using the O_DIRECT flag, on Linux, at least. Let me explain my circumstance: I use rsync to migration VMware disk images from one machine to another while the VM is running. This works really well: you take a snapshot of the VM, run rsync, take another snapshot, run it again, then finally suspend

On Fri, Aug 07, 2020 at 05:29:24PM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > These ones? > > https://www.redhat.com/archives/libguestfs/2020-August/msg00078.html > > No, we had a bug when copying image from glance caused sanlock timeouts > because of the unpredictable page cache flushes. > > We

Hi all, In a conversation between me, Milind and Csaba, Milind pointed out fadvise(2) [1] and its potential benefits to Glusterfs' caching translators like read-ahead etc. After discussing about it, we agreed that our performance translators can leverage the hints to provide better performance. Now the question is how commonly applications actually provide hints? Is it something that is used

Hi List, I am using rsync for hard-link backup. I found that there is a major problem with frequent backup filling up the file system cache with all the data from the files being backed up. The effect is that all the other 'sensible' data in the cache gets thrown out in the process. This is rather unfortunate as the performance of the system becomes very bad after running rsync. Some

On Sat, Aug 08, 2020 at 02:14:22AM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020 at 5:36 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > On Fri, Aug 07, 2020 at 05:29:24PM +0300, Nir Soffer wrote: > > > On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > > These ones? > > > >

On Fri, Aug 07, 2020 at 04:43:12PM +0300, Nir Soffer wrote: > On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > > I'm not sure if or even how we could ever do a robust O_DIRECT > > > > We can let the plugin an filter deal with that. The simplest solution is to > drop it on the user and require aligned requests. I mean this is very error

On Fri, Aug 7, 2020, 16:16 Richard W.M. Jones <rjones@redhat.com> wrote: > On Fri, Aug 07, 2020 at 07:53:13AM -0500, Eric Blake wrote: > > >$ free -m; time ./nbdkit file /var/tmp/random fadvise=sequential > cache=none --run 'qemu-img convert -n -p -m 16 -W $nbd >

This is a quick hack to experiment with parallel threading model in the python plugin. Changes: - Use aligned buffers to make it possible to use O_DIRECT. Using parallel I/O does not buy us much when using buffered I/O. pwrite() copies data to the page cache, and pread() reads data from the page cache. - Disable extents in the file plugin. This way we can compare it with the python

On 8/7/20 6:31 AM, Richard W.M. Jones wrote: > You can use these flags as described in the manual page to optimize > access patterns, and to get better behaviour with the page cache in > some scenarios. And if you guess wrong, it is only a performance penalty, not a correctness issue. > > For my testing I used the cachedel and cachestats utilities written by > Julius Plenz

On Fri, Aug 7, 2020 at 5:36 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > On Fri, Aug 07, 2020 at 05:29:24PM +0300, Nir Soffer wrote: > > On Fri, Aug 7, 2020 at 5:07 PM Richard W.M. Jones <rjones@redhat.com> wrote: > > > These ones? > > > https://www.redhat.com/archives/libguestfs/2020-August/msg00078.html > > > > No, we had a bug when

https://bugzilla.samba.org/show_bug.cgi?id=9560 Summary: drop-cache option Product: rsync Version: 3.0.9 Platform: All OS/Version: All Status: NEW Severity: normal Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy: colundrum at gmail.com QAContact: rsync-qa at