samba-bugs at samba.org
2010-Dec-20 16:03 UTC
DO NOT REPLY [Bug 7876] New: please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 Summary: please implement o_direct Product: rsync Version: 3.1.0 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned at samba.org ReportedBy: costinel at gmail.com QAContact: rsync-qa at samba.org running nightly rsync leaves a lot of unneeded data in the filesystem cache. an "--o_direct" argument would solve this. thank you. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2011-Jan-25 11:33 UTC
DO NOT REPLY [Bug 7876] please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 ------- Comment #1 from samba-bugs at thequod.de 2011-01-25 05:33 CST -------
samba-bugs at samba.org
2011-Jan-25 19:35 UTC
DO NOT REPLY [Bug 7876] please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 ------- Comment #2 from samba-bugs at thequod.de 2011-01-25 13:35 CST ------- Created an attachment (id=6227) --> (https://bugzilla.samba.org/attachment.cgi?id=6227&action=view) Proposed patch This is a noobish attempt at getting this fixed. I have forgotten to diff it using "-p", but the context is "struct map_struct *map_file", where every file appears to get mapped through. Please note that this is a very amateurish attempt, but I would really like to get feedback to improve on this. Of course the best would be to have an official patch for this.. :) -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2011-Jan-25 20:30 UTC
DO NOT REPLY [Bug 7876] please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 ------- Comment #3 from wayned at samba.org 2011-01-25 14:30 CST ------- (In reply to comment #2)> Please note that this is a very amateurish attempt, but I would really like to > get feedback to improve on this.See the patch drop-cache.diff in older releases. There is also a newer version of the patch that someone else wrote. This won't be in the official release until OS support is better. At present the POSIX_FADV_DONTNEED option in linux does all kinds of weird things (or did at the time the patch was written by Tobi Oetiker), such as uncaching files that someone else put into the cache (so, if you copy a file that someone else is using heavily, it totally vanishes from the disk cache). I still say that a better solution than modifying various copy programs would be to have a helper app (ala nice, etc.) that would use a pre-loaded library to do whatever the current OS needs to do to make the file reading not flood the cache. Then it could be used with cp, tar, etc. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2011-Jan-25 21:10 UTC
DO NOT REPLY [Bug 7876] please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 ------- Comment #4 from samba-bugs at thequod.de 2011-01-25 15:10 CST ------- Thanks for your feedback! I had just found the patch by Tobi myself, and it is using mincore (now) to check if a file has been cached before rsync touched it (and then skip removing it from the cache). Post: http://insights.oetiker.ch/linux/fadvise.html Patch for 3.0.7: http://tobi.oetiker.ch/patches/rsync-3.0.7-fadvise.patch I agree that something like "nice" and "ionice" would be the best approach, and I find it rather unusual to not have something like that available already. I have found the following wrapper, which is using O_DIRECT however: http://arighi.blogspot.com/2007/04/how-to-bypass-buffer-cache-in-linux.html For what it's worth, here's a link to the POSIX_FADV_DONTNEED implementation in the current kernel: http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.37.y.git;a=blob;f=mm/fadvise.c;hb=refs/heads/master#l118 -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
samba-bugs at samba.org
2011-Jan-25 23:15 UTC
DO NOT REPLY [Bug 7876] please implement o_direct
https://bugzilla.samba.org/show_bug.cgi?id=7876 ------- Comment #5 from samba-bugs at thequod.de 2011-01-25 17:15 CST ------- Just for information: rsync builds using the patch are available from my PPA ("Lucid", works on Squeeze, too): https://launchpad.net/~blueyed/+archive/ppa -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug, or are watching the QA contact.
https://bugzilla.samba.org/show_bug.cgi?id=7876 --- Comment #6 from costin gusa <costinel at gmail.com> 2011-04-30 12:55:16 UTC --- (In reply to comment #4)> Thanks for your feedback! > > I had just found the patch by Tobi myself, and it is using mincore (now) to > check if a file has been cached before rsync touched it (and then skip removing > it from the cache). > > Post: http://insights.oetiker.ch/linux/fadvise.html > Patch for 3.0.7: http://tobi.oetiker.ch/patches/rsync-3.0.7-fadvise.patch > > I agree that something like "nice" and "ionice" would be the best approach, and > I find it rather unusual to not have something like that available already. > I have found the following wrapper, which is using O_DIRECT however: > http://arighi.blogspot.com/2007/04/how-to-bypass-buffer-cache-in-linux.html > > For what it's worth, here's a link to the POSIX_FADV_DONTNEED implementation in > the current kernel: > http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.37.y.git;a=blob;f=mm/fadvise.c;hb=refs/heads/master#l118To me as an end user is irrelevant what method is chosen to avoid filling up the cache. The LD_PRELOAD library wrapper solution looks like a more logic approach; however besides the mentioned blogspot article, are there out successful usage reports based on it? Second, if the answer to the previous question is "yes", how can I enable it remotely ? - My backup solution is a "pull" type - to avoid a bunch of servers potentially abuse the backup machine, only the backup machine connects to servers. So if I on the backup machine would be able to run rsync with LD_PRELOAD wrapper, how do I run the remote '-e ssh' rsync with the LD_PRELOAD wrapper? Thank you. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug.
https://bugzilla.samba.org/show_bug.cgi?id=7876 --- Comment #7 from Tomasz Chmielewski <mangoo at wpkg.org> 2011-06-15 14:47:25 UTC --- There is also this tool by Andrew Morton, which can be used with any arbitrary application: http://lwn.net/Articles/224653/ http://code.google.com/p/pagecache-mangagement/ Other than that, it would be of course useful to have such a "--direct" switch in rsync, as using LD_PRELOAD with remote servers may not be feasible. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the QA contact for the bug.
Possibly Parallel Threads
- Announce: rsync fadvise (cache dropping) patch updated
- [RFC PATCH] fadvise support in rsync
- DO NOT REPLY [Bug 7004] New: Use posix_fadvise to free cached file contents when done
- Re: [PATCH nbdkit] file: Implement cache=none and fadvise=normal|random|sequential.
- [PATCH nbdkit] file: Implement cache=none and fadvise=normal|random|sequential.