While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed to disk. This is an extremely poor trade-off in the case of a backup but the user-mode had no way to communicate this to the kernel as few POSIX implementations had working implementations of fadvise. While there was at least one[1] attempt at making kernels' limited fadvise implementations work for rsync, it was quite awkward and it was understandably deemed inappropriate for merge. As a result of discussions with Minchan Kim and KOSAKI Motohiro, Linux has had reasonable support for handling POSIX_FADV_DONTNEED[3]. In particular, issuing this hint will compel the kernel to try reclaiming the affected pages more quickly, which should reduce memory pressure by dirty pages. The accompanying patches adds support for this hint when available. Currently DONTNEED is issued unconditionally although it could be conditioned on a flag if this is deemed more inappropriate. Cheers, - Ben [1] http://marc.info/?l=rsync&m=128885034930933&w=2 [2] http://insights.oetiker.ch/linux/fadvise.html [3] http://kernel.opensuse.org/cgit/kernel/commit/?id=315601809d124d046abd6c3ffa346d0dbd7aa29d&ignorews=1
With recent discussion on the LKML[1], it seems likely that Linux will finally support posix_fadvise in a useful way with the FADV_DONTNEED flag. This should allow us to minimize the effect of rsync on the system's working set. Add the necessary wrapper to syscall.c. [1] http://lkml.org/lkml/2010/11/21/59 --- syscall.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/syscall.c b/syscall.c index c85f73e..bec9ca8 100644 --- a/syscall.c +++ b/syscall.c @@ -28,6 +28,7 @@ #ifdef HAVE_SYS_ATTR_H #include <sys/attr.h> #endif +#include <fcntl.h> extern int dry_run; extern int am_root; @@ -374,3 +375,13 @@ int do_utime(const char *fname, time_t modtime, UNUSED(uint32 mod_nsec)) #else #error Need utimes or utime function. #endif + +#if _XOPEN_SOURCE >= 600 +int do_fadvise(int fd, OFF_T offset, OFF_T len, int advise) +{ + return posix_fadvise(fd, offset, len, advise); +} +#else +#define do_fadvise() +#endif + -- 1.7.5.4
Ben Gamari
2012-Feb-18 00:54 UTC
[PATCH 2/3] Inform kernel of FADV_DONTNEED hint in sender
Use the FADV_DONTNEED fadvise hint after finishing reading an origin fd in the sender. --- sender.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/sender.c b/sender.c index 59dae7d..e442933 100644 --- a/sender.c +++ b/sender.c @@ -338,6 +338,12 @@ void send_files(int f_in, int f_out) if (do_progress) end_progress(st.st_size); + if (do_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) != 0) { + rsyserr(FERROR_XFER, errno, + "fadvise failed in sending %s", + full_fname(fname)); + } + log_item(log_code, file, &initial_stats, iflags, NULL); if (mbuf) { -- 1.7.5.4
Ben Gamari
2012-Feb-18 00:54 UTC
[PATCH 3/3] Inform kernel of FADV_DONTNEED hint in receiver
Use the FADV_DONTNEED fadvise hint after finishing writing to a destinataion fd in the receiver. --- receiver.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/receiver.c b/receiver.c index d90fa25..b151cf8 100644 --- a/receiver.c +++ b/receiver.c @@ -742,6 +742,12 @@ int recv_files(int f_in, char *local_name) recv_ok = receive_data(f_in, fnamecmp, fd1, st.st_size, fname, fd2, F_LENGTH(file)); + if (do_fadvise(fd2, 0, 0, POSIX_FADV_DONTNEED) != 0) { + rsyserr(FERROR_XFER, errno, + "fadvise failed in writing %s", + full_fname(fname)); + } + log_item(log_code, file, &initial_stats, iflags, NULL); if (fd1 != -1) -- 1.7.5.4
Ben Gamari <bgamari.foss at gmail.com> writes:> While going through an old todo list I found that these patches had fallen by > the way-side. About a year ago I initiated a discussion[1] with the Linux > kernel folks regarding the lack of any useable fadvise support on the kernel > side. As a result, I was observing extremely poor performance on my server > after backup as executable pages were being swapped out in favor of data > waiting to be flushed to disk. This is an extremely poor trade-off in the case > of a backup but the user-mode had no way to communicate this to the > kernel as few POSIX implementations had working implementations of > fadvise. >Any opinion on merging this? I'd be happy to port it forward if there are conflicts. Cheers, - Ben
Reasonably Related Threads
- [PATCH 2/3] Inform kernel of FADV_DONTNEED hint in sender
- [PATCH 3/3] Inform kernel of FADV_DONTNEED hint in receiver
- [RFC PATCH] fadvise support in rsync
- fadvise DONTNEED implementation (or lack thereof)
- [PATCH nbdkit] file: Implement cache=none and fadvise=normal|random|sequential.