search for: fadv_dontneed

...here was at least one[1] attempt at making kernels' limited fadvise implementations work for rsync, it was quite awkward and it was understandably deemed inappropriate for merge. As a result of discussions with Minchan Kim and KOSAKI Motohiro, Linux has had reasonable support for handling POSIX_FADV_DONTNEED[3]. In particular, issuing this hint will compel the kernel to try reclaiming the affected pages more quickly, which should reduce memory pressure by dirty pages. The accompanying patches adds support for this hint when available. Currently DONTNEED is issued unconditionally although it could be c...

Use the FADV_DONTNEED fadvise hint after finishing reading an origin fd in the sender. --- sender.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/sender.c b/sender.c index 59dae7d..a934bfe 100644 --- a/sender.c +++ b/sender.c @@ -338,6 +338,12 @@ void send_files(int f_in, int f_out) i...

Use the FADV_DONTNEED fadvise hint after finishing writing to a destinataion fd in the receiver. --- receiver.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/receiver.c b/receiver.c index 39c5e49..33b21fb 100644 --- a/receiver.c +++ b/receiver.c @@ -721,6 +721,12 @@ int recv_files(int f_...

...terface for this, posix_fadvise, has existed for some time, but there has been no useable implementation in any of the major operating systems. Attempts have been made in the past[1] to use the fadvise interface, but kernel limitations have made this quite messy. In particular, the kernel supports FADV_DONTNEED as a single-shot hint; i.e. if the page is clean when the hint is given it will be freed, otherwise the hint is ignored. For this reason it is necessary to fdatasync() against dirtied pages before giving the hint. This, however, requires that rsync do some accounting, calling fdatasync() and fadvis...

...50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern. It looks like a patch[1] was written (although never merged, it seems) incorporating fadvise support, but I found its implementation rather odd, using mincore() and FADV_DONTNEED to kick out only regions brought in by rsync. It seemed to me the simpler and more appropriate solution would be to simply flag every touched file with FADV_NOREUSE and let the kernel manage automatically expelling used pages. After looking deeper into the kernel implementation[2] of fadvise() the...

...error (EXIT_FAILURE, errno, "ftruncate: %s", outputfile); + if (fallocate (ofd, 0, 0, size) == -1) + error (EXIT_FAILURE, errno, "fallocate: %s", outputfile); /* Tell the kernel we won't read the output file. */ posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); /* Iterate over blocks. */ -- 1.8.4.1.563.g8e6fc32

With recent discussion on the LKML[1], it seems likely that Linux will finally support posix_fadvise in a useful way with the FADV_DONTNEED flag. This should allow us to minimize the effect of rsync on the system's working set. Add the necessary wrapper to syscall.c. [1] http://lkml.org/lkml/2010/11/21/59 --- syscall.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/syscall.c b/syscall.c index...

...but also problematic if we don't actually need that space due to sparseness. Do we get prior knowledge of the sparseness to be able to fallocate() and appropriate amount? > /* Tell the kernel we won't read the output file. */ > posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); I know this isn't related to your patch, but I don't think you can set a flag like this. These are rather operations that need to be done while writing I think. For ref I made the above call after writing various chunks to disk in: https://github.com/pixelb/dvd-vr/blob/master/dvd-vr.c#L...

...rg/?p=pxzcat.git;a=commitdiff;h=68640d56b2ea96401a1355ab56603b0837058d21 http://git.annexia.org/?p=pxzcat.git;a=commitdiff;h=05f0de58de6cbcbdc40f5a661d406b3fbe5a9060 > > /* Tell the kernel we won't read the output file. */ > > posix_fadvise (fd, 0, 0, POSIX_FADV_RANDOM|POSIX_FADV_DONTNEED); > > I know this isn't related to your patch, > but I don't think you can set a flag like this. > These are rather operations that need to be done > while writing I think. > > For ref I made the above call after writing various chunks to disk in: > https://github....