similar to: [PATCH 3/3] Inform kernel of FADV_DONTNEED hint in receiver

Use the FADV_DONTNEED fadvise hint after finishing reading an origin fd in the sender. --- sender.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) diff --git a/sender.c b/sender.c index 59dae7d..a934bfe 100644 --- a/sender.c +++ b/sender.c @@ -338,6 +338,12 @@ void send_files(int f_in, int f_out) if (do_progress) end_progress(st.st_size); + if

While going through an old todo list I found that these patches had fallen by the way-side. About a year ago I initiated a discussion[1] with the Linux kernel folks regarding the lack of any useable fadvise support on the kernel side. As a result, I was observing extremely poor performance on my server after backup as executable pages were being swapped out in favor of data waiting to be flushed

Some background: I have been assigned a task to implement one-way synchronization of a large file storage to another box. The problem is that on the target box a special directory has to be maintained in which symlinks to received files should be created using special logic. After some experimentation I have set up a named pipe which is used by the receiving rsync daemon for transfer logging;

This mode is useful when a process is monitoring the log for post-processing of transferred files. With --log-after in local mode, both sender and receiver log to the same log file, so it require --log-file with absolute path. We add %o to the default log format, so it will be easy to tell the logs of the sender from the logs of the receiver: 2023/02/14 14:40:25 [559755] building file list

With recent discussion on the LKML[1], it seems likely that Linux will finally support posix_fadvise in a useful way with the FADV_DONTNEED flag. This should allow us to minimize the effect of rsync on the system's working set. Add the necessary wrapper to syscall.c. [1] http://lkml.org/lkml/2010/11/21/59 --- syscall.c | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-)

fallocate() is linux specific and will preallocate the space on disk for the entire file. FALLOC_FL_KEEP_SIZE does not change the filesize as reported by stat(). An aborted transfer will have preallocated disk space which is not "visible" via stat(). This shouldn't matter unless the user does complet his transfer. An alternative would be to use ftruncate() and shorten the file to the

Warning for kernel folks: I'm not much of an mm person; let me know if I got anything horribly wrong. Many folks use rsync in their nightly backup jobs. In these applications, speed is of minimal concern and should be sacrificed in order to minimize the effect of rsync on the rest of the machine. When rsync is working on a large directory it can quickly fill the page cache with written data,

seems the behavior of rsync has changed when dealing with output and using both -v and -q at the same time ... for example: $ mkdir test1 $ touch test1/foo $ rsync-2.6.0 -avq test1 test2 $ rm -r test2 $ rsync-2.6.8 -avq test1 test2 test1/ test1/fo $ rm -r test2 $ rsync-cvs -avq test1 test2 building file list ... test1/ test1/fo $ rm -r test2 the new output in 2.6.8 comes from the calls to

I've recently been trying to track down the root cause of my server's persistent issue of thrashing horribly after being left inactive. It seems that the issue is likely my nightly backup schedule (using rsync) which traverses my entire 50GB home directory. I was surprised to find that rsync does not use fadvise to notify the kernel of its use-once data usage pattern. It looks like a

https://bugzilla.samba.org/show_bug.cgi?id=13526 Bug ID: 13526 Summary: Hard link creation time Product: rsync Version: 3.1.3 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter:

On 10/22/2013 05:56 PM, Gabriel de Perthuis wrote: > --- > pxzcat.c | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > diff --git a/pxzcat.c b/pxzcat.c > index 4ab8689..9bcdc36 100644 > --- a/pxzcat.c > +++ b/pxzcat.c > @@ -29,10 +29,11 @@ > * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT > * OF THE USE OF THIS SOFTWARE,

https://bugzilla.samba.org/show_bug.cgi?id=11683 Bug ID: 11683 Summary: hang on select when send many files Product: rsync Version: 3.1.2 Hardware: All OS: All Status: NEW Severity: normal Priority: P5 Component: core Assignee: wayned at samba.org Reporter: tom916 at

On Wed, Oct 23, 2013 at 03:38:30AM +0100, Pádraig Brady wrote: [...] By the way, Eric Sandeen solved the problem. It's a genuine misfeature in ext4 called auto_da_alloc which causes a flush on close if the file has been truncated (ftruncate or O_TRUNC) and the file size is zero bytes. I added these patches which work around the issue:

Similar to --omit-{dir,link}-times: --omit-device-times omit device files from --times --omit-special-times omit sockets and fifos from --times Also, fix corner case that allows --omit-dir-times to be ignored. See unchanged_attrs() and recv_generator()'s call to try_dests_non(). Marc. diff -aNpRruz -X /etc/diff.excludes rsync-3.2.7/generator.c devel-3.2.7/generator.c ---

https://bugzilla.samba.org/show_bug.cgi?id=10494 Summary: remove-source-files fails with symlinks Product: rsync Version: 3.1.0 Platform: x64 OS/Version: Linux Status: NEW Severity: major Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy: afried at deteque.com

--- pxzcat.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pxzcat.c b/pxzcat.c index 4ab8689..9bcdc36 100644 --- a/pxzcat.c +++ b/pxzcat.c @@ -29,10 +29,11 @@ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ +#define _GNU_SOURCE #include <config.h>

https://bugzilla.samba.org/show_bug.cgi?id=10518 Summary: rsync hangs (100% cpu) Product: rsync Version: 3.1.1 Platform: All OS/Version: Linux Status: NEW Severity: critical Priority: P5 Component: core AssignedTo: wayned at samba.org ReportedBy: syzop at vulnscan.org QAContact:

Hiya, I can see it's a regular subject on this list. I, like others wanted to use rsync to synchronise two block devices (as it happens one lvm volume and one nbd device served by qemu-img on a remote host from a qcow2 disk image so that I can keep the old versions) As I couldn't find any report of it being done successfully, I'm just sharing my findings as it might benefit others.

The 0.9.8 release breaks API, requiring a number of changes: - Use symbolic constants instead of magic numbers/open-coded strings (well, the string for "base:allocation" was present before this libnbd bump) - Change callbacks to drop the valid_flag parameter - Add _is to nbd_read_only call - Drop the _callback suffix on nbd_aio_FOO calls - Add a struct for managing callback/user_data

3.1, including the latest version, produces bogus ---stats output: % rsync -r --stats . > /tmp/3.1-n % tail -n 17 /tmp/3.1-n Number of files: 3,159 (reg: 2,984, dir: 173, link: 2) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: -6,417,895,484,864,819,269 bytes ************************************************* Total transferred file