Hello, I've been using rsync on some backup servers for years. In 2011 we had a situation where the FS of the backup server was behaving strange, even thou there was enough available I/O, the fs(ext4 on 16TB partition with a lot of inodes) was lagging. After much testing we found that rsync was hammering the fs too hard. At that point I patched rsync to add a stupid option which will sleep for a while before creating new file. That helped us a lot to keep the same rate of concurrent rsyncs to the same machine. I would love your comments on this, generally stupid approach to the problem :) Best regards, Marian -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-options.c-Add-the-slow-down-option.patch Type: text/x-patch Size: 1840 bytes Desc: not available URL: <http://lists.samba.org/pipermail/rsync/attachments/20140403/be6941c2/attachment.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: 0002-flist.c-Implementing-the-slow-down-option.patch Type: text/x-patch Size: 1097 bytes Desc: not available URL: <http://lists.samba.org/pipermail/rsync/attachments/20140403/be6941c2/attachment-0001.bin>
Marian Marinov wrote...> I've been using rsync on some backup servers for years. In 2011 we > had a situation where the FS of the backup server was behaving > strange, even thou there was enough available I/O, the fs(ext4 on > 16TB partition with a lot of inodes) was lagging. After much testing > we found that rsync was hammering the fs too hard.I'd like to learn more about that scenario. Mostly, I'm curious whether these file transfers involved creation of a *lot* and probably rather small files. Christoph
This is way beyond my level of expertise, but wouldn't something like ionice help with that? Also, check out: 2 more pipe utilities Viewer & throttle http://www.ivarch.com/programs/pv.shtml Throttle - limits bandwidth of a pipe - for use with network transfers http://linux.die.net/man/1/throttle HTH Joe On 04/03/2014 07:41 AM, Marian Marinov wrote:> Hello, > > I've been using rsync on some backup servers for years. In 2011 we had > a situation where the FS of the backup server was behaving strange, > even thou there was enough available I/O, the fs(ext4 on 16TB > partition with a lot of inodes) was lagging. After much testing we > found that rsync was hammering the fs too hard. > > At that point I patched rsync to add a stupid option which will sleep > for a while before creating new file. > That helped us a lot to keep the same rate of concurrent rsyncs to the > same machine. > > I would love your comments on this, generally stupid approach to the > problem :) > > Best regards, > Marian > > > >