I have a backup server now restoring 6TB of data to a client machine. This has been going on for four days now, and no sign of getting close to completion. The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It has far more capacity than that. The only limiting factor I can see is on the backup server one core of the CPU is running 100% rsync. Clearly rsync is not multi-threaded. In researching this I find that a change to multi-threaded goodness would require a massive rewrite, and would only be considered for an rsync replacement. Is there such a replacement in the offing? Is there any sort of workaround for the time being? ATM the large amount of data is the /home directory on the client machine.
I have no idea about potential rsync modifications, but you might try FDT to solve this problem: http://monalisa.cern.ch/FDT/ I have found it easy to use and it absolutely maxes all available bandwidth for the fastest possible data transfer. It doesn't have rsync goodies like update and only changed data, but it's multithreaded, multi-stream, and all around a really fast data mover. Chris ----- Original Message -----> I have a backup server now restoring 6TB of data to a client machine. > This has been going on for four days now, and no sign of getting close > to completion. > > The connexion is Gb enet end-to-end, and is running at only 40Mb/s. It > has far more capacity than that. The only limiting factor I can see is > on the backup server one core of the CPU is running 100% rsync. > Clearly rsync is not multi-threaded. > > In researching this I find that a change to multi-threaded goodness > would require a massive rewrite, and would only be considered for an > rsync replacement. > > Is there such a replacement in the offing? Is there any sort of > workaround for the time being? ATM the large amount of data is the > /home directory on the client machine. > > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: > http://www.catb.org/~esr/faqs/smart-questions.html
On Tue, 2011-05-17 at 08:45 -0700, CACook at quantum-sci.com wrote:> The connexion is Gb enet end-to-end, and is running at only 40Mb/s. > It has far more capacity than that. The only limiting factor I can > see is on the backup server one core of the CPU is running 100% rsync. > Clearly rsync is not multi-threaded.That's probably the delta-transfer algorithm. Turning it off with --whole-file should help somewhat. -- Matt
On Tue, 17 May 2011, CACook at quantum-sci.com wrote:> In researching this I find that a change to multi-threaded goodness > would require a massive rewrite, and would only be considered for an > rsync replacement.Abstracting the core functionality into a librsync.so would be really spiffy too... ..Ch:W.. -- "An idea does not gain truth as it gains followers." Amanda Bloom
On Tuesday 17 May, 2011 10:43:21 you wrote:> Wow! That's a lot of data you're transferring into one server, even with a GB pipe you're using, which appears to be using a fraction of it.Yes, it's my home theater computer with all the movies. I've just set up a RAID array and need to move my data there.> At any rate, I was wondering can you break up that /home directory into several parts in terms of your "rsync" command line. > > Example: rsync... /home/a... > rsync... /home/b... > rsync... /home/c...I thought of this, but so many subdirectories are tiny, and it would be a fussy operation to transfer them individually. If I needed this as a matter of course I might go to the trouble, but I am just waiiting for this to complete so I can watch some decent shows. (MythTV) I've removed the -compress flag and added --whole-file, and am waiting for it to recalculate the transfer. It has already done some small files, and one large file which transferred at 18.65MB/s, a big improvement over 4.5. Still running CPU at 100% on one core. On Tuesday 17 May, 2011 10:46:12 Carlos Carvalho wrote:> You might want to check the read rate at the server, it should be > in the hundreds of MB/s. If not rsync got stuck somewhere.Carlos as I am CPU-bound on the server, I don't really take the disk read rate very seriously, but iotop says rsync is doing around 85M/s on these three WD Green 2TB drives.