Hi, first a short success report: rsync 3.0.0 works as far as I can tell flawless for me. My performance question: Below are of some recent backup stats one, using the old protocol (29), the other the latest cvs (30). As you see, the file creation time has dropped significantly (0.01 sec vs 80.2 sec). Still the overall performance hasn't increased much. This may not be surprising as very few data (42 and 11 files) are actually transferred. Anyway, I am wondering why it is taking full 12 minutes to complete the rsync. The connection link is a GigE LAN. Thus most time is spent comparing the file lists at sender and receiver. However, a comparison rate of 217293 files / 670 sec = 325 files/sec seems pretty low, no? Regards, Matt With new protocol 30: ---------------8<--------------- Mon Feb 5 13:44:54 PST 2007 /usr/local/bin/rsync -ax --delete --stats --numeric-ids --port=8730 --password-file=xxxx --link-dest=/path/to/backup/xx.old user@machine::xx /path/to/xx Number of files: 217293 Number of files transferred: 42 Total file size: 223074167501 bytes Total transferred file size: 97022268 bytes Literal data: 6545079 bytes Matched data: 90477189 bytes File list size: 5722327 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 145920 Total bytes received: 12369920 sent 145920 bytes received 12369920 bytes 54064.10 bytes/sec total size is 223074167501 speedup is 17823.35 Mon Feb 5 13:56:04 PST 2007 --------------->8--------------- With protocol 29: ---------------8<--------------- Sat Feb 3 03:15:50 PST 2007 /usr/local/bin/rsync -ax --delete --stats --numeric-ids --port=8730 --password-file=xxxx --protocol=29 --link-dest=/path/to/backup/xx.old user@machine::xx /path/to/xx Number of files: 217290 Number of files transferred: 11 Total file size: 223073128036 bytes Total transferred file size: 92543007 bytes Literal data: 4854237 bytes Matched data: 87688770 bytes File list size: 5932539 File list generation time: 80.197 seconds File list transfer time: 0.000 seconds Total bytes sent: 66791 Total bytes received: 10825958 sent 66791 bytes received 10825958 bytes 40720.56 bytes/sec total size is 223073128036 speedup is 20479.05 Sat Feb 3 03:27:18 PST 2007 --------------->8---------------
On Mon 05 Feb 2007, Matt wrote:> > Anyway, I am wondering why it is taking full 12 minutes to complete the > rsync. The connection link is a GigE LAN. Thus most time is spent > comparing the file lists at sender and receiver. However, a comparison > rate of 217293 files / 670 sec = 325 files/sec seems pretty low, no?If your network is fast, try using --whole-file to prevent a lot of IO while rsync is trying to find matching blocks when a file has been updated. Paul Slootman
On Mon, Feb 05, 2007 at 05:06:31PM -0800, Matt wrote:> Anyway, I am wondering why it is taking full 12 minutes to complete the > rsync.There are several things to check. Try timing an rsync run with -n to see how quickly it runs without doing any data transfer. Try doing an rsync of the destination dir (with -n --protocol=29) to some other local dir just to see how many seconds it takes to scan through all the files on the receiving side. Try using --del instead of --delete to ensure that you're not doing a separate delete pass. Do keep in mind that performance turning of protocol 30 hasn't been done yet. The largest potential for gain is when scanning huge numbers of files, as it should help to have both the sender and the receiver (generator, actually) scanning through the hierarchy at the same time rather than separately. It should also help that it scans and transfers a directory more closely together, so that it is more likely that the directory information is in the cache on the sending side. Beyond that, there might be a win due to less memory swapping (for really, really large transfers). If there are a large number of files to transfer, the win won't really be that much, as the generator will be waiting around for new file-list info a lot, and that data channel will be clogged up with file data. Thus, it might degrade back to waiting for the sender to finish the file list, but since the vast portion of such a transfer will be limited by the throughput of the data arriving from the sender, it is hard to imagine a way to improve that further (since the generator would be waiting a lot anyway for the sender to receive its data). ..wayne..