hi all- been reading through the archives but I still can't seem to find a solution to my problem. I am using rsync to keep mirror copies of content which is being served (via http) on both the sender and receiver. The files on average are 20-50mb each, (mostly already in a compressed format... mp3, etc) and both sender and receiver pushes around 30mbps on average. the sender is a Celeron 1.3GHz with 1GB RAM receiver is an AMD 1.2GHz with 1GB RAM Both have dual IDE drives configured for software RAID0 When I initiate an rsync from the receiver, the file list is built in around / under 1 minute, and the transfer proceeds at around 500K/s This seems very slow since the machines are on the same subnet and sitting right next to each other in the same datacenter. While this rsync is running, I open a second terminal and do an scp, again initiated from the receiver copying from the sender, and the transfer proceeds at around 5MB/s! Why is scp /so much faster/ than rsync here? Is there anything I can do to improve the speed of these transfers? Fwiw, this is the rsync command I'm issuing: rsync -azL --whole-file --stats --progress --delete sender::my_files /test_destination and rsyncd.conf on the sender looks like: use chroot = no max connections = 10 pid file = /var/run/rsyncd.pid motd file = /etc/rsync/rsyncd.motd timeout = 300 transfer logging = yes log file = /var/log/rsyncd.log log format = %t %h (%a) %m %l %b %o %f [my_files] uid = root path = /my_files max connections = 5 read only = true hosts allow = 192.168.0.0/24 dont compress = * list = false MUCH respect and appreciation to anyone who can help! Thanks, Marty -------------- next part -------------- HTML attachment scrubbed and removed
Oh, and in case it helps:
# uname -a
Linux mydomain.com 2.6.10-1.771_FC2 #1 Mon Mar 28 00:50:14 EST 2005 i686
i686 i386 GNU/Linux
# rsync --version
rsync version 2.6.2 protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
IPv6, 64-bit system inums, 64-bit internal inums
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
You should get some better answers, but a couple of points jump out at me. If the files are already compressed, "small" changes result in very different files, so the business of reading both the target and the source to find common stuff is kinda counterproductive. Also the -z (compressing something already compressed) can't really do much good, particularly with LAN speeds Methinks in this case, you want to disable a bunch of stuff that makes rsync great. Seriously. Basically, you want rsync to copy the entire file if the timestamp is different. You do not want thingees that are extremely useful for transferring big files over bad connections. (I suspect that here is a case where you use rsync because it's easier to figure out rsync that to use something (actually quite different) that is more "appropriate". You discover that "copy stuff from here to there (or vice-versa) is actually incredibly complicated as to exactly what you (should) mean. -----Original Message----- From: rsync-bounces+tony=servacorp.com@lists.samba.org [mailto:rsync-bounces+tony=servacorp.com@lists.samba.org]On Behalf Of Marty Mulligan Sent: Wednesday, May 24, 2006 2:29 PM To: rsync@lists.samba.org Subject: rsync shows poor throughput vs. scp hi all- been reading through the archives but I still can't seem to find a solution to my problem. I am using rsync to keep mirror copies of content which is being served (via http) on both the sender and receiver. The files on average are 20-50mb each, (mostly already in a compressed format... mp3, etc) and both sender and receiver pushes around 30mbps on average. the sender is a Celeron 1.3GHz with 1GB RAM receiver is an AMD 1.2GHz with 1GB RAM Both have dual IDE drives configured for software RAID0 When I initiate an rsync from the receiver, the file list is built in around / under 1 minute, and the transfer proceeds at around 500K/s This seems very slow since the machines are on the same subnet and sitting right next to each other in the same datacenter. While this rsync is running, I open a second terminal and do an scp, again initiated from the receiver copying from the sender, and the transfer proceeds at around 5MB/s! Why is scp so much faster than rsync here? Is there anything I can do to improve the speed of these transfers? Fwiw, this is the rsync command I'm issuing: rsync -azL --whole-file --stats --progress --delete sender::my_files /test_destination and rsyncd.conf on the sender looks like: use chroot = no max connections = 10 pid file = /var/run/rsyncd.pid motd file = /etc/rsync/rsyncd.motd timeout = 300 transfer logging = yes log file = /var/log/rsyncd.log log format = %t %h (%a) %m %l %b %o %f [my_files] uid = root path = /my_files max connections = 5 read only = true hosts allow = 192.168.0.0/24 dont compress = * list = false MUCH respect and appreciation to anyone who can help! Thanks, Marty -------------- next part -------------- HTML attachment scrubbed and removed
On Wed, May 24, 2006 at 03:29:12PM -0400, Marty Mulligan wrote:> Why is scp /so much faster/ than rsync here? Is there anything I can do > to improve the speed of these transfers? > > Fwiw, this is the rsync command I'm issuing: > > rsync -azL --whole-file --stats --progress --delete sender::my_files > /test_destinationSomeone else pointed this out but it bears repeating since you never replied saying you had this figuredout. i don't know whether things will get compressed or not based on your configuration (-z vs. dont compress) but try without the -z. If it *is* trying to compress that could easily explain the slow speed of your transfers. danno -- dan pritts - systems administrator - internet2 734/352-4953 office 734/834-7224 mobile