I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex switched network with both boxes on the SAME subnet. Am I missing something? Should I expect these speeds? Could there be something on the App switch throttling rsync for some reason? Any thought will help, Ben Ricker System Administrator Wellinx.com
You might try putting the rsycd (server) on the source. rsync as a server is really designed for read-only. The read-write support is an afterthough, and very flaky. Martin, Tridge, Wayne: correct me if that's not true any more. Make sure you're not using --bwlimit=, unless you need to parcel your bandwidth (In my situation, I do). Don't use -H, unless you actually use multiple hard links to files, as that adds tremendous overhead on large filesystems. If the files are nfs mounted, use -W, else you will read both files completely, before you transfer anything... might as well just send it if datestamp or size differs. I doubt ssh was giving you any slowdown, unless you're cpu-limited on one or the other box. If you don't need the security, and policy permits, plain rsh transport might be a bit faster. Tim Conway tim.conway@philips.com 303.682.4917 Philips Semiconductor - Longmont TC 1880 Industrial Circle, Suite D Longmont, CO 80501 Available via SameTime Connect within Philips Available as n9hmg on AIM perl -e 'print pack(nnnnnnnnnnnn, 19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), ".\n" ' "There are some who call me.... Tim?" Ben Ricker <rickerb@mindspring.com>@lists.samba.org on 10/08/2001 06:50:02 PM Sent by: rsync-admin@lists.samba.org To: rsync@lists.samba.org cc: (bcc: Tim Conway/LMT/SC/PHILIPS) Subject: Rsync Throughput issues Classification: I am using Rsync between a Redhat Linux box and an AIX RS600. We have a about 30gb of database we need to sync to a backup server. Sounds good, right? The problem is that Rsync is so slow when we do the initial dump. We have files that are 1 - 5gb. It takes around 14-20 hours to Rsync the file structure, which means our daily backup will miss that days data because we are waiting for the data to rsync. At first we were using ssh as the agent but we thought that might be slowing us down. We then tried to use rsync as a server and push the data to the module/zone of the server. Consistently, it takes roughly 2 minutes to transfer 10meg of data I thought it was a network problem so we tried ftp. A 10 mb file took 90% the time (about 20 seconds. This is on a 100bit Full Duplex switched network with both boxes on the SAME subnet. Am I missing something? Should I expect these speeds? Could there be something on the App switch throttling rsync for some reason? Any thought will help, Ben Ricker System Administrator Wellinx.com
Ben Ricker wrote:> > I am using Rsync between a Redhat Linux box and an AIX RS600. We have > a about 30gb of database we need to sync to a backup server. Sounds > good, right? The problem is that Rsync is so slow when we do the > initial dump. We have files that are 1 - 5gb. It takes around 14-20 > hours to Rsync the file structure, which means our daily backup will > miss that days data because we are waiting for the data to rsync.How many files are in the tree? Are all the files 1-5G? Are the source files on new/fast disk? For older systems, the 100Mbit network is often faster than disk. I have a 40G tree with 1 million files in it. I just killed the tree on a development box and restarted a rsync to it. I think it might take 10+ hours mainly because of how many files are in the tree -- not so much because of the total size. I'll let you know. I know it takes about 2 hours just to verify that all 1 million files are in sync even when there are no changes to the tree.> At first we were using ssh as the agent but we thought that might be > slowing us down. We then tried to use rsync as a server and push the > data to the module/zone of the server. Consistently, it takes roughly > 2 minutes to transfer 10meg of dataThat seems way too long. Sounds like a reverse dns lookup failure. Long shot debug question -- Do you have dns/nis/files all working properly? Sometimes reverse dns lookups can take a few minutes to fail or time out and thus affect the overall speed of transfers. Repeated reverse dns failures can add up fast.> I thought it was a network problem so we tried ftp. A 10 mb file took > 90% the time (about 20 seconds. This is on a 100bit Full Duplex > switched network with both boxes on the SAME subnet.I think you still have other problems. A ftp of a 10Mbyte file takes my boxes about 1s on a 100Mbit wire. Be sure to try ftp both get and put. Oftentimes a duplex mismatch or speed problem will only affect traffic in one direction and not the other. I think you might have some network issues. Do a ifconfig on the linux box to see what sort of errors and collisions you are seeing.> Am I missing something? Should I expect these speeds? Could there be > something on the App switch throttling rsync for some reason? > > Any thought will help, > > Ben Ricker > System Administrator > Wellinx.com
Quick question(s) I am trying to improve the speed for an rsync backup. At the moment, we have workstations running windows 98 and I have a Linux 2.2.19 server who accesses the client disks using samba to take the backup. rsync is running in local mode, e.g. accesses directories only. Each client uses ssh to start a server script in their home directories, which in turn mounts the client drive to perform the actual backup. The backup is done to a rather large local drive on the server, such that the user can have up to two weeks of full and incremental backup using a RO samba mounted directory. This works like charm, except that each Monday, the system will take a full backup of the client data ( which is Identities and "My Documents" ), but the data rate is rather low ( 600Kbyte+ ) on a full duplex 100Mbits network ( no other traffic ). On other days, I use rsync -avW, ( which is marginally better than -av ), together with a --compare-dest to the Monday backup, as to limit the number of files selected for backup. Of course, ftp is fast, real fast. So is xcopy from the client to a samba mounted server ( when smb debugging is disabled ). Any hints?
Yes, built. Compiled with specific options. Do you have any questions about building on AIX? (again, I mention 4.1.5 and Apple hardware because it is older and different, but it should be easier on your newer, standard boxes) Your I/O wait was on the box being read from or written to? How are you viewing the wait (something like monitor or top?)? Is one of the boxes running rsync as a deamon? Do you experience the drag on any read/write or only the large files you mention (or only on specific files)? You mentioned ftp at some point, yes? Was there a performance hit for that service as well? Did you try the -u flag? Do you have space to try the sync locally (take out the network and second host variables in order to narrow down the troubleshooting)? Again, let us know if you have questions specific to the tasks on AIX and good luck. At 3:10 PM -0500 10/9/01, Ben Ricker wrote:>On Tue, 2001-10-09 at 14:51, meg wrote: >> we use rsync on aix 4.1.5 (apple hardware). it builds very easily. >> do you have specific questions? >> >> goodluck! > >We are running on an RS6000 (H80 with single CPU and 1gb memory). We are >also running a rather large Oracle database on it. I noticed that when >rsync was running, there was a considerable I/O wait (at one point, it >was at 98%!). We installed Rsync from a binary dist, not compiled. I >assume by your use of "build" that you compiled from source? > >Ben Ricker >System Administrator >Wellinx.com