We have an interesting quandry here. When I'm rsync'ing my directory tree (100 directories each containing 1000 files) I see some strange results: All of these machines are on their own network segment (100t) in our QA lab Solaris->Solaris - time taken: 11m32s Solaris->RH Linux 7.2 - time taken: 206s RH Linux->Rn Linux - time taken 1m59s In each instance I have rsync running as a server on the target machine (Linux in each case). I just can't explain why the performance is just so AMAZING on Linux. Any ideas? Thanks Dave Dave North Solutions Engineering SIGNIANT Inc. www.signiant.com Phone: 613-599-2158 Mobile: 613-294-3231 Fax: 613-599-2169 Email: dnorth@signiant.com
> -----Original Message----- > From: rsync-admin@lists.samba.org [mailto:rsync-admin@lists.samba.org]On > Behalf Of Dave North > Subject: Linux and Solaris performance > > We have an interesting quandry here. When I'm rsync'ing my directory > tree (100 directories each containing 1000 files) I see some strange > results: > > All of these machines are on their own network segment (100t) in our QA > lab > > Solaris->Solaris - time taken: 11m32s > Solaris->RH Linux 7.2 - time taken: 206s > RH Linux->Rn Linux - time taken 1m59s > > In each instance I have rsync running as a server on the target machine > (Linux in each case). I just can't explain why the performance is just > so AMAZING on Linux. Any ideas? > > ThanksJust guessing... Are you _doing_ the same thing in each case? That is, not, "did you type the same command", but does rsync perform the exact same block transfers? You use rsync to move changes from one machine to another, are you moving the same changes? Maybe there isn't much difference between the files on the linix machines.
-----Original Message----- From: Bernard A Badger [mailto:bab@vx.com] Sent: Wednesday, July 17, 2002 2:35 PM To: rsync@lists.samba.org Subject: RE: Linux and Solaris performance> In each instance I have rsync running as a server on the targetmachine> (Linux in each case). I just can't explain why the performance isjust> so AMAZING on Linux. Any ideas? > > ThanksJust guessing... Are you _doing_ the same thing in each case? That is, not, "did you type the same command", but does rsync perform the exact same block transfers? You use rsync to move changes from one machine to another, are you moving the same changes? Maybe there isn't much difference between the files on the linix machines. DN: Sorry, I forgot to mention. NONE of the files exist on the target machine (that is, it's a straight copy)...there is no realy sync'ing (in the true sense of the word) to be done for each file. Cheers Dave
-----Original Message----- From: Daniel Veillard [mailto:daniel@veillard.com] Sent: Wednesday, July 17, 2002 3:22 PM To: Dave North Cc: Bernard A Badger; rsync@lists.samba.org Subject: Re: Linux and Solaris performance <SNIP>> DN: Sorry, I forgot to mention. NONE of the files exist on the target > machine (that is, it's a straight copy)...there is no realy sync'ing(in> the true sense of the word) to be done for each file.Then the reason is that metadata filesystem operations on linux filesystems are quite faster than on Solaris. You can see the same effect when exploding a tar archive on a filesystem on both system. DN: You're onto something here...I just did an un-tar of the file containing my data (100,000 files) and saw: Solaris: 11m:28.5s Linux: 1m26.923s That's a hell of a difference. Very peculiar how the sun FS can be so incredibly slow compared to Linux - and these are reasonably high end sun Netra machines. Very strange. I wonder how reliable the Linux FS is then if it's extracting the files so fast...does sun have some ultra-realiable FS features or something I wonder. Thanks for the reply Cheers Dave
On Wed, Jul 17, 2002 at 01:52:15PM -0400, Dave North wrote:> We have an interesting quandry here. When I'm rsync'ing my directory > tree (100 directories each containing 1000 files) I see some strange > results: > > All of these machines are on their own network segment (100t) in our QA > lab > > Solaris->Solaris - time taken: 11m32s > Solaris->RH Linux 7.2 - time taken: 206s > RH Linux->Rn Linux - time taken 1m59s > > In each instance I have rsync running as a server on the target machine > (Linux in each case). I just can't explain why the performance is just > so AMAZING on Linux. Any ideas?I can immediately see three factors that may be affecting this. 1. Cache. I'm not certain of Solaris but linux uses almost all free memory for caching so there may be much less disk I/O in your situation on the linux boxes. This will be affected by kernel version, amount of memory, other activity on system. 2. I/O and network subsystems. I can't remember the name but you may recall a notorious benchmark pitting Linux against NT a couple of years ago. Linux has been undergoing focused effort to improve performance both disk I/O (esp 2.5) and networking(2.3). Solaris has not been held under the cross-hairs and if you talk to an SGI fan he'll tell you Solaris has poor network performance. 1+1=2 Rsync is a poor choice for unit and system benchmarks. Other tools would be better for checking whether I/O or network are affecting things. 3. Filesystem design. You don't say what filesystem types are involved. I'm assuming ext2 on Linux and UFS on Solaris. This might not be a major factor but could be. The design of a filesystem involving directory layout, block allocation and tracking schemes can affect the time taken to add, delete and grow files. Witness early XFS slowness in mass file deletion. I can immediately see FOUR factors. 4. Filesystem build options. Block and fragment size affect not only allocation frequency but the presence of indirect blocks affects speed of file access. Less important for speed would be bytes per inode and cylinders per group. Historically Solaris defaults have been piss-poor for performance so i haven't used defaults on that platform in more than 10 years. I can immediately see three, no five factors... Damn, come in again. 5. Synchronous directory updates. UNIX has traditionally done directory ops synchronously. Linux, on the other hand, does them asynchronously. In English what this means is that on UNIX the system calls unlink(2), rename(2) or open(2) will not return until any change to the directory file has been written to disk; but linux will will return from these system calls as soon as the in-memory structures are updated. Rsync is pretty heavy into making directory changes. Every file updated causes two directory changes. On UNIX this means that you have to pause for the disk I/O to complete twice for every updated file. This may be what accounts for the big jump when you change the destination from Solaris to Linux. Mount options may affect significantly. A fairly good description of this can be found at: http://marc.theaimsgroup.com/?l=linux-kernel&m=102676438229862&w=2 -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
> In each instance I have rsync running as a server on the target machine > (Linux in each case). I just can't explain why the performance is just > so AMAZING on Linux. Any ideas?Because Linux Rocks !! M@
Dave North wrote:> > We have an interesting quandry here. When I'm rsync'ing my directory > tree (100 directories each containing 1000 files) I see some strange > results: > > All of these machines are on their own network segment (100t) in our QA > lab > > Solaris->Solaris - time taken: 11m32s > Solaris->RH Linux 7.2 - time taken: 206s > RH Linux->Rn Linux - time taken 1m59s > > In each instance I have rsync running as a server on the target machine > (Linux in each case). I just can't explain why the performance is just > so AMAZING on Linux. Any ideas? >You did not mention what hardware you are running. If you are running Ultra-10 boxes then they have a slow IDE hard disk controller chipset (CMD640 I think). If your linux box has a faster IDE/SCSI controller (as long as it runs faster than 12.5Mbytes/s) then that would explain some of the difference. Linux->linux rsync I'd guess you are seeing the network as the bottleneck. Solaris->solaris I'd guess you are seeing the disk (or some other HW) as the bottleneck. Of course if your solaris HW is an V880 then I'm out to lunch on my guess. eric