The short version. Copying over SFTP to our centos boxes maxes out at 2MB a sec. The Question: Is there some sort of speed limitation somewhere in the ssh/sftp daemon? Does anyone have any idea why this could be happening? The long version We have centos 4.6 installed on 2 different machines, different hardware, different nics. Even on 2 different subnets. I've tried different switches, 1000 Full/Half 100 Full/Half and still get the same speed. It maxes out at 2MB a sec. Same switch, network, client using sftp to our xserves gets 10MB a sec. Basically filling the pipe of the network. The only thing that is the same on both the linux boxes is the distro. So i thought maybe someone may have come across this before. I haven't tried other protocols yet (FTP and HTTP) I'll check that tomorrow. Thanks, Jamie
-----Original Message----- From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On Behalf Of Jamie Lists Sent: Tuesday, May 01, 2007 12:35 AM To: centos at centos.org Subject: [CentOS] Sftp slow on both our centos installs The short version. Copying over SFTP to our centos boxes maxes out at 2MB a sec. The Question: Is there some sort of speed limitation somewhere in the ssh/sftp daemon? Does anyone have any idea why this could be happening? The long version We have centos 4.6 installed on 2 different machines, different hardware, different nics. Even on 2 different subnets. I've tried different switches, 1000 Full/Half 100 Full/Half and still get the same speed. It maxes out at 2MB a sec. Same switch, network, client using sftp to our xserves gets 10MB a sec. Basically filling the pipe of the network. The only thing that is the same on both the linux boxes is the distro. So i thought maybe someone may have come across this before. I haven't tried other protocols yet (FTP and HTTP) I'll check that tomorrow. Thanks, Jamie -----Reply Message----- You mentioned you tried different switch speeds/duplexes. You didn't mention if you tried manually setting the speed/duplex on the NIC's rather than use auto-negotiate. If you have the ethtools package installed, and type "ethtool eth0" what is the speed/duplex set to? Does it match what you have set on the switch? If not, try forcing eth0 (or whatever your NIC is) to the speed/duplex you have on your switch and are cabled up for. (e.g. in /etc/sysconfig/network-scripts/ifcfg-eth0 we have: ETHTOOL_OPTS="speed 100 duplex full autoneg off" ) ----------------------------------------- This email (including any attachments) is confidential. If you are not the intended recipient you must not copy, use, disclose, distribute or rely on the information contained in it. If you have received this email in error, please notify the sender immediately by reply email and delete the email from your system. Confidentiality and legal privilege attached to this communication are not waived or lost by reason of mistaken delivery to you. Lend Lease does not guarantee that this email or the attachment(s) are unaffected by computer virus, corruption or other defects. Lend Lease may monitor incoming and outgoing emails for compliance with its Email Policy. Please note that our servers may not be located in your country.
On Tue, May 1, 2007 12:34 am, Jamie Lists wrote:> The short version. > Copying over SFTP to our centos boxes maxes out at 2MB a sec. > > The Question: > Is there some sort of speed limitation somewhere in the ssh/sftp daemon? > > Does anyone have any idea why this could be happening? > > The long version > We have centos 4.6 installed on 2 different machines, different > hardware, different nics. Even on 2 different subnets. > > I've tried different switches, 1000 Full/Half 100 Full/Half and still > get the same speed. It maxes out at 2MB a sec. > > Same switch, network, client using sftp to our xserves gets 10MB a > sec. Basically filling the pipe of the network. > > The only thing that is the same on both the linux boxes is the distro. > So i thought maybe someone may have come across this before. > > I haven't tried other protocols yet (FTP and HTTP) I'll check that > tomorrow. > > Thanks, > JamieI did some quick research on google a few min ago. Thinking back I have never seen higher than 2MB/s transfer speed using sftp between a Windows and Linux machine. I have however seen much higher transfer speeds using sftp/scp between 2 linux machines. A few people mentioned they believe the slowness is with the implementation of sftp/scp/ssh in the Windows app. Try doing an sftp or scp transfer between the 2 CentOS machines and look at the transfer rate. You could also try doing a transfer using Cygwin or maybe some other win32 ports. However it may just be somehting strange with Windows. ...While typing this I figured I would do some additional testing. I downloaded VanDyke SecureFX (VanDyke has some of the best SSH implementations I have seen for windows). My transfer rate was approx 10MB/s on a 100Mbit connection. Whereas with WinSCP and FileZilla it was only approx 2MB/s -- Matt Martz CentOS Mirror Admin mdmartz at gflug.net
On Mon, 30 Apr 2007, Jamie Lists wrote:> The short version. > Copying over SFTP to our centos boxes maxes out at 2MB a sec. > > [....] > > I haven't tried other protocols yet (FTP and HTTP) I'll check that > tomorrow.This is a pretty crucial test. It'll be difficult to diagnose the issue until you know whether the protocol is contributing to the problem or merely symptomatic of it. Also, I assume that you're scp-ing files to and from local disk, not filesystems with network dependencies (nfs or cifs), right? -- Paul Heinlein <> heinlein at madboa.com <> www.madboa.com
On Apr 30, 2007, at 9:34 PM, Jamie Lists wrote:> The Question: > Is there some sort of speed limitation somewhere in the ssh/sftp > daemon?Here's a couple potential problems 1. CPU usage. sftp is more intensive than CPU. Don't run out the Mac even though it worked fine with another machine - the algorithm negotiation might have gone differently. Look at the CPU utilization with a tool like top. The pessimistic assumption is that no bytes are being sent during the percentage of the time the CPU is busy. 2. TCP windows. This one's a bit complex to expain: The deal is that between sftp on machine A sending bytes and sftp on machine B receiving them, machine A's kernel is responsible for retrying sends until machine B's kernel acknowledges them (thus it needs to keep them in a send buffer), and machine B's kernel is responsible for buffering them until the sftp process is ready to deal with them. Machine B advertises how many bytes are free in its receive buffer through the TCP window. This number is important for performance: that's the number of bytes machine B can send before receiving an acknowledgement, so if it's less than (ping time) * (bandwidth), it can limit throughput. So first, what is the ping time between these two machines? Calculate this number accordingly, which is usually called the bandwidth*delay product. Now run sftp between the machines while running tcpdump. On your Macintosh, try something like this: $ sudo /usr/sbin/tcpdump -ni en0 'port ssh and tcp-syn != 0' That will dump out the initiating packets of the connection only. For me, an ssh connection from my OS X 10.4.9 machine to my CentOS 5 machine looks like this: 18:09:48.824252 IP 10.4.253.189.53581 > 10.4.254.87.22: S 1900985803:1900985803(0) win 65535 <mss 1460,nop,wscale 0,nop,nop,timestamp 468743802 0,sackOK,eol> 18:09:48.824434 IP 10.4.254.87.22 > 10.4.253.189.53581: S 4196469947:4196469947(0) ack 1900985804 win 5792 <mss 1460,sackOK,timestamp 280331387 468743802,nop,wscale 7> My Macintosh (10.4.253.189) is advertising a TCP window size of 65535 bytes. My Linux box (10.4.254.87) is doing something a bit more complicated. When TCP was originally created, they didn't foresee the need for window sizes greater than 65535. It's using an option called TCP window scaling to multiply its advertised window size by 2^7. So it's advertising a window size of 741k...if the Mac supports window scaling as well...otherwise, a dismal 5792 bytes. In this case, my Mac does claim support for window scaling, so the full size is used. I'm using 100 Mbps Ethernet and my round-trip ping times between the machines are at most 0.3 ms, so full utilization requires this window size: 100 * 10^6 bits/sec * (byte / 8 bits) * (0.3 * 10^-3 s) = 3000 bytes. so both window sizes are plenty. But if I were using gigabit Ethernet with a full millisecond of latency: 100 * 10^9 bits/sec * (byte / 8 bits) * (10^-3 s) = 1000000 bytes = 100 kbytes I could send at full speed from the Mac to the Linux machine but not the other way around, as the Mac's receive window is too small. -- Scott Lamb <http://www.slamb.org/>