openssh unix dev - Jan 2006 - sftp performance problem, cured by TCP_NODELAY

In certain situations sftp download speed can be much less than that
of scp.

After many days of trying to find the cause finally I found it to be
the tcp nagle algorithm, which if turned off with TCP_NODELAY
eliminates the problem.

Now I see it being discussed back in 2002, but it still unresolved in
openssh-4.2 :(

Simple solution would be to add a NoDelay option to ssh which sftp
would set.  The proprietary ssh seems to already have this option.

Attached an LD_PRELOAD hack to set TCP_NODELAY.  For me this hack
doubles the download speed of a 100MB file over a 100Mbps network.

Miklos

-----------------------------------------------------------------------
/* gcc -Wall -W -o sshnodelay.so --shared -ldl sshnodelay.c */
#define _GNU_SOURCE
#include <dlfcn.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>

int connect(int sock, const struct sockaddr *addr, socklen_t addrlen)
{
    int (*next_connect)(int, const struct sockaddr *, socklen_t)        
dlsym(RTLD_NEXT, "connect");
    int res = next_connect(sock, addr, addrlen);
    if (addr->sa_family == AF_INET) {
        int opt = 1;
        setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &opt, sizeof(opt));
    }
    return res;
}
-----------------------------------------------------------------------

$ time sftp -b /tmp/big100M.b server
sftp> get big100M /dev/null
Fetching /tmp/big100M to /dev/null

real    0m18.231s
user    0m6.248s
sys     0m1.684s
$ time LD_PRELOAD=/tmp/sshnodelay.so sftp -b /tmp/big100M.b server
sftp> get big100M /dev/null
Fetching /tmp/big100M to /dev/null

real    0m9.001s
user    0m6.320s
sys     0m1.840s

Miklos Szeredi wrote:
> In certain situations sftp download speed can be much less than that
> of scp.
> 
> After many days of trying to find the cause finally I found it to be
> the tcp nagle algorithm, which if turned off with TCP_NODELAY
> eliminates the problem.
> 
> Now I see it being discussed back in 2002, but it still unresolved in
> openssh-4.2 :(
> 
> Simple solution would be to add a NoDelay option to ssh which sftp
> would set.  The proprietary ssh seems to already have this option.
What sort of socket calling pattern was sftp making that had it running into 
issues with Nagle and delayed ACK?

rick jones

Miklos Szeredi wrote:
>>In broad handwaving terms, applications that try to do "write write
>>read" can run afoul of interactions between Nagle and delayed ACK.
> 
> 
> Which is exactly what sftp does.
> 
> Excerpt from sftp(1):
> 
> |      -R num_requests
> |              Specify how many requests may be outstanding at any one
time.
> |              Increasing this may slightly improve file transfer speed but
> |              will increase memory usage.  The default is 16 outstanding
> |              requests.
> 
> I also noticed, that -R 1 actually increases throughput in some cases,
> which is nicely explained by the fact, that then it will always be
> "write read" pairs.
> 
> Of course -R 1 will still result in suboptimal throughput, so that is
> not the solution.
So, if the code knows it is going to inject N requests at one time, why not use 
writev() or sendmsg()?

rick jones
still interested in the packet trace though

> So, if the code knows it is going to inject N requests at one time,
> why not use writev() or sendmsg()?
No, it keeps N requests outstanding all the time, meaning, that when a
reply comes back, it sends out another request.  I think this is
causing the problem.

However the problem is more general, in that ssh should be able to
efficiently transport _any_ kind of protocol, no matter what I/O
pattern it uses.

Does Nagle make sense for anything other than interactive sessions?

Maybe turning it off if input/output is not a terminal makes sense, no?

Miklos

Miklos Szeredi wrote:
>>So, if the code knows it is going to inject N requests at one time,
>>why not use writev() or sendmsg()?
> 
> 
> No, it keeps N requests outstanding all the time, meaning, that when a
> reply comes back, it sends out another request.  I think this is
> causing the problem.
> 
> However the problem is more general, in that ssh should be able to
> efficiently transport _any_ kind of protocol, no matter what I/O
> pattern it uses.
> 
> Does Nagle make sense for anything other than interactive sessions?
> 
> Maybe turning it off if input/output is not a terminal makes sense, no?
http://bugzilla.mindrot.org/show_bug.cgi?id=556 has been there for a
while, but nobody commented on it.