openssh bugs - Apr 2018 - [Bug 2848] New: Addition of an option to control Nagle's algorithm (enable/disable) can lead to significant speedups

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

            Bug ID: 2848
           Summary: Addition of an option to control Nagle's algorithm
                    (enable/disable) can lead to significant speedups
           Product: Portable OpenSSH
           Version: 7.6p1
          Hardware: All
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: ttsiodras at gmail.com

Hello, OpenSSH guys.

As you can see here...

   
https://unix.stackexchange.com/questions/434825/ssh-speed-greatly-improved-via-proxycommand-but-why

...I've spent a good part of the last two days tracing a weird
performance issue with SSH. The executive summary is that I was able to
significantly increase the performance of SSH (as used by rsync and
borgbackup) by delegating the actual connection to netcat:

    $ dd if=/dev/urandom bs=1M count=50 of=sample.data
    50+0 records in
    50+0 records out
    52428800 bytes (52 MB, 50 MiB) copied, 0.24135 s, 217 MB/s

    $ rsync -av --progress ./sample.data user at 192.168.1.150:/dev/shm/
    sending incremental file list
    sample.data
         52,428,800 100%  575.23kB/s    0:01:28 (xfr#1, to-chk=0/1)

    sent 52,458,757 bytes  received 35 bytes  543,614.42 bytes/sec
    total size is 52,428,800  speedup is 1.00

    $ ssh user at 192.168.1.150 'rm /dev/shm/sample.data'

    $ rsync -av --progress -e 'ssh -o "ProxyCommand nc %h %p"'
./sample.data user at 192.168.1.150:/dev/shm/
    sending incremental file list
    sample.data
         52,428,800 100%    3.40MB/s    0:00:14 (xfr#1, to-chk=0/1)

    sent 52,458,757 bytes  received 35 bytes  2,690,194.46 bytes/sec
    total size is 52,428,800  speedup is 1.00

As you can see, this is a 5x speedup - basically, the use of nc as a
ProxyCommand 
boosted the speed to the same levels reported by iperf3 for the link.

The reason behind this, ended up being the call to enable TCP_NODELAY -
which if I understood correctly, disables Nagle's algorithm to increase
the responsiveness of SSH. Which is good; SSH is supposed to be used
for interactive shells, so I understand that decision.

But since rsync and borgbackup and many others are using SSH as a
universal transport system, I believe it makes sense to provide a
generic option to enable/disable Nagle - e.g. 

    --nodelay (default)
    --delay 

...to control Nagle - so that tools using SSH for bulk transfers can
benefit from "full" packets sent on the wire, and avoid wasting
bandwidth. Especially over slow links - wifi in my case, but it can be
any kind of slow link - this will have significant benefits - as you
can see in my aforementioned (extensive) report in Unix Stack Exchange.

I hope you will consider adding this option to OpenSSH... My
understanding from the feedback I got talking about this in other
forums is that other SSH implementations did offer such an option - and
it shouldn't be too difficult to add... (we just want to control the
invocation of the related setsockopt via a run-time controlled
boolean).

Thanks for you time - I hope you will consider this.

Kind regards,
Thanassis Tsiodras, Dr.-Ing.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

Thanassis Tsiodras <ttsiodras at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Addition of an option to    |Adding an option to control
                   |control Nagle's algorithm   |Nagle's algorithm
                   |(enable/disable) can lead   |(enable/disable) can lead
                   |to significant speedups     |to significant speedups

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

Kristian Klausen <kristian at klausen.dk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kristian at klausen.dk

--- Comment #1 from Kristian Klausen <kristian at klausen.dk> ---
I think it is worth mentioning this very old patch [1], it contains
some good input regards a "NoDelay" option.

[1]
https://lists.mindrot.org/pipermail/openssh-unix-dev/2002-January/010245.html

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

--- Comment #2 from Thanassis Tsiodras <ttsiodras at gmail.com> ---
@Kristian: Unless I am mistaken, that post argues from the other side -
that of a fast link, where they *want* to disable Nagle. As you can see
in my case (slow wifi link), enabling Nagle dramatically increases the
transfer bandwidth...

...which basically reinforces why we need to have this option as a
user-configurable one: as is right now, the call to setsockopt that
enables TCP_NODELAY is unconditional. IMHO it shouldn't be - the
end-user should be able to control this, so that when SSH is used for
transport, users with fast links can choose to disable Nagle - and
users like me (with slow links) can enable it and avoid wasting
bandwidth by sending half-empty packets.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
I'm not sure that I understand: Nagle's algorithm is only disabled for
interactive connections. Are you asking it be disabled for
non-interactive connections too?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

--- Comment #4 from Thanassis Tsiodras <ttsiodras at gmail.com> ---
@Damien Miller: What I am saying (please see the linked post in UNIX
Stack Exchange for details) is that when used as a transport, SSH
cannot predict what is best; in my case, stracing showed that
TCP_NODELAY was enabled, greatly impacting the speed over my congested
WiFi link. 

    # from strace output - for details see the linked StackExchange
post
    1522665534.007899 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
<0.000021>

A simple addition of two options that control the setting...

    --nodelay (default)
    --delay 

...would allow tools like borgbackup and rsync (that use SSH as a
transport) to try out and adapt to whatever works best. Note that in
the link from Kristian posted above, you can see arguments from the
other side of the Nagle spectrum - basically confirming that direct
control of Nagle's setup is a desired trait! 

And an easy one to add, I believe... (it's just a global boolean flag).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2848

joel.schaerer at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |joel.schaerer at gmail.com

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.