openssh unix dev - Jun 2005 - New Set of High Performance Networking Patches Available

http://www.psc.edu/networking/projects/hpn-ssh/

Mike Stevens and I just released a new set of high performance 
networking patches for OpenSSH 3.9p1, 4.0p1, and 4.1p1. These patches 
will provide the same set of functionality across all 3 revisions. New 
functionality includes

1) HPN performance even without both sides of the connection being HPN 
enabled. As long as the bulk data flow is in the direction of the HPN 
side you should see improved performance. I've measure 200Mb/s to an HPN 
server from a non HPN client and vice versa.

2) HPN client can now set the local tcp receive buffer on a per 
connection basis. Using the -w option allows the client to override the 
local tcp receive window settings up to the maximum tcp buffer size. 
This is just a setsockopt() call really.

3) NONE cipher switching now available for all revisions. This *is* a 
separate set of patches so it doesn't have to be part of your HPN 
install if you don't want the additional risk associated with NONE. It 
is important to note that the NONE cipher is only used *after* 
authentication takes place. During authentication the default or user 
specified cipher will be used. It should not be very easy to accidental 
switch over to NONE in an interactive session. Of course, the user has 
to accept that there are additional risks.

We're seeing very good throughput - up to 280Mb/s with the arcfour 
cipher. We're actually being limited by the disk speed in this case as 
cpu load is still under 60% on our test machines.

You can find the patches here
http://www.psc.edu/networking/projects/hpn-ssh/

Note: All we are doing is allowing SSH to make better use of the network 
available to it. People on networking paths with a bandwidth delay 
product lower that 64KB won't see an improvement in throughput. You 
should also make sure that the TCP stack on both sides are properly 
tuned for high performance networking.

Comments, questions, observations, bug reports, and the like are 
welcome. We'll have some more stuff out in the next month or so 
hopefully (not patches).

Chris Rapier, PSC
Mike Stevens, CMU

Chris Rapier wrote:
> http://www.psc.edu/networking/projects/hpn-ssh/
Looking at these has been on my to-do list for a while and I finally 
took a look.
> 1) HPN performance even without both sides of the connection being HPN 
> enabled. As long as the bulk data flow is in the direction of the HPN 
> side you should see improved performance. I've measure 200Mb/s to an
HPN
> server from a non HPN client and vice versa.
I've been testing with tunbridge[1] on OpenBSD to add latency.  I've 
seen an improvement of around 50% throughput on scp with 100ms of 
latency (each way, ie 200ms rtt) simulated link with Linux endpoints.

Using -w doesn't seem to make any difference (or sometimes it's a net 
loss) although it's quite possible something in my test environment is 
responsible for that.  (Yes, I did the stack tuning, both netstat and 
getsockopt show the buffers are 1MB or more.)
> 2) HPN client can now set the local tcp receive buffer on a per 
> connection basis. Using the -w option allows the client to override the 
> local tcp receive window settings up to the maximum tcp buffer size. 
> This is just a setsockopt() call really.
I think this should be a ssh_config(5) option (maybe
"TCPReceiveBuffer"
?) rather than a command-line switch (ssh already has enough switches...)

This would allow it to be set either per-connection or globally, and may 
be passed through from the scp command line with the "-o" option.

The latter would also mean that scp would need less modification (and 
scp's code is mostly shared with rcp, so that's also a plus).

Attached is a diff relative to openssh-4.1p1-hpn11.diff with a couple of 
proposed changes:
* move the sshconnect.c setsockopt code into its own function
* make that function style(9) compliant
* fix a bug where strerror was used on the non-error path
* make BUFFER_MAX_HPN_LEN an unsigned to placate gcc -Wsign-compare
* replace magic numbers in channels.h with symbolic names

I don't think I changed any functionality (but I could have missed 
something...)

[1] http://www.iijlab.net/~kjc/software/dist/tunbridge-0.1.tar.gz

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: openssh-hpn.patch
Url:
http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20050804/f556df8c/attachment.ksh

Darren Tucker wrote:
> +#define CHAN_SES_WINDOW_DEFAULT	(BUFFER_MAX_LEN/2)
Thinking about it, those ought to be (BUFFER_MAX_LEN - BUFFER_MAX_CHUNK) 
since in 4.0 and up, the buffers will be compacted once the buffer 
offset is beyond BUFFER_MAX_CHUNK, rather than half of the allocated 
size in previous versions.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Darren Tucker wrote:
> Chris Rapier wrote:
> [throughput]
> 
>>Yeah, thats definitely low. Can you try a loopback to your localhost and
>>dump the data into /dev/null to see what sort of limit the CPU is 
>>imposing? Also, what cipher? 3des is justa nightmare. Most of our tests 
>>are using arcfour and blowfish.
> 
> 
> Tried all that.  All tests done with arcfour, same (low) throughput 
> copying to /dev/null.  And as I mentioned, systems did 6MB/s throughput 
> w/out the high-latency network (which was disk-to-disk).
> 
> I will probably try measuring with ttcp or netperf next.
You might want to try iPerf too. Its a good tool. Better in some 
environments not as good in others.