I have a customer who is complaining about slow SFTP transfers over a long haul connection. The current transfer rate is limited by the TCP window size and the RTT. I looked at HPN-SSH, but that won't work because we don't control what software the peer is using. I was thinking about coding a much more modest enhancement that just does SO_RCVBUF for specific subsystems. In the interest of adding something that the OpenSSH community would take back into the source base, what do people think is a better fit in the configuration file? A single keyword for specifying connection options: SubsystemOptions <subsystem-name> [command-line-option...] The subsystem name would be scp, sftp-server, etc. The command line options would be -r <n> or --rcvbuf <n> to allow one to specify the number passed to SO_RCVBUF. This would allow one to easily add other options (like SO_SNDBUF support). An alternative would be to add the following option: SubsystemRcvbuf <subsystem-name> <n> This would be much easier to implement, but doesn't allow for more options to be added without another keyword. I guess this is the way I'm leaning, but I can be swayed.
On Thu, Dec 12, 2019 at 12:00:40AM +0000, Robinson, Herbie wrote:> I have a customer who is complaining about slow SFTP transfers over a > long haul connection. The current transfer rate is limited by the TCP > window size and the RTT. I looked at HPN-SSH, but that won't work > because we don't control what software the peer is using. I was > thinking about coding a much more modest enhancement that just does > SO_RCVBUF for specific subsystems. In the interest of adding > something that the OpenSSH community would take back into the source > base, what do people think is a better fit in the configuration file?It is possible to simply tweak the kernel default instead, at least on Linux (tcp_rmem). Is this not workable in your case? David
>On Thu, Dec 12, 2019 at 12:00:40AM +0000, Robinson, Herbie wrote:> > I have a customer who is complaining about slow SFTP transfers over a > > long haul connection. The current transfer rate is limited by the TCP > > window size and the RTT. I looked at HPN-SSH, but that won't work > > because we don't control what software the peer is using. I was > > thinking about coding a much more modest enhancement that just does > > SO_RCVBUF for specific subsystems. In the interest of adding > > something that the OpenSSH community would take back into the source > > base, what do people think is a better fit in the configuration file?David Wilson replied:> It is possible to simply tweak the kernel default instead, at least on Linux (tcp_rmem). Is this not workable in your case?It's not Linux, but there is a mechanism and we have used it to get them a small performance boost. Unfortunately, the customer only wants the large window size for the sftp upload, not all connections. They are looking at window sizes somewhere between 250K and 500K to get the speed they really want. They would overload the long haul line if they did that for all connections.>On Thu, Dec 12, 2019 at 12:00:40AM +0000, Robinson, Herbie wrote:> > A single keyword for specifying connection options: > > > > SubsystemOptions <subsystem-name> [command-line-option...] > > > > The subsystem name would be scp, sftp-server, etc. The command line options would be -r <n> or --rcvbuf <n> to allow one to specify the number passed to SO_RCVBUF. This would allow one to easily add other options (like SO_SNDBUF support). > > > > An alternative would be to add the following option: > > > > SubsystemRcvbuf <subsystem-name> <n> > > > > This would be much easier to implement, but doesn't allow for more options to be added without another keyword. I guess this is the way I'm leaning, but I can be swayed.Darren Tucker Replied:> Sending, receiving or both?Server (sftp-server) receiving.> What's the speed and RTT on the link?The RTT is between 40 and 50 msec. The window when the customer measured was 32K. The throughput they saw was 6mbit/sec (which implies an RTT of 44 msec). We worked with them to adjust kernel parameters so they get 65K windows and that's kept them happy in the short term, but they also indicated they were hoping for 100mb; so, I'm trying to get ahead of them before they come back for more.> If your limiting factor is really the TCP BDP that's between the ssh and sshd process, and that's independent of subsystem.Correct. The actual TCP socket is not available to the subsystem. If it was, I could just add options to the sftp-server code. Given that upping the window size to a really big number is not a good idea for all connections, I wanted to have sshd decide what to use based on the subsystem. Perhaps it might also be useful to select the window size based on input and/or output subnet masks, too. Opinions?> There is no scp subsystem.I guess I'm taking a little poetic license there. If it turns out to be difficult to include scp, as a selector, I would just drop it.> If you're talking about the client side you could implement whatever you want in a ProxyCommand dialer then possibly pass the open fd back via ProxyUseFdpass.I am looking to address a server issue.
On Fri, 13 Dec 2019 at 03:16, Robinson, Herbie <Herbie.Robinson at stratus.com> wrote:> [...]Darren Tucker Replied:> > Sending, receiving or both? > Server (sftp-server) receiving. >Earlier you said "I looked at HPN-SSH, but that won't work because we don't control what software the peer is using" but you're now talking about modifying the server. Last time I looked at HPN it'd interop with other implementations so any changes it did to its socket buffers would also affect any other client.> What's the speed and RTT on the link? > > The RTT is between 40 and 50 msec. The window when the customer measured > was 32K. The throughput they saw was 6mbit/sec (which implies an RTT of 44 > msec). We worked with them to adjust kernel parameters so they get 65K > windows and that's kept them happy in the short term, but they also > indicated they were hoping for 100mb; so, I'm trying to get ahead of them > before they come back for more. >The other limiting factor you may see is the number and size of SFTP operations in flight. OpenSSH's sftp defaults to 64 requests of 32k each but other clients may vary.> If your limiting factor is really the TCP BDP that's between the ssh and > sshd process, and that's independent of subsystem. > > Correct. The actual TCP socket is not available to the subsystem. If it > was, I could just add options to the sftp-server code. Given that upping > the window size to a really big number is not a good idea for all > connections, I wanted to have sshd decide what to use based on the > subsystem.The problem with doing it per-subsystem is that subsystem is a per ssh channel thing and tcp socket buffers are a per connection thing. There are zero or more channels per TCP connection and it's possible to have both interactive and sftp channels in a single connection, either serially or concurrently.> Perhaps it might also be useful to select the window size based on input > and/or output subnet masks, too. Opinions? >If it was a standard sshd option then it could be conditionalized using Match, which would let you select based on any of: User, Group, Host, LocalAddress, LocalPort, RDomain, Address. Would that address your use case? There was a request to do "Match subsystem" in the past however I didn't think it was a good architectural fit since all the other match things are per-connection and set early in the connection process. -- Darren Tucker (dtucker at dtucker.net) GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA (new) Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
Darren Tucker Wrote:> On Fri, 13 Dec 2019 at 03:16, Robinson, Herbie <mailto:Herbie.Robinson at stratus.com> wrote: > [...] > Darren Tucker Replied: > > Sending, receiving or both?? > Server (sftp-server) receiving. > > Earlier you said "I looked at HPN-SSH, but that won't work because we don't control what software the peer is using" but you're now talking about modifying the server.? Last time I looked at HPN it'd interop with other implementations so any changes it did to its socket buffers would also affect any other client.I looked at the comments around the code that did setsockopt for SO_RCVBUF and it appeared to only apply when the peer -- the comments implied that there are implementations out there that will choke on large window sizes. I will take a more careful look at it (there are other reasons I was shying away that are probably OT here).> > What's the speed and RTT on the link? > > The RTT is between 40 and 50 msec.? The window when the customer measured was 32K.? The throughput they saw was 6mbit/sec (which implies an RTT of 44 msec).? We worked with them to adjust kernel parameters so they get 65K windows and that's kept them happy in the short term, but they also indicated they were hoping for 100mb; so, I'm trying to get ahead of them before they come back for more. > > > If your limiting factor is really the TCP BDP that's between the ssh and sshd process, and that's independent of subsystem. > > Correct.? The actual TCP socket is not available to the subsystem.? If it was, I could just add options to the sftp-server code.? Given that upping the window size to a really big number is not a good idea for all connections, I wanted to have sshd decide what to use based on the subsystem. > > The problem with doing it per-subsystem is that subsystem is a per ssh channel thing and tcp socket buffers are a per connection thing.? There are zero or more channels per TCP connection and it's possible to have both interactive and sftp channels in a single connection, either serially or concurrently. >? >? Perhaps it might also be useful to select the window size based on input and/or output subnet masks, too.? Opinions? > > If it was a standard sshd option then it could be conditionalized using Match, which would let you select based on any of: User, Group, Host, LocalAddress, LocalPort, RDomain, Address.? Would that address your use case?I missed "Match" when I was looking through the documentation. That is a much better way to go. Now, I need to go look at whether HPN-SSH operates under Match or not. Thanks. ?