openssh unix dev - Nov 2013 - Re: VPN MTU limit breaks ssh connection to openssh 6.2p2 server

On Thu, 7 Nov 2013, Ernst Kratschmer wrote:
> Dear openssh developer, 
> 
> I want to use a Win7 client with putty to access a Linux host running an 
> openssh 6.2p2 through a VPN connection. These connection worked relatively 
> flawless with  all versions of openssh up until openssh 6.1p1. Since the 
> openssh 6.2p2 upgrade the ssh connection fail consistently with a message: 
> Network error, connection reset by peer. After inspecting the tcp 
> transmission between the putty client and openssh server it appears that 
> the maximum MTU limit of 1362 of the VPN connection, causes the server to 
> break the 1460 byte cipher string into two packets. At that point the ssh 
> client, putty or openssh, resets the connection. 
> 
> This was not a problem with openssh 6.1p1 since the cipher string was only 
> 1106 bytes and therefore transmitted in one packet. Since I am stuck with 
> the VPN MTU limit of 1362, I am hoping that you could help in some form to 
> get the ssh connection working again by, e.g., limit the cipher string to 
> less than 1322 bytes?
I don''t think this is something we can fix in OpenSSH. You could work
around it by shrinking the list of ciphers/MACs/key exchange algorithms
that are offered using sshd_config''s "Ciphers",
"MACs" and "KexAlgorithms"
options.

-d

> On Thu, 7 Nov 2013, Ernst Kratschmer wrote:
> 
> > Dear openssh developer, 
> > 
> > I want to use a Win7 client with putty to access a Linux host running
an
> > openssh 6.2p2 through a VPN connection. These connection worked
relatively
> > flawless with  all versions of openssh up until openssh 6.1p1. Since
the
> > openssh 6.2p2 upgrade the ssh connection fail consistently with a
message:
> > Network error, connection reset by peer. After inspecting the tcp 
> > transmission between the putty client and openssh server it appears
that
> > the maximum MTU limit of 1362 of the VPN connection, causes the server
to
> > break the 1460 byte cipher string into two packets. At that point the
ssh
> > client, putty or openssh, resets the connection. 
> > 
> > This was not a problem with openssh 6.1p1 since the cipher string was
only
> > 1106 bytes and therefore transmitted in one packet. Since I am stuck
with
> > the VPN MTU limit of 1362, I am hoping that you could help in some
form to
> > get the ssh connection working again by, e.g., limit the cipher string
to
> > less than 1322 bytes?
Damien answered :
> I don''t think this is something we can fix in OpenSSH.
I don''t get it. Could you elaborate on this ?

Please correct me if I''m wrong, but from my point of view, TCP
splitting packets to respect MTU somewhere in transit should not have
any impact at the functionnal level. We should not expect the values
returned by read()/recv() to match those we gave to write()/send().
When read()-ing on a socket, the size returned is at the kernel (or
libc) discretion. If it prefers to send us the data byte by byte, it
is allowed to, and the process is supposed to wait for more data if
it expects more.

I''m not telling it''s easy to implement in OpenSSH as I never
seriously
looked at the code but saying it cannot be done surprises me.

Thanks for your clues !

Flavien.

On Fri, Nov 08, 2013 at 11:52:11AM +0100, Flavien Lebarbe
wrote:
> > On Thu, 7 Nov 2013, Ernst Kratschmer wrote:
[vpn + mtu problem]
> Damien answered :
> > I don''t think this is something we can fix in OpenSSH.
> 
> I don''t get it. Could you elaborate on this ?
> 
> Please correct me if I''m wrong, but from my point of view, TCP
> splitting packets to respect MTU somewhere in transit should not have
> any impact at the functionnal level. We should not expect the values
> returned by read()/recv() to match those we gave to write()/send().
it doesn''t.  it does expect that they make it to the other end, though,
which doesn''t seem to happen in this case.
> When read()-ing on a socket, the size returned is at the kernel (or
> libc) discretion. If it prefers to send us the data byte by byte, it
> is allowed to, and the process is supposed to wait for more data if
> it expects more.
> 
> I''m not telling it''s easy to implement in OpenSSH as I
never seriously
> looked at the code but saying it cannot be done surprises me.
TCP is supposed to be a bi-directional reliable byte stream and the ssh
protocol doesn''t rely on it being anything more than that.

The network in this case seems broken.  It sounds like the classic "MTU
causes IP fragmentation then something else drops the fragments" problem.
Often something doing this will also reduce the TCP MSS so that the
endpoints don''t emit IP packets big enough to fragment, but it seems
not in this case.

It can be worked around in the config as described by Damien but any
attempt to work around it in openssh would be an ugly hack at best and
would likely hurt performace on networks that aren''t broken.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On 8 Nov 2013, at 15:26, Ernst Kratschmer wrote:
> If I understand Darren correctly, he is concerned that the packet 
> fragmentation causes fragments to get dropped. From my debug, I can see 
> that the 1460 byte packet get split into two packets exactly as expected 
> from the MTU limit.
Run a ping with DF set across your tunnel and sweep the packet size. This
should stop working at one particular packet size.

Now run a ping without DF set across your tunnel and repeat. This should
work for all packet sizes. I bet one particular size fails. This will be
breaking pMTU discovery.

-- 
Alex Bligh

Ernst Kratschmer wrote:
> From my debug, I can see that the 1460 byte packet get split into
> two packets exactly as expected from the MTU limit.
Nobody is challenging this.

> And both packets get sent to the client.
Nor this. But what really matters is that they are both well-formed
and that they *arrive* at the client. I''m not sure you''ve
provided
information about whether that is the case.

> The client then resets the communication.
If either packet is malformed then that''s the right thing to do.

> This happens regardless of the type of client software used,
> Cygwin, putty, openssh any version, etc.
As you know, TCP is not implemented by the client software, but by
the operating system network stack. And given an invalid packet most
operating systems will do the same (right) thing.


Please analyze what the client receives, if it is two packets then
investigate whether they are well-formed. If they are, then look
further up the layers.


//Peter

Ernst Kratschmer wrote:
> I am sorry, but I don''t know what "ping with DF set" is.
> I can run "ping -l size". This fails if size is greater than
1252, over
> VPN or LAN. 
> Not sure if this helps. 
On Linux:

$ man ping | grep -A 1 -B 5 DF

       -M pmtudisc_opt
              Select  Path  MTU  Discovery  strategy.   pmtudisc_option may be
              either do (prohibit fragmentation, even  local  one),  want  (do
              PMTU  discovery, fragment locally when packet size is large), or
              dont (do not set DF flag).

$


//Peter