openssh unix dev - May 2016 - On automatic MAC selection in OpenSSH_6.7p1 + OpenSSL 1.0.1k

Dear All,

I am writing this email because of some observed strange behavior of ssh. I
think this is some sort of complement of the issue that was discussed in
[1] ([2, 3, 4, 5, 6] were related to the issue).

Long story short, I have a client and a server, both running the latest
raspbian jessie with all the relevant updates in two different countries
(i.e. both OpenSSH_6.7p1 Raspbian-5+deb8u2, OpenSSL 1.0.1k 8 Jan 2015).
When I attempt to ssh from one to the other one (direction: country B -->
country A) using the command
$ ssh -p PPPP user at host
then the connection times out after about 2 minutes and I get the error
message `Connection closed by <host>`. (So, it is not the message
`connection reset by peer` that was present in [1].)

Inspecting the configuration for the established link using `-v` I can see
that the cipher selected is `aes128-ctr`, the message authentication code
(mac) is `umac-64-etm at openssh.com` and the compression is  `none`. If I now
go ahead and use the command

$ ssh -m umac-64-etm at openssh.com -p PPPP user at host

quite surprisingly, the key exchange issue that I have above has
disappeared and I can connect to the remote machine from country B to
country A. Further, specifying only the cipher on command line does not
resolve the issue; i.e. using

$ ssh -c aes128-ctr -p PPPPP user at host

the key exchange times out again. Finally, the combined

$ ssh -c aes128-ctr -m umac-64-etm at openssh.com -p PPPPP user at host

also works.


To make things even weirder, when I attempt to ssh between the two machines
in the opposite direction (country A --> country B), then the simple

$ ssh -p PPPP user at host

command works as expected using exactly the same configuration. That is, I
do not have to specify the message authentication code (recall that both
machines are raspberry pi's running the latest raspbian jessie,
OpenSSH_6.7p1 and OpenSSL 1.0.1k).



On another note, lowering the MTU size (which was another workaround for
[1]) also allows me to pass successfully the key exchange phase in the
direction where I normally have an issue (that is, country B --> country
A). The maximum MTU size that would allow me to pass the key exchange
negotiation was 1458 (that is, with a size of 1459 the key exchange got
stuck).



My first attempt to resolve the situation was in Raspberry Pi Stack
Exchange in thread [7]. Since the issue was not resolved there, I posted
the issue in Super User Stack Exchange in thread [8]. It was this thread
that made me aware of the issue in [1] where by lowering the MTU size was a
viable solution. However, this may not be the only issue here, because, as
explained above, automatic configuration of MAC leads to failure (but then
again, only in one direction of establishing the link).

If you need more details on the output or the configurations that I am
using I am happy to provide you with all the relevant information.
Meanwhile, if you would like to read an extended version with debug
information (`ssh -vv ...`), I would suggest [8] which was written more
recently and I think has a better description of the problem.



Best regards,
Dimitris



References

[1] ssh 'connection reset by peer' problem since 5.8p1, url:
http://lists.mindrot.org/pipermail/openssh-unix-dev/2011-March/029432.html
[2] The mysterious case of broken SSH client (?connection reset by peer?),
url:
http://www.held.org.il/blog/2011/05/the-myterious-case-of-broken-ssh-client-connection-reset-by-peer/
[3] Can't login anymore: Read from socket failed: Connection reset by peer,
url: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/708493
[4] Connecting to older ssh version has cipher negotiation problem, url:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=612607
[5] FS#22897 - [openssh] SSH is falling to connect to some server Read from
socket failed: Connection reset by peer, url:
https://bugs.archlinux.org/task/22897
[6] ssh client problem: Connection reset by peer [closed], url:
http://serverfault.com/questions/265244/ssh-client-problem-connection-reset-by-peer

[7] Can not ssh on remote host, url:
http://raspberrypi.stackexchange.com/questions/48089/can-not-ssh-on-remote-host
[8] Strange ssh connection issue, url:
http://superuser.com/questions/1082529/strange-ssh-connection-issue/

On Tue 2016-05-31 10:59:51 -0400, Dimitris Diochnos
wrote:
> On another note, lowering the MTU size (which was another workaround for
> [1]) also allows me to pass successfully the key exchange phase in the
> direction where I normally have an issue (that is, country B --> country
> A). The maximum MTU size that would allow me to pass the key exchange
> negotiation was 1458 (that is, with a size of 1459 the key exchange got
> stuck).
This is the relevant hint for your connection.  It sounds like some
element along the network path from B to A is silently dropping packets
that are larger than 1458, and your network stack has not detected this
situation.

When you force the MAC algorithm to be the specific one, you are
probably making the ssh handshake negotiation packets each be small
enough to fit into the smaller MTU.

As such, i think this is a networking configuration issue, and not
something for ssh to try to fix.  Maybe the fix belongs in your TCP
stack, or in your network configuration?

Sounds frustrating!

          --dkg

I can understand that some element along the path from B to A is
silently dropping packets larger than 1458. However, the other point
that I am making is that when MTU is set to the default 1500 (as
reported by ifconfig) then, the command

ssh -p PPPP user at host

fails (selecting automatically for mac umac-64-etm at openssh.com), while
the command

ssh -m umac-64-etm at openssh.com -p PPPP user at host

succeeds.

In this sense, both commands are executed with MTU=1500 but ssh does
behave differently in these two situations without me having to change
anything in my network configuration. Thus a reasonable (?) guess is
that perhaps ssh does not set all the necessary flags and options
correctly when umac-64-etm at openssh.com is set automatically during the
negotiation, but does set all the necessary flags and options
correctly when the same mac is specified as a command line parameter.

Perhaps my guess is incorrect (after all I have no real knowledge on
this matter), but I thought it would be best if I report this. My
apologies if this is spamming the mailing list.

Best regards,
Dimitris


On Tue, May 31, 2016 at 12:20 PM, Daniel Kahn Gillmor
<dkg at fifthhorseman.net> wrote:
>
> On Tue 2016-05-31 10:59:51 -0400, Dimitris Diochnos wrote:
> > On another note, lowering the MTU size (which was another workaround
for
> > [1]) also allows me to pass successfully the key exchange phase in the
> > direction where I normally have an issue (that is, country B -->
country
> > A). The maximum MTU size that would allow me to pass the key exchange
> > negotiation was 1458 (that is, with a size of 1459 the key exchange
got
> > stuck).
>
> This is the relevant hint for your connection.  It sounds like some
> element along the network path from B to A is silently dropping packets
> that are larger than 1458, and your network stack has not detected this
> situation.
>
> When you force the MAC algorithm to be the specific one, you are
> probably making the ssh handshake negotiation packets each be small
> enough to fit into the smaller MTU.
>
> As such, i think this is a networking configuration issue, and not
> something for ssh to try to fix.  Maybe the fix belongs in your TCP
> stack, or in your network configuration?
>
> Sounds frustrating!
>
>           --dkg