openssh bugs - May 2020 - [Bug 3170] New: Sometimes sshd responds with different server signature

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

            Bug ID: 3170
           Summary: Sometimes sshd responds with different server
                    signature
           Product: Portable OpenSSH
           Version: 8.1p1
          Hardware: ARM
                OS: Other
            Status: NEW
          Severity: normal
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: steven at c88.org

Created attachment 3397
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3397&action=edit
sshd and ssh logs during good and bad transactions

I am running openssh-server 8.1p-1 on a Netgear R7800 router (running
DD-WRT).

I am having a problem with basic SSH operation with openssh. I have a
separate dropbear SSH server running on a different port (which has
been rock solid) and configured openssh to use another for testing
purposes.

I find that about 80% of the time, I can do an ssh operation from my
Cygwin laptop to the openssh server with no problem. But the other
times I get an error "incorrect signature" which tells me the server
sometimes returns a different signature than my client expected.

I was able to capture debug logging on both the SSH client and the SSH
server when I executed "ssh router ls -a" for both good and bad
(failed
with incorrect signature) transactions.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
This could be a problem with libcrypto's signature generation or
verification. IMO the first step would be to figure out whether it is
the client or the server that is going wrong.

Could you try a different client (e.g. openssh on Linux, or PuTTY on
Windows)? If the problem persists then it's likely the server is at
fault.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #2 from Steven C <steven at c88.org> ---
I did repeated sessions with Windows PuTTY release 7.0.

I got about the same frequency of failures with the message: "Server's
host key did not match the signature supplied."

So it sounds like the issue is with the server.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
Did you compile openssh/openssl yourself or did you use DD-WRT's
packages?

If you compiled OpenSSL yourself, then please run its self-tests and
see if they catch anything. Likewise OpenSSH ("make tests" after
building.

If you're using DD-WRT's pre-built packages then I recommend either
filing a bug on their bug tracking system or building your own
openssl/openssh so you can run the above self-tests.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #4 from Steven C <steven at c88.org> ---
I used the pre-compiled package provided by the Entware system in
DD-WRT (https://github.com/Entware/Entware).

I will enter a bug in their system, but I fear they don't do much
except compile pre-existing applications and make them available
through the "opkg" command.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #5 from Steven C <steven at c88.org> ---
By the way, regarding "make tests" - does that work in a cross-compile
environment?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #6 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Steven C from comment #5)
> By the way, regarding "make tests" - does that work in a
> cross-compile environment?
No, the tests rely on being able to run the built executables from the
Makefiles.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #7 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Steven C from comment #4)
> I used the pre-compiled package provided by the Entware system in
> DD-WRT (https://github.com/Entware/Entware).
> 
> I will enter a bug in their system, but I fear they don't do much
> except compile pre-existing applications and make them available
> through the "opkg" command.
They apply a dozen patches to their openssl, including some to the
crypto engines:
https://github.com/Entware/Entware/tree/master/package/libs/openssl/patches,
some of which invoke /dev/crypto and based on the kernel logs from
https://openwrt.org/toh/netgear/r7800 it looks like your device has
crypto hardware, so all of libcrypto, the kernel and the hardware are
potential causes too.

I'd suggest trying the other host key types and see if the problem
occurs with all of them or only a subset.

I've also seen similar problems caused by bad ram and buggy compilers. 
There's an awful lot of variables, and if you can't change sshd you
won't be able to eliminate many of them.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

--- Comment #8 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Darren Tucker from comment #6)
> No, the tests rely on being able to run the built executables from
> the Makefiles.
actually, in theory if you copied the build directory in its entirety
and had the required tools (at least make, but probably others) then it
might be possible to run it on the device.  I have in the past done
native builds and tests on openwrt, but it took some setting up and
it's far from an ideal platform.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3170

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
closing for lack of followup

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.