openssh bugs - May 2021 - [Bug 3279] UpdateHostKeys triggers "client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0" error message

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

            Bug ID: 3279
           Summary: UpdateHostKeys triggers
                    "client_global_hostkeys_private_confirm: server gave
                    bad signature for RSA key 0" error message
           Product: Portable OpenSSH
           Version: 8.5p1
          Hardware: Other
                OS: Windows 10
            Status: NEW
          Severity: normal
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: thomas.braun at byte-physics.de

Upstream issue: https://github.com/git-for-windows/git/issues/3108

$ ssh -V
OpenSSH_8.5p1, OpenSSL 1.1.1j  16 Feb 2021

Platform:
Windows 10

The following recipe requires a gitlab.com account with ssh key
attached.

```
$ ssh git at gitlab.com -i ~/.ssh/XXX
client_global_hostkeys_private_confirm: server gave bad signature for
RSA key 0
PTY allocation request failed on channel 0
Welcome to GitLab, @XXX!
Connection to gitlab.com closed.
```

As found on the internet disabling UpdateHostKeys turns the error
message off:

```
$ ssh -o UpdateHostKeys=no git at gitlab.com -i ~/.ssh/github_ed255519
PTY allocation request failed on channel 0
Welcome to GitLab, @t-b!
Connection to gitlab.com closed.
```

The ssh server:

```
debug1: Remote protocol version 2.0, remote software version
OpenSSH_7.9p1 Debian-10+deb10u2
debug1: compat_banner: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat
OpenSSH* compat 0x04000000
````

Host Key of the server:

```
gitlab.com ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
```

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
Could you please attach a complete debug trace (ssh -vvv ...)?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #2 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3505
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3505&action=edit
stderr output

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #3 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3506
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3506&action=edit
stdout output

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #4 from Thomas Braun <thomas.braun at byte-physics.de> ---
Done.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3505|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3506|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
Are you able to test OpenSSH git head or otherwise apply commit
ac31aa3c63 ? It adds some debugging that might be useful in figuring
out what is going wrong.

Also a workaround: add

Host gitlab.com
    UpdateHostkeys no

to your ~/.ssh/config

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
Created attachment 3507
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3507&action=edit
use old-style RSA signature algorithm for SSH_BUG_SIGTYPE servers

Please also try this patch

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #7 from Thomas Braun <thomas.braun at byte-physics.de> ---
Yes I should be able compile openssh HEAD. I presume we are talking
about https://github.com/openssh/openssh-portable?

Do you think I can test that on linux as well or is that specific to
Windows?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Richard W.M. Jones <rjones at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rjones at redhat.com

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #8 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3513
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3513&action=edit
stderr with ac31aa3c63 applied

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #9 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3514
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3514&action=edit
stdout with ac31aa3c63 applied

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #10 from Thomas Braun <thomas.braun at byte-physics.de> ---
I've applied the SSH_BUG_SIGTYPE fix but that did not solve the issue.

I also added the requested debug output with ac31aa3c63 applied.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3513|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3514|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #11 from Damien Miller <djm at mindrot.org>
---
> client_global_hostkeys_private_confirm: server gave bad signature for RSA
key 0: error in libcrypto
hmm, this is not what I expected. This particular error can only occur
during RSA verification here:
https://github.com/openssh/openssh-portable/blob/e86968280e358e62649d268d41f698d64d0dc9fa/ssh-rsa.c#L429
and indicates an RSA decryption failure in OpenSSL libcrypto.

Moreover I can't reproduce the same problem with OpenSSH 7.9 sshd
locally - the hostkey update signature function fine for RSA keys.

This makes me suspect that either gitlab.com is returning an incorrect
signature, or OpenSSL libcrypto is failing to verify a good one on your
platform.

I don't know much about how the ssh client in git-for-windows works. Is
it built from Cygwin, Microsoft's OpenSSH port or something else?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #12 from Thomas Braun <thomas.braun at byte-physics.de>
---
> I don't know much about how the ssh client in git-for-windows works. Is
it built from Cygwin, Microsoft's OpenSSH port or something else?
It's basically built from cygwin. It's called MSYS which is a
derivation of cygwin.

The sources are available at
https://github.com/git-for-windows/MSYS2-packages/tree/main/openssh. It
is not particular easy to build though.

Is there a way I can store the failing RSA key in a file?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #13 from Damien Miller <djm at mindrot.org> ---
Created attachment 3521
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3521&action=edit
dump failed key and signature

This will log the failing key and signature.

You can convert the key to an OpenSSL PEM format key using something
like:

ssh-keygen -ef /path/key.pub -m pem

Verifying the contents of the signature blob is more difficult. Some
extra debug logging in ssh-rsa.c might be required there

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #14 from Damien Miller <djm at mindrot.org> ---
Created attachment 3522
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3522&action=edit
debug failed libcrypto call

This will dump the actual data passed to RSA_public_decrypt() and the
detailed errors from libcrypto when it fails

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #15 from Damien Miller <djm at mindrot.org> ---
Any update on this?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #16 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3535
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3535&action=edit
debug output with latest patches

Sorry for taking so long to respond.

Attached is the output generated with `ssh.exe -vv git at gitlab.com -i
~/.ssh/github_ed255519 2> output-with-debug-info.txt`.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #17 from Thomas Braun <thomas.braun at byte-physics.de> ---
I've applied both the " dump failed key and signature" and the
" dump
failed key and signature" patches.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #18 from Damien Miller <djm at mindrot.org> ---
The debugging contains a signature from the server, being (hex
encoded):

484a9f2d247575206758581a99945cc0964dcb38e83ec57204d10f3421180131677082cb4aaa90a0bd2df2927e40382903866677caf2553f152156a77fe8462ce20466a08ec530ee3be6f9780e39b9f112d98db389e225d2025ad0b2a27498b56f36ce322f03124f911f4bb92311d3870dbfb988b46898d6a7b94833a6af9c9eefab7c77ca082aba62161c79d7f73842681e8da68d4b527b27104701f5e74136028550ccca3f55a1c0cafa35be7afdda609712d5d278122977867c9d1925cfee89687950be9837e696a472f9cd92a9dd7afbf4eae25d225b497da87d24af99ead3302dfbbf2085d04a91277e850ebf46828da7e5b04a9938649fe0c94a240d42

I retrieved gitlab.com's rsa key. It's:

ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9

With these, I hacked up a small program to load the key and run the 
libcrypto RSA_public_decrypt() operation that fails in your case.

It worked for me and yielded a decrypted signature:

decrypted: len=35
0000: 30 21 30 09 06 05 2b 0e 03 02 1a 05 00 04 14 59  0!0...+........Y
0016: 90 c1 b8 16 fd f3 aa a4 d8 a6 3f 94 e0 21 03 c5  ..........?..!..
0032: e4 c2 c7   

This is a structurally valid PKCS#1 1.5 rsa-sha1 padded hash.

So I think that something is wrong inside your libcrypto/OpenSSL

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #19 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Damien Miller from comment #18)
[...]
> So I think that something is wrong inside your libcrypto/OpenSSL
If you built your own libcrypto, try running its self-tests (OpenSSL:
"make test", LibreSSL: "make check") and see if those pass.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #20 from Thomas Braun <thomas.braun at byte-physics.de> ---
Created attachment 3536
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3536&action=edit
openssl test result

Thanks both.

I just rebuilt openssl and ran it's tests and all pass.

Just out of curiosity:
Why is the error I'm seeing only present when "UpdateHostKeys" is
turned on? And why does the decryption error not influence my ability
to connect to the server?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #21 from Thomas Braun <thomas.braun at byte-physics.de> ---
@Damien Can you post your small program so that I can run it here as
well?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

--- Comment #22 from Damien Miller <djm at mindrot.org> ---
Created attachment 3537
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3537&action=edit
test "program"

This is what I used, I basically hacked it in to ssh-keygen.c. It's the
opposite of pretty :)

You'll also need to repack the dumped signature into base64 and paste
it in to the sshbuf_b64tod() call. I used something like:

$ python3
>>> h='484a9f2d24757...' # Line from RSA_public_decrypt:
sig=...
>>> import base64
>>> b=base64.b16decode(h, True)
>>> base64.b64encode(b)
b'SEqfLSR1dS...'

As to why ssh works while this particular operation is failing while
your ssh connection remains successful - it's probably because you're
not using RSA for the regular key exchange signature, but Ed25519:

debug1: kex: host key algorithm: ssh-ed25519


If you try something like:

for x in rsa-sha2-512 rsa-sha2-256 ssh-rsa ; do ssh
-oHostkeyAlgorithms=$x -oStrictHostkeyChecking=no
-oUserKnownHostsFile=/tmp/gitlab git at gitlab.com ; done

Then you can exercise RSA (across its variants) in the signature path
too

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3522|0                           |1
        is obsolete|                            |

--- Comment #23 from Damien Miller <djm at mindrot.org> ---
Created attachment 3538
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3538&action=edit
Better RSA verification debugging

Actually, instead of messing around with python please replace the
previous debugging diff with this one. It dumps the key in usable
format and the signature blob in base64

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3279

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WORKSFORME

--- Comment #24 from Damien Miller <djm at mindrot.org> ---
closing for lack of followup

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.