openssh bugs - Dec 2022 - [Bug 3506] Permission denied (publickey) with two -i identity files

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

            Bug ID: 3506
           Summary: Permission denied (publickey) with two -i identity
                    files
           Product: Portable OpenSSH
           Version: 9.1p1
          Hardware: ARM64
                OS: Mac OS X
            Status: NEW
          Severity: critical
          Priority: P5
         Component: Miscellaneous
          Assignee: unassigned-bugs at mindrot.org
          Reporter: andy.klier at zuar.com

Created attachment 3627
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3627&action=edit
verbose output of ssh attempt

attempting to ssh with two identity files fails with `Permission denied
(publickey)`

for example:
```
ssh -i signed-cert.pub -i ~/.ssh/id_rsa username at 10.0.23.5
```

only able to resolve by downgrading to 8.6.

this is a fairly common process to ssh with two `-i` identity files,
for example if you're using vault as a certificate authority. as
described here:
https://developer.hashicorp.com/vault/docs/secrets/ssh/signed-ssh-certificates#client-ssh-authentication

full ssh with -vvv attached

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
Comment on attachment 3627
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3627
verbose output of ssh attempt

The handling of -i hasn't changed as far as I know.

[...]
>debug1: identity file
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert type 4
>debug1: identity file
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert-cert type -1
>debug1: identity file /Users/steve/.ssh/vault type 0
>debug1: identity file /Users/steve/.ssh/vault-cert type -1
This doesn't exactly match the example invocation, but it indicates
that two keys were loaded.

[...]
>debug1: Offering public key:
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
>debug1: send_pubkey_test: no mutual signature algorithm
I think this is your problem: ssh-rsa was disabled by default in 8.8
(https://www.openssh.com/releasenotes.html#8.8).  You can test this by
adding "-oPubkeyAcceptedAlgorithms=+ssh-rsa" to your command line. 
I'm
not sure why it didn't try one of the stronger RSA SHA256/512 variants.
>debug1: Offering public key: /Users/steve/.ssh/vault RSA
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
>debug3: send packet: type 50
>debug2: we sent a publickey packet, wait for reply
>debug3: receive packet: type 51
>debug1: Authentications that can continue: publickey
This key is not in the server's authorized_keys.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
Tne problem is here:
> debug1: Offering public key:
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
> debug1: send_pubkey_test: no mutual signature algorithm
What does `ssh -G username at 10.0.23.5 | grep ^pubkeyacceptedalgorithms`
report?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #3 from andy klier <andy.klier at zuar.com> ---
```
pubkeyacceptedalgorithms
ssh-ed25519-cert-v01 at openssh.com,ecdsa-sha2-nistp256-cert-v01 at
openssh.com,ecdsa-sha2-nistp384-cert-v01 at
openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,sk-ssh-ed25519-cert-v01
at openssh.com,sk-ecdsa-sha2-nistp256-cert-v01 at
openssh.com,rsa-sha2-512-cert-v01 at openssh.com,rsa-sha2-256-cert-v01 at
openssh.com,ssh-rsa-cert-v01 at
openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519
at openssh.com,sk-ecdsa-sha2-nistp256 at
openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
```
> Tne problem is here:
> 
> > debug1: Offering public key:
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
> > debug1: send_pubkey_test: no mutual signature algorithm
> 
> What does `ssh -G username at 10.0.23.5 | grep
> ^pubkeyacceptedalgorithms` report?
-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #4 from andy klier <andy.klier at zuar.com> ---
(In reply to Darren Tucker from comment #1)
> Comment on attachment 3627 [details]
> verbose output of ssh attempt
> 
> The handling of -i hasn't changed as far as I know.
> 
> [...]
> >debug1: identity file
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert type 4
> >debug1: identity file
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert-cert type -1
> >debug1: identity file /Users/steve/.ssh/vault type 0
> >debug1: identity file /Users/steve/.ssh/vault-cert type -1
> 
> This doesn't exactly match the example invocation, but it indicates
> that two keys were loaded.
> 
> [...]
> >debug1: Offering public key:
/Users/steve/.config/zaccess/penguin.randomhostname.com.cert RSA-CERT
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
> >debug1: send_pubkey_test: no mutual signature algorithm
> 
> I think this is your problem: ssh-rsa was disabled by default in 8.8
> (https://www.openssh.com/releasenotes.html#8.8).  You can test this
> by adding "-oPubkeyAcceptedAlgorithms=+ssh-rsa" to your command
> line.  I'm not sure why it didn't try one of the stronger RSA
> SHA256/512 variants.
> 
> >debug1: Offering public key: /Users/steve/.ssh/vault RSA
SHA256:v1zotU9ug24hc109SJwmsnWA0JQHABY/t2NjMn/SDVM explicit
> >debug3: send packet: type 50
> >debug2: we sent a publickey packet, wait for reply
> >debug3: receive packet: type 51
> >debug1: Authentications that can continue: publickey
> 
> This key is not in the server's authorized_keys.
TBC the example command is from the vault docs. the command we are
running is: 
ssh -i /Users/steve/.config/zaccess/penguin.randomhostname.com.cert -i
/Users/steve/.ssh/vault ubuntu at penguin.randomhostname.com

the pub key for the CA is in `TrustedUserCAKeys` in
`/etc/ssh/sshd_config`.
we sign a cert using `/Users/steve/.ssh/vault.pub` and then ssh with
the cert and it's private key.

with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #5 from Darren Tucker <dtucker at dtucker.net>
---
> with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails.
oh, it's a cert not a plain key.  Try CASignatureAlgorithms=+ssh-rsa
instead.  (same goes for Damien's suggestion: what does "ssh -G
username at 10.0.23.5 | grep ^casignaturealgoritms" output?)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
I think the problem is that OpenSSH 7.6 is marked as having
SSH_BUG_SIGTYPE:

https://github.com/openssh/openssh-portable/blob/f84b9cffd52c9c5c359a54a1929f9948e803ab1d/compat.c#L74

and so falls in to this special-case:

https://github.com/openssh/openssh-portable/blob/f84b9cffd52c9c5c359a54a1929f9948e803ab1d/sshconnect2.c#L1200

That was introduced in
https://github.com/openssh/openssh-portable/commit/4ba0d54794814ec0de1ec87987d0c3b89379b436

I don't remember exactly which bug(s) SSH_BUG_SIGTYPE was added to
cover, but I do note that it was subsequently split into
SSH_BUG_SIGTYPE and SSH_BUG_SIGTYPE74. Maybe 7.6 could be made to work
with some more compat glue?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #7 from Damien Miller <djm at mindrot.org> ---
Here's an excerpt from an review email on that commit:
> This re-adds the agent signature type checking fallback, but also moves
> forward with the foreshadowed stricter signature type checking. sshd
> will check that the type of a pubkey signature is consistent with the
> key type advertised in the USERAUTH_REQUEST packet.
> 
> Because OpenSSH >= 7.6 were lax with this in some bases (legacy agents
> being one), I've put this stricter checking behind a compat flag and
> cranked the version number.
> 
> This also switches the various key algorithm knobs
> (PubkeyAcceptedKeyTypes, etc.) to check the pktype as is present in the
> userauth packet and not the type of the deserialised key blob as they
> currently do. The effect of this is to allow users to ban ssh-rsa if
> they want. More users have been asking to be able to do this to prevent
> the use of SHA-1 but still use RSA keys (via the rsa-sha2-* sigtypes).
> 
> This tweak to semantics has some possibility of breaking existing
> configurations, e.g. if they had "PubkeyAcceptedKeyTypes=ssh-rsa"
in
> sshd_config. It might be able to fix this risk with some creative
> filtering of proposal strings.
So the bug that is being avoided (badly as far as certs are concerned)
is that <=7.6 would use an incorrect "public key algorithm name"
field
in the SSH2_USERAUTH_REQUEST message
(https://www.rfc-editor.org/rfc/rfc4252#section-7) - it should actually
be the _signature_ algorithm being used and not the type of the public
key (this is arguably an error in the RFC)

Making this work for back-compat would require arranging for
sshconnect2.c:key_sig_algorithm() to return two strings: the actual
signing algorithm identifier as it does currently plus the algorithm
identifier to use in when the SSH2_AUTH_REQUEST is assembled in
sign_and_send_pubkey().

However. it's highly likely that additional compat problems would
remain. IIRC ssh-agent had problems wrt RSA/SHA2 signatures around then
too.

None of this helps you right now unfortunately. If you need something
sooner than someone getting motivated to fix this (which, honestly, may
never happen) than you options are 1) update the server to a
version
>=7.8 or change to a different key type - ECDSA or ED25519 are fine
choices.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #8 from andy klier <andy.klier at zuar.com> ---
(In reply to Darren Tucker from comment #5)
> > with -oPubkeyAcceptedAlgorithms=+ssh-rsa also fails.
> 
> oh, it's a cert not a plain key.  Try CASignatureAlgorithms=+ssh-rsa
> instead.  (same goes for Damien's suggestion: what does "ssh -G
> username at 10.0.23.5 | grep ^casignaturealgoritms" output?)
hey sorry for the late response. `-oCASignatureAlgorithms=+ssh-rsa`
doesnt work either, and `grep ^casignaturealgoritms` returns nothing.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

--- Comment #9 from andy klier <andy.klier at zuar.com> ---
(In reply to Damien Miller from comment #7)
> Here's an excerpt from an review email on that commit:
> 
> > This re-adds the agent signature type checking fallback, but also
moves
> > forward with the foreshadowed stricter signature type checking. sshd
> > will check that the type of a pubkey signature is consistent with the
> > key type advertised in the USERAUTH_REQUEST packet.
> > 
> > Because OpenSSH >= 7.6 were lax with this in some bases (legacy
agents
> > being one), I've put this stricter checking behind a compat flag
and
> > cranked the version number.
> > 
> > This also switches the various key algorithm knobs
> > (PubkeyAcceptedKeyTypes, etc.) to check the pktype as is present in
the
> > userauth packet and not the type of the deserialised key blob as they
> > currently do. The effect of this is to allow users to ban ssh-rsa if
> > they want. More users have been asking to be able to do this to
prevent
> > the use of SHA-1 but still use RSA keys (via the rsa-sha2-* sigtypes).
> > 
> > This tweak to semantics has some possibility of breaking existing
> > configurations, e.g. if they had
"PubkeyAcceptedKeyTypes=ssh-rsa" in
> > sshd_config. It might be able to fix this risk with some creative
> > filtering of proposal strings.
> 
> So the bug that is being avoided (badly as far as certs are
> concerned) is that <=7.6 would use an incorrect "public key
> algorithm name" field in the SSH2_USERAUTH_REQUEST message
> (https://www.rfc-editor.org/rfc/rfc4252#section-7) - it should
> actually be the _signature_ algorithm being used and not the type of
> the public key (this is arguably an error in the RFC)
> 
> Making this work for back-compat would require arranging for
> sshconnect2.c:key_sig_algorithm() to return two strings: the actual
> signing algorithm identifier as it does currently plus the algorithm
> identifier to use in when the SSH2_AUTH_REQUEST is assembled in
> sign_and_send_pubkey().
> 
> However. it's highly likely that additional compat problems would
> remain. IIRC ssh-agent had problems wrt RSA/SHA2 signatures around
> then too.
> 
> None of this helps you right now unfortunately. If you need
> something sooner than someone getting motivated to fix this (which,
> honestly, may never happen) than you options are 1) update the
> server to a version >=7.8 or change to a different key type - ECDSA
> or ED25519 are fine choices.
we confirmed we can ssh to a server running `OpenSSH_8.2p1` thanks.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|critical                    |normal

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

Celeste Liu <CoelacanthusHex at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |CoelacanthusHex at gmail.com

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3506

Celeste Liu <CoelacanthusHex at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|CoelacanthusHex at gmail.com   |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.