openssh bugs - May 2023 - [Bug 3572] New: ssh-agent refused operation when using FIDO2 with -O verify-required

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

            Bug ID: 3572
           Summary: ssh-agent refused operation when using FIDO2 with -O
                    verify-required
           Product: Portable OpenSSH
           Version: 9.3p1
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: minor
          Priority: P5
         Component: ssh-agent
          Assignee: unassigned-bugs at mindrot.org
          Reporter: bluebird090909 at proton.me

When using FIDO2 keys in combination with the option verify-required,
using ssh-agent will fail with the error message: 

sign_and_send_pubkey: signing failed for ED25519-SK
"/home/user/.ssh/id_ed25519_sk" from agent: agent refused operation



When the ssh-agent is not used or the key has not yet been cached, the
login operation works as expected, asking the passphrase for the local
identity key, followed by the FIDO2 device PIN, followed by a request
to touch the device.

running ssh-add -l will list the key as expected as well.

After closing the ssh connection and connecting again (with ssh-agent
running) the operation will fail with the following:

...
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_ed25519_sk ED25519-SK
SHA256:nHEA..................... explicit authenticator agent
debug1: Server accepts key: /home/user/.ssh/id_ed25519_sk ED25519-SK
SHA256:nHEA..................... explicit authenticator agent
sign_and_send_pubkey: signing failed for ED25519-SK
"/home/user/.ssh/id_ed25519_sk" from agent: agent refused operation
debug1: No more authentication methods to try.
root at testhost: Permission denied (publickey)


To reproduce:

1. ssh-keygen -t ed25519-sk -O application=ssh:mytestkey -O
verify-required
2. copy public key to authorized_keys
3. login: ssh -i ~/.ssh/id_ed25519_sk root at testhost (config has
AddKeysToAgent yes)
4. exit ssh shell
5. login again 



When using Fido2 keys generated without -O verify-required, ssh-agent
works as expected, asking only for touch verification when the local
passphrase has been cached.


Expected behavior:

ssh-agent should ask for the Fido2 device Pin to be entered when the
local identity key is already cached


Tested with Nitrokey 3, running firmware 1.4.0 and libfido2 1.13.0
OS: Arch Linux, OpenSSH_9.3p1, OpenSSL 3.0.8 7 Feb 2023

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

xspielinbox+mindrot at protonmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xspielinbox+mindrot at protonm
                   |                            |ail.com

--- Comment #1 from xspielinbox+mindrot at protonmail.com ---
I came across the same issue using Fedora Linux 38 with OpenSSH_9.0p1,
OpenSSL 3.0.9 and a YubiKey 5 NFC with firmware version 5.2.7 and
libfido2 1.12.0-3.fc38

When creating the ssh-key of type ed25519-sk (regardless of the options
given) it asks for the FIDO pin of the security key.
In the default config (which uses an ssh-agent) it does never
thereafter again ask for the FIDO pin of the security key, no matter
how it has been created (resident or not; with verify-required or not).

When explicitly running ssh with an invalid ssh-agent socket, it does
correctly ask for the pin, if the key has been created with
verify-required, but not if it hasn't:
$ SSH_AUTH_SOCK=/tmp/ssh.sock ssh user at host -i id_ed25519_sk_verify
Enter PIN for ED25519-SK key id_ed25519_sk_verify:
Confirm user presence for key ED25519-SK SHA256:[...]

$ SSH_AUTH_SOCK=/tmp/ssh.sock ssh user at host -i id_ed25519_sk
Confirm user presence for key ED25519-SK SHA256:[...]

When using a key without verify-required with the default ssh-agent
config and specifying verify-required in the authorized_keys of the
target system, it does correctly deny the key with 'debug3: receive
packet: type 51', though this is an pretty ambiguous failure message...

All in all, I can reproduce the reproduce the reported issue. To an
user at first it seems like the verify-required option is useless or
broken, as is does not change anything.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #2 from bluebird090909 at proton.me ---
A workaround for this issue is to disable the ssh-agent for the
relevant connections using the option -o IdentityAgent=none

Alternatively add this to your ~/.ssh/config

Host myserver.tld
  IdentityAgent none

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
Is your ssh-agent configured to use $SSH_ASKPASS? The agent should
display a PIN entry dialog for verify-required keys if the askpass
program is available.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #4 from xspielinbox+mindrot at protonmail.com ---
I do not have openssh-askpass installed, but I do have pinentry,
pinentry-gnome3, gnome-keyring and gnome-keyring-pam installed.
$SSH_ASKPASS and SSH_AGENT_PID seem to be unset. $SSH_AUTH_SOCK is set
to: /run/user/1000/keyring/ssh
I am using the default configuration of Fedora Workstation what
SSH-Agent / SSH-Askpass is concerned.
I do get graphical dialogs to unlock my password-protected SSH-Keys
without any issues.
After running ssh-add an additional process is running:
/usr/bin/ssh-agent -D -a /run/user/1000/keyring/.ssh

I hope this helped. I apologize, if it was useless information. I am
not that familiar with ssh-agent/ssh-askpass workings.

If you need any other information please let me know. I will do my best
to answer.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
This looks like it is a problem with how Fedora is running/configuring
ssh-agent. You can test this using something like:

sudo yum install openssh-askpass
env SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass ssh-agent $SHELL
-l
ssh-add ~/.ssh/id_ed25519_sk
ssh-add -T ~/.ssh/id_ed25519_sk.pub

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #6 from bluebird090909 at proton.me ---
I didn't have ssh-askpass installed either, but even after installing
it and using the steps above, the result was the same.

Running on Arch Linux:

sudo pacman -S x11-ssh-askpass
env SSH_ASKPASS=/usr/lib/ssh/x11-ssh-askpass ssh-agent $SHELL -l
ssh-add ~/.ssh/id_ed25519_sk
  Identity added: /home/user/.ssh/id_ed25519_sk
ssh-add -T ~/.ssh/id_ed25519_sk.pub
  Agent signature failed for /home/user/.ssh/id_ed25519_sk.pub: agent
refused operation


Shouldn't entering the pin on the terminal work as well? It works
during key registration at least, so I don't get why ssh-askpass would
be required?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #7 from Damien Miller <djm at mindrot.org> ---
(In reply to bluebird090909 from comment #6)
> I didn't have ssh-askpass installed either, but even after
> installing it and using the steps above, the result was the same.
Well, you didn't follow my instructions so that's not surprising.
> Running on Arch Linux:
> 
> sudo pacman -S x11-ssh-askpass
> env SSH_ASKPASS=/usr/lib/ssh/x11-ssh-askpass ssh-agent $SHELL -l
That's not the right path. I had the correct path in the instructions
in comment #5. Try replacing /usr/lib/ssh/x11-ssh-askpass with
/usr/libexec/openssh/x11-ssh-askpass.
> Shouldn't entering the pin on the terminal work as well? It works
> during key registration at least, so I don't get why ssh-askpass
> would be required?
Because ssh-agent is a daemon process that isn't connected to the
terminal.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #8 from bluebird090909 at proton.me ---
The path /usr/libexec/ does not exist on arch linux but
/usr/lib/ssh/x11-ssh-askpass is available 

However I did manage to get the pin entry to work on arch using the
x11-ssh-askpass package on a fresh arch installation.

Your instructions also worked on a fresh Debian Bookworm after
installing the ssh-askpass-gnome package and I can use the agent with
the fido2 key and pin verification.

In both cases I had to define SSH_ASKPASS first.


Eventually I found out that the reason ssh-askpass didn't work
initially on my arch setup was because I had this set in my bashrc:

export SSH_AUTH_SOCK="$XDG_RUNTIME_DIR/ssh-agent.socket"

while I also had this systemd service:

[Unit]
Description=SSH key agent

[Service]
Type=simple
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -D -t 1h -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

Removing this export from my bashrc results in ssh-askpass successfully
requesting the pin. (And I'm very confused why that is)
Note that SSH_AUTH_SOCK is available as environment variable in both
cases, but setting it in bashrc seems to prevent askpass from working
for some reason.

To conclude, setting SSH_ASKPASS allows the agent to successfully
request the pin when using fido2 keys with verify-required

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #9 from xspielinbox+mindrot at protonmail.com ---
(In reply to Damien Miller from comment #5)
> This looks like it is a problem with how Fedora is
> running/configuring ssh-agent. You can test this using something
> like:
> 
> sudo yum install openssh-askpass
> env SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass ssh-agent
> $SHELL -l
> ssh-add ~/.ssh/id_ed25519_sk
> ssh-add -T ~/.ssh/id_ed25519_sk.pub
Sorry, for the delay. I did quite some testing:

I first tried the test as is without openssh-askpass installed:

When just running $ ssh-add -T ~/.ssh/id_ed25519_sk-pin.pub
I get a (gnome-builtin) graphical dialog prompting for the
password/passphrase of the key, then have to tap the authenticator and
the shell prompt returns (so it is successful).
When testing it again, it has remembered the passphrase, and I only
have to tap the authenticator.
$ ssh-add -T ~/.ssh/id_ed25519_sk-pin.pub
results in the same graphical dialog for the password/passphrase, but
after that it just fails with:
"Agent signature failed for [full path of key]: agent refused
operation"
One never get's the opportunity to tap the authenticator or enter the
pin for user verification. The authenticator also never lights up to
indicate that user interaction is required.

When adding the keys to the ssh-agent:
$ ssh-add ~/.ssh/id_ed25519_sk-pin
Enter passphrase for [full path of key]: 
Identity added: [full path of key] (pin)
$ ssh-add ~/.ssh/id_ed25519_sk-verify-pin
Enter passphrase for [full path of key]: 
Identity added: [full path of key] (verify-pin)

I always get the prompt for the passphrase in the terminal and also
have to enter it, even if I already entered it in the graphical dialog
or ran ssh-add already before. When after that testing the signatures
again with ssh-add -T nothing has changed. I still have to enter the
password/passphrase in the graphical dialog, when I ran ssh-add -D or
killing the ssh-agent before adding the the keys to the ssh-agent.
Verification succeeds for the key with the pin, but not for the one,
with verify-required.


After installing the openssh-askpass package, echo $SSH_ASKPASS
returns:
/usr/libexec/openssh/gnome-ssh-askpass
This binary also does indeed exist.

After running env SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
ssh-agent $SHELL -l
there is an additional process running: ssh-agent /bin/bash -l
For every time, I run this command, an additional process get's
created.

When row running: ssh-add -T ~/.ssh/id_ed25519_sk-pin.pub or ssh-add -T
~/.ssh/id_ed25519_sk-verify-pin.pub
I get: "Agent signature failed for [full path of key]: agent refused
operation"
It does not prompt me for the passphrase anymore.

$ ssh-add -L
returns: "The agent has no identities.", whereas the default agent has
all identities from the hard disk preloaded, even when the first
ssh-add command I enter is ssh-add -L. When opening a new terminal
windows, it again defaults to the default agent and I have to manually
execute above command again to enable openssh-askpass.

I first have to add the keys to the agent with ssh-add as above and
when then running $ ssh-add -T ~/.ssh/id_ed25519_sk-pin.pub
after a tap on the authenticator the shell prompt returns (so it is
successful).
But $ ssh-add -T ~/.ssh/id_ed25519_sk-verify-pin.pub
still fails with:
"Agent signature failed for [full path of key]: agent refused
operation"
When checking with ssh-add -L, one can see that the latter key also was
not added to the agent, despite the ssh-add command not giving an error
(though also not asking for the pin, as it should have).

Also: when running ssh-add ~/.ssh/id_ed25519_sk-pin or ssh-add
~/.ssh/id_ed25519_sk-verify-pin
just hitting enter directly and not entering any passphrase seems to
abort it the same way as doing a keyboard interrupt. It does not show
the message of a bad passphrase, and there is no new key shown in
ssh-add -L

When then running ssh-add after killing the ssh-agent I receive "Error
connecting to agent: No such file or directory". I have to rerun env
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass ssh-agent $SHELL -l
before the error goes away. When opening a new terminal window or
exiting the new login shell, the command creates, this error disappears
too.

After some wild testing in different terminal tabs, I somehow then got
to a point, that now, when running ssh-add
~/.ssh/id_ed25519_sk-verify-pin it adds the key to the ssh-agent, even
though it did not prompt for a pin.
When having added the key to the ssh-agent and then running the
signature test, it now shows a (gnome-builtin) graphical dialog, that
openssh-askpass wants to inhibit shortcuts. When allowing that, I get
to see a new application window "openssh" that asks me to confirm user
presence for the respective key and in case of the verfiy-pin key also
asks me to enter the PIN.
For the pin key, it succeeds after taping the authenticator, but for
the verify-pin key no matter what I enter as the pin (the actual FIDO2
pin of the authenticator, something completely wrong, nothing or the
passphrase of the key), it immediately fails with "Agent signature
failed for [full path of key]: agent refused operation", not even
giving me any chance to confirm my user presence. trying to confirm the
user presence before entering the pin also does not work. In fact, the
autenticator never lights up the light to indicate, that interaction is
required.

I don't have anything SSH-related in my ~/.bashrc, /etc/bashrc,
/etc/profile or ~/.bash_profile.
There also isn't any ssh-askpass or ssh-agent related systemd service
on Fedora.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

personal at ilanjoselevich.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |personal at ilanjoselevich.com

--- Comment #10 from personal at ilanjoselevich.com ---
Created attachment 3778
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3778&action=edit
two terminals running ssh and ssh-agent

I really want to use ssh-agent with my FIDO2 (PIN) protected ssh key
but it seems that it requires you to use a GUI SSH_ASKPASS. I played
around with it a bit and figured out that the reason why it needs a GUI
SSH_ASKPASS is because it executes it inside the ssh-agent's process
and tty, meaning that it will just fail once it asks for a password on
the terminal because it is non-interactive.

I wonder if it's possible for ssh to run the SSH_ASKPASS on the
client's terminal rather than on the agent and pass the output back to
the agent.

I attached an image of the two terminals to the side, one running the
agent, and one running `ssh`.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

chn at chn.moe changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |chn at chn.moe

--- Comment #11 from chn at chn.moe ---
(In reply to personal from comment #10)
> Created attachment 3778 [details]
> two terminals running ssh and ssh-agent
> 
> I really want to use ssh-agent with my FIDO2 (PIN) protected ssh key
> but it seems that it requires you to use a GUI SSH_ASKPASS. I played
> around with it a bit and figured out that the reason why it needs a
> GUI SSH_ASKPASS is because it executes it inside the ssh-agent's
> process and tty, meaning that it will just fail once it asks for a
> password on the terminal because it is non-interactive.
> 
> I wonder if it's possible for ssh to run the SSH_ASKPASS on the
> client's terminal rather than on the agent and pass the output back
> to the agent.
> 
> I attached an image of the two terminals to the side, one running
> the agent, and one running `ssh`.
Have you finally made it works? It seems you are using NixOS, would you
mind to share the corresponding configs?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #12 from personal at ilanjoselevich.com ---
(In reply to chn from comment #11)
> Have you finally made it works? It seems you are using NixOS, would
> you mind to share the corresponding configs?
I have managed to make it work with a GUI based SSH_ASKPASS, are you
interested in seeing the config?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #13 from chn at chn.moe ---
(In reply to personal from comment #12)
> (In reply to chn from comment #11)
> > Have you finally made it works? It seems you are using NixOS, would
> > you mind to share the corresponding configs?
> 
> I have managed to make it work with a GUI based SSH_ASKPASS, are you
> interested in seeing the config?
Yes of course, it will be great if you share you config. Thank you very
much!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #14 from personal at ilanjoselevich.com ---
(In reply to chn from comment #13)
> Yes of course, it will be great if you share you config. Thank you
> very much!
```
  programs.ssh = {
    startAgent = true;
    enableAskPassword = true;
    askPassword "${pkgs.gnome.seahorse}/libexec/seahorse/ssh-askpass";
  };
```

That's all I needed, besides forwardAgent and addKeysToAgent on the
client.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #15 from chn at chn.moe ---
(In reply to personal from comment #14)
> (In reply to chn from comment #13)
> > Yes of course, it will be great if you share you config. Thank you
> > very much!
> 
> ```
>   programs.ssh = {
>     startAgent = true;
>     enableAskPassword = true;
>     askPassword >
"${pkgs.gnome.seahorse}/libexec/seahorse/ssh-askpass";
>   };
> ```
> 
> That's all I needed, besides forwardAgent and addKeysToAgent on the
> client.
Thank you, I tried it and it works. I also tried
`systemd-ask-password`, it works too.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #16 from personal at ilanjoselevich.com ---
(In reply to chn from comment #15)
> Thank you, I tried it and it works. I also tried
> `systemd-ask-password`, it works too.
So you have ssh-agent running in a systemd service and it asks you for
the password using systemd-ask-password on the terminal executing
`ssh`?
This is exactly what I want but I could never get it to work. Can you
show me your config?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #17 from chn at chn.moe ---
(In reply to personal from comment #16)
> (In reply to chn from comment #15)
> > Thank you, I tried it and it works. I also tried
> > `systemd-ask-password`, it works too.
> 
> So you have ssh-agent running in a systemd service and it asks you
> for the password using systemd-ask-password on the terminal
> executing `ssh`?
> This is exactly what I want but I could never get it to work. Can
> you show me your config?
Sorry for the late response, I have not check my email during holiday.
Happy new year!

I am not sure if my ssh-agent was running in a systemd service or not.
I am using ed25519-sk with discoverable (resident) credentials without
`verify-required` (everytime I use my ssh key, I need to tap the metal
thing on my Yubikey 5 NFC, but do not need to enter PIN code). Here is
my config, hope it is useful.

https://github.com/CHN-beta/nixos/blob/d3f38c3b55676e5a746666a5d07c6d137759066b/modules/packages/server/ssh/default.nix#L103

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3572

--- Comment #18 from personal at ilanjoselevich.com ---
(In reply to chn from comment #17)
> I am not sure if my ssh-agent was running in a systemd service or
> not. I am using ed25519-sk with discoverable (resident) credentials
> without `verify-required` (everytime I use my ssh key, I need to tap
> the metal thing on my Yubikey 5 NFC, but do not need to enter PIN
> code). Here is my config, hope it is useful.
> 
> https://github.com/CHN-beta/nixos/blob/
> d3f38c3b55676e5a746666a5d07c6d137759066b/modules/packages/server/ssh/
> default.nix#L103
It is running inside a systemd service. Since you are not using a FIDO2
PIN you don't need an askpass, I do use a FIDO2 PIN, which is why I
can't get systemd-ask-password to work since it's a terminal-based
askpass and it gets executed in the systemd service as I've previously.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.