openssh unix dev - Feb 2020 - Re-adding PKCS#11 key in ssh-agent produces "agent refused operation" error.

Hi all,

Thanks for all your hard work! I was particularly excited to see
FIDO/U2F support in the latest release.

I'd like to make the following bug report in ssh-agent's PKCS#11
support:

Steps to reproduce:

1. Configure a smart card (e.g. Yubikey in PIV mode) as an SSH key.
2. Add that key to ssh-agent.
3. Remove that key from ssh-agent.
4. Add that key to ssh-agent.

Expected results:

Key is successfully added to ssh-agent.

Actual results:

ssh-add fails with "agent refused operation".

I've looked at the code, and it appears that register_pkcs11_provider
(https://github.com/openssh/openssh-portable/blob/master/ssh-pkcs11.c#L1470)
fails if a PKCS#11 provider already exists. However, PKCS#11 providers
are never unloaded. There is a pkcs11_del_provider but it is never called.

That means that after deleting a key, there is no way to re-add it. Also, since
removing a USB smartcard reader results in ssh-agent losing its session, the
user must call ssh-add again after reinserting the USB card reader, and that
second ssh-add will fail in the same way.

I think the best fix here is to treat "provider already exists" as a
non-error.
There is no need to unload providers when they become unused because they
don't use very much memory, and because it is uncommon to have more than one
provider on any given system. Also, a user is likely to reuse a provider they
have previously used.

If a maintainer can confirm that this is an acceptable fix, I may be able to
write a patch.

Environments reproduced on: Ubuntu 19.10, Fedora
Version of OpenSSH: git commit b2491c28, latest at time of writing.

Example output demonstrating the problem (with a Yubikey in PIV mode inserted):

 $ SSH_AUTH_SOCK=/tmp/ssh-dhfNCpXwSk8B/agent.21022; export SSH_AUTH_SOCK;
 $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so": agent
refused operation
 $ SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK;
 $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Card added: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
 $ ssh-add -D
All identities removed.
 $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
Enter passphrase for PKCS#11:
Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so": agent
refused operation

In a separate terminal:

 $ ./ssh-agent -d
SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK;
echo Agent pid 21116;
debug2: fd 3 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug1: process_add_smartcard_key: add
/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
debug1: pkcs11_start_helper: starting /usr/local/libexec/ssh-pkcs11-helper -vvv
debug1: process_add
debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so:
manufacturerID <OpenSC Project> cryptokiVersion 2.20
libraryDescription <OpenSC smartcard framework> libraryVersion 0.19
debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0:
label <SSH key> manufacturerID <piv_II> model <PKCS#15
emulate> serial
<00000000> flags 0x40d
debug1: have 1 keys
debug1: pkcs11_k11_free: parent 0x5598092d2f30 ptr 0x5598092d2b10 idx 1
debug1: pkcs11_provider_unref: 0x559809258df0 refcount 2
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 19
debug1: process_message: socket 1 (fd=4) type 9
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug1: process_add_smartcard_key: add
/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
debug1: process_add
debug1: check 0x559809258df0 /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
debug1: pkcs11_register_provider: provider already registered:
/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

As a side note, OpenSC is looking at issues with using tokens vs separate
readers and smart cards. The code paths in PKCS#11 differ. Removing a card
from a reader leaves the pkcs#11 slot still available. Removing a token
(Yubikey)
removes both the reader and and its builtin smart card. Firefox has a similar
problem.

See
https://github.com/OpenSC/OpenSC/pull/1947 and #1945, #1935 and #1908
https://bugzilla.mozilla.org/show_bug.cgi?id=1613632

#1947 may be withdrawn and replaced with a different approach.

On 2/21/2020 9:53 PM, Jacob Hoffman-Andrews wrote:
> Hi all,
> 
> Thanks for all your hard work! I was particularly excited to see
> FIDO/U2F support in the latest release.
> 
> I'd like to make the following bug report in ssh-agent's PKCS#11
support:
> 
> Steps to reproduce:
> 
> 1. Configure a smart card (e.g. Yubikey in PIV mode) as an SSH key.
> 2. Add that key to ssh-agent.
> 3. Remove that key from ssh-agent.
> 4. Add that key to ssh-agent.
> 
> Expected results:
> 
> Key is successfully added to ssh-agent.
> 
> Actual results:
> 
> ssh-add fails with "agent refused operation".
> 
> I've looked at the code, and it appears that register_pkcs11_provider
>
(https://github.com/openssh/openssh-portable/blob/master/ssh-pkcs11.c#L1470)
> fails if a PKCS#11 provider already exists. However, PKCS#11 providers
> are never unloaded. There is a pkcs11_del_provider but it is never called.
> 
> That means that after deleting a key, there is no way to re-add it. Also,
since
> removing a USB smartcard reader results in ssh-agent losing its session,
the
> user must call ssh-add again after reinserting the USB card reader, and
that
> second ssh-add will fail in the same way.
> 
> I think the best fix here is to treat "provider already exists"
as a non-error.
> There is no need to unload providers when they become unused because they
> don't use very much memory, and because it is uncommon to have more
than one
> provider on any given system. Also, a user is likely to reuse a provider
they
> have previously used.
> 
> If a maintainer can confirm that this is an acceptable fix, I may be able
to
> write a patch.
> 
> Environments reproduced on: Ubuntu 19.10, Fedora
> Version of OpenSSH: git commit b2491c28, latest at time of writing.
> 
> Example output demonstrating the problem (with a Yubikey in PIV mode
inserted):
> 
>   $ SSH_AUTH_SOCK=/tmp/ssh-dhfNCpXwSk8B/agent.21022; export SSH_AUTH_SOCK;
>   $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> Enter passphrase for PKCS#11:
> Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so":
agent
> refused operation
>   $ SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK;
>   $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> Enter passphrase for PKCS#11:
> Card added: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
>   $ ssh-add -D
> All identities removed.
>   $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> Enter passphrase for PKCS#11:
> Could not add card "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so":
agent
> refused operation
> 
> In a separate terminal:
> 
>   $ ./ssh-agent -d
> SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export SSH_AUTH_SOCK;
> echo Agent pid 21116;
> debug2: fd 3 setting O_NONBLOCK
> debug2: fd 4 setting O_NONBLOCK
> debug1: process_message: socket 1 (fd=4) type 20
> debug1: process_add_smartcard_key: add
> /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> debug1: pkcs11_start_helper: starting /usr/local/libexec/ssh-pkcs11-helper
-vvv
> debug1: process_add
> debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so:
> manufacturerID <OpenSC Project> cryptokiVersion 2.20
> libraryDescription <OpenSC smartcard framework> libraryVersion 0.19
> debug1: provider /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so slot 0:
> label <SSH key> manufacturerID <piv_II> model <PKCS#15
emulate> serial
> <00000000> flags 0x40d
> debug1: have 1 keys
> debug1: pkcs11_k11_free: parent 0x5598092d2f30 ptr 0x5598092d2b10 idx 1
> debug1: pkcs11_provider_unref: 0x559809258df0 refcount 2
> debug2: fd 4 setting O_NONBLOCK
> debug1: process_message: socket 1 (fd=4) type 19
> debug1: process_message: socket 1 (fd=4) type 9
> debug2: fd 4 setting O_NONBLOCK
> debug1: process_message: socket 1 (fd=4) type 20
> debug1: process_add_smartcard_key: add
> /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> debug1: process_add
> debug1: check 0x559809258df0 /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> debug1: pkcs11_register_provider: provider already registered:
> /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
> 
-- 

  Douglas E. Engert  <DEEngert at gmail.com>

On Sat, 2020-02-22 at 10:50 -0600, Douglas E Engert
wrote:
> As a side note, OpenSC is looking at issues with using tokens vs
> separate
> readers and smart cards. The code paths in PKCS#11 differ. Removing a
> card
> from a reader leaves the pkcs#11 slot still available. Removing a
> token (Yubikey)
> removes both the reader and and its builtin smart card. Firefox has a
> similar
> problem.
> 
> See
> https://github.com/OpenSC/OpenSC/pull/1947 and #1945, #1935 and #1908
> https://bugzilla.mozilla.org/show_bug.cgi?id=1613632
> 
> #1947 may be withdrawn and replaced with a different approach.
Right,
I tried to address transparent smart card/yubikey removal in the
OpenSSH before [1], but it still had some issues inside of OpenSC [2]
that should be hopefully addressed by this time (though some more
referenced by previous mail might still be present).

I would like to get back to this in coming months as it is popping up
as a common pain point over and over. Rebasing/testing would be
welcomed.

[1] https://bugzilla.mindrot.org/show_bug.cgi?id=2890
[2] https://github.com/OpenSC/OpenSC/issues/1822

On 2/21/2020 9:53 PM, Jacob Hoffman-Andrews wrote:
> > Hi all,
> > 
> > Thanks for all your hard work! I was particularly excited to see
> > FIDO/U2F support in the latest release.
> > 
> > I'd like to make the following bug report in ssh-agent's
PKCS#11
> > support:
> > 
> > Steps to reproduce:
> > 
> > 1. Configure a smart card (e.g. Yubikey in PIV mode) as an SSH key.
> > 2. Add that key to ssh-agent.
> > 3. Remove that key from ssh-agent.
> > 4. Add that key to ssh-agent.
> > 
> > Expected results:
> > 
> > Key is successfully added to ssh-agent.
> > 
> > Actual results:
> > 
> > ssh-add fails with "agent refused operation".
> > 
> > I've looked at the code, and it appears that
> > register_pkcs11_provider
> > (
> >
https://github.com/openssh/openssh-portable/blob/master/ssh-pkcs11.c#L1470
> > )
> > fails if a PKCS#11 provider already exists. However, PKCS#11
> > providers
> > are never unloaded. There is a pkcs11_del_provider but it is never
> > called.
> > 
> > That means that after deleting a key, there is no way to re-add it.
> > Also, since
> > removing a USB smartcard reader results in ssh-agent losing its
> > session, the
> > user must call ssh-add again after reinserting the USB card reader,
> > and that
> > second ssh-add will fail in the same way.
> > 
> > I think the best fix here is to treat "provider already
exists" as
> > a non-error.
> > There is no need to unload providers when they become unused
> > because they
> > don't use very much memory, and because it is uncommon to have
more
> > than one
> > provider on any given system. Also, a user is likely to reuse a
> > provider they
> > have previously used.
> > 
> > If a maintainer can confirm that this is an acceptable fix, I may
> > be able to
> > write a patch.
> > 
> > Environments reproduced on: Ubuntu 19.10, Fedora
> > Version of OpenSSH: git commit b2491c28, latest at time of writing.
> > 
> > Example output demonstrating the problem (with a Yubikey in PIV
> > mode inserted):
> > 
> >   $ SSH_AUTH_SOCK=/tmp/ssh-dhfNCpXwSk8B/agent.21022; export
> > SSH_AUTH_SOCK;
> >   $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> > Enter passphrase for PKCS#11:
> > Could not add card
"/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so":
> > agent
> > refused operation
> >   $ SSH_AUTH_SOCK=/tmp/ssh-RORElJeiiHBc/agent.21116; export
> > SSH_AUTH_SOCK;
> >   $ ssh-add -s /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> > Enter passphrase for PKCS#11:
> > Card added: /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so
> >   $ ssh-add -D
> > All identities removed.
I think the problem here is that the -D switch is not smartcards aware.
PKCS#11 modules should be removed using -e switch, which works fine to
my testing.

The correct fix would be for the -D switch to remove the pkcs11
providers too.

Regards,
-- 
Jakub Jelen
Senior Software Engineer
Security Technologies
Red Hat, Inc.

Note I'm no maintainer here, just someone on the list.

Jacob Hoffman-Andrews wrote:
> I think the best fix here is to treat "provider already exists"
as a
> non-error.
This seems logically OK to me.

> There is no need to unload providers when they become unused
I disagree with this for reasons already mentioned, and

> because it is uncommon to have more than one provider on any given system.
It may be uncommon, but that is no reason to make it impossible!

> Also, a user is likely to reuse a provider they have previously used.
Disagree again - consider an interactive session, where a user logs
in, performs a system update of either ssh-agent, p11, or both, and
then wants to use the newer versions.

The scope of the ssh-agent is the user's session. The scope of the
p11 is the loaded provider in the agent. It's simply ugly to force
the user to restart the agent process if she really only wants to
replace the p11.


Jacob Hoffman-Andrews wrote:
> Indeed, `ssh-add -e` does fix this issue for me on the latest release
> 
> I realized there's a similar problem with the `-d` flag: If you delete
> an identity backed by a PKCS#11 device, it will remove the identity
> and report success but not remove the provider.
Intuitively I would expect -d (and -D) to remove the provider when the
last key from that provider is removed.

> Is it desirable in the future to have multiple identities offered by the
> same provider?
I for one would like that to work.

> For instance, multiple instances of the same smartcard reader?
Sure. Or a device making more than one key available through the same
interface, thus controlled by one (and only one) provider.

> If so, we would need to have some facility to keep track of already-loaded
> providers and reuse them, as well as do reference counting for removed
> identities.
I think this would make sense.

> That's why I was suggesting it would be more straightforward
> to never unload providers (or in other words, require a restart of
> ssh-agent if user requires that provider to be non-resident,
Again, I disagree strongly with forcing this onto us users. Consider a
system where an agent also has many other, unrelated keys. It would be
really painful and annoying to ditch all that other setup just because
some p11 provider needs to be reloaded. Windows does still get away
with requiring a reboot now and then, but let's not copy that pattern
if we can avoid it in any way.


//Peter