openssh bugs - Oct 2016 - [Bug 2635] New: Unable to use SSH Agent and user level PKCS11Provider configuration directive

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

            Bug ID: 2635
           Summary: Unable to use SSH Agent and user level PKCS11Provider
                    configuration directive
           Product: Portable OpenSSH
           Version: 7.3p1
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: jamin.collins at gmail.com

I have found that I am unable to connect to an ssh host if I have both
my user's ssh config set to use a PCKS11 library and my yubikey based
keys loaded into my ssh agent.

I have tried both the opensc and yubico pcks11 libraries for accessing
the card.  The results differ slightly, but both ultimately fail to
authenticate if my user's ssh config is set to use the PCKS11 library
and the keys have been added to my ssh agent.

** using libykcs11.so from yubico-piv-tool 1.4.2

$ ssh-add -s /usr/lib/libykcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/lib/libykcs11.so

$ ssh-add -L | awk {'print $1,$3}'
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so

$ ssh -vv $REMOTEHOST
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Connecting to $REMOTEHOST [$REMOTEIP] port 22.
debug1: Connection established.

At this point the connection attempt simply hangs.  Hoever if I remove
the libykcs11.so library (and keys) from the ssh agent with the
following:

$ ssh-add -e /usr/lib/libykcs11.so
Card removed: /usr/lib/libykcs11.so
$ ssh-add -l
The agent has no identities.

The connection attempt proceeds and I get prompted for my pin:

Enter PIN for 'YubiKey PIV': 

** using opensc-pkcs11.so from opensc 0.16.0

$ ssh-add -s /lib/pkcs11/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /lib/pkcs11/opensc-pkcs11.so

$ ssh-add -L | awk {'print $1,$3}'
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so

$ ssh -vv $REMOTEHOST 
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Offering RSA public key: /usr/lib/libykcs11.so
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:...
sign_and_send_pubkey: signing failed: agent refused operation
...
debug1: Next authentication method: password
$USER@$REMOTEHOST's password:

If I remove the library (and keys) and try the connection again:

$ ssh-add -e /lib/pkcs11/opensc-pkcs11.so
Card removed: /lib/pkcs11/opensc-pkcs11.so
$ ssh-add -l
The agent has no identities.

$ ssh -vv $REMOTEHOST 
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Offering RSA public key: /usr/lib/libykcs11.so
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:...
Enter PIN for 'PIV_II (PIV Card Holder pin)': 

If I remove the PKCS11Provider directive from my user's ssh config, the
keys loaded in the agent are used and everything works fine.  However,
if I then attempt to connect to the host without first loading the keys
into the agent, I am not prompted for my yubikey pin.  Ideally, I
should be able to have both the user level PKCS11Provider directive and
my keys loaded in the ssh agent.  However, it appears that the user
level directive is being attempted before trying to use the keys from
the agent.


I have found that I am unable to connect to an ssh host if I have both
my user's ssh config set to use a PCKS11 library and my yubikey based
keys loaded into my ssh agent.

I have tried both the opensc and yubico pcks11 libraries for accessing
the card.  The results differ slightly, but both ultimately fail to
authenticate if my user's ssh config is set to use the PCKS11 library
and the keys have been added to my ssh agent.

** using libykcs11.so from yubico-piv-tool 1.4.2

$ ssh-add -s /usr/lib/libykcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/lib/libykcs11.so

$ ssh-add -L | awk {'print $1,$3}'
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so
ssh-rsa /usr/lib/libykcs11.so

$ ssh -vv $REMOTEHOST
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Connecting to $REMOTEHOST [$REMOTEIP] port 22.
debug1: Connection established.

At this point the connection attempt simply hangs.  Hoever if I remove
the libykcs11.so library (and keys) from the ssh agent with the
following:

$ ssh-add -e /usr/lib/libykcs11.so
Card removed: /usr/lib/libykcs11.so
$ ssh-add -l
The agent has no identities.

The connection attempt proceeds and I get prompted for my pin:

Enter PIN for 'YubiKey PIV': 

** using opensc-pkcs11.so from opensc 0.16.0

$ ssh-add -s /lib/pkcs11/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /lib/pkcs11/opensc-pkcs11.so

$ ssh-add -L | awk {'print $1,$3}'
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so
ssh-rsa /lib/pkcs11/opensc-pkcs11.so

$ ssh -vv $REMOTEHOST 
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Offering RSA public key: /usr/lib/libykcs11.so
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:...
sign_and_send_pubkey: signing failed: agent refused operation
...
debug1: Next authentication method: password
$USER@$REMOTEHOST's password:

If I remove the library (and keys) and try the connection again:

$ ssh-add -e /lib/pkcs11/opensc-pkcs11.so
Card removed: /lib/pkcs11/opensc-pkcs11.so
$ ssh-add -l
The agent has no identities.

$ ssh -vv $REMOTEHOST 
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
...
debug1: Offering RSA public key: /usr/lib/libykcs11.so
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp SHA256:...
Enter PIN for 'PIV_II (PIV Card Holder pin)': 

If I remove the PKCS11Provider directive from my user's ssh config, the
keys loaded in the agent are used and everything works fine.  However,
if I then attempt to connect to the host without first loading the keys
into the agent, I am not prompted for my yubikey pin.  Ideally, I
should be able to have both the user level PKCS11Provider directive and
my keys loaded in the ssh agent.  However, it appears that the user
level directive is being attempted before trying to use the keys from
the agent.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

--- Comment #1 from Jamin Collins <jamin.collins at gmail.com> ---
It's been quite a while since this was reported and there has been no
update.  Is any further information needed?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

--- Comment #2 from Jamin Collins <jamin.collins at gmail.com> ---
Any update?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
Sorry, but there isn't enough information to figure out what is going
wrong. Please attach a full debug log from the client (ssh -vvv ...) as
well as your ~/.ssh/config

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

--- Comment #4 from Jamin Collins <jamin.collins at gmail.com> ---
Created attachment 3034
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3034&action=edit
the requested full debug log

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

--- Comment #5 from Jamin Collins <jamin.collins at gmail.com> ---
Created attachment 3035
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3035&action=edit
the requested ssh config

Attached you should find both the requested full debug log and ssh
config.  Please let me know if there is any additional information I
can provide.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Marc 'Zugschlus' Haber <mh+openssh-bugzilla at zugschlus.de>
changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mh+openssh-bugzilla at zugschl
                   |                            |us.de

--- Comment #6 from Marc 'Zugschlus' Haber <mh+openssh-bugzilla at
zugschlus.de> ---
I have exactly the same issue, on Debian unstable, using OpenSSH 7.5p1
from the Debian packages, and a yubikey 4 Nano. My ssh -vvvv output is
the same as Jamin's.

I can provide additional information:
(1) My second Yubikey, a Yubikey Neo, works fine even with the agent
loaded and the PKCS11Provider option in the config.
(2) When using the agent without the PKCS11Provider option, the ssh
-vvv output is the identical same until:

debug3: sign_and_send_pubkey: RSA <deleted>
sign_and_send_pubkey: signing failed: agent refused operation
debug1: Offering RSA public key:
/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so

On the working client, things are:
debug3: sign_and_send_pubkey: RSA <same-deleted-as-above>
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([127.0.0.1]:10022).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

--- Comment #7 from Marc 'Zugschlus' Haber <mh+openssh-bugzilla at
zugschlus.de> ---
And, after trying with PKCS11Provider option in place, and the agent
refusing operation for the first time, I need to do the ssh-add -D
ssh-add -e, ssh-add -s routine, or the agent will refuse operation even
after removing the PKCS11Provider option:

ssh -F config-with-PKSCS11Provider 
=> agent refused operation
ssh -F config-without-PKCS11Provider
=> agent refused operation
ssh-add -D
ssh-add -e
ssh-add -s
ssh -F config-without-PKCS11Provider
=> works

Whenever ssh says "agent refused operations", ssh-agent started with
-d
logs "process_sign_request2: sshkey_sign: error in libcrypto".

Hope this helps.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Jakub Jelen <jjelen at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jjelen at redhat.com

--- Comment #8 from Jakub Jelen <jjelen at redhat.com> ---
Created attachment 3126
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3126&action=edit
Tail of openSC debug log

I believe this is not a problem of OpenSSH, but of the PKCS#11 module,
which is not correctly handling the concurrent access from two separate
processes (ssh and ssh-pkcs11-helper of ssh-agent).

I can reproduce the same issue with latest OpenSC and OpenSSH. Running
the current OpenSC in debug mode, shows similar errors as in the
attachment, while running the ssh-agent in debug mode and adding the
latest OpenSC pkcs11 module:

    OPENSC_DEBUG=9 ssh-agent -d

I just tested the same case with the patch proposed in OpenSC upstream
PR [1] and it seems to resolving the problem.

This is also related to the recent change in OpenSC upstream, which is
setting disconnect_action=leave by default (previously, it was
"reset",
which was also breaking long-running sessions such as ssh-agent).

You can try if this will help you to resolve your problems. If not,
please, provide also the debug logs from OpenSC as shown above.

[1] https://github.com/OpenSC/OpenSC/pull/1256
[2] https://github.com/OpenSC/OpenSC/pull/1242

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |pkcs11

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
Looks like this is an OpenSC bug. If it still persists with the
patch/version that Jakub recommended then please reopen this bug.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2635

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
close bugs that were resolved in OpenSSH 8.5 release cycle

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.