openssh bugs - Sep 2023 - [Bug 3613] New: Unable to sign using certificates and PKCS#11

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

            Bug ID: 3613
           Summary: Unable to sign using certificates and PKCS#11
           Product: Portable OpenSSH
           Version: 8.9p1
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh-keygen
          Assignee: unassigned-bugs at mindrot.org
          Reporter: aim at orbit.online
>From my own experimentation and from looking at the code and some of
the reported bugs here I believe it is currently not possible to sign
arbitrary data with ssh-keygen and an SSH certificate (e.g. for git
commit signing, verified using @cert-authority).

I have tried specifying the certificate when invoking ssh-keygen with
```
$ ssh-add -e /usr/lib/x86_64-linux-gnu/p11-kit-proxy.so
Enter passphrase for PKCS#11:
Card added: /usr/lib/x86_64-linux-gnu/p11-kit-proxy.so
$ ssh-keygen -Y sign -f ~/.ssh/id_rsa-cert.pub -n file test.txt
debug2: hash_file: hashed 3401 bytes
debug3: hash_file: final hash:
1239125ebf618d51bfe64e65dce15530a7a3c9c230438b537564261473c050cd915185a8c19dbb85f40e4faf4367a9779fc54564bcc8de0824e42004c3e3777f
Couldn't sign message (signer): agent refused operation
Signing config/git/config failed: agent refused operation
```

though the `-f` option seems to be ignored and the `ssh-agent` looks
for an RSA-CERT when only RSA keys are loaded:

```
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug2: process_add_smartcard_key: entering
debug1: process_add_smartcard_key: add
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
debug1: process_add
debug1: provider /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0:
manufacturerID <PKCS#11 Kit> cryptokiVersion 2.40 libraryDescription
<PKCS#11 Kit Proxy Module> libraryVersion 1.1
debug1: provider /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0:
label <YubiKey PIV #19258332> manufacturerID <Yubico
(www.yubico.com)>
model <YubiKey YK5> serial <19258332> flags 0x40d
debug2: pkcs11_fetch_keys: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug1: have 1 keys
debug2: pkcs11_fetch_keys: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:54W0/9rkv84M3pwsFa7qvWkCeQGbkWlwSkvk1fcsrV0
debug1: have 2 keys
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878dee71a0 ptr 0x55878dee5e90 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:54W0/9rkv84M3pwsFa7qvWkCeQGbkWlwSkvk1fcsrV0
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878dee68c0 ptr 0x55878dee6290 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878dee7640 ptr 0x55878dee5f20 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug2: process_request_identities: entering
debug3: identity_permitted: entering: key RSA comment "Public key for
PIV Authentication", 0 socket bindings, 0 constraints
debug3: identity_permitted: entering: key RSA comment "Public key for
PIV Attestation", 0 socket bindings, 0 constraints
debug2: process_request_identities: replying with 2 allowed of 2
available keys
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign_request2: entering
process_sign_request2: RSA-CERT key not found
```

It is also not possible to get `ssh-agent` to load the certificate
with:
```
$ ssh-add -s /usr/lib/x86_64-linux-gnu/p11-kit-proxy.so
~/.ssh/id_rsa-cert.pub
Enter passphrase for PKCS#11:
Card added: /usr/lib/x86_64-linux-gnu/p11-kit-proxy.so
```

Where the `ssh-agent` looks like this:
```
debug1: new_socket: type = CONNECTION
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug2: process_add_smartcard_key: entering
debug1: process_add_smartcard_key: add
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
debug1: process_add
debug1: provider /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0:
manufacturerID <PKCS#11 Kit> cryptokiVersion 2.40 libraryDescription
<PKCS#11 Kit Proxy Module> libraryVersion 1.1
debug1: provider /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0:
label <YubiKey PIV #19258332> manufacturerID <Yubico
(www.yubico.com)>
model <YubiKey YK5> serial <19258332> flags 0x40d
debug2: pkcs11_fetch_keys: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug1: have 1 keys
debug2: pkcs11_fetch_keys: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:54W0/9rkv84M3pwsFa7qvWkCeQGbkWlwSkvk1fcsrV0
debug1: have 2 keys
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878dee9c50 ptr 0x55878dee87d0 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:54W0/9rkv84M3pwsFa7qvWkCeQGbkWlwSkvk1fcsrV0
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878dee83b0 ptr 0x55878dee8c90 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
debug2: pkcs11_fetch_certs: provider
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0 slot 0: RSA
SHA256:vRdQ4M0pBHf4Cb4pqxFGVTeJmqRTzRusvXxu7vRrjNk
debug2: pkcs11_fetch_certs: key already included
debug1: pkcs11_k11_free: parent 0x55878deea160 ptr 0x55878dee8cc0 idx 1
debug1: pkcs11_provider_unref: provider
"/usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0" refcount 3
```
```

A workaround would be to somehow support the `-O CertificateFile`
option in `ssh-keygen` like `ssh` does. 
A more robust way to solve this would of course be to support loading
certificate files into the ssh-agent.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
Created attachment 3730
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3730&action=edit
Attempt to lookup plain private key in agent

I think this should fix it, but I'm unable to test ATM.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
Another way to fix it would be to allow adding p11 keys to the agent
while specifying a certificate to graft to them.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #3 from aim at orbit.online ---
Created attachment 3734
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3734&action=edit
Self-contained testscript for cert signing via HSM

First of all thank you for the quick response and a potential fix! And
second of all my apologies on dragging my feet to get this tested!

OK. So it still fails with "process_sign_request2: RSA-CERT key not
found". However, I'm 50/50 on whether I'm using ssh-keygen
correctly.
It's a... rather large tool :-)

I have attached a testing script that applies your patch and then tests
everything automatically using SoftHSMv2. It's self-contained and
cleans up after itself. So you should be able to just run it (if you
have docker installed).

Do note that I'm applying the patch to and testing with 9.0p1, which is
the latest version available on Ubuntu. The patch applies cleanly, so I
don't think that that's the issue.

p.s.: Even though the script is a bit quick & dirty I hope this is
usable as a template for an eventual regression test :-)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

aim at orbit.online changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |aim at orbit.online

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #4 from Damien Miller <djm at mindrot.org> ---
IMO it would be better to get the existing regress/agent-pkcs11.sh test
working for certs, we'll need to do this anyway

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #5 from aim at orbit.online ---
(In reply to Damien Miller from comment #4)
> IMO it would be better to get the existing regress/agent-pkcs11.sh
> test working for certs, we'll need to do this anyway
Oh yeah, I can see it already uses softhsm. Should be easy enough to
port. I can try giving it a go if you like? Have you made any progress
on the patch, is there anything I can help with?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
(In reply to aim from comment #5)
> Oh yeah, I can see it already uses softhsm. Should be easy enough to
> port. I can try giving it a go if you like? Have you made any
> progress on the patch, is there anything I can help with?

Sorry, I've been away and haven't had time to look at it. Getting the
agent-pkcs11.sh regress test going (and failing) with certs would be a
great help if you're able.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #7 from Damien Miller <djm at mindrot.org> ---
Created attachment 3743
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3743&action=edit
allow grafting certs to PKCS#11 keys in ssh-agent

(In reply to Damien Miller from comment #2)
> Another way to fix it would be to allow adding p11 keys to the agent
> while specifying a certificate to graft to them.
Here's an untested prototype of this approach. It's a little more work
but is more general than just doing it in ssh-keygen.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #8 from aim at orbit.online ---
Thank you Damien! I have modified agent-pkcs11.sh to also test signing
with a certificate, but I can't for the life of me figure out how to
run the "t-extra" test target. It looks like you were the one who
added
it back in 2019. Any tips on how to run that test specifically?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net
   Attachment #3744|                            |ok?(dtucker at dtucker.net)
              Flags|                            |

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
Created attachment 3744
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3744&action=edit
expose extra tests

hmm, it looks like there is no easy way to run them. This patch should
fix that. It will also run them by default, which might expose new
problems by we can deal with those if/when we see them.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3744|ok?(dtucker at dtucker.net)    |ok+
              Flags|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
Thanks Darren - the Makefile fixed have been committed

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

aim at orbit.online changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3734|0                           |1
        is obsolete|                            |

--- Comment #11 from aim at orbit.online ---
Created attachment 3745
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3745&action=edit
patch for agent pkcs11 testsuite that tests signing with a certificate

OK. Here is the diff for the test. I couldn't get the pinentry working
through the pipe when using ssh-keygen, so I create a little askpass
script instead. Do ssh-keygen and ssh-add behave the same way in those
regards?

Anyways, I still can't get the test to pass.

If you'd like to see ssh-agent debug messages in the output, just use
the commented-out section right above where the agent is started and
comment out the original `eval` instead.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3730|0                           |1
        is obsolete|                            |
   Attachment #3743|0                           |1
        is obsolete|                            |

--- Comment #12 from Damien Miller <djm at mindrot.org> ---
Created attachment 3752
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3752&action=edit
Allow loading certificates alongside PKCS#11 keys in ssh-agent

This is revised diff to allow loading of certificates alongside PKCS#11
keys into ssh-agent. The certificates are grafted to their
corresponding PKCS#11 private keys inside the agent and can be used as
normal thereafter.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #13 from Damien Miller <djm at mindrot.org> ---
Created attachment 3753
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3753&action=edit
regression test for agent PKCS#11 certificates

This is a regression test for PKCS#11 certificates in ssh-agent. It
will load some certs and verify that they are usable. (You'll need to
add the test to the LTESTS list in regress/Makefile)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |peter at pean.org

--- Comment #14 from Damien Miller <djm at mindrot.org> ---
*** Bug 2808 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |thomas.jarosch at intra2net.co
                   |                            |m

--- Comment #15 from Damien Miller <djm at mindrot.org> ---
*** Bug 2472 has been marked as a duplicate of this bug. ***

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #16 from aim at orbit.online ---
Created attachment 3810
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3810&action=edit
test-pkcs11-cert-sign.sh

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #17 from aim at orbit.online ---
Created attachment 3811
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3811&action=edit
Dockerfile

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3613

--- Comment #18 from aim at orbit.online ---
Yes!! Thank you Damien. This works perfectly!

I only just now had the extra time to get back to it.

I can confirm that I am now able to sign a peer PKCS#11 pubkey with a
CA PKCS#11 key, use the resulting certificate and the peer PKCS#11 key
to sign a file, and then verify that the file has been signed by the
peer and that the peer is trusted through a "cert-authority" in the
allow signers file.

I have attached a Dockerfile and a test script which functionally tests
everything and also demos how it all works together. It can be run with
`docker run --rm $(docker build -q .)`.

The "Good "file" signature for Peer with RSA-CERT key
SHA256:..." is
what to look for in the logs.

Again, thank you for your hard work Damien, in a corporate context we
can now do short lived ssh-certs for git commit signing and pushing
while the key itself can reside on a e.g. a YubiKey or a TPM.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.