CentOS - Jul 2016 - CentOS 6.7->6.8, ssh-add issue, followup, more info

> -----Original Message-----
> From: m.roth at 5-cent.us [mailto:m.roth at 5-cent.us]
> Sent: Friday, July 22, 2016 4:15 PM
> To: CentOS mailing list
> Subject: Re: [CentOS] CentOS 6.7->6.8, ssh-add issue, followup, more
info
> 
> m.roth at 5-cent.us wrote:
> > Folks,
> >
> >    I am perplexed. I updated my workstation at work Wed before I left,
> > from 6.7 to 6.8. Then, yesterday, I went to use ssh-add -s
> > libcoolkeypk11.so, which I've done many times before to add the
certs
> > from my PIV card... and 100% of the time if fails, letting me
> > SSH_AGENT_FAILURE, cannot add card.
> >
> >    Now, using a script called sccr, which uses my public and private
key
> > to generate a one-time password (we use the to sudo to root), works
> > with no problem. I used my card to go into the data center this
> > morning, which also reads my card, and had no problem. I've tried
eval
> > $(ssh-agent) to start a new instance. Nothing works.
> >
> >    Also, pklogin-finder finds the cards, asks for my PIN< and it
works.
> >
> >    Clues for the poor?
> >
> I just tried ssh -I libcoolkeypk11.so <servername> and in messages,
it
> reports "ssh-pkcs11-helper: errror:no slots" before failing to
let me log
> on.
> 
>      mark
> 
Assuming
1) that /etc/pki/nssdb/ has been populated with all the appropriate and current
gov certificate authorities (CA).
certutil -L -d /etc/pki/nssdb/ #list the CAs
2) that you are using the RH/CentOS stock openssh*rpm files.
3) that you have not also gotten a newer card in the same time period, which
happens to use a CA that is not in /etc/pki/nssdb/

Have you tried a third different set of ssh commands to use the cac:
ln -s /etc/pki/nssdb/*  ~/.ssh/ #make the certificate authorities available to
ssh*
ssh-add -D #clear out any existing sigs
ssh-add -n #use nss to access the cac

Also on some boxes coolkey gets disassociated from nss, and I have found the
simple
yum reinstall coolkey
fixes it, may need to logout/reboot as it affects a bunch O'stuff (and been
a while since I had the problem).


Even when this disclaimer is not here:
I am not a contracting officer. I do not have authority to make or modify the
terms of any contract.

Denniston, Todd A CIV NAVSURFWARCENDIV Crane wrote:
>> From: m.roth at 5-cent.us [mailto:m.roth at 5-cent.us]
>> Sent: Friday, July 22, 2016 4:15 PM
>> m.roth at 5-cent.us wrote:
>> >
>> >    I am perplexed. I updated my workstation at work Wed before I
left,
>> > from 6.7 to 6.8. Then, yesterday, I went to use ssh-add -s
>> > libcoolkeypk11.so, which I've done many times before to add
the certs
>> > from my PIV card... and 100% of the time if fails, letting me
>> > SSH_AGENT_FAILURE, cannot add card.
>> >
>> > Now, using a script called sccr, which uses my public and private
key
>> > to generate a one-time password (we use the to sudo to root),
works
>> > with no problem. I used my card to go into the data center this
>> > morning, which also reads my card, and had no problem. I've
tried eval
>> > $(ssh-agent) to start a new instance. Nothing works.
>> >
>> >    Also, pklogin-finder finds the cards, asks for my PIN< and
it
>> works.
>> >
>> >    Clues for the poor?
>> >
>> I just tried ssh -I libcoolkeypk11.so <servername> and in
messages, it
>> reports "ssh-pkcs11-helper: errror:no slots" before failing
to let me
>> log on.
> Assuming
> 1) that /etc/pki/nssdb/ has been populated with all the appropriate and
> current gov certificate authorities (CA).
> certutil -L -d /etc/pki/nssdb/ #list the CAs
> 2) that you are using the RH/CentOS stock openssh*rpm files.
> 3) that you have not also gotten a newer card in the same time period,
> which happens to use a CA that is not in /etc/pki/nssdb/
>
> Have you tried a third different set of ssh commands to use the cac:
> ln -s /etc/pki/nssdb/*  ~/.ssh/ #make the certificate authorities
> available to ssh*
> ssh-add -D #clear out any existing sigs
I tried ssh-add -e, and it also said "unable to connect to agent".
> ssh-add -n #use nss to access the cac
>
> Also on some boxes coolkey gets disassociated from nss, and I have found
> the simple
> yum reinstall coolkey
> fixes it, may need to logout/reboot as it affects a bunch O'stuff (and
> been a while since I had the problem).
>
I could try the reinstall, but it's very odd - everything worked, and now,
after the upgrade, it doesn't. Oh, and here's another twist on this:
Under
6.7, if I'd logged into my webmail via firefox, and while that was
happening, I stuck my AA ("Logical token" card) into the keyboard
slot,
and used it in logging onto one server (using sccr), it just chugged
along. Now, at 6.8, *everything* in firefox hangs - even a google search,
until I pull the card out of the keyboard. It's clearly trying to
authenticate the client, not just the server....

         mark

Tue Jul 26 21:17:40 UTC 2016,  m.roth at 5-cent.us
wrote:
> I could try the reinstall, but it's very odd - everything worked, and
now,
> after the upgrade, it doesn't. Oh, and here's another twist on
this: Under
> 6.7, if I'd logged into my webmail via firefox, and while that was
> happening, I stuck my AA ("Logical token" card) into the keyboard
slot,
> and used it in logging onto one server (using sccr), it just chugged
> along. Now, at 6.8, *everything* in firefox hangs - even a google search,
> until I pull the card out of the keyboard. It's clearly trying to
> authenticate the client, not just the server....

rpm -qa --last \*pcsc\* \*cool\* \*nss-\* \*ssh\* ccid\* 

in 6.7 the last nss was 3.21.0-0.3.el6_7 in 6.8 it is 3.21.0-8.el6
--changelog is confusing, as there is no 3.21.0-0.3, and there is a "Rebase
RHEL 6.8 to NSS 3.21" even though we where ALREADY on 3.21?

--changelog ccid shows changes the Omnikey 3022 behavior, and added more ccids
and "Allow longer ccid messages"

Fortunately/Unfortunately the problems you are seeing are not affecting me on
the RHEL box I am currently using with the nss access method I sent the other
day, and I don't have an AA card, so it is a little hard to figure out what
broke.

I suspect that for us to figure out what exactly happened (which I would like to
know) we would have downgrade some components back to their 6.7 state on your
box and see  which component had the bad change.  And then figure out what those
changes were.  I've done it before to help RH get CAC with ssh going again
in 6.x series (6.[34] IIRC, broke something).

Remember the software stack is IIRC:
firefox
nss
coolkey
pcscd
ccid

I think your use of openssh -s libcoolkeypk11.so makes it look like:
ssh
coolkey
pcscd
ccid
 

and for your sccr script probably looks a bit like:
sccr
?nss??openssl?
coolkey
pcscd
ccid

I would likely** yum downgrade ccid pcscd coolkey   and then see if the lock
still happened between sccr and firefox, then kick nss (or do douwngrade in the
other order).  After it is working, I would upgrade each of those components
until it broke again... and depending on how pedantic I was being I might
downgrade just the component that I think broke it and upgrade the rest.

**downgrade is a little trickier using CentOS when crossing point releases...
you may need to build your own 'updates' repo, containing all the 6.7
updates and the 6.8 stuff.

Good hunting.

BTW: if you drop us/me the whole ssh invocation using -s, I might give that a go
here and see if it works here on RHEL compared to my normal nss invocation.

Even when this disclaimer is not here:
I am not a contracting officer. I do not have authority to make or modify the
terms of any contract.

Denniston, Todd A CIV NAVSURFWARCENDIV Crane wrote:
> Tue Jul 26 21:17:40 UTC 2016,  m.roth at 5-cent.us wrote:
>> I could try the reinstall, but it's very odd - everything worked,
and
>> now, after the upgrade, it doesn't. Oh, and here's another
twist on this:
>> Under 6.7, if I'd logged into my webmail via firefox, and while
that was
>> happening, I stuck my AA ("Logical token" card) into the
keyboard slot,
>> and used it in logging onto one server (using sccr), it just chugged
>> along. Now, at 6.8, *everything* in firefox hangs - even a google
>> search, until I pull the card out of the keyboard. It's clearly
trying to
>> authenticate the client, not just the server....
>
> I think your use of openssh -s libcoolkeypk11.so makes it look like:
> ssh
> coolkey
> pcscd
> ccid
<snip>
Some minor corrections, and more data: as I was typing, before I pulled my
PIV card out of the slot, and a second or two later, firefox crashed, so I
have to retype this....

I was doing ssh -i libcoolkeypk11.so <host> for debugging purposes. What I
used to do was ssh-add -s libcoolkeypk11.so. It would then ask for a PIN,
and add it. Now, it still asks for the PIN, but then announces that it
failed to add it to the agent. From messages, I see
ssh-pkcs11-helper[14669]: error: no slots, which I suspect is suggesting
some weird misconfiguration, because my manager, who originally got the
configuration working, tells me it shouldn't be using that.

Also, for a reader, lsusb shows  Dell Computer Corp. SmartCard Reader
Keyboard. More: when I used to insert the card, it would blink a bit,
maybe half a dozen times or so, and then stop. Now, it blinks for longer
(10?15?), then stops... but once I've tried to add the card and it fails,
it starts blinking, and doesn't stop.

       mark

Thu Jul 28 17:39:16 UTC 2016 ,  m.roth at 5-cent.us
wrote:
> What I
> used to do was ssh-add -s libcoolkeypk11.so. It would then ask for a PIN,
> and add it. Now, it still asks for the PIN, but then announces that it
> failed to add it to the agent.
Not sure if this is good or bad news :-/
On up to date RHEL6.8 the following looks like it worked.

$ ssh-add -D
All identities removed.
$ ssh-add -s libcoolkeypk11.so
Enter passphrase for PKCS#11:
Card added: libcoolkeypk11.so
$ ssh-add -l
#lists all three expected finger prints.
$ ssh -XA PKIneedingUser at localhost
[PKIneedingUser at localhost]$

Nothing good is ever easy.

Even when this disclaimer is not here:
I am not a contracting officer. I do not have authority to make or modify the
terms of any contract.