openssh unix dev - Jul 2023 - Ten second intermittent delay on login

Hi, folks,

     We're experiencing an odd ten-second delay intermittently when logging
into any of our Linux boxes which authenticate against LDAP. Here's where
it happens:

Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
uid\gid> (e=0/0)

Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
file>

     My assumption is there's something in sssd slowing it down, but I'm
having a heck of a time figuring out what or why. Any guidance would be
greatly appreciated.

Thanks,

     John A

-- 
John Adams
Senior Linux/Middleware Administrator  | Information Technology Services
+1-501-916-3010 | jxadams at ualr.edu | http://ualr.edu/itservices
*UA Little Rock*

Reminder:  IT Services will never ask for your password over the phone or
in an email. Always be suspicious of requests for personal information that
come via email, even from known contacts.  For more information or to
report suspicious email, visit IT Security
<http://ualr.edu/itservices/security/>.

On Thu, 20 Jul 2023 at 10:54, Johnnie W Adams <jxadams at ualr.edu>
wrote:
>      We're experiencing an odd ten-second delay intermittently when
logging
> into any of our Linux boxes [...]   I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.
You can strace sshd to see what it's doing when it stalls.  Assuming
you can connect to another port, you can do something like this to run
sshd on 2222, tracing both parent and child sshds:

$ sudo strace -f /usr/sbin/sshd -De -ologlevel=fatal -p 2222

(note that the trace may contain sensitive information so treat it with caution)

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

> On 20 Jul 2023, at 19:45, Johnnie W Adams <jxadams at ualr.edu>
wrote:
> 
> Hi, folks,
> 
>     We're experiencing an odd ten-second delay intermittently when
logging
> into any of our Linux boxes which authenticate against LDAP. Here's
where
> it happens:
> 
> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
> uid\gid> (e=0/0)
> 
> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my
key
> file>
> 
>     My assumption is there's something in sssd slowing it down, but
I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.
The current (by default for me, after DNS is fixed) guilty party since Debian
11, had been dbus on especially LXC containers.
Disabling that had been one of the easy fixes for me when I get hit by that
"hang" .. its actually, so frequent for me, it's part of my script
to copy-paste when prepping a LXC DEbian 11&12 instance

On Thu, Jul 20, 2023 at 1:49?PM Johnnie W Adams <jxadams at ualr.edu>
wrote:
>
> Hi, folks,
>
>      We're experiencing an odd ten-second delay intermittently when
logging
> into any of our Linux boxes which authenticate against LDAP. Here's
where
> it happens:
>
> Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
> uid\gid> (e=0/0)
>
> Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my
key
> file>
>
>      My assumption is there's something in sssd slowing it down, but
I'm
> having a heck of a time figuring out what or why. Any guidance would be
> greatly appreciated.
>
> Thanks,
>
>      John A
sssd is a pretty aggressively "optimized" tool. It's designed, not
to
issue LDAP queries, but to pull from a locally stashed copy of the
*entire* upstream LDAP directory, or at least enough of the LDAP
directory to contain every dolder it may reference. The result can be
really nasty when the VPN connection between an internal AD and a
cloud environment, especially when it thinks it has to refresh that
cache. All of it. Without notice. And crash, if it doesn't succeed
within the hard-coded and un-tunable timeout periods.

I'm not happy with some of sssd's behavior, especially the head games
it plays with systemd about "I'm started, I'm running, I'm
allowing
logins via SSH, la-la-la-la-la, I failed to cache the full LDAP and
now I will crash hard with systemd not noticing and recovering the
service". It's an unpleasant problem.