Hi, folks,
We're experiencing an odd ten-second delay intermittently when logging
into any of our Linux boxes which authenticate against LDAP. Here's where
it happens:
Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my
uid\gid> (e=0/0)
Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key
file>
My assumption is there's something in sssd slowing it down, but I'm
having a heck of a time figuring out what or why. Any guidance would be
greatly appreciated.
Thanks,
John A
--
John Adams
Senior Linux/Middleware Administrator | Information Technology Services
+1-501-916-3010 | jxadams at ualr.edu | http://ualr.edu/itservices
*UA Little Rock*
Reminder: IT Services will never ask for your password over the phone or
in an email. Always be suspicious of requests for personal information that
come via email, even from known contacts. For more information or to
report suspicious email, visit IT Security
<http://ualr.edu/itservices/security/>.
On Thu, 20 Jul 2023 at 10:54, Johnnie W Adams <jxadams at ualr.edu> wrote:> We're experiencing an odd ten-second delay intermittently when logging > into any of our Linux boxes [...] I'm > having a heck of a time figuring out what or why. Any guidance would be > greatly appreciated.You can strace sshd to see what it's doing when it stalls. Assuming you can connect to another port, you can do something like this to run sshd on 2222, tracing both parent and child sshds: $ sudo strace -f /usr/sbin/sshd -De -ologlevel=fatal -p 2222 (note that the trace may contain sensitive information so treat it with caution) -- Darren Tucker (dtucker at dtucker.net) GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860 37F4 9357 ECEF 11EA A6FA Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.
> On 20 Jul 2023, at 19:45, Johnnie W Adams <jxadams at ualr.edu> wrote: > > Hi, folks, > > We're experiencing an odd ten-second delay intermittently when logging > into any of our Linux boxes which authenticate against LDAP. Here's where > it happens: > > Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my > uid\gid> (e=0/0) > > Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key > file> > > My assumption is there's something in sssd slowing it down, but I'm > having a heck of a time figuring out what or why. Any guidance would be > greatly appreciated.The current (by default for me, after DNS is fixed) guilty party since Debian 11, had been dbus on especially LXC containers. Disabling that had been one of the easy fixes for me when I get hit by that "hang" .. its actually, so frequent for me, it's part of my script to copy-paste when prepping a LXC DEbian 11&12 instance
On Thu, Jul 20, 2023 at 1:49?PM Johnnie W Adams <jxadams at ualr.edu> wrote:> > Hi, folks, > > We're experiencing an odd ten-second delay intermittently when logging > into any of our Linux boxes which authenticate against LDAP. Here's where > it happens: > > Jul 13 11:54:23 console2 sshd[1853]: debug1: temporarily_use_uid: <my > uid\gid> (e=0/0) > > Jul 13 11:54:35 console2 sshd[1853]: debug1: trying public key file <my key > file> > > My assumption is there's something in sssd slowing it down, but I'm > having a heck of a time figuring out what or why. Any guidance would be > greatly appreciated. > > Thanks, > > John Asssd is a pretty aggressively "optimized" tool. It's designed, not to issue LDAP queries, but to pull from a locally stashed copy of the *entire* upstream LDAP directory, or at least enough of the LDAP directory to contain every dolder it may reference. The result can be really nasty when the VPN connection between an internal AD and a cloud environment, especially when it thinks it has to refresh that cache. All of it. Without notice. And crash, if it doesn't succeed within the hard-coded and un-tunable timeout periods. I'm not happy with some of sssd's behavior, especially the head games it plays with systemd about "I'm started, I'm running, I'm allowing logins via SSH, la-la-la-la-la, I failed to cache the full LDAP and now I will crash hard with systemd not noticing and recovering the service". It's an unpleasant problem.