freebsd stable - Dec 2019 - ldapsearch stops working after ~4-12 hours (one host of 4)

I have four hosts all running 11.3-RELEASE-p5 on the same subnet.

Yesterday I did the usual "freebsd-update fetch && freebsd-update
install" on all four. They came back up fine but about 4 hours later one
of them started reporting problems with ldap search. I poked at it a bit
but it was struggling to do anything so I had to reboot it. It came back
up clean but the problem resurfaced early this morning.

from /var/log/messages:

Dec 27 03:30:00 [redacted] root: 3:30AM up 11:16, 1 user, load averages:
0.04, 0.07, 0.08
Dec 27 03:35:00 [redacted] root: 3:35AM up 11:21, 1 user, load averages:
0.21, 0.23, 0.15
Dec 27 03:35:10 [redacted] chgrp: nss_ldap: could not search LDAP server
- Server is unavailable
Dec 27 03:35:12 [redacted] top: nss_ldap: could not search LDAP server -
Server is unavailable
Dec 27 03:35:35 [redacted] nrpe[76163]: nss_ldap: could not search LDAP
server - Server is unavailable

Both times I observed this:

: ldapsearch -v -LLL -x -h [redacted].ee.washington.edu -b
dc=ee,dc=washington,dc=edu uid=[redacted]
ldap_initialize( ldap://[redacted].ee.washington.edu )
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

The other hosts on the same subnet have no such problems.

I re-ran fetch & update but it said there was nothing to do.

Any hints where I should start poking this?

thanks,
nomad

On Fri, Dec 27, 2019 at 12:10 PM Lee Damon <nomad at castle.org> wrote:
>
> Both times I observed this:
>
> : ldapsearch -v -LLL -x -h [redacted].ee.washington.edu -b
> dc=ee,dc=washington,dc=edu uid=[redacted]
> ldap_initialize( ldap://[redacted].ee.washington.edu )
> ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Do you have connection/access logs on the LDAP server to verify whether a
connection is even being established? Also, are you able to try running
those same ldapsearch queries with the IP address(es) rather than DNS names
for your server? The ?can?t contact? initially seems more like potentially
DNS resolution or firewall/connectivity than something LDAP related like
failure to bind successfully?


Thanks,
-Matt