samba - Apr 2016 - winbindd becomes unresponsive on member server

Hello,

I've been working on converting our OmniOS home directory member
servers from Samba 3.6.25 to 4.3.8.  For the first few hours after
startup, everything is accessible and works as expected.  Eventually,
winbindd stops responding and smbd starts logging this error:

  domain_client_validate: Domain password server not available.

At this point, authentications no longer work.  "wbinfo -p" fails.  If
I run it with debug logging, winbindd only logs these messages every 5
minutes post-failure:


[2016/04/25 15:47:10.676347, 10, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cm.c:385(winbind_msg_domain_online)
  Domain ad is marked as online now.
[2016/04/25 15:47:10.682186, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain BUILTIN () SID S-1-5-32, flags = 0x0,
attribs = 0x0, type = 0x0
[2016/04/25 15:47:10.682226, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain YUKON () SID
S-1-5-21-1178196917-3343520102-2534146612, flags = 0x0, attribs = 0x0, type =
0x0
[2016/04/25 15:47:10.682257, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain ad (ad.cpp.edu) SID
S-1-5-21-2732431017-2472381161-1794148792, flags = 0x1d, attribs = 0x0, type =
0x2
[2016/04/25 15:47:10.682285, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain WIN (win.csupomona.edu) SID
S-1-5-21-117609710-706699826-1801674531, flags = 0x22, attribs = 0x48, type =
0x2
[2016/04/25 15:52:10.651117,  5, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_dual.c:579(winbind_child_died)
  Already reaped child 2409 died
[2016/04/25 15:52:10.684484, 10, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cm.c:385(winbind_msg_domain_online)
  Domain ad is marked as online now.
[2016/04/25 15:52:10.689225, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain BUILTIN () SID S-1-5-32, flags = 0x0,
attribs = 0x0, type = 0x0
[2016/04/25 15:52:10.689260, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain YUKON () SID
S-1-5-21-1178196917-3343520102-2534146612, flags = 0x0, attribs = 0x0, type =
0x0
[2016/04/25 15:52:10.689283, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain ad (ad.cpp.edu) SID
S-1-5-21-2732431017-2472381161-1794148792, flags = 0x1d, attribs = 0x0, type =
0x2
[2016/04/25 15:52:10.689306, 11, pid=1739, effective(0, 0), real(0, 0),
class=winbind] ../source3/winbindd/winbindd_cache.c:4597(unpack_tdc_domains)
  unpack_tdc_domains: Unpacking domain WIN (win.csupomona.edu) SID
S-1-5-21-117609710-706699826-1801674531, flags = 0x22, attribs = 0x48, type =
0x2


Note that the host is joined to the ad.cpp.edu domain and there's trust
to win.csupomona.edu.  Before the failure, I authenticated using both
domains.


Has anyone seen something like this before?  What should be my next
steps?


And here's our config:


[global]
        allow trusted domains = yes
        enable privileges = no
        deadtime = 10
        debug pid = yes
        disable netbios = yes
        idmap config * : backend = nss
        idmap config * : range = 1000-2147483648
        lanman auth = no
        load printers = no
        log level = 1
        map archive = no
        name resolve order = host
        realm = ad.cpp.edu
        restrict anonymous = 1
        security = ads
        server signing = auto
        show add printer wizard = no
        workgroup = ad
        writable = yes
        max log size = 512000
        unix extensions = no
        veto files = /$RECYCLE.BIN/
        vfs objects = shadow_copy2 zfsacl
        shadow: snapdir = .zfs/snapshot
        shadow: format = backup-%Y.%m.%d-%H.%M.%S
        shadow: sort = desc
        shadow: localtime = yes
        nfs4: mode = special
        multicast dns register = no
        wide links = yes
        private dir = /etc/samba/private
        logging = file

[homes] 
        browseable = no
        path = /export/user/%S

On Mon, 25 Apr 2016 18:33:05 -0700
Brian De Wolf <bldewolf at cpp.edu> wrote:
> I've been working on converting our OmniOS home directory member
> servers from Samba 3.6.25 to 4.3.8.  For the first few hours after
> startup, everything is accessible and works as expected.  Eventually,
> winbindd stops responding and smbd starts logging this error:
> 
>   domain_client_validate: Domain password server not available.
> 
In case anyone finds my post and wonders what happened, I tested
previous versions of samba (4.3.3 and 4.2.11) and was not able to
reproduce the issue.  While I was deciding what to do, 4.3.9 was
released and apparently fixed the bug I was running into.  We're now
moving forward with deploying 4.3.9.