[Samba] Problems with TDBs on CTDB-managed Samba instance

Jeremy Allison
2015-10-18T00:31:47Z

> Hi Jeremy,
>
> Thanks so much for your reply!  As a matter of fact, we did just that
around 3:45p yesterday when our CTDB cluster was unable to self-heal from the
latest in this series of failover events.  Here's how the situation went
down:
>
> 1)  We saw flapping identical to that described in my original post
>
> 2)  After about 30 minutes of waiting, CTDB was just spinning with `smbd`
repeatedly failing its health checks.
>
> 3)  We stopped CTDB on the cluster with `onnode all service ctdb stop`
>
> 4)  We moved the following files out of the way:
>
> gencache_notrans.tdb
> gencache.tdb
> mutex.tdb
>
> 5)  We started CTDB again with `onnode all service ctdb start`
>
> At that point, the CTDB cluster came back up successfully and all has been
quiet since yesterday afternoon.  We notice that gencache_notrans.tdb has not
even begun to move toward its (supposedly) pathological high-water-mark size of
~4G.  In fact, we have a script watching its size right now and it is steady at
~500K since the restart yesterday:
>
> """
> [root@<HOST> lock]# ll gencache_notrans.tdb
> -rw-r--r-- 1 root root 532480 Oct 17 12:01 gencache_notrans.tdb
> """
>
> Because of the current steady size of this file compared to its repeated,
intermittent, and rapid inflation, we suspect that there is some operational
condition which *causes* the corruption and which we're running into with
some regularity.  Our cluster is attached to a rather large ADS domain and
`strings gencache_notrans.tdb|less` during the trouble reveals a long series of
Windows SID entries followed eventually by a *very large* number of the ASCII
character "B" (presumably going all the way to the end of the file.
Our current suspicion in that there is some ADS user whose record, when
ingested, somehow corrupts the TDB.  Our investigations into the last
*successfully-ingested* SID in the corrupt TDB will continue on Monday morning.
>
> Of course, we may be wrong in this hypothesis.  Can you (or anyone else on
the list) comment on the apparent correctness of this assessment and possibly
shed light on what scenarios are known to result in TDB corruption?  The actions
we took yesterday seem to have alleviated the problem for now, however, we are
very interested in taking any steps necessary to prevent its recurrence in the
future.
>
> Again, thank you so much for your time  :)

So you have a copy of the corrupted tdb files ?

I don't know of any outstanding bugs that can cause
tdb corruption I'm afraid.