samba - Mar 2016 - Samba AD/DC crashed again, third time in as many months

Samba is compiled from source.
Samba DB is stored on local RAID array.
Changes to AD are done using ADUC from a Windows 7 box.
AD is used for authentication, user shares (folder redirection), and shared
folders.

JS
On Mar 2, 2016 12:40 PM, "Marc Muehlfeld" <mmuehlfeld at
samba.org> wrote:
> Am 02.03.2016 um 18:07 schrieb IT Admin:
> > No, no schema adjustments.  This is the only domain controller, there
> have
> > been no changes to this domain since I rebuilt it about a month ago.
> Samba
> > keeps failing like this almost every month.
>
> Can you tell us some more information about your environment
> * Samba DBs are stored on a local HDD, NFS, Gluster, iSCSI,...
> * Is the AD only used for authentication? Or do other applications are
> connected, too? Especially if there are any that are trying to write to AD
> * How do you do changes in AD? ADUC? samba-tool, other tools?
> * etc.
>
>
> Regards,
> Marc
>

Am 02.03.2016 um 18:52 schrieb IT Admin:
> Samba is compiled from source.
> Samba DB is stored on local RAID array.
> Changes to AD are done using ADUC from a Windows 7 box.
> AD is used for authentication, user shares (folder redirection), and shared
> folders.
Can you add
  log level = 10
to your smb.conf, empty your log directory and start Samba. It should
generate a new log, that captures all output. Then put it to cpaste.org
or some other paste service and share the link with us. Maybe we see
something interesting.


Does something changed when the problem occured the first time? Some
package updates, crashes, etc.?


Can you make sure that no kind of Samba package (daemon, libs, etc.) is
installed on the system? Maybe your selfcompiled version overwrites some
stuff and your OS installed an update, that mixes now with the self
compiled version. Just a guess.

Regards,
Marc

Hi Marc,

I added "log level = 10" to my smb.conf, the new log is here:
http://www.anonpaste.net/?p=f1ec8

The initial failure of this domain occurred in mid-december, I associated
it with a failure of the raid array after an ubuntu upgrade, however, I
suspect that wasn't really the case.  I rebuilt the domain, it failed again
about a week later with roughly the same behaviour as I'm seeing now.  I
worked with the samba-users group over a period of about three weeks to
troubleshoot the issue, we settled on my database being corrupted beyond
repair, at the time I believed CrashPlan backup was to blame, but again I
suspect that may not have been the case, or at least it wasn't the only
issue in play.  I removed all Canonical samba packages, downloaded the
latest samba source from samba.org, compiled and installed samba 4.3.3.  I
then provisioned the current domain and rebuilt their infrastructure after
which I worked with Rowland a bit to implement the samba_backup script.
The domain seemed fine up until about two weeks ago at which time I started
seeing failures in log.samba, typically referencing dns update failures and
NT_STATUS_UNAVAILABLE etc.  A reboot of the DC always brought samba back, I
had hoped upgrading to the latest release would alleviate those issues.
Unfortunately the entire thing fell apart this morning, samba failing to
start, their entire domain is noon-functional.

I'm not sure how to verify all other unwanted samba packages are gone, what
would you suggest?

I look forward to any guidance you may be able to offer.

On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org>
wrote:
> Am 02.03.2016 um 18:52 schrieb IT Admin:
> > Samba is compiled from source.
> > Samba DB is stored on local RAID array.
> > Changes to AD are done using ADUC from a Windows 7 box.
> > AD is used for authentication, user shares (folder redirection), and
> shared
> > folders.
>
> Can you add
>   log level = 10
> to your smb.conf, empty your log directory and start Samba. It should
> generate a new log, that captures all output. Then put it to cpaste.org
> or some other paste service and share the link with us. Maybe we see
> something interesting.
>
>
> Does something changed when the problem occured the first time? Some
> package updates, crashes, etc.?
>
>
> Can you make sure that no kind of Samba package (daemon, libs, etc.) is
> installed on the system? Maybe your selfcompiled version overwrites some
> stuff and your OS installed an update, that mixes now with the self
> compiled version. Just a guess.
>
> Regards,
> Marc
>

I poked around the system using locate, I don't think there are any
packages installed on this system other than those associated with the
4.3.3 build I compiled.

smbd

/usr/lib/python2.7/dist-packages/samba/samba3/smbd.so
/usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0
/usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0
/usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0
/usr/local/samba/lib/private/libsmbd-base-samba4.so
/usr/local/samba/lib/private/libsmbd-conn-samba4.so
/usr/local/samba/lib/private/libsmbd-shim-samba4.so
/usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so
/usr/local/samba/private/smbd.tmp
/usr/local/samba/sbin/smbd
/usr/local/samba/share/man/man8/smbd.8
/usr/local/samba/var/log.smbd
/usr/local/samba/var/log.smbd.old
/usr/local/samba/var/run/smbd.pid
/var/log/upstart/smbd.log.1.gz

nmbd

/usr/local/samba/sbin/nmbd
/usr/local/samba/share/man/man8/nmbd.8
/var/log/upstart/nmbd.log.1.gz

samba

/usr/local/samba/sbin/nmbd
/usr/local/samba/sbin/samba
/usr/local/samba/sbin/samba_dnsupdate
/usr/local/samba/sbin/samba_kcc
/usr/local/samba/sbin/samba_spnupdate
/usr/local/samba/sbin/samba_upgradedns
/usr/local/samba/sbin/smbd
/usr/local/samba/sbin/winbind


If I'm overlooking something obvious please let me know.

JS

On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org>
wrote:
> Am 02.03.2016 um 18:52 schrieb IT Admin:
> > Samba is compiled from source.
> > Samba DB is stored on local RAID array.
> > Changes to AD are done using ADUC from a Windows 7 box.
> > AD is used for authentication, user shares (folder redirection), and
> shared
> > folders.
>
> Can you add
>   log level = 10
> to your smb.conf, empty your log directory and start Samba. It should
> generate a new log, that captures all output. Then put it to cpaste.org
> or some other paste service and share the link with us. Maybe we see
> something interesting.
>
>
> Does something changed when the problem occured the first time? Some
> package updates, crashes, etc.?
>
>
> Can you make sure that no kind of Samba package (daemon, libs, etc.) is
> installed on the system? Maybe your selfcompiled version overwrites some
> stuff and your OS installed an update, that mixes now with the self
> compiled version. Just a guess.
>
> Regards,
> Marc
>