samba - Mar 2016 - Samba AD/DC crashed again, third time in as many months

Well, this puts me in a catch-22 situation... I can see the benefit of
spinning up VMs as ADCs, unfortunately this machine is already leveraged to
the limit and there aren't any resources available to support a single
additional VM, let alone two of them.  And I'm scratching my head a bit
here as I have another Samba ADC deployed on another network, similar host
OS, similar RAID setup, and it has been rock solid...

I think the best course of action for this specific situation short-term is
to restore one of the samba backups to get their domain up and running
ASAP.  I'll then have to figure out how to shuffle resources to get a
second ADC running in a VM and with any luck redundancy will mitigate
future corruption issues.  That being said I'm very keen to understand what
the source of this corruption is, as I mentioned earlier I haven't had any
other issues with data corruption on this host and suspect that mismatched
libraries are a big part of the problem.

So, I need to verify, what is the proper way to remove the unwanted
packages in /usr/lib?  Am I trying to remove the correct packages with
apt?  Can I simply rename the offending files and reboot?

Once I've gotten rid of those files I'll follow the restore procedure
and
attempt to get samba running again with last week's backup, that will allow
the client to move forward with end of year accounting work and give me a
chance to figure out how to shuffle resources so I can implement the dual
VM architecture.

Please advise, I really need to get this domain functional again by end of
day, I've got about 5 hours to do so.

Thanks again everyone for all of your help.

JS
On Mar 3, 2016 11:30 AM, "mathias dufresne" <infractory at
gmail.com> wrote:
>
>
> 2016-03-03 16:32 GMT+01:00 IT Admin <it at cliffbells.com>:
>
>> Apt and to think those packages aren't installed:
>>
>> sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> Package 'libtalloc-dev' is not installed, so not removed
>> Package 'libtdb-dev' is not installed, so not removed
>> Package 'python-talloc-dev' is not installed, so not removed
>> The following packages were automatically installed and are no longer
>> required:
>>   linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic
>> Use 'apt-get autoremove' to remove them.
>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>
>> Should I just rename the files?
>>
>> If the database is corrupt my only recourse is to follow the restore
>> guidelines in the Samba wiki and roll back to a previous version of the
>> db,
>> correct?
>>
>> JS
>>
>
> I'd say that depend on how complex is that corruption and how much you
> have around you to understand that corruption.
>
> If it is too complex (main reason to be too complex is a boss in a hurry)
> restore the whole thing.
>
> During restoration what you need is the same kind of system, you don't
> really need to restore on the very same system. What I mean is you can take
> advantage of that issue to deploy a new system (using same version of that
> system, ie if it was centos6, use centos6).
> As you seem to have a big system doing lot of things, part of these things
> is running a Samba and even that software is configured to do several
> things (AD + file server) which is not advised, I would use that down time
> to switch Samba from that big system to some virtual machine with
> minimalistic system doing one and only one thing: Samba as AD. Then in a
> second VM I would install file server. In fact before installing file
> server I would create another VM to host a second DC.
>
> AD with one DC is not advised, anywhere. If you were having several DC
> perhaps you would not have the whole AD broken but only one DC broken and
> the other(s) one(s) working well. This was already explained to you today,
> I was just insisting a little bit : )
>
>
>
>
>> On Mar 3, 2016 7:40 AM, "Sketch" <smblist at
rednsx.org> wrote:
>>
>> > I'd remove the distro packages providing those libs in
/usr/lib, as they
>> > could possibly cause problems.  One more I forgot, which might
possibly
>> be
>> > responsible for corrupting your ldb database if the wrong one is
loaded
>> by
>> > samba at runtime, is libldb.
>> >
>> > From the log, I'm guessing your database is corrupt if it
can't read the
>> > schema, but someone else might have more insight.
>> >
>> > On Wed, 2 Mar 2016, IT Admin wrote:
>> >
>> > pytalloc:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2
>> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2
>> >> /usr/local/samba/include/pytalloc.h
>> >> /usr/local/samba/lib/private/libpytalloc-util.so.2
>> >> /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3
>> >>
>> >> libtalloc:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2
>> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2
>> >> /usr/local/samba/lib/private/libtalloc-report-samba4.so
>> >> /usr/local/samba/lib/private/libtalloc.so.2
>> >> /usr/local/samba/lib/private/libtalloc.so.2.1.3
>> >>
>> >>
>> >> libtdb:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1
>> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
>> >> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
>> >> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
>> >> /usr/local/samba/lib/private/libtdb-wrap-samba4.so
>> >> /usr/local/samba/lib/private/libtdb.so.1
>> >> /usr/local/samba/lib/private/libtdb.so.1.3.7
>> >>
>> >>
>> >> I also noticed that I've got a log.smbd in
/usr/local/samba/var:
>> >> http://www.anonpaste.net/?p=bdfa3
>> >>
>> >> JS
>> >>
>> >>
>> >>
>> >> On Wed, Mar 2, 2016 at 9:00 PM, Sketch <smblist at
rednsx.org> wrote:
>> >>
>> >> Also check for pytalloc|libtalloc and libtdb.
>> >>>
>> >>>
>> >>> On Wed, 2 Mar 2016, IT Admin wrote:
>> >>>
>> >>> I poked around the system using locate, I don't think
there are any
>> >>>
>> >>>> packages installed on this system other than those
associated with
>> the
>> >>>> 4.3.3 build I compiled.
>> >>>>
>> >>>> smbd
>> >>>>
>> >>>> /usr/lib/python2.7/dist-packages/samba/samba3/smbd.so
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0
>> >>>> /usr/local/samba/lib/private/libsmbd-base-samba4.so
>> >>>> /usr/local/samba/lib/private/libsmbd-conn-samba4.so
>> >>>> /usr/local/samba/lib/private/libsmbd-shim-samba4.so
>> >>>>
/usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so
>> >>>> /usr/local/samba/private/smbd.tmp
>> >>>> /usr/local/samba/sbin/smbd
>> >>>> /usr/local/samba/share/man/man8/smbd.8
>> >>>> /usr/local/samba/var/log.smbd
>> >>>> /usr/local/samba/var/log.smbd.old
>> >>>> /usr/local/samba/var/run/smbd.pid
>> >>>> /var/log/upstart/smbd.log.1.gz
>> >>>>
>> >>>> nmbd
>> >>>>
>> >>>> /usr/local/samba/sbin/nmbd
>> >>>> /usr/local/samba/share/man/man8/nmbd.8
>> >>>> /var/log/upstart/nmbd.log.1.gz
>> >>>>
>> >>>> samba
>> >>>>
>> >>>> /usr/local/samba/sbin/nmbd
>> >>>> /usr/local/samba/sbin/samba
>> >>>> /usr/local/samba/sbin/samba_dnsupdate
>> >>>> /usr/local/samba/sbin/samba_kcc
>> >>>> /usr/local/samba/sbin/samba_spnupdate
>> >>>> /usr/local/samba/sbin/samba_upgradedns
>> >>>> /usr/local/samba/sbin/smbd
>> >>>> /usr/local/samba/sbin/winbind
>> >>>>
>> >>>>
>> >>>> If I'm overlooking something obvious please let me
know.
>> >>>>
>> >>>> JS
>> >>>>
>> >>>> On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld
<mmuehlfeld at samba.org
>> >
>> >>>> wrote:
>> >>>>
>> >>>> Am 02.03.2016 um 18:52 schrieb IT Admin:
>> >>>>
>> >>>>>
>> >>>>> Samba is compiled from source.
>> >>>>>> Samba DB is stored on local RAID array.
>> >>>>>> Changes to AD are done using ADUC from a
Windows 7 box.
>> >>>>>> AD is used for authentication, user shares
(folder redirection),
>> and
>> >>>>>>
>> >>>>>> shared
>> >>>>>
>> >>>>> folders.
>> >>>>>>
>> >>>>>>
>> >>>>> Can you add
>> >>>>>   log level = 10
>> >>>>> to your smb.conf, empty your log directory and
start Samba. It
>> should
>> >>>>> generate a new log, that captures all output. Then
put it to
>> >>>>> cpaste.org
>> >>>>> or some other paste service and share the link
with us. Maybe we see
>> >>>>> something interesting.
>> >>>>>
>> >>>>>
>> >>>>> Does something changed when the problem occured
the first time? Some
>> >>>>> package updates, crashes, etc.?
>> >>>>>
>> >>>>>
>> >>>>> Can you make sure that no kind of Samba package
(daemon, libs,
>> etc.) is
>> >>>>> installed on the system? Maybe your selfcompiled
version overwrites
>> >>>>> some
>> >>>>> stuff and your OS installed an update, that mixes
now with the self
>> >>>>> compiled version. Just a guess.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Marc
>> >>>>>
>>
>

On 03/03/2016 06:34 PM, IT Admin wrote:
> So, I need to verify, what is the proper way to remove the unwanted
> packages in /usr/lib?  Am I trying to remove the correct packages with
> apt?  Can I simply rename the offending files and reboot?
To be absolutely sure to have about a clean and non-corrupt system, I'd 
reccommend to reinstall on a CLEAN os, without manually 
renaming/removing files.

Just install whatever OS you use, like and know on a empty fresh machine 
(I DO like mdadm raid1, and have only good experiences with it, contrary 
to Rowland) One tip: don't use btrfs for your AD server, use ext4, or xfs.

I really advise to NOT start manually deleting stuff etc on your current 
install. You want to be as safe as possible after your misery. Even a 
unused desktop machine with raid1 will be better than patching your 
current misbehaving machine, is my advise.

That way you'd also be able to seperate AD DC functionality from your 
fileserver. If you want to be REALLY cheap, you could even start like this:

desktop machine, raid one, install kvm, and run TWO dc's on that 
machine. It's very easy to move around those kvm machines to different 
hosts, if you get some budget, or another spare machine.

(but: running two DCs on the same host of course does NOT give you 
redundancy if that host goes down, it could just help against the kind 
of corruption you experienced)

MJ

Thanks for your input, I could spin up a VM on a workstation but a) it
would be 32bit, and b) current samba ADC is on a 64 bit host... Not sure if
I could restore my backup there it not.

I think I can swap the roles of the current file server and another machine
to get greater resources to play with, so I likely will be implementing the
dual VM approach you outlined in the future but my primary concern at this
moment is to get the domain up again, they need access no later than 8am
tomorrow.

If I can get answers re removing the offending libs from /usr/lib I'll
hopefully me moving forward here in the next couple of hours.

JS
On Mar 3, 2016 2:35 PM, "mj" <lists at merit.unu.edu> wrote:
>
>
> On 03/03/2016 06:34 PM, IT Admin wrote:
>
>> So, I need to verify, what is the proper way to remove the unwanted
>> packages in /usr/lib?  Am I trying to remove the correct packages with
>> apt?  Can I simply rename the offending files and reboot?
>>
>
> To be absolutely sure to have about a clean and non-corrupt system, I'd
> reccommend to reinstall on a CLEAN os, without manually renaming/removing
> files.
>
> Just install whatever OS you use, like and know on a empty fresh machine
> (I DO like mdadm raid1, and have only good experiences with it, contrary to
> Rowland) One tip: don't use btrfs for your AD server, use ext4, or xfs.
>
> I really advise to NOT start manually deleting stuff etc on your current
> install. You want to be as safe as possible after your misery. Even a
> unused desktop machine with raid1 will be better than patching your current
> misbehaving machine, is my advise.
>
> That way you'd also be able to seperate AD DC functionality from your
> fileserver. If you want to be REALLY cheap, you could even start like this:
>
> desktop machine, raid one, install kvm, and run TWO dc's on that
machine.
> It's very easy to move around those kvm machines to different hosts, if
you
> get some budget, or another spare machine.
>
> (but: running two DCs on the same host of course does NOT give you
> redundancy if that host goes down, it could just help against the kind of
> corruption you experienced)
>
> MJ
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>