samba - Mar 2016 - Samba AD/DC crashed again, third time in as many months

On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> I built this machine, and while it isn't the most robust box in the
> world
> it has been stable otherwise.  The RAID array is configured RAID1, I
> can't
> see how that could cause corruption issues and I haven't experienced
> any
> other data corruption issues apart from SAMBA collapsing
I know it is hard to swallow, but I really think this is hardware, or
the OS configuration under it, combined with unexpected shutdown or
some other corruption vector.

We have at this stage 10,000 or more domains running Samba4, and this
is only the second I've heard of with this kind of symptom.  The first
I blamed on a use of DRDB that I postulated was not preserving 'write
barriers' (that is, the thing that makes fsync() work) and a poweroff,
but I didn't really have any proof.

You do need to run a second DC, as well as run tools like memcheck on
this DC.  Make sure you regularly run the backup script, so you can
work out when the corruption happens, and verify your DB with dbcheck.

The second DC has the advantage that this kind of low-level corruption
doesn't easily spread across DRS replication (it would instead fail
replication). 

The error shown indicates that for some reason or other, it can't read
the schema.  This is very odd, as the schema doesn't change!  

We would love to get to the bottom of this. 

Unlike others, I don't think this has anything to do with packaging
(that would just make us not start at all), but a clean install on a
clean machine is my best advise, keeping the rest aside (and off) for
forensics if you have the patience. 

Finally, always keep the steps simple - otherwise we might start
confusing admin errors for hardware errors or vice verca.  The things
we all do in the panic are always the hardest to de-construct in the
cold light of day.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba

Hi Andrew,

Thanks for your input.  I intend to take a look at the memory on this
machine asap to see if that is the cause of my issues, and I figure I may
as well swap out the data cables while I'm there for good measure.

I didn't think package conflicts could be an issue as I spent a fair amount
of time double checking for conflicts when I initially moved from
canonical's distribution of samba to a compiled build as the domain had
been functional for awhile.

My only frustration at this point is that at no point does the wiki state
that in order to stably deploy active directory using samba one MUST deploy
at least to ADCs.  If that is true it would be helpful to state the
requirement in the docs.  I've got a few ADs deployed with single ADCs, now
I feel compelled to make those environments more robust and in many cases
lack the resources to do so without added cost for the client and/or a fair
amount of labor on my part.

I'll update this thread once I've had an opportunity to inspect the
hardware.

Kind regards,

JS
On Mar 6, 2016 6:56 PM, "Andrew Bartlett" <abartlet at
samba.org> wrote:
> On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> > I built this machine, and while it isn't the most robust box in
the
> > world
> > it has been stable otherwise.  The RAID array is configured RAID1, I
> > can't
> > see how that could cause corruption issues and I haven't
experienced
> > any
> > other data corruption issues apart from SAMBA collapsing
>
> I know it is hard to swallow, but I really think this is hardware, or
> the OS configuration under it, combined with unexpected shutdown or
> some other corruption vector.
>
> We have at this stage 10,000 or more domains running Samba4, and this
> is only the second I've heard of with this kind of symptom.  The first
> I blamed on a use of DRDB that I postulated was not preserving 'write
> barriers' (that is, the thing that makes fsync() work) and a poweroff,
> but I didn't really have any proof.
>
> You do need to run a second DC, as well as run tools like memcheck on
> this DC.  Make sure you regularly run the backup script, so you can
> work out when the corruption happens, and verify your DB with dbcheck.
>
> The second DC has the advantage that this kind of low-level corruption
> doesn't easily spread across DRS replication (it would instead fail
> replication).
>
> The error shown indicates that for some reason or other, it can't read
> the schema.  This is very odd, as the schema doesn't change!
>
> We would love to get to the bottom of this.
>
> Unlike others, I don't think this has anything to do with packaging
> (that would just make us not start at all), but a clean install on a
> clean machine is my best advise, keeping the rest aside (and off) for
> forensics if you have the patience.
>
> Finally, always keep the steps simple - otherwise we might start
> confusing admin errors for hardware errors or vice verca.  The things
> we all do in the panic are always the hardest to de-construct in the
> cold light of day.
>
> Thanks,
>
> Andrew Bartlett
>
> --
> Andrew Bartlett
> https://samba.org/~abartlet/
> Authentication Developer, Samba Team         https://samba.org
> Samba Development and Support, Catalyst IT
> https://catalyst.net.nz/services/samba
>
>
>
>
>
>
>

As advised I have begun the process of adding ADCs to this domain and
currently have a second samba ADC joined to the domain.  I would like to
demote the initial ADC and make this secondary ADC the primary as the
problematic machine is still crashing and when samba has failed DNS fails
across the domain.  I'd also like to know if it is possible to create
redundancy when using SAMBA_INTERNAL as a DNS backend.

Please advise.

JS
On Mar 6, 2016 6:56 PM, "Andrew Bartlett" <abartlet at
samba.org> wrote:
> On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> > I built this machine, and while it isn't the most robust box in
the
> > world
> > it has been stable otherwise.  The RAID array is configured RAID1, I
> > can't
> > see how that could cause corruption issues and I haven't
experienced
> > any
> > other data corruption issues apart from SAMBA collapsing
>
> I know it is hard to swallow, but I really think this is hardware, or
> the OS configuration under it, combined with unexpected shutdown or
> some other corruption vector.
>
> We have at this stage 10,000 or more domains running Samba4, and this
> is only the second I've heard of with this kind of symptom.  The first
> I blamed on a use of DRDB that I postulated was not preserving 'write
> barriers' (that is, the thing that makes fsync() work) and a poweroff,
> but I didn't really have any proof.
>
> You do need to run a second DC, as well as run tools like memcheck on
> this DC.  Make sure you regularly run the backup script, so you can
> work out when the corruption happens, and verify your DB with dbcheck.
>
> The second DC has the advantage that this kind of low-level corruption
> doesn't easily spread across DRS replication (it would instead fail
> replication).
>
> The error shown indicates that for some reason or other, it can't read
> the schema.  This is very odd, as the schema doesn't change!
>
> We would love to get to the bottom of this.
>
> Unlike others, I don't think this has anything to do with packaging
> (that would just make us not start at all), but a clean install on a
> clean machine is my best advise, keeping the rest aside (and off) for
> forensics if you have the patience.
>
> Finally, always keep the steps simple - otherwise we might start
> confusing admin errors for hardware errors or vice verca.  The things
> we all do in the panic are always the hardest to de-construct in the
> cold light of day.
>
> Thanks,
>
> Andrew Bartlett
>
> --
> Andrew Bartlett
> https://samba.org/~abartlet/
> Authentication Developer, Samba Team         https://samba.org
> Samba Development and Support, Catalyst IT
> https://catalyst.net.nz/services/samba
>
>
>
>
>
>
>

Answering to previous mail:
AD is hearth of infrastructure. That's where all accounts are stored. That
last affirmation implies few times after you start deploying AD most of
your IT infrastructure depends on AD (all applications need accounts, they
are in AD, no AD, no accounts, nothing work) and that you take security in
consideration and that you do that seriously: an attacker with
administrator account can do almost everything everywhere on machines
joined to AD.

So redundancy, every times.

You could also think about your own issue: is it the whole DB which is
broken or is it the DB on the broken DC? With one DC, the whole DB is the
one DC, so you always break the whole DB.
With several you get a chance to break only one DC and to have others with
a coherent DB. That do not means you will never break the whole DB (backup
and a working process to restore is still needed).

Second mail:
You want to remove your FSMO owner. The FSMO owner is SOA.
These two are really important notion in AD:
- FSMO is kind of PDC in NT4 domain, these roels must belong to one DC.
Seize role before demoting the old one.
- SOA is about DNS, it refers the one server where some client can push DNS
modification. Change SOA before you try to add a replacement server to the
one you demoted. If you don't the DC you would join to replace demoted DC
won't be able to send DNS update!

And yes it possible to get redundancy with dns-backend=SAMBA_INTERNAL.

How to test your DNS servers are well configured: samba_dnsupdate gives no
error on all DC (this a test related to DNS service only).



2016-03-07 11:47 GMT+01:00 IT Admin <it at cliffbells.com>:
> As advised I have begun the process of adding ADCs to this domain and
> currently have a second samba ADC joined to the domain.  I would like to
> demote the initial ADC and make this secondary ADC the primary as the
> problematic machine is still crashing and when samba has failed DNS fails
> across the domain.  I'd also like to know if it is possible to create
> redundancy when using SAMBA_INTERNAL as a DNS backend.
>
> Please advise.
>
> JS
> On Mar 6, 2016 6:56 PM, "Andrew Bartlett" <abartlet at
samba.org> wrote:
>
> > On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> > > I built this machine, and while it isn't the most robust box
in the
> > > world
> > > it has been stable otherwise.  The RAID array is configured
RAID1, I
> > > can't
> > > see how that could cause corruption issues and I haven't
experienced
> > > any
> > > other data corruption issues apart from SAMBA collapsing
> >
> > I know it is hard to swallow, but I really think this is hardware, or
> > the OS configuration under it, combined with unexpected shutdown or
> > some other corruption vector.
> >
> > We have at this stage 10,000 or more domains running Samba4, and this
> > is only the second I've heard of with this kind of symptom.  The
first
> > I blamed on a use of DRDB that I postulated was not preserving
'write
> > barriers' (that is, the thing that makes fsync() work) and a
poweroff,
> > but I didn't really have any proof.
> >
> > You do need to run a second DC, as well as run tools like memcheck on
> > this DC.  Make sure you regularly run the backup script, so you can
> > work out when the corruption happens, and verify your DB with dbcheck.
> >
> > The second DC has the advantage that this kind of low-level corruption
> > doesn't easily spread across DRS replication (it would instead
fail
> > replication).
> >
> > The error shown indicates that for some reason or other, it can't
read
> > the schema.  This is very odd, as the schema doesn't change!
> >
> > We would love to get to the bottom of this.
> >
> > Unlike others, I don't think this has anything to do with
packaging
> > (that would just make us not start at all), but a clean install on a
> > clean machine is my best advise, keeping the rest aside (and off) for
> > forensics if you have the patience.
> >
> > Finally, always keep the steps simple - otherwise we might start
> > confusing admin errors for hardware errors or vice verca.  The things
> > we all do in the panic are always the hardest to de-construct in the
> > cold light of day.
> >
> > Thanks,
> >
> > Andrew Bartlett
> >
> > --
> > Andrew Bartlett
> > https://samba.org/~abartlet/
> > Authentication Developer, Samba Team         https://samba.org
> > Samba Development and Support, Catalyst IT
> > https://catalyst.net.nz/services/samba
> >
> >
> >
> >
> >
> >
> >
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>