samba - Oct 2020 - Replication issues / local DRS authentication failure

Hello,

I've having recurring issues with the second DC in my Samba AD domain.
I've
demoted/removed the second DC a number of times and re-provisioned it.
It'll work for a bit and then replication breaks.

Since replication is broken I've been doing the following to remove it:

DC-02
systemctl stop samba
systemctl disable samba

DC-01
samba-tool domain demote --remove-other-dead-server=h-msn-smbdc-02
--verbose --username=administrator
samba-tool dbcheck --cross-ncs --fix --yes
systemctl restart samba
samba-tool domain tombstones expunge


Before re-provisioning DC-02 yet again, I wanted to make sure DC-01 is
truly healthy, and it appears that it may not be.

[root at h-msn-smbdc-01 ~]# samba-tool drs showrepl
Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for
ncacn_ip_tcp:10.10.50.10[49153,seal,target_hostnameh-msn-smbdc-01.dom.local.com,abstract_syntax=e3514235-4b06-11d1-ab04-00c04fc2dcd2/0x00000004,localaddress=10.10.50.10]
NT_STATUS_LOGON_FAILURE
ERROR(<class 'samba.drs_utils.drsException'>): DRS connection to
h-msn-smbdc-01.dom.local.com failed - drsException: DRS connection to
h-msn-smbdc-01.dom.local.com failed: (3221225581, 'The attempted logon is
invalid. This is either due to a bad username or authentication
information.')
  File "/usr/lib64/python3.8/site-packages/samba/netcmd/drs.py", line
55,
in drsuapi_connect
    (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions)
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
  File "/usr/lib64/python3.8/site-packages/samba/drs_utils.py", line
63, in
drsuapi_connect
    raise drsException("DRS connection to %s failed: %s" % (server,
e))


[root at h-msn-smbdc-01 ~]# net ads testjoin
kerberos_kinit_password LOCAL at DOM.LOCAL.COM failed: Client not found in
Kerberos database
kerberos_kinit_password LOCAL at DOM.LOCAL.COM failed: Client not found in
Kerberos database
Join to domain is not valid: The name provided is not a properly formed
account name.


[root at h-msn-smbdc-01 ~]# net rpc testjoin
Join to domain 'LOCAL' is not valid: NT_STATUS_ACCESS_DENIED


It looks like there's some sort of authentication issue with authenticating
to itself?

I've considered re-provisioning DC-02 and stealing all the FMSO roles,
demoting/removing DC-01, and re-provisioning DC-01. Not sure if that'd fix
the issue, but I'd rather understand what the root cause of this is.

Any help would be greatly appreciated.

Thanks,
Derek

On 19/10/2020 18:43, Derek Lambert via samba wrote:
> Hello,
>
> I've having recurring issues with the second DC in my Samba AD domain.
I've
> demoted/removed the second DC a number of times and re-provisioned it.
> It'll work for a bit and then replication breaks.
>
> Since replication is broken I've been doing the following to remove it:
>
> DC-02
> systemctl stop samba
> systemctl disable samba
>
> DC-01
> samba-tool domain demote --remove-other-dead-server=h-msn-smbdc-02
> --verbose --username=administrator
> samba-tool dbcheck --cross-ncs --fix --yes
> systemctl restart samba
> samba-tool domain tombstones expunge
>
>
> Before re-provisioning DC-02 yet again, I wanted to make sure DC-01 is
> truly healthy, and it appears that it may not be.
>
> [root at h-msn-smbdc-01 ~]# samba-tool drs showrepl
> Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for
> ncacn_ip_tcp:10.10.50.10[49153,seal,target_hostname>
h-msn-smbdc-01.dom.local.com,abstract_syntax=e3514235-4b06-11d1-ab04-00c04fc2dcd2/0x00000004,localaddress=10.10.50.10]
> NT_STATUS_LOGON_FAILURE
> ERROR(<class 'samba.drs_utils.drsException'>): DRS connection
to
> h-msn-smbdc-01.dom.local.com failed - drsException: DRS connection to
> h-msn-smbdc-01.dom.local.com failed: (3221225581, 'The attempted logon
is
> invalid. This is either due to a bad username or authentication
> information.')
>    File "/usr/lib64/python3.8/site-packages/samba/netcmd/drs.py",
line 55,
> in drsuapi_connect
>      (ctx.drsuapi, ctx.drsuapi_handle, ctx.bind_supported_extensions) >
drs_utils.drsuapi_connect(ctx.server, ctx.lp, ctx.creds)
>    File "/usr/lib64/python3.8/site-packages/samba/drs_utils.py",
line 63, in
> drsuapi_connect
>      raise drsException("DRS connection to %s failed: %s" %
(server, e))
>
>
> [root at h-msn-smbdc-01 ~]# net ads testjoin
> kerberos_kinit_password LOCAL at DOM.LOCAL.COM failed: Client not found in
> Kerberos database
> kerberos_kinit_password LOCAL at DOM.LOCAL.COM failed: Client not found in
> Kerberos database
> Join to domain is not valid: The name provided is not a properly formed
> account name.
>
>
> [root at h-msn-smbdc-01 ~]# net rpc testjoin
> Join to domain 'LOCAL' is not valid: NT_STATUS_ACCESS_DENIED
>
>
> It looks like there's some sort of authentication issue with
authenticating
> to itself?
>
> I've considered re-provisioning DC-02 and stealing all the FMSO roles,
> demoting/removing DC-01, and re-provisioning DC-01. Not sure if that'd
fix
> the issue, but I'd rather understand what the root cause of this is.
>
> Any help would be greatly appreciated.
>
> Thanks,
> Derek
There is a possible vital piece of information missing from the above, 
the name 'Fedora'. I take it you are using the distro Samba packages, if
so, don't. The Fedora distro packages use MIT for kerberos and are not 
recommended for production use.

Can I suggest you use Debian on your computer (rpi ?) instead and this 
will use Heimdal for kerberos.

Rowland

On 19/10/2020 20:37, Derek Lambert wrote:
> Yes, I'm running Fedora 32 on RPi 4's.
>
> I'll need to rework my Ansible role but Debian should be doable.
>
> But that alone won't resolve the existing issue will it? I'd rather
> not recreate the domain unless it's completely?unavoidable.
>
There are two benefits of using Raspbian, one is that is doesn't use 
MIT, the second is that you can get up to date Samba packages here:? 
http://apt.van-belle.nl/

I would add a new 'Debian' DC to your domain and then transfer all your 
FSMO roles to this and demote all the other DC's. You may be able fix 
your old domain once this has been done and you wont be any worse off 
than you are now.

There is a least one other user who posts regularly on here that uses 
raspberrypi's as DC's, so it is known to work.

Rowland

On 20/10/2020 21:09, Derek Lambert wrote:
> I really dislike Raspian, and there's not a 64bit release yet.
>
> Does the official Debian distro use Heimdal? Ubuntu?
Cannot really speak for the rpi, I just use mine as a Unix domain member 
running raspbian, but it is basically only the distros based on red-hat 
distros that use MIT. If the Arm version of Ubuntu is like the X86-64 
version (and I no reason to believe otherwise) it will use Heimdal. Not 
sure about the Suse Arm distro (which uses RPMs), but it may be worth 
considering.

Rowland

On 10/20/20 3:25 PM, Rowland penny via samba wrote:
> On 20/10/2020 21:09, Derek Lambert wrote:
>> I really dislike Raspian, and there's not a 64bit release yet.
>>
>> Does the official Debian distro use Heimdal? Ubuntu?
> 
> Cannot really speak for the rpi, I just use mine as a Unix domain member
running
> raspbian, but it is basically only the distros based on red-hat distros
that use
> MIT. If the Arm version of Ubuntu is like the X86-64 version (and I no
reason to
> believe otherwise) it will use Heimdal. Not sure about the Suse Arm distro 
> (which uses RPMs), but it may be worth considering.
openSUSE does have libheimdal and libheimdal-devel, but samba and default 
kerberos... all MIT.

With that said, way back in 2016, Red Hat promised to get problems using MIT 
kerberos resolved with regards to having a Samba DC.  In their words, 
"...Andreas, Guenther and Alexander at Red Hat are working diligently every
day
towards this. We're planning to get to that sooner rather than later."
(again,
from 2016)

Others pointed out that Red Hat's "working diligently" meant work
on sssd though.