samba - Dec 2018 - After upgrade to 4.9.4, internal DNS no longer working

Ok, you have found something, and i've seen a bit more. 

First, this. 
  /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
version `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
[2018/12/22 19:13:45.004497,  0]

There are left overs from a previous install. 

You are on Arch Linux as i did read. 
With what i've seen. 
Stop samba and all related services, if needed stop everything except ssh. 

Now, backup /var/lib/samba /var/cache/samba /etc/samba at least. 
Now remove every thing releated to samba., packages and any fine in a samba
subfolders.
Install mlocate or use find, and once you have removed all samba related the
packages
run updatedb
locate samba
And check again and remove the left overs, once you 100% sure there are no left
over.

The next thing to do. 
As for i've seen in your logs, you systemd service is not detecting your
hostname.
And if empty it falles back to localhost.

Use these steps for a possible fix. 

If you using systemd-resolved 
Disabled it, because your setup is not using it. 

Your OS is also upgraded also for a lower version, correct ? 
Small check,  What is in : /etc/hostname  and  /etc/machine-info
If thats correct, you can run : hostnamectl set-hostname server-name
Server-name only not FQDN here. 
If its not correct, correct it first, check /etc/hosts also then run the
hostnamectl command and set it.

Disable you systemd-resovled 
( since your not using it.) 

systemctl disable systemd-resolved
systemctl stop systemd-resolved
Systemctl mask systemd-resolved 

review /etc/resolv.conf and set it like this. 
search your.primary.domain.tld 
Servername this_DC_FIRST
Servername other_DC_second

Check if : /etc/NetworkManager/NetworkManager.conf exists.
If so, set in [main] 
dns=none

When this is all done, reboot the server. 
If its not ok now, make only this change. 

review /etc/resolv.conf and set it like this. 
search your.primary.domain.tld 
Servername this_DC_WITH_FSMO_ROLES_FIRST 
Servername this_DC
Servername other_DCs


Above looks like a lot but is really a 5-10 min fix. 
Most important backup the samba data and config. 

And ignore the out of memory, thats imo due to wrong samba libs. ( wrong
versions.. )


Greetz, 

Louis


 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Viktor Trojanovic via samba
> Verzonden: woensdag 26 december 2018 19:03
> Aan: Rowland Penny
> CC: samba at lists.samba.org
> Onderwerp: Re: [Samba] After upgrade to 4.9.4, internal DNS 
> no longer working
> 
> Hi Rowland,
> 
> I went back to the logs from before the upgrade to see if we 
> didn't miss
> anything.
> 
> If I understand the logs correctly, in the early morning of 
> December 21
> (3am), the server started running out of memory as was 
> confirmed in the
> logs with several Samba components complaining about 
> NT_STATUS_NO_MEMORY.
> Here just one line as an example.
> 
> [2018/12/21 09:19:38.518542,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_NO_MEMORY
> 
> By the time I was able to work on this (around 9:30), the 
> machine could no
> longer be accessed and had to be rebooted via hard reset.
> 
> However, Samba started after this hard reset without any errors. Samba
> actually remains error free for the next 1.5 days.
> 
> In the evening of the next day, I started the upgrade, at 19:10 to be
> exact. And the following lines stands out:
> 
> [2018/12/22 19:13:44.916007,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc: Traceback (most recent call last):
> [2018/12/22 19:13:44.925489,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, 
> in <module>
> [2018/12/22 19:13:44.925556,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:     from samba import getopt as options
> [2018/12/22 19:13:44.925595,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:   File
> "/usr/lib/python2.7/site-packages/samba/__init__.py", line 
> 29, in <module>
> [2018/12/22 19:13:44.999014,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:     import samba.param
> [2018/12/22 19:13:44.999099,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc: ImportError: 
> /usr/lib/samba/libreplace-samba4.so:
> version `SAMBA_4.7.4' not found (required by 
> /usr/lib/libsamba-util.so.0)
> [2018/12/22 19:13:45.004497,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_ACCESS_DENIED
> 
> These and some SIGTERM messages are the last lines in the 
> Samba log before
> the server is rebooted, and Samba is then started in version 
> 4.9.4 - but
> now with several errors.
> 
> While I really don't know if the lines above have any 
> relevance, I believe
> we may need to correct the assumption that something was 
> wrong before the
> update, it now much rather looks as if the update broke my DC. Do you
> agree?
> 
> 
> 
> On Wed, 26 Dec 2018 at 16:29, Viktor Trojanovic 
> <viktor at troja.ch> wrote:
> 
> >
> >
> > On Wed, 26 Dec 2018 at 15:04, Rowland Penny via samba <
> > samba at lists.samba.org> wrote:
> >
> >> On Wed, 26 Dec 2018 14:40:10 +0100
> >> Viktor Trojanovic <viktor at troja.ch> wrote:
> >>
> >> > Hi Rowland,
> >> >
> >> > Thanks for taking an interest.
> >> >
> >> > On Wed, 26 Dec 2018 at 14:27, Rowland Penny via samba
> >> > <samba at lists.samba.org> wrote:
> >> >
> >> > > On Wed, 26 Dec 2018 11:43:37 +0100
> >> > > Viktor Trojanovic <viktor at troja.ch> wrote:
> >> > >
> >> > > > I could really use some support with this. I
understand it's
> >> > > > always possible to just restore from a backup but
the more
> >> > > > interesting question is if something can be done 
> with the data at
> >> > > > hand. Basically, I'm trying to understand how
it's
> possible that
> >> > > > a dbcheck shows no errors, an ldbsearch is 
> successful, and yet
> >> > > > it's not possible to start the AD properly.
What
> else is there
> >> > > > that could be corrupted, and is there a way to
repair it?
> >> > > >
> >> > >
> >> > > OK, I have been reviewing all the posts in this thread 
> and I have a
> >> > > few questions ;-)
> >> > >
> >> > > You posted that you are running the DC using the
internal DNS
> >> > > server, but you also posted this:
> >> > >
> >> > > All checks on the flat files work fine
> >> > >
> >> > > What 'flat files' ?
> >> > > Are you referring to Bind9 flat files ?
> >> > >
> >> > >
> >> > No. I meant the *.tdb and *.ldb files in the samba directory.
I'm
> >> > using the internal DNS server, I don't have Bind 
> installed, never had.
> >>
> >> Good, 'flat files' usually refers to the Bind9 zones conf
files.
> >>
> >> >
> >> >
> >> > > There is also this:
> >> > >
> >> > > kdc_task_init: Cannot determine if we are an RODC: 
> operations error
> >> > > at ../source4/dsdb/common/util.c:3534
> >> > > task_server_terminate: task_server_terminate: [kdc:
> >> > > krb5_init_context samdb RODC connect failed]
> >> > >
> >> > > Why is 'RODC' getting mentioned ?
> >> > >
> >> > >
> >> > I don't even know what RODC is. :) This is a regular AD 
> DC install as
> >> > it is described on the Wiki, I didn't do anything exotic
there.
> >>
> >> A normal DC is an RWDC (Read Write Domain Controller), an RODC is
a
> >> 'Read Only Domain Controller'
> >>
> >> For some reason, Your Samba is getting confused
> >>
> >>
> > I wouldn't know why, I didn't make any changes in this
respect.
> >
> >
> >> >
> >> >
> >> > > Can you post the contents of:
> >> > >
> >> > > /etc/hostname
> >> > >
> >> >
> >> > DC1
> >> >
> >> >
> >> > > /etc/nsswitch.conf
> >> > >
> >> > >
> >> > passwd: files winbind mymachines systemd
> >> > group: files winbind mymachines systemd
> >> > shadow: files
> >> >
> >> > publickey: files
> >> >
> >> > hosts: files mymachines myhostname resolve [!UNAVAIL=return]
dns
> >>
> >> Try the above line like this:
> >>
> >> hosts: files dns
> >>
> >> > networks: files
> >> >
> >> > protocols: files
> >> > services: files
> >> > ethers: files
> >> > rpc: files
> >> >
> >> > netgroup: files
> >> >
> >> >
> >> > Could the separate Samba daemons (smbd, nmbd, winbind) be
being
> >> > started
> >> > > instead of/as well as the 'samba' daemon ?
> >> > >
> >> > >
> >> > No, it's really just the "Samba AD Daemon"
(samba.service). smbd
> >> > (smb.service), nmbd (nmb.service), as well as winbindd
> >> > (winbind.service) are not started. As you could see from 
> the log, the
> >> > binaries such as smbd and winbindd are indeed started but I
guess
> >> > that is done by Samba automatically?
> >>
> >> Yes, the 'samba' daemon does start smbd & winbind, but
it
> was just a
> >> thought.
> >>
> >> Check if systemd is starting any other dns server (resolved ?)
> >>
> >
> > Yes, it is starting resolved but that was always the case. 
> Resolved reads
> > from the network file and adds the correct nameserver 
> (itself) and search
> > string (samdom.example.com).
> >
> > Just to rule out resolved as the culprit, I now disabled 
> resolved entirely
> > and created a static resolv.conf file, then rebooted the server.
> >
> > After doing all of this, this is the Samba log:
> >
> > [2018/12/26 16:28:57.872726,  0]
> > ../source4/smbd/server.c:510(binary_smbd_main)
> >   samba version 4.9.4 started.
> >   Copyright Andrew Tridgell and the Samba Team 1992-2018
> > [2018/12/26 16:28:58.179432,  0]
> > ../source4/smbd/server.c:696(binary_smbd_main)
> >   binary_smbd_main: samba: using 'standard' process model
> > [2018/12/26 16:28:58.192497,  0]
> > ../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
> >   Failed to find object DC=samdom,DC=example,DC=com for attribute
> > fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com 
> to get attribute
> > fsmoRoleOwner for reference dn: No such Base DN: 
> DC=samdom,DC=example,DC=com
> > [2018/12/26 16:28:58.208645,  0]
> > ../source4/smbd/service_task.c:36(task_server_terminate)
> > [2018/12/26 16:28:58.211103,  0]
> > ../source4/smbd/service_task.c:36(task_server_terminate)
> >   task_server_terminate: task_server_terminate: [kdc: 
> krb5_init_context
> > samdb RODC connect failed]
> >   task_server_terminate: task_server_terminate: [dreplsrv: Failed to
> > connect to local samdb: WERR_DS_UNAVAILABLE
> >   ]
> > [2018/12/26 16:28:58.235875,  0]
> > ../source4/smbd/service_task.c:36(task_server_terminate)
> > [2018/12/26 16:28:58.241918,  0]
> > ../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
> >   task_server_terminate: task_server_terminate: [kccsrv: 
> Failed to connect
> > to local samdb: WERR_DS_UNAVAILABLE
> >   ]
> > [2018/12/26 16:28:58.274730,  0]
> > ../source4/smbd/service_task.c:36(task_server_terminate)
> >   task_server_terminate: task_server_terminate: [dns: 
> failed to load DNS
> > zones]
> >
> >
> >
> >
> >
> >
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

On Thu, 27 Dec 2018 09:34:03 +0100
"L.P.H. van Belle via samba" <samba at lists.samba.org> wrote:
> Ok, you have found something, and i've seen a bit more. 
> 
> First, this. 
>   /usr/bin/samba_kcc:
> ImportError: /usr/lib/samba/libreplace-samba4.so: version
> `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
> [2018/12/22 19:13:45.004497,  0]
> 
> There are left overs from a previous install. 
Good catch Louis, totally missed that, it is possible that not all the
Samba packages got upgraded, perhaps checking for old Samba packages
might help. It may be be this is all that is required.

Rowland

On Thu, 27 Dec 2018 08:52:02 +0000
Rowland Penny via samba <samba at lists.samba.org> wrote:
> On Thu, 27 Dec 2018 09:34:03 +0100
> "L.P.H. van Belle via samba" <samba at lists.samba.org>
wrote:
> 
> > Ok, you have found something, and i've seen a bit more. 
> > 
> > First, this. 
> >   /usr/bin/samba_kcc:
> > ImportError: /usr/lib/samba/libreplace-samba4.so: version
> > `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
> > [2018/12/22 19:13:45.004497,  0]
> > 
> > There are left overs from a previous install. 
> 
> Good catch Louis, totally missed that, it is possible that not all the
> Samba packages got upgraded, perhaps checking for old Samba packages
> might help. It may be be this is all that is required.
> 
> Rowland
> 
It looks like 'libsamba-util.so.0' is part of 'smbclient', so
was
smbclient upgraded ?

Rowland

Gooood morning Rowland, :-) 

Your late ;-).. 
What i also did see, so its more clear for others also. 
> Dez 22 21:08:31 dc1 systemd[1]: Starting Samba AD Daemon...
> Dez 22 21:08:31 dc1 kernel: audit: type=1131 audit(1545509311.984:52):
> pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=samba
comm="systemd"
> exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
res=failed'
> Dez 22 21:08:32 dc1 samba[733]: root process[733]: [2018/12/22
This line:  exe="/usr/lib/systemd/systemd" hostname=? addr=?
terminal=? res=failed'

So incorrect hostname/resolving resulting in this problem. 


Greetz, 

Louis


> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Rowland Penny via samba
> Verzonden: donderdag 27 december 2018 9:52
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] After upgrade to 4.9.4, internal DNS 
> no longer working
> 
> On Thu, 27 Dec 2018 09:34:03 +0100
> "L.P.H. van Belle via samba" <samba at lists.samba.org>
wrote:
> 
> > Ok, you have found something, and i've seen a bit more. 
> > 
> > First, this. 
> >   /usr/bin/samba_kcc:
> > ImportError: /usr/lib/samba/libreplace-samba4.so: version
> > `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
> > [2018/12/22 19:13:45.004497,  0]
> > 
> > There are left overs from a previous install. 
> 
> Good catch Louis, totally missed that, it is possible that not all the
> Samba packages got upgraded, perhaps checking for old Samba packages
> might help. It may be be this is all that is required.
> 
> Rowland
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

On Thu, 27 Dec 2018 11:07:08 +0100
"L.P.H. van Belle via samba" <samba at lists.samba.org> wrote:
> Gooood morning Rowland, :-) 
> 
> Your late ;-).. 
> What i also did see, so its more clear for others also. 
> 
> > Dez 22 21:08:31 dc1 systemd[1]: Starting Samba AD Daemon...
> > Dez 22 21:08:31 dc1 kernel: audit: type=1131
> > audit(1545509311.984:52): pid=1 uid=0 auid=4294967295
> > ses=4294967295 msg='unit=samba comm="systemd"
> > exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
> > res=failed' Dez 22 21:08:32 dc1 samba[733]: root process[733]:
> > [2018/12/22
> 
> This line:  exe="/usr/lib/systemd/systemd" hostname=? addr=?
> terminal=? res=failed' 
> 
> So incorrect hostname/resolving resulting in this problem. 
I actually think it could be a symptom and not the root cause. It
could be that two main things happened, systemd was upgraded and with
it 'resolved' was installed and smbclient wasn't upgraded.

I think that if 'resolved' is removed and ALL Samba packages are
upgraded, he might get it to work again.

Rowland