samba - Dec 2018 - After upgrade to 4.9.4, internal DNS no longer working

I was too quick to send my previous email. It can't really be a problem of
the package since I have, as mentioned, another system with the exact same
setup and the exact same package versions and it all works there...

On Sun, 23 Dec 2018 at 11:18, Viktor Trojanovic <viktor at troja.ch>
wrote:
> Yes, I tried running dbcheck (see first email), it showed 0 errors from
> the start, I never had to fix anything. All checks on the flat files work
> fine, SMB basic access works fine too, but as soon as I introduce
> authentication or ldap or DNS, it fails.
>
> ch/com, just a typo. No firewall set up on this server so that's not it
> either.
>
> Is there anything else we could try? Or do you think this warrants raising
> a bug with the package maintainer at Arch?
>
>
>
> On Sun, 23 Dec 2018 at 11:10, Rowland Penny via samba <
> samba at lists.samba.org> wrote:
>
>> On Sun, 23 Dec 2018 10:38:05 +0100
>> Viktor Trojanovic <viktor at troja.ch> wrote:
>>
>> > I'm not aware of a new folder being created. I can confirm
that
>> > /var/lib/samba/private/sam.ldb is the only file with that name on
my
>> > system. How could I check if Samba indeed looks up this file and
is
>> > not looking for it somewhere else?
>> >
>> > Some additional information that might be relevant or not. I can
run
>> > ldbsearch -H on sam.ldb without errors. I can even query specific
>> > information, such as '(objectclass=person)' and the result
list looks
>> > accurate. Doesn't this mean that my sam.ldb is actually in
order and
>> > the error lies elsewhere?
>> >
>>
>> It sort of sounded like your latest Samba was using a different folder,
>> but it seems it isn't.
>>
>> If everything in sam.ldb is readable, then it is probably okay, have
>> you tried running 'samba-tool dbcheck' on it ?
>>
>> I wonder if your old Samba was <= 4.7.x. A new GUID index mode was
>> introduced at 4.8.0, but this should just slow things down at first
>> start up.
>>
>> There was also a change of ports used at 4.7.0, so if there is a
>> firewall in use, this could be your problem, see here:
>>
>> https://wiki.samba.org/index.php/Samba_AD_DC_Port_Usage
>>
>> You also posted in your smb.conf:
>>
>> realm = samdom.example.com
>>
>> Yet, in your other posts, you have this:
>>
>> DC=samdom,DC=example,DC=ch
>>
>> Which would make your dns domain (and realm)
'samdom.example.ch', I
>> take it this is a typo.
>>
>> Rowland
>>
>> --
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/options/samba
>
>

On Sun, 23 Dec 2018 11:19:37 +0100
Viktor Trojanovic <viktor at troja.ch> wrote:
> I was too quick to send my previous email. It can't really be a
> problem of the package since I have, as mentioned, another system
> with the exact same setup and the exact same package versions and it
> all works there...
> 
Upgrading a running Samba server on an OS should work without error,
unless there was an underlying problem already.

Is your other system running as a DC in the same domain, or is it a
totally separate domain ?

It might help if you could remember what version you upgraded from, was
it the version it was provisioned at ?
If so, there is an attribute in the base 'dc=samdom,dc=example,dc',
this attribute 'oEMInformation' contains the Samba version at provision.

Rowland

The two systems I'm referring to are on different domains, different
locations. No connection whatsoever except me being the one administering
both.

According to oEMInformation the server was provisioned by 4.3.3. I just
checked the Samba log file and it seems that version 4.7.4 was running
before the update.

I'm now looking at the logs right before the upgrade and indeed, there have
been errors so it looks like we found the reason for my issues. As a matter
of fact, just a day before the upgrade, we had an out of memory error on
the server and got into the situation that we had to make a hard reset. As
I can see now, Samba didn't go through this chain of events unharmed. It
was careless of me not to check but since the clients seemed to be able to
talk to the AD, I just didn't think there is an issue with Samba.

Here are the respective log entries. In the beginning, they refer to
version 4.7.4.:

[... prior log entry from a month earlier, no issues. Errors and warning
start after out of memory issue on the server ...]

[2018/12/21 03:06:02.974778,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.413186,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.741426,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:26:03.562027,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:26:03.831110,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:27:58.903344,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT
[2018/12/21 03:32:59.630451,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT

  [... same errors repeat about 100 times, deleted for better legibility
...]

[2018/12/21 09:17:59.389008,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
12
[2018/12/21 09:19:38.518542,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_NO_MEMORY
[2018/12/21 09:22:52.899142,  0]
../source4/smbd/server.c:450(binary_smbd_main)
  samba version 4.7.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2017
[2018/12/21 09:22:55.544887,  0]
../source4/smbd/server.c:622(binary_smbd_main)
  samba: using 'standard' process model
[2018/12/21 09:22:55.565451,  0]
../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'samba' finished starting up and ready to serve
connections
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.

[... warning above repeats about 10 times ...]

[2018/12/22 19:13:44.916007,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: Traceback (most recent call last):
[2018/12/22 19:13:44.925489,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, in
<module>
[2018/12/22 19:13:44.925556,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     from samba import getopt as options
[2018/12/22 19:13:44.925595,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File
"/usr/lib/python2.7/site-packages/samba/__init__.py", line 29, in
<module>
[2018/12/22 19:13:44.999014,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     import samba.param
[2018/12/22 19:13:44.999099,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
version `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
[2018/12/22 19:13:45.004497,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_ACCESS_DENIED
[2018/12/22 19:31:48.283948,  0]
../source4/smbd/process_standard.c:86(sigterm_signal_handler)
  Exiting pid 391 on SIGTERM

  [... error above repeats about 10 times ...]

  [2018/12/22 19:31:48.446449,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447387,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447428,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 367 on SIGTERM
[2018/12/22 19:31:48.464044,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 391 on SIGTERM
[2018/12/22 19:31:48.806419,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 394 on SIGTERM
[2018/12/22 19:31:48.806490,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 395 on SIGTERM
[2018/12/22 19:31:48.806540,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 399 on SIGTERM
[2018/12/22 19:31:48.806590,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 392 on SIGTERM
[2018/12/22 19:31:48.806634,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 458 on SIGTERM
[2018/12/22 19:31:48.806679,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 393 on SIGTERM
[2018/12/22 19:31:48.811314,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811348,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 401 on SIGTERM
  Exiting pid 396 on SIGTERM
[2018/12/22 19:31:48.811404,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 400 on SIGTERM
[2018/12/22 19:31:48.811451,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 402 on SIGTERM
[2018/12/22 19:31:48.811511,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811546,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 398 on SIGTERM
[2018/12/22 19:31:48.811605,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 390 on SIGTERM
  Exiting pid 397 on SIGTERM
[2018/12/22 19:35:38.400284,  0]
../source4/smbd/server.c:510(binary_smbd_main)

[... clearly, I made an upgrade at this point, not aware I had problems.
what follows are issues I already described ...]

  samba version 4.9.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2018
[2018/12/22 19:35:40.471526,  0]
../source4/smbd/server.c:696(binary_smbd_main)
  binary_smbd_main: samba: using 'standard' process model
[2018/12/22 19:35:40.647282,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.647380,  0]
../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
  Failed to find object DC=samdom,DC=example,DC=com for attribute
fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com to get attribute
fsmoRoleOwner for reference dn: No such Base DN: DC=samdom,DC=example,DC=com
[2018/12/22 19:35:40.648352,  0]
../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
  task_server_terminate: task_server_terminate: [kdc: krb5_init_context
samdb RODC connect failed]
[2018/12/22 19:35:40.674777,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.674823,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
  task_server_terminate: task_server_terminate: [dreplsrv: Failed to
connect to local samdb: WERR_DS_UNAVAILABLE


It looks like the memory issue corrupted something in Samba. But what and
why? I'm sure I didn't make it better by upgrading, I assume this
reindexing you referred to didn't really happen if Samba was already in a
degraded state. On the other hand, if the ldb-files can be queried
properly, where is the corruption? When it comes to these details, Samba is
still pretty much a black box to me.

Thanks a lot for bearing with me so far. I hope this can be solved somehow.
I guess the good thing is that, if worse comes to worst, I could just
downgrade back to 4.7. and restore a backup from before the memory issue.
Would it be alright, in that worst case, if I just overwrite all contents
of /var/lib/samba with the files I have in the backup?


On Sun, 23 Dec 2018 at 11:36, Rowland Penny via samba <samba at
lists.samba.org>
wrote:
> On Sun, 23 Dec 2018 11:19:37 +0100
> Viktor Trojanovic <viktor at troja.ch> wrote:
>
> > I was too quick to send my previous email. It can't really be a
> > problem of the package since I have, as mentioned, another system
> > with the exact same setup and the exact same package versions and it
> > all works there...
> >
>
> Upgrading a running Samba server on an OS should work without error,
> unless there was an underlying problem already.
>
> Is your other system running as a DC in the same domain, or is it a
> totally separate domain ?
>
> It might help if you could remember what version you upgraded from, was
> it the version it was provisioned at ?
> If so, there is an attribute in the base 'dc=samdom,dc=example,dc',
> this attribute 'oEMInformation' contains the Samba version at
provision.
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba