samba - Dec 2018 - After upgrade to 4.9.4, internal DNS no longer working

The two systems I'm referring to are on different domains, different
locations. No connection whatsoever except me being the one administering
both.

According to oEMInformation the server was provisioned by 4.3.3. I just
checked the Samba log file and it seems that version 4.7.4 was running
before the update.

I'm now looking at the logs right before the upgrade and indeed, there have
been errors so it looks like we found the reason for my issues. As a matter
of fact, just a day before the upgrade, we had an out of memory error on
the server and got into the situation that we had to make a hard reset. As
I can see now, Samba didn't go through this chain of events unharmed. It
was careless of me not to check but since the clients seemed to be able to
talk to the AD, I just didn't think there is an issue with Samba.

Here are the respective log entries. In the beginning, they refer to
version 4.7.4.:

[... prior log entry from a month earlier, no issues. Errors and warning
start after out of memory issue on the server ...]

[2018/12/21 03:06:02.974778,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.413186,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:16:03.741426,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:26:03.562027,  0]
../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
  ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error code
110
[2018/12/21 03:26:03.831110,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
110
[2018/12/21 03:27:58.903344,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT
[2018/12/21 03:32:59.630451,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_IO_TIMEOUT

  [... same errors repeat about 100 times, deleted for better legibility
...]

[2018/12/21 09:17:59.389008,  0]
../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
  ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error code
12
[2018/12/21 09:19:38.518542,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_NO_MEMORY
[2018/12/21 09:22:52.899142,  0]
../source4/smbd/server.c:450(binary_smbd_main)
  samba version 4.7.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2017
[2018/12/21 09:22:55.544887,  0]
../source4/smbd/server.c:622(binary_smbd_main)
  samba: using 'standard' process model
[2018/12/21 09:22:55.565451,  0]
../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'samba' finished starting up and ready to serve
connections
samba: setproctitle not initialized, please either call setproctitle_init()
or link against libbsd-ctor.

[... warning above repeats about 10 times ...]

[2018/12/22 19:13:44.916007,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: Traceback (most recent call last):
[2018/12/22 19:13:44.925489,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, in
<module>
[2018/12/22 19:13:44.925556,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     from samba import getopt as options
[2018/12/22 19:13:44.925595,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:   File
"/usr/lib/python2.7/site-packages/samba/__init__.py", line 29, in
<module>
[2018/12/22 19:13:44.999014,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc:     import samba.param
[2018/12/22 19:13:44.999099,  0]
../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
  /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
version `SAMBA_4.7.4' not found (required by /usr/lib/libsamba-util.so.0)
[2018/12/22 19:13:45.004497,  0]
../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
  ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
NT_STATUS_ACCESS_DENIED
[2018/12/22 19:31:48.283948,  0]
../source4/smbd/process_standard.c:86(sigterm_signal_handler)
  Exiting pid 391 on SIGTERM

  [... error above repeats about 10 times ...]

  [2018/12/22 19:31:48.446449,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447387,  0] ../source4/smbd/server.c:131(sig_term)
  SIGTERM: killing children
[2018/12/22 19:31:48.447428,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 367 on SIGTERM
[2018/12/22 19:31:48.464044,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 391 on SIGTERM
[2018/12/22 19:31:48.806419,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 394 on SIGTERM
[2018/12/22 19:31:48.806490,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 395 on SIGTERM
[2018/12/22 19:31:48.806540,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 399 on SIGTERM
[2018/12/22 19:31:48.806590,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 392 on SIGTERM
[2018/12/22 19:31:48.806634,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 458 on SIGTERM
[2018/12/22 19:31:48.806679,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 393 on SIGTERM
[2018/12/22 19:31:48.811314,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811348,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 401 on SIGTERM
  Exiting pid 396 on SIGTERM
[2018/12/22 19:31:48.811404,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 400 on SIGTERM
[2018/12/22 19:31:48.811451,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 402 on SIGTERM
[2018/12/22 19:31:48.811511,  0] ../source4/smbd/server.c:135(sig_term)
[2018/12/22 19:31:48.811546,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 398 on SIGTERM
[2018/12/22 19:31:48.811605,  0] ../source4/smbd/server.c:135(sig_term)
  Exiting pid 390 on SIGTERM
  Exiting pid 397 on SIGTERM
[2018/12/22 19:35:38.400284,  0]
../source4/smbd/server.c:510(binary_smbd_main)

[... clearly, I made an upgrade at this point, not aware I had problems.
what follows are issues I already described ...]

  samba version 4.9.4 started.
  Copyright Andrew Tridgell and the Samba Team 1992-2018
[2018/12/22 19:35:40.471526,  0]
../source4/smbd/server.c:696(binary_smbd_main)
  binary_smbd_main: samba: using 'standard' process model
[2018/12/22 19:35:40.647282,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.647380,  0]
../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
  Failed to find object DC=samdom,DC=example,DC=com for attribute
fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com to get attribute
fsmoRoleOwner for reference dn: No such Base DN: DC=samdom,DC=example,DC=com
[2018/12/22 19:35:40.648352,  0]
../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
  task_server_terminate: task_server_terminate: [kdc: krb5_init_context
samdb RODC connect failed]
[2018/12/22 19:35:40.674777,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
[2018/12/22 19:35:40.674823,  0]
../source4/smbd/service_task.c:36(task_server_terminate)
  task_server_terminate: task_server_terminate: [dreplsrv: Failed to
connect to local samdb: WERR_DS_UNAVAILABLE


It looks like the memory issue corrupted something in Samba. But what and
why? I'm sure I didn't make it better by upgrading, I assume this
reindexing you referred to didn't really happen if Samba was already in a
degraded state. On the other hand, if the ldb-files can be queried
properly, where is the corruption? When it comes to these details, Samba is
still pretty much a black box to me.

Thanks a lot for bearing with me so far. I hope this can be solved somehow.
I guess the good thing is that, if worse comes to worst, I could just
downgrade back to 4.7. and restore a backup from before the memory issue.
Would it be alright, in that worst case, if I just overwrite all contents
of /var/lib/samba with the files I have in the backup?


On Sun, 23 Dec 2018 at 11:36, Rowland Penny via samba <samba at
lists.samba.org>
wrote:
> On Sun, 23 Dec 2018 11:19:37 +0100
> Viktor Trojanovic <viktor at troja.ch> wrote:
>
> > I was too quick to send my previous email. It can't really be a
> > problem of the package since I have, as mentioned, another system
> > with the exact same setup and the exact same package versions and it
> > all works there...
> >
>
> Upgrading a running Samba server on an OS should work without error,
> unless there was an underlying problem already.
>
> Is your other system running as a DC in the same domain, or is it a
> totally separate domain ?
>
> It might help if you could remember what version you upgraded from, was
> it the version it was provisioned at ?
> If so, there is an attribute in the base 'dc=samdom,dc=example,dc',
> this attribute 'oEMInformation' contains the Samba version at
provision.
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

I could really use some support with this. I understand it's always
possible to just restore from a backup but the more interesting question is
if something can be done with the data at hand. Basically, I'm trying to
understand how it's possible that a dbcheck shows no errors, an ldbsearch
is successful, and yet it's not possible to start the AD properly. What
else is there that could be corrupted, and is there a way to repair it?

Thanks.

On Sun, 23 Dec 2018 at 13:12, Viktor Trojanovic <viktor at troja.ch>
wrote:
> The two systems I'm referring to are on different domains, different
> locations. No connection whatsoever except me being the one administering
> both.
>
> According to oEMInformation the server was provisioned by 4.3.3. I just
> checked the Samba log file and it seems that version 4.7.4 was running
> before the update.
>
> I'm now looking at the logs right before the upgrade and indeed, there
> have been errors so it looks like we found the reason for my issues. As a
> matter of fact, just a day before the upgrade, we had an out of memory
> error on the server and got into the situation that we had to make a hard
> reset. As I can see now, Samba didn't go through this chain of events
> unharmed. It was careless of me not to check but since the clients seemed
> to be able to talk to the AD, I just didn't think there is an issue
with
> Samba.
>
> Here are the respective log entries. In the beginning, they refer to
> version 4.7.4.:
>
> [... prior log entry from a month earlier, no issues. Errors and warning
> start after out of memory issue on the server ...]
>
> [2018/12/21 03:06:02.974778,  0]
> ../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
>   ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error
> code 110
> [2018/12/21 03:16:03.413186,  0]
> ../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
>   ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error
> code 110
> [2018/12/21 03:16:03.741426,  0]
> ../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
>   ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error
> code 110
> [2018/12/21 03:26:03.562027,  0]
> ../source4/dsdb/dns/dns_update.c:290(dnsupdate_nameupdate_done)
>   ../source4/dsdb/dns/dns_update.c:290: Failed DNS update - with error
> code 110
> [2018/12/21 03:26:03.831110,  0]
> ../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
>   ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error
> code 110
> [2018/12/21 03:27:58.903344,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_IO_TIMEOUT
> [2018/12/21 03:32:59.630451,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_IO_TIMEOUT
>
>   [... same errors repeat about 100 times, deleted for better legibility
> ...]
>
> [2018/12/21 09:17:59.389008,  0]
> ../source4/dsdb/dns/dns_update.c:313(dnsupdate_spnupdate_done)
>   ../source4/dsdb/dns/dns_update.c:313: Failed SPN update - with error
> code 12
> [2018/12/21 09:19:38.518542,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_NO_MEMORY
> [2018/12/21 09:22:52.899142,  0]
> ../source4/smbd/server.c:450(binary_smbd_main)
>   samba version 4.7.4 started.
>   Copyright Andrew Tridgell and the Samba Team 1992-2017
> [2018/12/21 09:22:55.544887,  0]
> ../source4/smbd/server.c:622(binary_smbd_main)
>   samba: using 'standard' process model
> [2018/12/21 09:22:55.565451,  0]
> ../lib/util/become_daemon.c:124(daemon_ready)
>   STATUS=daemon 'samba' finished starting up and ready to serve
connections
> samba: setproctitle not initialized, please either call
> setproctitle_init() or link against libbsd-ctor.
>
> [... warning above repeats about 10 times ...]
>
> [2018/12/22 19:13:44.916007,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc: Traceback (most recent call last):
> [2018/12/22 19:13:44.925489,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:   File "/usr/bin/samba_kcc", line 45, in
<module>
> [2018/12/22 19:13:44.925556,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:     from samba import getopt as options
> [2018/12/22 19:13:44.925595,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:   File
> "/usr/lib/python2.7/site-packages/samba/__init__.py", line 29, in
<module>
> [2018/12/22 19:13:44.999014,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc:     import samba.param
> [2018/12/22 19:13:44.999099,  0]
> ../lib/util/util_runcmd.c:327(samba_runcmd_io_handler)
>   /usr/bin/samba_kcc: ImportError: /usr/lib/samba/libreplace-samba4.so:
> version `SAMBA_4.7.4' not found (required by
/usr/lib/libsamba-util.so.0)
> [2018/12/22 19:13:45.004497,  0]
> ../source4/dsdb/kcc/kcc_periodic.c:693(samba_kcc_done)
>   ../source4/dsdb/kcc/kcc_periodic.c:693: Failed samba_kcc -
> NT_STATUS_ACCESS_DENIED
> [2018/12/22 19:31:48.283948,  0]
> ../source4/smbd/process_standard.c:86(sigterm_signal_handler)
>   Exiting pid 391 on SIGTERM
>
>   [... error above repeats about 10 times ...]
>
>   [2018/12/22 19:31:48.446449,  0] ../source4/smbd/server.c:131(sig_term)
>   SIGTERM: killing children
> [2018/12/22 19:31:48.447387,  0] ../source4/smbd/server.c:131(sig_term)
>   SIGTERM: killing children
> [2018/12/22 19:31:48.447428,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 367 on SIGTERM
> [2018/12/22 19:31:48.464044,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 391 on SIGTERM
> [2018/12/22 19:31:48.806419,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 394 on SIGTERM
> [2018/12/22 19:31:48.806490,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 395 on SIGTERM
> [2018/12/22 19:31:48.806540,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 399 on SIGTERM
> [2018/12/22 19:31:48.806590,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 392 on SIGTERM
> [2018/12/22 19:31:48.806634,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 458 on SIGTERM
> [2018/12/22 19:31:48.806679,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 393 on SIGTERM
> [2018/12/22 19:31:48.811314,  0] ../source4/smbd/server.c:135(sig_term)
> [2018/12/22 19:31:48.811348,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 401 on SIGTERM
>   Exiting pid 396 on SIGTERM
> [2018/12/22 19:31:48.811404,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 400 on SIGTERM
> [2018/12/22 19:31:48.811451,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 402 on SIGTERM
> [2018/12/22 19:31:48.811511,  0] ../source4/smbd/server.c:135(sig_term)
> [2018/12/22 19:31:48.811546,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 398 on SIGTERM
> [2018/12/22 19:31:48.811605,  0] ../source4/smbd/server.c:135(sig_term)
>   Exiting pid 390 on SIGTERM
>   Exiting pid 397 on SIGTERM
> [2018/12/22 19:35:38.400284,  0]
> ../source4/smbd/server.c:510(binary_smbd_main)
>
> [... clearly, I made an upgrade at this point, not aware I had problems.
> what follows are issues I already described ...]
>
>   samba version 4.9.4 started.
>   Copyright Andrew Tridgell and the Samba Team 1992-2018
> [2018/12/22 19:35:40.471526,  0]
> ../source4/smbd/server.c:696(binary_smbd_main)
>   binary_smbd_main: samba: using 'standard' process model
> [2018/12/22 19:35:40.647282,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
> [2018/12/22 19:35:40.647380,  0]
> ../source4/dsdb/common/util.c:1815(samdb_reference_dn_is_our_ntdsa)
>   Failed to find object DC=samdom,DC=example,DC=com for attribute
> fsmoRoleOwner - Cannot find DN DC=samdom,DC=example,DC=com to get attribute
> fsmoRoleOwner for reference dn: No such Base DN:
DC=samdom,DC=example,DC=com
> [2018/12/22 19:35:40.648352,  0]
> ../source4/dsdb/dns/dns_update.c:127(dnsupdate_rebuild)
>   task_server_terminate: task_server_terminate: [kdc: krb5_init_context
> samdb RODC connect failed]
> [2018/12/22 19:35:40.674777,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
> [2018/12/22 19:35:40.674823,  0]
> ../source4/smbd/service_task.c:36(task_server_terminate)
>   task_server_terminate: task_server_terminate: [dreplsrv: Failed to
> connect to local samdb: WERR_DS_UNAVAILABLE
>
>
> It looks like the memory issue corrupted something in Samba. But what and
> why? I'm sure I didn't make it better by upgrading, I assume this
> reindexing you referred to didn't really happen if Samba was already in
a
> degraded state. On the other hand, if the ldb-files can be queried
> properly, where is the corruption? When it comes to these details, Samba is
> still pretty much a black box to me.
>
> Thanks a lot for bearing with me so far. I hope this can be solved
> somehow. I guess the good thing is that, if worse comes to worst, I could
> just downgrade back to 4.7. and restore a backup from before the memory
> issue. Would it be alright, in that worst case, if I just overwrite all
> contents of /var/lib/samba with the files I have in the backup?
>
>
> On Sun, 23 Dec 2018 at 11:36, Rowland Penny via samba <
> samba at lists.samba.org> wrote:
>
>> On Sun, 23 Dec 2018 11:19:37 +0100
>> Viktor Trojanovic <viktor at troja.ch> wrote:
>>
>> > I was too quick to send my previous email. It can't really be
a
>> > problem of the package since I have, as mentioned, another system
>> > with the exact same setup and the exact same package versions and
it
>> > all works there...
>> >
>>
>> Upgrading a running Samba server on an OS should work without error,
>> unless there was an underlying problem already.
>>
>> Is your other system running as a DC in the same domain, or is it a
>> totally separate domain ?
>>
>> It might help if you could remember what version you upgraded from, was
>> it the version it was provisioned at ?
>> If so, there is an attribute in the base
'dc=samdom,dc=example,dc',
>> this attribute 'oEMInformation' contains the Samba version at
provision.
>>
>> Rowland
>>
>> --
>> To unsubscribe from this list go to the following URL and read the
>> instructions:  https://lists.samba.org/mailman/options/samba
>
>

On Wed, 26 Dec 2018 11:43:37 +0100
Viktor Trojanovic <viktor at troja.ch> wrote:
> I could really use some support with this. I understand it's always
> possible to just restore from a backup but the more interesting
> question is if something can be done with the data at hand.
> Basically, I'm trying to understand how it's possible that a
dbcheck
> shows no errors, an ldbsearch is successful, and yet it's not
> possible to start the AD properly. What else is there that could be
> corrupted, and is there a way to repair it?
> 
OK, I have been reviewing all the posts in this thread and I have a few
questions ;-)

You posted that you are running the DC using the internal DNS server,
but you also posted this:

All checks on the flat files work fine

What 'flat files' ?
Are you referring to Bind9 flat files ?

There is also this:

kdc_task_init: Cannot determine if we are an RODC: operations error at
../source4/dsdb/common/util.c:3534
task_server_terminate: task_server_terminate: [kdc: krb5_init_context samdb RODC
connect failed]

Why is 'RODC' getting mentioned ?

Can you post the contents of:

/etc/hostname
/etc/nsswitch.conf

Could the separate Samba daemons (smbd, nmbd, winbind) be being started
instead of/as well as the 'samba' daemon ?

Rowland