samba - Apr 2023 - samba on ubuntu 22 lts breaks after unattended-upgrade

Hello, this morning we had the unpleasant experience of two DCs breaking 
at the same time, no logins, no DNS resolution, any call to samba 
binaries fails with an error message similar to :

libldbsamba.so.0 : ldb_msg_element_is_inaccessible
(cannot copy&paste as I disabled network access to this server).

We first thought we had a crashed AD structure. Reverting to a copy one 
of the DCs from 06:00 this morning worked, alls services available, then 
after 30 mins again similar problems (but logins continued to work this 
time). After a lot of trial and failure we could conclude that the error 
happened some time this morning, but as soon as I activate an older copy 
of the server, after some time the same symptoms appear. I then started 
to look at updates, and found that the server had applied 
unattended-upgrades this morning. After applying these updates, the 
samba binaries break. It seems to be a security update (see 
https://bugs.launchpad.net/ubuntu/jammy/+source/samba/+bug/2014052). I 
could not find any further descriptions for this error, so it seems to 
be a specific case. We had used the van Belle repos, then after the got 
"stalled" we switched to the classical U22 samba, maybe there is a 
reason to find?

Samba version is 4.15.13-Ubuntu.

I now disabled unattended-upgrades and the DC is up and running since 2 
hours, which was never achieved since this morning.

Does this ring a bell with anybody? My current recovery plan is to setup 
a new "DC3" with U22 and join it to the domain, then demote the old
"DC1".

Best regards, JC

Samba strictly requires the matching version of ldb, you have mixed
packages with the Samba security update but not the matching ldb
security update.

I have attempted to re-bundle ldb back into Samba in the past, but was
unable to obtain consensus.  However, Debian (and so Ubuntu) has taken
this task on in a private patch for new versions, which is hopeful.

This doesn't help you however, you need to source your samba from a
single place and apply the whole security update, including ldb and
samba packages at a single time.

Andrew Bartlett

On Thu, 2023-04-06 at 00:05 +0200, Jakob Curdes via samba
wrote:
> Hello, this morning we had the unpleasant experience of two DCs
> breaking 
> at the same time, no logins, no DNS resolution, any call to samba 
> binaries fails with an error message similar to :
> 
> libldbsamba.so.0 : ldb_msg_element_is_inaccessible
> (cannot copy&paste as I disabled network access to this server).
> 
> We first thought we had a crashed AD structure. Reverting to a copy
> one 
> of the DCs from 06:00 this morning worked, alls services available,
> then 
> after 30 mins again similar problems (but logins continued to work
> this 
> time). After a lot of trial and failure we could conclude that the
> error 
> happened some time this morning, but as soon as I activate an older
> copy 
> of the server, after some time the same symptoms appear. I then
> started 
> to look at updates, and found that the server had applied 
> unattended-upgrades this morning. After applying these updates, the 
> samba binaries break. It seems to be a security update (see 
> https://bugs.launchpad.net/ubuntu/jammy/+source/samba/+bug/2014052
> ). I 
> could not find any further descriptions for this error, so it seems
> to 
> be a specific case. We had used the van Belle repos, then after the
> got 
> "stalled" we switched to the classical U22 samba, maybe there is
a
> reason to find?
> 
> Samba version is 4.15.13-Ubuntu.
> 
> I now disabled unattended-upgrades and the DC is up and running since
> 2 
> hours, which was never achieved since this morning.
> 
> Does this ring a bell with anybody? My current recovery plan is to
> setup 
> a new "DC3" with U22 and join it to the domain, then demote the
old
> "DC1".
> 
> Best regards, JC
> 
> 
> 
-- 
Andrew Bartlett (he/him)       https://samba.org/~abartlet/
Samba Team Member (since 2001) https://samba.org
Samba Team Lead, Catalyst IT   https://catalyst.net.nz/services/samba

Samba Development and Support, Catalyst.Net Limited

Catalyst.Net Ltd - a Catalyst IT group company - Expert Open Source
Solutions

06.04.2023 01:05, Jakob Curdes via samba wrote:
> Hello, this morning we had the unpleasant experience of two DCs breaking at
the same time, no logins, no DNS resolution, any call to samba binaries
> fails with an error message similar to :
> 
> libldbsamba.so.0 : ldb_msg_element_is_inaccessible
> (cannot copy&paste as I disabled network access to this server).
This is a packaging error.

As Andrew pointed out, ldb is packaged separately. This changed in a
later samba release for debian (4.16+, ubuntu usually follows debian
here), now it should not happen anymore.

Even before the merge of ldb source package in debian/ubuntu back into
samba source, like in this case, it is possible to get the deps right.
But someone needs to keep an eye on this, and to properly fix the deps
manually when necessary, again like in this case. The recent security
update of samba changed *both* libldb and its users, and this has been
apparent because ldb version is increased and new symbols appeared there
which needed to be updated in the debian packaging.  This means older
version of libldb can not be used. However, whomever did the update
for ubuntu, forgot to use stricter Depends: fields for samba-libldb
dependency.  Hence the error.

..
> I now disabled unattended-upgrades and the DC is up and running since 2
hours, which was never achieved since this morning.
> 
> Does this ring a bell with anybody? My current recovery plan is to setup a
new "DC3" with U22 and join it to the domain, then demote the old
"DC1".
You have to upgrade libldb2 before or together with upgrading
samba. If this doesn't work automatically for whatever reason
(I don't know why, I haven't looked at ubuntu samba update),
just upgrade it before you upgrade samba.

Hopefully next version of ubuntu will pick up current samba
packaging from debian and this problem will not occur anymore.

/mjt