samba - Feb 2019 - winbind causing huge timeouts/delays since 4.8

On 24.02.2019 20:22, Rowland Penny via samba wrote:
> On Sun, 24 Feb 2019 19:56:27 +0100
> Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
>
>> On 24.02.2019 19:25, Ralph Böhme via samba wrote:
>>> Am 24.02.2019 um 18:48 schrieb Rowland Penny via samba
>>> <samba at lists.samba.org>:
>>>> On Sun, 24 Feb 2019 18:28:43 +0100 Ralph Böhme <slow at
samba.org>
>>>> wrote:
>>>>> Am 24.02.2019 um 16:42 schrieb Rowland Penny via samba
>>>>> <samba at lists.samba.org>:
>>>>>> On Sun, 24 Feb 2019 15:58:39 +0100 Ralph Böhme <slow
at samba.org>
>>>>>> wrote:
>>>>>>> Another thing that a customer has just been bitten
by, was a
>>>>>>> subtle bug in winbindd's idmap cache that
resulted in all
>>>>>>> xid2sid requests going through the idmap backend,
iow winbindd
>>>>>>> issued LDAP requests. With a few thousand users,
things came to
>>>>>>> a grinding halt.
>>>>>>>
>>>>>>> https://bugzilla.samba.org/show_bug.cgi?id=13802
>>>>>>>
>>>>>>> Patch just landed upstream.
>>>>>> That is the bug I was referring to and probably
(amongst all the
>>>>>> other cruft) what was causing the OP's problem.
>>>>> Unlikely.
>>>> It is was I thought, but as the OP's setup is so
convoluted, it is
>>>> hard to say.
>>> I don't think it's convoluted, it's certainly beyond
the simple
>>> standard setup we all wish everybody was using, but I don't
think
>>> it is broken as is. I just think an appropriate analysis requires
>>> more resources then is available on the list.
>>>
>>>>>> However, this has nothing to
>>>>>> do with using the 'ad' backend with Active
Directory. We keep
>>>>>> dancing around this problem, saying things like 'we
need to fix
>>>>>> this', we have been saying this since Samba 4 was
released.
>>>>> Which problem? Fix what? Been saying what?
>>>> There have been numerous discussions about the 'ad'
backend over
>>>> the years and they have all gone nowhere. The 'ad'
backend still
>>>> works in the same way as it did when Samba 4 was released and
you
>>>> still have to store the next uidNumber & gidNumber outside
AD if
>>>> you use the Samba tools.
>>> Looks like you're mixing AD DC use case with member server use
>>> case. Can we please keep that seperate? Afaict, the one has nothing
>>> to do with the other.
>> I'm confused.. how is the choice of the idmap backend related to an
>> AD DC use case?
> Only in the case of wanting the same ID everywhere.
In my understanding, the idmap backend only needs to be specified in the 
smb.conf for member servers. That's why I still don't see how it is 
related to a AD DC use case. I take it I'm missing something crucial
here.
>>
>>>>>> Windows Uses the SID-RID to identify the user and the
domain it
>>>>>> comes from, surely we can find a way to do this for
Samba, we are
>>>>>> half way there with the 'rid' backend.
>>>>> I'm not really what "there" implies for you,
but it seems
>>>>> idmap_autorid is eventually the backend that takes you
"there". :)
>>>> No it doesn't, at the moment, the only way to get the same
ID on
>>>> all Unix machines (this includes DC's) is to use the
'ad' backend.
>>> Sure. But only certain use cases require the same id on all
>>> machines, many don't. I'm just saying that you should
better not
>>> use idmap_ad, but instead use eg idmap_autorid unless you're
setup
>>> requires idmap_ad.
>> Would you, or someone else mind sharing some of these use cases when
>> idmap_ad would be necessary and when idmap_autorid would suffice?
> My understanding is, like the 'rid' backend, if you use the same
> smb.conf on all Unix domain members, you will get the same ID.
>
>> Specifically, in which situations do I absolutely need the ID to be
>> the same on each member, and in which cases could I actually go
>> without this? For example, if my AD is managed by Samba only but I
>> only have Windows users who will never have to log in to a unix box,
>> are there still advantages of the ad backend over the (auto)rid one?
> If your users never log into a Unix domain member or store files on
> one, then do not even need to consider a winbind backend.
In a Windows clients only environment, I thought that having AD member 
servers implied that files were stored on them.

But I take it from what you write that as soon as some kind of access is 
needed, we are actually speaking about a "login", even if it's
just on
the share level and the user never sees a shell.

In which case, yet again, the question comes up which back end to use. 
I'll get to that further below.
>> I assume that most readers of the wiki will, like me, find that
>> "central administration of ID's inside the AD" and
"ID's not stored
>> in a local database that can corrupt with lost file ownership"
seem
>> like really important arguments (btw, the last point is not stated as
>> a disadvantage for the rid/autorid backend in the wiki). Reading
>> this, it just seems that the ad backend is always the right one
>> except that it's a headache to manage.
> It isn't really a headache to manage, once you get your head around
> it ;-)
For very large environments, I guess you would just script it either 
with samba-tool or powershell. But having to do it manually and remember 
that for every user, computer, and group 1-2 attributes need to be set, 
and kept track of, does seem like a nuisance.

>> To put it differently, if Samba was improved in such a way that we
>> could use the ad backend without having to manually manage the
>> rfc2307 attributes, wouldn't this be the best if not only solution
we
>> needed?
> I think you are mistaking using a subset of the RFC2307 attributes with
> using most of them. If you use any backend other than 'ad' (and I
> include a DC here) all you get is a uidNumber for a user, or a
> gidNumber for a group. Using the 'ad' backend gets an ID number
that
> you set, plus individual login shells, Unix home dirs etc
>
I guess I was. Are you saying, then, that if I have no need for 
individual login shells and *nix home dirs (which in a pure Windows 
client environment I clearly have not), the ad back end isn't really 
offering any advantage for me?

What still confuses me here is how the wiki states that in case of 
rid/autorid information about file ownership is lost if the local tdb 
fails and cannot be restored. I assume this does not mean that the 
actual ownership information is stored in tdb (or in the AD, for that 
matter) but that the user ID used in the ACL needs to be retrieved from 
somewhere, and in case of the ad back end this ID is being retrieved 
from the AD directly, in case of the rid back end it's retrieved from 
the local store only. And yet you said that on member servers with 
identical smb.conf files (I assume you were talking about the global 
section only), IDs would always be the same. Naturally, two questions 
come up:

1) If one member server fails, wouldn't it be possible and sufficient to 
just regenerate the tdb using the same smb.conf? If this yields the same 
results every time, then why would I even worry about a corrupted tdb?

2) If IDs were the same on all member servers, wouldn't that also mean 
that I should be able to restore the corrupted tdb on one member server 
using a valid tdb from another member server? And how would I go about 
that in practice?

I hope this makes sense.

Viktor

On Mon, 25 Feb 2019 09:24:24 +0100
Viktor Trojanovic via samba <samba at lists.samba.org> wrote:


> >> I'm confused.. how is the choice of the idmap backend related
to an
> >> AD DC use case?
> > Only in the case of wanting the same ID everywhere.
> In my understanding, the idmap backend only needs to be specified in
> the smb.conf for member servers. That's why I still don't see how
it
> is related to a AD DC use case. I take it I'm missing something
> crucial here.
A Samba AD DC uses idmap.ldb by default, this means that you get
xidNumbers in the '3000000' range, but if you use the 'ad'
backend on
Unix domain members, these xidNumbers get overridden by the uidNumber's
and gidNumber's set in AD. It also turns some groups from being both
users and groups into just groups.

> > If your users never log into a Unix domain member or store files on
> > one, then do not even need to consider a winbind backend.
> 
> In a Windows clients only environment, I thought that having AD
> member servers implied that files were stored on them.
> 
> But I take it from what you write that as soon as some kind of access
> is needed, we are actually speaking about a "login", even if
it's
> just on the share level and the user never sees a shell.
> 
> In which case, yet again, the question comes up which back end to
> use. I'll get to that further below.
Then I will answer it below ;-)
> 
> >> I assume that most readers of the wiki will, like me, find that
> >> "central administration of ID's inside the AD" and
"ID's not stored
> >> in a local database that can corrupt with lost file
ownership" seem
> >> like really important arguments (btw, the last point is not stated
> >> as a disadvantage for the rid/autorid backend in the wiki).
Reading
> >> this, it just seems that the ad backend is always the right one
> >> except that it's a headache to manage.
> > It isn't really a headache to manage, once you get your head
around
> > it ;-)
> 
> For very large environments, I guess you would just script it either 
> with samba-tool or powershell. But having to do it manually and
> remember that for every user, computer, and group 1-2 attributes need
> to be set, and kept track of, does seem like a nuisance.
Ah, that type of headache, what do I need to ad and what was the last
uidNumber I used.
> 
> 
> >> To put it differently, if Samba was improved in such a way that we
> >> could use the ad backend without having to manually manage the
> >> rfc2307 attributes, wouldn't this be the best if not only
solution
> >> we needed?
> > I think you are mistaking using a subset of the RFC2307 attributes
> > with using most of them. If you use any backend other than
> > 'ad' (and I include a DC here) all you get is a uidNumber for
a
> > user, or a gidNumber for a group. Using the 'ad' backend gets
an ID
> > number that you set, plus individual login shells, Unix home dirs
> > etc
> >
> I guess I was. Are you saying, then, that if I have no need for 
> individual login shells and *nix home dirs (which in a pure Windows 
> client environment I clearly have not), the ad back end isn't really 
> offering any advantage for me?
If your users do not need to actually log into a Unix machine, then
there is no need to set a login shell, they will get the default
'bin/false'. If your users do not store anything in a Unix homedir
(only really needed if they log in) then you don't need to set the
template homedir, though there is the default /home/DOMAIN/user.
> 
> What still confuses me here is how the wiki states that in case of 
> rid/autorid information about file ownership is lost if the local tdb 
> fails and cannot be restored. 
Where does it say that ?
>I assume this does not mean that the 
> actual ownership information is stored in tdb (or in the AD, for that 
> matter) but that the user ID used in the ACL needs to be retrieved
> from somewhere, and in case of the ad back end this ID is being
> retrieved from the AD directly, in case of the rid back end it's
> retrieved from the local store only. And yet you said that on member
> servers with identical smb.conf files (I assume you were talking
> about the global section only), IDs would always be the same.
The 'DOMAIN' ID's on Unix domain members are always stored in AD,
just
not always in a way you expect ;-)

For the 'ad' backend, the uidNumber & gidNumber attributes are used,
most others calculate the ID from the 'RID'. This means that if
something goes wrong and AD is okay, recreating the Unix domain member
should be enough. If the '*' tdb gets corrupt, you can delete it and it
will get recreated and whilst I cannot totally guarantee the ID's will
be the same, it doesn't really matter because the users and groups in
there are from the Well Known SIDs and are not used by the normal
users.
 
> Naturally, two questions come up:
> 
> 1) If one member server fails, wouldn't it be possible and sufficient
> to just regenerate the tdb using the same smb.conf? If this yields
> the same results every time, then why would I even worry about a
> corrupted tdb?
Yes and, on a Unix domain member, you don't have to worry
> 
> 2) If IDs were the same on all member servers, wouldn't that also
> mean that I should be able to restore the corrupted tdb on one member
> server using a valid tdb from another member server? And how would I
> go about that in practice?
Never even thought of doing that, probably because you don't need to.
> 
> I hope this makes sense.
Perfect sense to myself.

Rowland

On 25.02.2019 10:20, Rowland Penny via samba wrote:
> On Mon, 25 Feb 2019 09:24:24 +0100
> Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
>
>
>
>>>> I'm confused.. how is the choice of the idmap backend
related to an
>>>> AD DC use case?
>>> Only in the case of wanting the same ID everywhere.
>> In my understanding, the idmap backend only needs to be specified in
>> the smb.conf for member servers. That's why I still don't see
how it
>> is related to a AD DC use case. I take it I'm missing something
>> crucial here.
> A Samba AD DC uses idmap.ldb by default, this means that you get
> xidNumbers in the '3000000' range, but if you use the 'ad'
backend on
> Unix domain members, these xidNumbers get overridden by the uidNumber's
> and gidNumber's set in AD. It also turns some groups from being both
> users and groups into just groups.
>
Can you just help me understand the relationship between these 
xidNumbers and the SID, if there is any?

Assuming just DCs and I create a new user (using ADUC), I understand 
this user will receive a xidNumber (I take it xid stands for both uid 
and gid?) that will automatically be set by the DC and is in the 3000000 
range. Where is this xidNumber stored, though? If it was stored in the 
AD, I assume member servers could just query it directly, right? But 
that doesn't seem to be the case since we still need to set these 
xidNumbers manually when using the ad back end, and when using the rid 
backend not the xidNumber but the rid is used to calculate the ID. Then 
what is the purpose of the xidNumber set by the DC? Honestly, I'm a bit 
lost here...

>>> If your users never log into a Unix domain member or store files on
>>> one, then do not even need to consider a winbind backend.
>> In a Windows clients only environment, I thought that having AD
>> member servers implied that files were stored on them.
>>
>> But I take it from what you write that as soon as some kind of access
>> is needed, we are actually speaking about a "login", even if
it's
>> just on the share level and the user never sees a shell.
>>
>> In which case, yet again, the question comes up which back end to
>> use. I'll get to that further below.
> Then I will answer it below ;-)
>
>>>> I assume that most readers of the wiki will, like me, find that
>>>> "central administration of ID's inside the AD"
and "ID's not stored
>>>> in a local database that can corrupt with lost file
ownership" seem
>>>> like really important arguments (btw, the last point is not
stated
>>>> as a disadvantage for the rid/autorid backend in the wiki).
Reading
>>>> this, it just seems that the ad backend is always the right one
>>>> except that it's a headache to manage.
>>> It isn't really a headache to manage, once you get your head
around
>>> it ;-)
>> For very large environments, I guess you would just script it either
>> with samba-tool or powershell. But having to do it manually and
>> remember that for every user, computer, and group 1-2 attributes need
>> to be set, and kept track of, does seem like a nuisance.
> Ah, that type of headache, what do I need to ad and what was the last
> uidNumber I used.
Well.. yes. What other headaches are there with the ad backend?
:-)
>>
>>>> To put it differently, if Samba was improved in such a way that
we
>>>> could use the ad backend without having to manually manage the
>>>> rfc2307 attributes, wouldn't this be the best if not only
solution
>>>> we needed?
>>> I think you are mistaking using a subset of the RFC2307 attributes
>>> with using most of them. If you use any backend other than
>>> 'ad' (and I include a DC here) all you get is a uidNumber
for a
>>> user, or a gidNumber for a group. Using the 'ad' backend
gets an ID
>>> number that you set, plus individual login shells, Unix home dirs
>>> etc
>>>
>> I guess I was. Are you saying, then, that if I have no need for
>> individual login shells and *nix home dirs (which in a pure Windows
>> client environment I clearly have not), the ad back end isn't
really
>> offering any advantage for me?
> If your users do not need to actually log into a Unix machine, then
> there is no need to set a login shell, they will get the default
> 'bin/false'. If your users do not store anything in a Unix homedir
> (only really needed if they log in) then you don't need to set the
> template homedir, though there is the default /home/DOMAIN/user.
>
>> What still confuses me here is how the wiki states that in case of
>> rid/autorid information about file ownership is lost if the local tdb
>> fails and cannot be restored.
> Where does it say that ?
It's implied here: https://wiki.samba.org/index.php/Idmap_config_ad

Advantages: "IDs are not stored in a local database that can corrupt and 
thus file ownerships are not lost.*"*

Either I'm misunderstanding this, or this statement is really giving the 
wrong picture.
**
>> I assume this does not mean that the
>> actual ownership information is stored in tdb (or in the AD, for that
>> matter) but that the user ID used in the ACL needs to be retrieved
>> from somewhere, and in case of the ad back end this ID is being
>> retrieved from the AD directly, in case of the rid back end it's
>> retrieved from the local store only. And yet you said that on member
>> servers with identical smb.conf files (I assume you were talking
>> about the global section only), IDs would always be the same.
> The 'DOMAIN' ID's on Unix domain members are always stored in
AD, just
> not always in a way you expect ;-)
>
> For the 'ad' backend, the uidNumber & gidNumber attributes are
used,
> most others calculate the ID from the 'RID'. This means that if
> something goes wrong and AD is okay, recreating the Unix domain member
> should be enough. If the '*' tdb gets corrupt, you can delete it
and it
> will get recreated and whilst I cannot totally guarantee the ID's will
> be the same, it doesn't really matter because the users and groups in
> there are from the Well Known SIDs and are not used by the normal
> users.
>   
Let me see if I understand this correctly. No matter if the ad or the 
rid backend is used, the member server queries the AD to check which ID 
belongs to which user and sets ACL accordingly. In the case of ad, the 
uidNumber is used, in case of rid the ID is calculated in a transparent 
way. I assume this means that each time someone access a member server 
(for example to access a file), the member server will query the DC to 
verify this user's ID. What happens if the DC cannot be reached, will a 
local cache be used or will the request remain pending until a DC answers?

>> Naturally, two questions come up:
>>
>> 1) If one member server fails, wouldn't it be possible and
sufficient
>> to just regenerate the tdb using the same smb.conf? If this yields
>> the same results every time, then why would I even worry about a
>> corrupted tdb?
> Yes and, on a Unix domain member, you don't have to worry
Why did you mention the *nix domain member specifically? In which other 
cases would I have to worry?

Questions over questions... thanks for bearing with me! :-)

Viktor