samba - Feb 2019 - winbind causing huge timeouts/delays since 4.8

On 24.02.2019 19:25, Ralph Böhme via samba wrote:
> Am 24.02.2019 um 18:48 schrieb Rowland Penny via samba <samba at
lists.samba.org>:
>> On Sun, 24 Feb 2019 18:28:43 +0100 Ralph Böhme <slow at
samba.org> wrote:
>>> Am 24.02.2019 um 16:42 schrieb Rowland Penny via samba <samba at
lists.samba.org>:
>>>> On Sun, 24 Feb 2019 15:58:39 +0100 Ralph Böhme <slow at
samba.org> wrote:
>>>>> Another thing that a customer has just been bitten by, was
a subtle
>>>>> bug in winbindd's idmap cache that resulted in all
xid2sid requests
>>>>> going through the idmap backend, iow winbindd issued LDAP
requests.
>>>>> With a few thousand users, things came to a grinding halt.
>>>>>
>>>>> https://bugzilla.samba.org/show_bug.cgi?id=13802
>>>>>
>>>>> Patch just landed upstream.
>>>> That is the bug I was referring to and probably (amongst all
the
>>>> other cruft) what was causing the OP's problem.
>>> Unlikely.
>> It is was I thought, but as the OP's setup is so convoluted, it is
hard
>> to say.
> I don't think it's convoluted, it's certainly beyond the simple
standard setup we all wish everybody was using, but I don't think it is
broken as is. I just think an appropriate analysis requires more resources then
is available on the list.
>
>>>> However, this has nothing to
>>>> do with using the 'ad' backend with Active Directory.
We keep
>>>> dancing around this problem, saying things like 'we need to
fix
>>>> this', we have been saying this since Samba 4 was released.
>>> Which problem? Fix what? Been saying what?
>> There have been numerous discussions about the 'ad' backend
over the
>> years and they have all gone nowhere. The 'ad' backend still
works in
>> the same way as it did when Samba 4 was released and you still have to
>> store the next uidNumber & gidNumber outside AD if you use the
Samba
>> tools.
> Looks like you're mixing AD DC use case with member server use case.
Can we please keep that seperate? Afaict, the one has nothing to do with the
other.
I'm confused.. how is the choice of the idmap backend related to an AD 
DC use case?

>>>> Windows Uses the SID-RID to identify the user and the domain it
>>>> comes from, surely we can find a way to do this for Samba, we
are
>>>> half way there with the 'rid' backend.
>>> I'm not really what "there" implies for you, but it
seems
>>> idmap_autorid is eventually the backend that takes you
"there". :)
>> No it doesn't, at the moment, the only way to get the same ID on
all
>> Unix machines (this includes DC's) is to use the 'ad'
backend.
> Sure. But only certain use cases require the same id on all machines, many
don't. I'm just saying that you should better not use idmap_ad, but
instead use eg idmap_autorid unless you're setup requires idmap_ad.
Would you, or someone else mind sharing some of these use cases when 
idmap_ad would be necessary and when idmap_autorid would suffice? 
Specifically, in which situations do I absolutely need the ID to be the 
same on each member, and in which cases could I actually go without 
this? For example, if my AD is managed by Samba only but I only have 
Windows users who will never have to log in to a unix box, are there 
still advantages of the ad backend over the (auto)rid one?

I assume that most readers of the wiki will, like me, find that "central 
administration of ID's inside the AD" and "ID's not stored in
a local
database that can corrupt with lost file ownership" seem like really 
important arguments (btw, the last point is not stated as a disadvantage 
for the rid/autorid backend in the wiki). Reading this, it just seems 
that the ad backend is always the right one except that it's a headache 
to manage.

To put it differently, if Samba was improved in such a way that we could 
use the ad backend without having to manually manage the rfc2307 
attributes, wouldn't this be the best if not only solution we needed?

Viktor

On Sun, 24 Feb 2019 19:56:27 +0100
Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
> 
> On 24.02.2019 19:25, Ralph Böhme via samba wrote:
> > Am 24.02.2019 um 18:48 schrieb Rowland Penny via samba
> > <samba at lists.samba.org>:
> >> On Sun, 24 Feb 2019 18:28:43 +0100 Ralph Böhme <slow at
samba.org>
> >> wrote:
> >>> Am 24.02.2019 um 16:42 schrieb Rowland Penny via samba
> >>> <samba at lists.samba.org>:
> >>>> On Sun, 24 Feb 2019 15:58:39 +0100 Ralph Böhme <slow at
samba.org>
> >>>> wrote:
> >>>>> Another thing that a customer has just been bitten by,
was a
> >>>>> subtle bug in winbindd's idmap cache that resulted
in all
> >>>>> xid2sid requests going through the idmap backend, iow
winbindd
> >>>>> issued LDAP requests. With a few thousand users,
things came to
> >>>>> a grinding halt.
> >>>>>
> >>>>> https://bugzilla.samba.org/show_bug.cgi?id=13802
> >>>>>
> >>>>> Patch just landed upstream.
> >>>> That is the bug I was referring to and probably (amongst
all the
> >>>> other cruft) what was causing the OP's problem.
> >>> Unlikely.
> >> It is was I thought, but as the OP's setup is so convoluted,
it is
> >> hard to say.
> > I don't think it's convoluted, it's certainly beyond the
simple
> > standard setup we all wish everybody was using, but I don't think
> > it is broken as is. I just think an appropriate analysis requires
> > more resources then is available on the list.
> >
> >>>> However, this has nothing to
> >>>> do with using the 'ad' backend with Active
Directory. We keep
> >>>> dancing around this problem, saying things like 'we
need to fix
> >>>> this', we have been saying this since Samba 4 was
released.
> >>> Which problem? Fix what? Been saying what?
> >> There have been numerous discussions about the 'ad'
backend over
> >> the years and they have all gone nowhere. The 'ad' backend
still
> >> works in the same way as it did when Samba 4 was released and you
> >> still have to store the next uidNumber & gidNumber outside AD
if
> >> you use the Samba tools.
> > Looks like you're mixing AD DC use case with member server use
> > case. Can we please keep that seperate? Afaict, the one has nothing
> > to do with the other.
> 
> I'm confused.. how is the choice of the idmap backend related to an
> AD DC use case?
Only in the case of wanting the same ID everywhere.
> 
> 
> >>>> Windows Uses the SID-RID to identify the user and the
domain it
> >>>> comes from, surely we can find a way to do this for Samba,
we are
> >>>> half way there with the 'rid' backend.
> >>> I'm not really what "there" implies for you, but
it seems
> >>> idmap_autorid is eventually the backend that takes you
"there". :)
> >> No it doesn't, at the moment, the only way to get the same ID
on
> >> all Unix machines (this includes DC's) is to use the
'ad' backend.
> > Sure. But only certain use cases require the same id on all
> > machines, many don't. I'm just saying that you should better
not
> > use idmap_ad, but instead use eg idmap_autorid unless you're setup
> > requires idmap_ad.
> 
> Would you, or someone else mind sharing some of these use cases when 
> idmap_ad would be necessary and when idmap_autorid would suffice? 
My understanding is, like the 'rid' backend, if you use the same
smb.conf on all Unix domain members, you will get the same ID.
> Specifically, in which situations do I absolutely need the ID to be
> the same on each member, and in which cases could I actually go
> without this? For example, if my AD is managed by Samba only but I
> only have Windows users who will never have to log in to a unix box,
> are there still advantages of the ad backend over the (auto)rid one?
If your users never log into a Unix domain member or store files on
one, then do not even need to consider a winbind backend.
> 
> I assume that most readers of the wiki will, like me, find that
> "central administration of ID's inside the AD" and
"ID's not stored
> in a local database that can corrupt with lost file ownership" seem
> like really important arguments (btw, the last point is not stated as
> a disadvantage for the rid/autorid backend in the wiki). Reading
> this, it just seems that the ad backend is always the right one
> except that it's a headache to manage.
It isn't really a headache to manage, once you get your head around
it ;-)
> 
> To put it differently, if Samba was improved in such a way that we
> could use the ad backend without having to manually manage the
> rfc2307 attributes, wouldn't this be the best if not only solution we
> needed?
I think you are mistaking using a subset of the RFC2307 attributes with
using most of them. If you use any backend other than 'ad' (and I
include a DC here) all you get is a uidNumber for a user, or a
gidNumber for a group. Using the 'ad' backend gets an ID number that
you set, plus individual login shells, Unix home dirs etc

Rowland

On 24.02.2019 20:22, Rowland Penny via samba wrote:
> On Sun, 24 Feb 2019 19:56:27 +0100
> Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
>
>> On 24.02.2019 19:25, Ralph Böhme via samba wrote:
>>> Am 24.02.2019 um 18:48 schrieb Rowland Penny via samba
>>> <samba at lists.samba.org>:
>>>> On Sun, 24 Feb 2019 18:28:43 +0100 Ralph Böhme <slow at
samba.org>
>>>> wrote:
>>>>> Am 24.02.2019 um 16:42 schrieb Rowland Penny via samba
>>>>> <samba at lists.samba.org>:
>>>>>> On Sun, 24 Feb 2019 15:58:39 +0100 Ralph Böhme <slow
at samba.org>
>>>>>> wrote:
>>>>>>> Another thing that a customer has just been bitten
by, was a
>>>>>>> subtle bug in winbindd's idmap cache that
resulted in all
>>>>>>> xid2sid requests going through the idmap backend,
iow winbindd
>>>>>>> issued LDAP requests. With a few thousand users,
things came to
>>>>>>> a grinding halt.
>>>>>>>
>>>>>>> https://bugzilla.samba.org/show_bug.cgi?id=13802
>>>>>>>
>>>>>>> Patch just landed upstream.
>>>>>> That is the bug I was referring to and probably
(amongst all the
>>>>>> other cruft) what was causing the OP's problem.
>>>>> Unlikely.
>>>> It is was I thought, but as the OP's setup is so
convoluted, it is
>>>> hard to say.
>>> I don't think it's convoluted, it's certainly beyond
the simple
>>> standard setup we all wish everybody was using, but I don't
think
>>> it is broken as is. I just think an appropriate analysis requires
>>> more resources then is available on the list.
>>>
>>>>>> However, this has nothing to
>>>>>> do with using the 'ad' backend with Active
Directory. We keep
>>>>>> dancing around this problem, saying things like 'we
need to fix
>>>>>> this', we have been saying this since Samba 4 was
released.
>>>>> Which problem? Fix what? Been saying what?
>>>> There have been numerous discussions about the 'ad'
backend over
>>>> the years and they have all gone nowhere. The 'ad'
backend still
>>>> works in the same way as it did when Samba 4 was released and
you
>>>> still have to store the next uidNumber & gidNumber outside
AD if
>>>> you use the Samba tools.
>>> Looks like you're mixing AD DC use case with member server use
>>> case. Can we please keep that seperate? Afaict, the one has nothing
>>> to do with the other.
>> I'm confused.. how is the choice of the idmap backend related to an
>> AD DC use case?
> Only in the case of wanting the same ID everywhere.
In my understanding, the idmap backend only needs to be specified in the 
smb.conf for member servers. That's why I still don't see how it is 
related to a AD DC use case. I take it I'm missing something crucial
here.
>>
>>>>>> Windows Uses the SID-RID to identify the user and the
domain it
>>>>>> comes from, surely we can find a way to do this for
Samba, we are
>>>>>> half way there with the 'rid' backend.
>>>>> I'm not really what "there" implies for you,
but it seems
>>>>> idmap_autorid is eventually the backend that takes you
"there". :)
>>>> No it doesn't, at the moment, the only way to get the same
ID on
>>>> all Unix machines (this includes DC's) is to use the
'ad' backend.
>>> Sure. But only certain use cases require the same id on all
>>> machines, many don't. I'm just saying that you should
better not
>>> use idmap_ad, but instead use eg idmap_autorid unless you're
setup
>>> requires idmap_ad.
>> Would you, or someone else mind sharing some of these use cases when
>> idmap_ad would be necessary and when idmap_autorid would suffice?
> My understanding is, like the 'rid' backend, if you use the same
> smb.conf on all Unix domain members, you will get the same ID.
>
>> Specifically, in which situations do I absolutely need the ID to be
>> the same on each member, and in which cases could I actually go
>> without this? For example, if my AD is managed by Samba only but I
>> only have Windows users who will never have to log in to a unix box,
>> are there still advantages of the ad backend over the (auto)rid one?
> If your users never log into a Unix domain member or store files on
> one, then do not even need to consider a winbind backend.
In a Windows clients only environment, I thought that having AD member 
servers implied that files were stored on them.

But I take it from what you write that as soon as some kind of access is 
needed, we are actually speaking about a "login", even if it's
just on
the share level and the user never sees a shell.

In which case, yet again, the question comes up which back end to use. 
I'll get to that further below.
>> I assume that most readers of the wiki will, like me, find that
>> "central administration of ID's inside the AD" and
"ID's not stored
>> in a local database that can corrupt with lost file ownership"
seem
>> like really important arguments (btw, the last point is not stated as
>> a disadvantage for the rid/autorid backend in the wiki). Reading
>> this, it just seems that the ad backend is always the right one
>> except that it's a headache to manage.
> It isn't really a headache to manage, once you get your head around
> it ;-)
For very large environments, I guess you would just script it either 
with samba-tool or powershell. But having to do it manually and remember 
that for every user, computer, and group 1-2 attributes need to be set, 
and kept track of, does seem like a nuisance.

>> To put it differently, if Samba was improved in such a way that we
>> could use the ad backend without having to manually manage the
>> rfc2307 attributes, wouldn't this be the best if not only solution
we
>> needed?
> I think you are mistaking using a subset of the RFC2307 attributes with
> using most of them. If you use any backend other than 'ad' (and I
> include a DC here) all you get is a uidNumber for a user, or a
> gidNumber for a group. Using the 'ad' backend gets an ID number
that
> you set, plus individual login shells, Unix home dirs etc
>
I guess I was. Are you saying, then, that if I have no need for 
individual login shells and *nix home dirs (which in a pure Windows 
client environment I clearly have not), the ad back end isn't really 
offering any advantage for me?

What still confuses me here is how the wiki states that in case of 
rid/autorid information about file ownership is lost if the local tdb 
fails and cannot be restored. I assume this does not mean that the 
actual ownership information is stored in tdb (or in the AD, for that 
matter) but that the user ID used in the ACL needs to be retrieved from 
somewhere, and in case of the ad back end this ID is being retrieved 
from the AD directly, in case of the rid back end it's retrieved from 
the local store only. And yet you said that on member servers with 
identical smb.conf files (I assume you were talking about the global 
section only), IDs would always be the same. Naturally, two questions 
come up:

1) If one member server fails, wouldn't it be possible and sufficient to 
just regenerate the tdb using the same smb.conf? If this yields the same 
results every time, then why would I even worry about a corrupted tdb?

2) If IDs were the same on all member servers, wouldn't that also mean 
that I should be able to restore the corrupted tdb on one member server 
using a valid tdb from another member server? And how would I go about 
that in practice?

I hope this makes sense.

Viktor