samba - Feb 2019 - winbind causing huge timeouts/delays since 4.8

On 25.02.2019 10:20, Rowland Penny via samba wrote:
> On Mon, 25 Feb 2019 09:24:24 +0100
> Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
>
>
>
>>>> I'm confused.. how is the choice of the idmap backend
related to an
>>>> AD DC use case?
>>> Only in the case of wanting the same ID everywhere.
>> In my understanding, the idmap backend only needs to be specified in
>> the smb.conf for member servers. That's why I still don't see
how it
>> is related to a AD DC use case. I take it I'm missing something
>> crucial here.
> A Samba AD DC uses idmap.ldb by default, this means that you get
> xidNumbers in the '3000000' range, but if you use the 'ad'
backend on
> Unix domain members, these xidNumbers get overridden by the uidNumber's
> and gidNumber's set in AD. It also turns some groups from being both
> users and groups into just groups.
>
Can you just help me understand the relationship between these 
xidNumbers and the SID, if there is any?

Assuming just DCs and I create a new user (using ADUC), I understand 
this user will receive a xidNumber (I take it xid stands for both uid 
and gid?) that will automatically be set by the DC and is in the 3000000 
range. Where is this xidNumber stored, though? If it was stored in the 
AD, I assume member servers could just query it directly, right? But 
that doesn't seem to be the case since we still need to set these 
xidNumbers manually when using the ad back end, and when using the rid 
backend not the xidNumber but the rid is used to calculate the ID. Then 
what is the purpose of the xidNumber set by the DC? Honestly, I'm a bit 
lost here...

>>> If your users never log into a Unix domain member or store files on
>>> one, then do not even need to consider a winbind backend.
>> In a Windows clients only environment, I thought that having AD
>> member servers implied that files were stored on them.
>>
>> But I take it from what you write that as soon as some kind of access
>> is needed, we are actually speaking about a "login", even if
it's
>> just on the share level and the user never sees a shell.
>>
>> In which case, yet again, the question comes up which back end to
>> use. I'll get to that further below.
> Then I will answer it below ;-)
>
>>>> I assume that most readers of the wiki will, like me, find that
>>>> "central administration of ID's inside the AD"
and "ID's not stored
>>>> in a local database that can corrupt with lost file
ownership" seem
>>>> like really important arguments (btw, the last point is not
stated
>>>> as a disadvantage for the rid/autorid backend in the wiki).
Reading
>>>> this, it just seems that the ad backend is always the right one
>>>> except that it's a headache to manage.
>>> It isn't really a headache to manage, once you get your head
around
>>> it ;-)
>> For very large environments, I guess you would just script it either
>> with samba-tool or powershell. But having to do it manually and
>> remember that for every user, computer, and group 1-2 attributes need
>> to be set, and kept track of, does seem like a nuisance.
> Ah, that type of headache, what do I need to ad and what was the last
> uidNumber I used.
Well.. yes. What other headaches are there with the ad backend?
:-)
>>
>>>> To put it differently, if Samba was improved in such a way that
we
>>>> could use the ad backend without having to manually manage the
>>>> rfc2307 attributes, wouldn't this be the best if not only
solution
>>>> we needed?
>>> I think you are mistaking using a subset of the RFC2307 attributes
>>> with using most of them. If you use any backend other than
>>> 'ad' (and I include a DC here) all you get is a uidNumber
for a
>>> user, or a gidNumber for a group. Using the 'ad' backend
gets an ID
>>> number that you set, plus individual login shells, Unix home dirs
>>> etc
>>>
>> I guess I was. Are you saying, then, that if I have no need for
>> individual login shells and *nix home dirs (which in a pure Windows
>> client environment I clearly have not), the ad back end isn't
really
>> offering any advantage for me?
> If your users do not need to actually log into a Unix machine, then
> there is no need to set a login shell, they will get the default
> 'bin/false'. If your users do not store anything in a Unix homedir
> (only really needed if they log in) then you don't need to set the
> template homedir, though there is the default /home/DOMAIN/user.
>
>> What still confuses me here is how the wiki states that in case of
>> rid/autorid information about file ownership is lost if the local tdb
>> fails and cannot be restored.
> Where does it say that ?
It's implied here: https://wiki.samba.org/index.php/Idmap_config_ad

Advantages: "IDs are not stored in a local database that can corrupt and 
thus file ownerships are not lost.*"*

Either I'm misunderstanding this, or this statement is really giving the 
wrong picture.
**
>> I assume this does not mean that the
>> actual ownership information is stored in tdb (or in the AD, for that
>> matter) but that the user ID used in the ACL needs to be retrieved
>> from somewhere, and in case of the ad back end this ID is being
>> retrieved from the AD directly, in case of the rid back end it's
>> retrieved from the local store only. And yet you said that on member
>> servers with identical smb.conf files (I assume you were talking
>> about the global section only), IDs would always be the same.
> The 'DOMAIN' ID's on Unix domain members are always stored in
AD, just
> not always in a way you expect ;-)
>
> For the 'ad' backend, the uidNumber & gidNumber attributes are
used,
> most others calculate the ID from the 'RID'. This means that if
> something goes wrong and AD is okay, recreating the Unix domain member
> should be enough. If the '*' tdb gets corrupt, you can delete it
and it
> will get recreated and whilst I cannot totally guarantee the ID's will
> be the same, it doesn't really matter because the users and groups in
> there are from the Well Known SIDs and are not used by the normal
> users.
>   
Let me see if I understand this correctly. No matter if the ad or the 
rid backend is used, the member server queries the AD to check which ID 
belongs to which user and sets ACL accordingly. In the case of ad, the 
uidNumber is used, in case of rid the ID is calculated in a transparent 
way. I assume this means that each time someone access a member server 
(for example to access a file), the member server will query the DC to 
verify this user's ID. What happens if the DC cannot be reached, will a 
local cache be used or will the request remain pending until a DC answers?

>> Naturally, two questions come up:
>>
>> 1) If one member server fails, wouldn't it be possible and
sufficient
>> to just regenerate the tdb using the same smb.conf? If this yields
>> the same results every time, then why would I even worry about a
>> corrupted tdb?
> Yes and, on a Unix domain member, you don't have to worry
Why did you mention the *nix domain member specifically? In which other 
cases would I have to worry?

Questions over questions... thanks for bearing with me! :-)

Viktor

On Mon, 25 Feb 2019 11:19:33 +0100
Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
> 
> On 25.02.2019 10:20, Rowland Penny via samba wrote:
> > On Mon, 25 Feb 2019 09:24:24 +0100
> > Viktor Trojanovic via samba <samba at lists.samba.org> wrote:
> >
> >
> >
> >>>> I'm confused.. how is the choice of the idmap backend
related to
> >>>> an AD DC use case?
> >>> Only in the case of wanting the same ID everywhere.
> >> In my understanding, the idmap backend only needs to be specified
> >> in the smb.conf for member servers. That's why I still
don't see
> >> how it is related to a AD DC use case. I take it I'm missing
> >> something crucial here.
> > A Samba AD DC uses idmap.ldb by default, this means that you get
> > xidNumbers in the '3000000' range, but if you use the
'ad' backend
> > on Unix domain members, these xidNumbers get overridden by the
> > uidNumber's and gidNumber's set in AD. It also turns some
groups
> > from being both users and groups into just groups.
> >
> Can you just help me understand the relationship between these 
> xidNumbers and the SID, if there is any?
I will try and there is a definite relationship between xidNumbers and
SID-RIDs
> 
> Assuming just DCs and I create a new user (using ADUC), I understand 
> this user will receive a xidNumber 
Yes
> (I take it xid stands for both uid and gid?)
No, I think it was chosen to differentiate them from uidNumber &
gidNumber attributes, they are similar but not the same. They also only
exist on DC's
> that will automatically be set by the DC and is in the 3000000 range.
They are set on a 'first come' basis.
> Where is this xidNumber stored, though?
They are stored in idmap.ldb, which on Debian based distros is usually
in /var/lib/samba/private/
> If it was stored in the AD, I assume member servers could just query
> it directly, right?
No, they are only used on DC's
> But that doesn't seem to be the case since we still need to set these
> xidNumbers manually when using the ad back end,
You cannot set xidNumbers manually, you set uidNumber and/or gidNumber
attributes.
> and when using the rid backend not the xidNumber but the rid is used
> to calculate the ID. 
This is correct on a Unix domain member.
>Then what is the purpose of the xidNumber set by the DC? Honestly, I'm
>a bit lost here...
Have you ever heard of 'Sysvol' ?

Windows has the interesting fact that groups can 'own' things, Unix
doesn't and domain groups need to 'own' things in 'Sysvol'.
To explain how this works, lets look at a fragment of idmap.ldb:

dn: CN=S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-500
cn: S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-500
objectClass: sidMap
objectSid: S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-500
type: ID_TYPE_UID
xidNumber: 0
distinguishedName: CN=S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-500

dn: CN=S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-512 
cn: S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-512 
objectClass: sidMap
objectSid: S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-512 
type: ID_TYPE_BOTH
xidNumber: 3000010
distinguishedName: CN=S-1-5-21-xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz-512 

xxxxxxxxxx-yyyyyyyyyy-zzzzzzzzzz replaces the domain SID
500 is the RID for Administrator
512 is the RID for Domain Admins

The first time a user or group contacts the DC, its SID-RID is mapped
to the next available xidNumber. there are a few exceptions to this,
Administrator being one of them, as you can see 'RID' 500
(Administrator) is mapped to the ID '0' (Unix root), it is also set to
be 'ID_TYPE_UID'.
Most others get the next xidNumber and 'ID_TYPE_BOTH', this means the
Unix OS identifies them by the xidNumber and thinks they are both a
user and a group. 
> > Ah, that type of headache, what do I need to ad and what was the
> > last uidNumber I used.
> Well.. yes. What other headaches are there with the ad backend? :-)
'need to ad' should have been 'need to add', that is the major
headache
if using the 'ad' backend, just remembering what options to add when
creating users and what values to use.
> >
> >> What still confuses me here is how the wiki states that in case of
> >> rid/autorid information about file ownership is lost if the local
> >> tdb fails and cannot be restored.
> > Where does it say that ?
> 
> It's implied here: https://wiki.samba.org/index.php/Idmap_config_ad
> 
> Advantages: "IDs are not stored in a local database that can corrupt
> and thus file ownerships are not lost.*"*
> 
> Either I'm misunderstanding this, or this statement is really giving
> the wrong picture.
Fair comment, I think what was meant was, the user & group data is
stored in AD and is easily retrievable in case of local failure.
 
> Let me see if I understand this correctly. No matter if the ad or the 
> rid backend is used, the member server queries the AD to check which
> ID belongs to which user and sets ACL accordingly. In the case of ad,
> the uidNumber is used, in case of rid the ID is calculated in a
> transparent way. I assume this means that each time someone access a
> member server (for example to access a file), the member server will
> query the DC to verify this user's ID. What happens if the DC cannot
> be reached, will a local cache be used or will the request remain
> pending until a DC answers?
Winbind caches ID's locally and refreshes the cache periodically and if
'winbind offline logon = yes' is set in a Unix domain members smb.conf,
they will allow login even if the DC cannot be contacted.
> Why did you mention the *nix domain member specifically? In which
> other cases would I have to worry?
As far as Samba and AD is concerned, there are only three posibilties,
DC's, Windows domain members and Unix domain members. Windows domain
members don't have 'tdb' files. If a DC goes down, you have bigger
worries. This just leaves Unix domain members and as long as you use
the same smb.conf as before you can easily recover from a Samba crash.
> 
> Questions over questions... thanks for bearing with me! :-)
Asking questions is the only way to get answers ;-)

Rowland

On 2019-02-25 at 11:32 +0000 Rowland Penny via samba sent
off:
> > (I take it xid stands for both uid and gid?)
> 
> No, I think it was chosen to differentiate them from uidNumber &
> gidNumber attributes, they are similar but not the same. They also only
> exist on DC's
in Windows the owner of a file can be a group.

In the unix world the main owner is always a user.

To reflect the fact that the owner can be a group also, winbind
can assign both a mapped uid number and a gid number for Windows
users and groups, both uid and gid have the same value and are the xid. That
way Samba can also assign the ownership of files to a group. The idmap backend
has to be able to support XID though, not all idmap backends do so.

Björn
-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: 0551-370000-0, mail: kontakt at sernet.de
Gesch.F.: Dr. Johannes Loxen & Reinhild Jung
AG Göttingen: HR-B 2816 - https://www.sernet.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190226/a29b53a1/signature.sig>