samba - Jan 2019 - Users created in last few years cannot login after 4.7 -> 4.8 + winbind

TLDR: after upgrading our CentOS 7.5 servers using Samba 4.7.x with
security = ads and no winbind to CentOS 7.6 with Samba 4.8.x with security = 
ads + winbind all users accounts created in the last few years can no longer 
login.

Explaining this requires a fairly long back story

Our corporate is primarily a Windows shop while our own research department
primarly uses Linux.  For over a decade we used our own account/group/file
namespace in our Linux infrastructure totally separate from corporate.

A couple years ago for new security hardening purposes corporate has
dictated all logins need to be based off their AD server so they
can manage/monitor/enforce password changes, access, etc.

The issue was we had petabytes of data using our accounts which had in most 
cases both different names and underlying user ids. For example, my Linux 
username is raines with ID 5829 and my corparte/AD username is per2 with ID 
2040470.  And groups have no relation whatsoever. Simply reconfiguring our 
Linux servers to do straight LDAP or winbind/nss to corporate AD was not 
possible without a wholesale painful re-ID-ing of files and breakage of
lots of apps that hard code usernames in settings.

For all non-Samba resources (login, web, LDAP-based apps, ...) I could solve 
this issue using LDAP SASL passthru.  In this scheme you set the user LDAP 
record the userPassword field to be something like

userPassword:: {SASL}per2

and any authentication to the LDAP server for user 'raines' is passed
through to the AD server as authentication for user 'per2'.

The issue was this did not work for Samba.  The solution I came up with
was to create a "username map = /etc/samba/users.map" with lines like

raines = MYDOMAIN\per2
aea32 = MYDOMAIN\aea32

and then have in smb.conf

 	workgroup = MYDOMAIN
         security = ads
         passdb backend = tdbsam
         realm = MYDOMAIN.ORG

         dedicated keytab file = /etc/krb5.keytab
         kerberos method = secrets and keytab
         preferred master = no
         encrypt passwords = yes

         socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 
SO_SNDBUF=65536 SO_KEEPALIVE

         idmap config *:backend = tdb
         idmap config *:range = 100-999999
         idmap config MYDOMAIN:backend = ad
         idmap config MYDOMAIN:schema_mode = rfc2307
         idmap config MYDOMAIN:range = 1000000-9999999

         username map = /etc/samba/users.map
         username map cache time = 60

         winbind nss info = rfc2307
         winbind trusted domains only = no
         winbind use default domain = yes
         winbind enum users = yes
         winbind enum groups = yes
         winbind nested groups = yes
         winbind refresh tickets = yes
         allow trusted domains = yes

         server signing = auto
         client signing = auto
         client ntlmv2 auth = yes
         ntlm auth = yes
         lanman auth = no
         max protocol = SMB2

         map acl inherit = yes
         nt acl support = yes
         map archive = no
         create mode = 0770
         directory mode = 0770

[posters]
     comment = poster volume
     path = /home/posters
     valid users = +webgp
     public = no
     writable = yes
     printable = no
     create mask = 664
     force create mode = 664
     directory mask = 2775
     force directory mode = 2775

but I would NOT run winbind.  When I did run winbind I found the behavior
I find now with 4.8.x + winbind, that many accounts could not login. Since
I did not need winbind and it worked without I just didn't run it and
didn't investigate further.

Also worth noting since this dictated policy all new accounts we create on
the Linux side have matching usernames and underlying UID to AD.  Groups
are still totally independent.

Now with the 4.8.x upgrade in CentOS 7.6 I have to run winbind.

(Actually curiously some users still COULD login with 4.8.x and no
  winbind running.  I have no idea how but it worked but it was a
  very small subset of very old users)

And now with 4.8.x and winbind running I have the situation where 
recently created users are unable to login.  Curiously it does not
coincide exactly with the username/UID matching change over.  Users
created in approximately the first year of that change over CAN login.
>From the log files for a user where things work I will see lines like
   check_ntlm_password:  authentication for user [per2] -> [per2] ->
[raines]
succeeded
   pdb_getsampwnam (TDB): error fetching database.
    Key: USER_raines
   Adding homes service for user 'raines' using home directory: 
'/homes/11/raines'


For a user where things do NOT work I see

   check_ntlm_password:  authentication for user [aea32] -> [aea32] ->
[aea32]
succeeded
   sid_to_gid(S-1-5-21-8915387-943144406-1916815836-513) failed
   Failed to generate session_info (user and group token) for session setup: 
NT_STATUS_NO_SUCH_USER
   NT error packet at ../source3/smbd/sesssetup.c(263) cmd=115 (SMBsesssetupX) 
NT_STATUS_LOGON_FAILURE

If I run 'strings -a /var/lib/samba/group_mapping.tdb' I see

TDB file
&MEMBEROF/S-1-5-21-8915387-943144406-1916815836-513
S-1-5-32-545
&UNIXGROUP/S-1-5-32-545
Users
&MEMBEROF/S-1-5-21-8915387-943144406-1916815836-512
S-1-5-32-544
&UNIXGROUP/S-1-5-32-544
Administrators
TDB file
&UNIXGROUP/S-1-5-32-545
Users
&MEMBEROF/S-1-5-21-8915387-943144406-1916815836-512
S-1-5-32-544
&UNIXGROUP/S-1-5-32-544
Administrators

Also I see

# wbinfo --domain=MYDOMAIN
--sid-to-name=S-1-5-21-8915387-943144406-1916815836-513
MYDOMAIN\Domain Users 2

When I downgrade back to 4.7.x with NO winbind everything works again
for everybody as it did before.

I have tried rejoining the domain.  I have tried wiping out everything
in /var/lib/samba.

Any clues as to what is going on?

To make things more confusing, there is one 4.8.x + winbind server
where new users CAN login fine.   I have tried recreating its setup
exactly on a different server but it behaves like the others denying all
accounts created in last few years.

P.S.

It might be related to this thread but maybe not

https://lists.samba.org/archive/samba/2018-November/219719.html

P.S.S.

Thought I had a solution

Changing my 'idmap config' stanza to simply be the two lines

         idmap config *:backend = tdb
         idmap config *:range = 100-9999999

covering the entire range.  This worked on one server for one new user
who could not login before.  But it did not work for that same user on
another server.  And when a second user who previously was unable
to login tried it failed on both servers.

The error message changes slightly in the log:

   check_ntlm_password:  authentication for user [aea32] -> [aea32] ->
[aea32]
succeeded
[2019/01/03 15:55:56.770349,  1] 
../source3/auth/token_util.c:561(add_local_groups)
   SID S-1-5-21-8915387-943144406-1916815836-1333656 -> getpwuid(100) failed
[2019/01/03 15:55:56.770401,  1] 
../source3/smbd/sesssetup.c:260(reply_sesssetup_and_X_spnego)
   Failed to generate session_info (user and group token) for session setup: 
NT_STATUS_NO_MEMORY
[2019/01/03 15:55:56.770488,  3] ../source3/smbd/error.c:82(error_packet_set)
   NT error packet at ../source3/smbd/sesssetup.c(263) cmd=115 (SMBsesssetupX) 
NT_STATUS_NO_MEMORY

I tried 'idmap config *:backend = nss' and randomly it worked on one
server
but failed on another with

   check_ntlm_password:  authentication for user [nmr27] -> [nmr27] ->
[nmr27]
succeeded
[2019/01/03 17:28:28.480615,  1] 
../source3/auth/token_util.c:561(add_local_groups)
   SID S-1-5-21-8915387-943144406-1916815836-949594 -> getpwuid(1000000)
failed
[2019/01/03 17:28:28.480673,  3] 
../source3/auth/token_util.c:400(create_local_nt_token_from_info3)
   Failed to add local groups
[2019/01/03 17:28:28.480738,  3] 
../source3/smbd/smb2_server.c:3171(smbd_smb2_request_error_ex)
   smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] 
status[NT_STATUS_UNSUCCESSFUL] || at ../source3/smbd/smb2_sesssetup.c:137

A lot of this just seems to be totally random.

On Thu, 3 Jan 2019 17:46:51 -0500 (EST)
Paul Raines via samba <samba at lists.samba.org> wrote:
> TLDR: after upgrading our CentOS 7.5 servers using Samba 4.7.x with
> security = ads and no winbind to CentOS 7.6 with Samba 4.8.x with
> security = ads + winbind all users accounts created in the last few
> years can no longer login.
> 
> Explaining this requires a fairly long back story
> 
> Our corporate is primarily a Windows shop while our own research
> department primarly uses Linux.  For over a decade we used our own
> account/group/file namespace in our Linux infrastructure totally
> separate from corporate.
> 
> A couple years ago for new security hardening purposes corporate has
> dictated all logins need to be based off their AD server so they
> can manage/monitor/enforce password changes, access, etc.
So go back to them, point out it isn't working and they need to
extend AD by adding the IDMU ldif and create your users and groups in
AD with the correct ID's ;-)
> 
> The issue was we had petabytes of data using our accounts which had
> in most cases both different names and underlying user ids. For
> example, my Linux username is raines with ID 5829 
I take it this ID is the Unix users ID found in /etc/passwd
>and my corparte/AD username is per2 with ID 2040470.
And this is the users AD RID
>  And groups have no relation
> whatsoever. Simply reconfiguring our Linux servers to do straight
> LDAP or winbind/nss to corporate AD was not possible without a
> wholesale painful re-ID-ing of files and breakage of lots of apps
> that hard code usernames in settings.
It wouldn't have been a problem if AD had been extended properly.
> 
> For all non-Samba resources (login, web, LDAP-based apps, ...) I
> could solve this issue using LDAP SASL passthru.  In this scheme you
> set the user LDAP record the userPassword field to be something like
> 
> userPassword:: {SASL}per2
> 
> and any authentication to the LDAP server for user 'raines' is
passed
> through to the AD server as authentication for user 'per2'.
> 
> The issue was this did not work for Samba.  The solution I came up
> with was to create a "username map = /etc/samba/users.map" with
lines
> like
> 
> raines = MYDOMAIN\per2
> aea32 = MYDOMAIN\aea32
That's one way of doing it, it isn't how I would have done it.
> 
> and then have in smb.conf
> 
>  	workgroup = MYDOMAIN
>          security = ads
>          passdb backend = tdbsam
>          realm = MYDOMAIN.ORG
> 
>          dedicated keytab file = /etc/krb5.keytab
>          kerberos method = secrets and keytab
>          preferred master = no
>          encrypt passwords = yes
> 
>          socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 
> SO_SNDBUF=65536 SO_KEEPALIVE
> 
>          idmap config *:backend = tdb
>          idmap config *:range = 100-999999
That is a foolish range, it interferes with the local system users.
The '*' domain is for the Well Known Sids and anything outside the
'MYDOMAIN' domain.
>          idmap config MYDOMAIN:backend = ad
>          idmap config MYDOMAIN:schema_mode = rfc2307
>          idmap config MYDOMAIN:range = 1000000-9999999
Sorry, but this is wrong for your Samba version, see here:

https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member

and here:

https://wiki.samba.org/index.php/Idmap_config_ad

The range must reflect the uidNumber & gidNumber attributes in AD.

Rowland

On Fri, 4 Jan 2019 4:57am, Rowland Penny wrote:
> On Thu, 3 Jan 2019 17:46:51 -0500 (EST)
> Paul Raines via samba <samba at lists.samba.org> wrote:
>
>> TLDR: after upgrading our CentOS 7.5 servers using Samba 4.7.x with
>> security = ads and no winbind to CentOS 7.6 with Samba 4.8.x with
>> security = ads + winbind all users accounts created in the last few
>> years can no longer login.
>>
>> Explaining this requires a fairly long back story
>>
>> Our corporate is primarily a Windows shop while our own research
>> department primarly uses Linux.  For over a decade we used our own
>> account/group/file namespace in our Linux infrastructure totally
>> separate from corporate.
>>
>> A couple years ago for new security hardening purposes corporate has
>> dictated all logins need to be based off their AD server so they
>> can manage/monitor/enforce password changes, access, etc.
>
> So go back to them, point out it isn't working and they need to
> extend AD by adding the IDMU ldif and create your users and groups in
> AD with the correct ID's ;-)
This is not an option for me.  Corporate is not going to change anything
on AD for me.  And this would create the opposite problem were my users
with resources on the corporate instrastructure will have things break
if the IDs are changed.
>
>>
>> The issue was we had petabytes of data using our accounts which had
>> in most cases both different names and underlying user ids. For
>> example, my Linux username is raines with ID 5829
>
> I take it this ID is the Unix users ID found in /etc/passwd
>
My LDAP server, but yes
>> and my corparte/AD username is per2 with ID 2040470.
>
> And this is the users AD RID
uidNumber you get when searching account on AD via LDAP
>
>>  And groups have no relation
>> whatsoever. Simply reconfiguring our Linux servers to do straight
>> LDAP or winbind/nss to corporate AD was not possible without a
>> wholesale painful re-ID-ing of files and breakage of lots of apps
>> that hard code usernames in settings.
>
> It wouldn't have been a problem if AD had been extended properly.
>
I really don't see how that was realistically possible.  ANd I am positive
the
managers would not allow the UIDs in the range 5000-10000 I had use for my 
users.  I have a handful of users from the early 2000s with UIDs in the 
100-5000 range (predates me -- this is before distros started enforcing user 
creation >500). It most likely conflicts with system accounts they had 
created.
>>
>> For all non-Samba resources (login, web, LDAP-based apps, ...) I
>> could solve this issue using LDAP SASL passthru.  In this scheme you
>> set the user LDAP record the userPassword field to be something like
>>
>> userPassword:: {SASL}per2
>>
>> and any authentication to the LDAP server for user 'raines' is
passed
>> through to the AD server as authentication for user 'per2'.
>>
>> The issue was this did not work for Samba.  The solution I came up
>> with was to create a "username map = /etc/samba/users.map"
with lines
>> like
>>
>> raines = MYDOMAIN\per2
>> aea32 = MYDOMAIN\aea32
>
> That's one way of doing it, it isn't how I would have done it.
>
Except for "extending AD" as above is there another way.

All I want and need from corporate AD is password authentication.  I
don't want or need anything else, including user or group ids. I
want Samba to get everything else (user id, group ids, shell, homedir, ...)
from the local NIS backend (i.e. my LDAP server)
>>
>> and then have in smb.conf
>>
>>  	workgroup = MYDOMAIN
>>          security = ads
>>          passdb backend = tdbsam
>>          realm = MYDOMAIN.ORG
>>
>>          dedicated keytab file = /etc/krb5.keytab
>>          kerberos method = secrets and keytab
>>          preferred master = no
>>          encrypt passwords = yes
>>
>>          socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536
>> SO_SNDBUF=65536 SO_KEEPALIVE
>>
>>          idmap config *:backend = tdb
>>          idmap config *:range = 100-999999
>
> That is a foolish range, it interferes with the local system users.
> The '*' domain is for the Well Known Sids and anything outside the
> 'MYDOMAIN' domain.
Again, I am not using WINBIND for actual NIS in the system.  It
will not be in /etc/nsswitch.conf, pam, etc.  I never used it before
in Samba 4.7.x or before.  I am only running it because it is now
required to use security=ads for authentication.  I DO want any
login on the samba server of any system account to always fail.
>
>>          idmap config MYDOMAIN:backend = ad
>>          idmap config MYDOMAIN:schema_mode = rfc2307
>>          idmap config MYDOMAIN:range = 1000000-9999999
>
> Sorry, but this is wrong for your Samba version, see here:
>
> https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member
>
> and here:
>
> https://wiki.samba.org/index.php/Idmap_config_ad
>
> The range must reflect the uidNumber & gidNumber attributes in AD.
>
I have tried using the nss backend and the results are random.
Sometimes it works.  Sometimes it doesn't.  It still always works
for my "old" account.  Just not for new accounts.

Do you have any configuration suggestion?

Is there a way to have winbind not fail a login just because it
cannot map a SID to GID? Is there a way for me to force
a mapping for S-1-5-21-8915387-943144406-1916815836-513?
I tried wbinfo --set-gid-mapping but just got WBC_ERR_NOT_IMPLEMENTED

I guess for now I will continue to run Samba 4.7.x as long as I can still get
it to run