samba - Feb 2019 - winbind causing huge timeouts/delays since 4.8

On Fri, 22 Feb 2019 16:40:46 +0100
Alexander Spannagel via samba <samba at lists.samba.org> wrote:
> Am 22.02.19 um 15:42 schrieb Rowland Penny via samba:
> > On Fri, 22 Feb 2019 15:35:53 +0100
> > Ralph Böhme via samba <samba at lists.samba.org> wrote:
> > 
> >> Hi,
> >>
> >> On Fri, Feb 22, 2019 at 01:59:15PM +0100, Alexander Spannagel via
> >> samba wrote:
> s.
> >>
> >> hm, can't reproduce:
> >>
> >> slow at titan:~/git/samba/scratch$ git describe
> >> samba-4.8.3
> >>
> >> slow at titan:~/git/samba/scratch$ sudo bin/net cache flush
> >>
> >> slow at titan:~/git/samba/scratch$ time bin/wbinfo -i foo
> >> failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND
> >> Could not get info for user foo
> >>
> >> real    0m0.025s
> >> user    0m0.004s
> >> sys     0m0.004s
> >>
> >> Can you share your full smb.conf?
> 
> Here is the extraction of the global section from our smb.conf:
> [root at centos7dev64 ~]# testparm --section-name=global 2>/dev/null
<
> /dev/null
> # Global parameters
> [global]
>          dedicated keytab file = /etc/krb5.keytab
>          disable spoolss = Yes
>          domain master = No
>          kerberos method = secrets and keytab
>          ldap connection timeout = 10
>          ldap timeout = 30
>          load printers = No
>          local master = No
>          log file = /var/log/samba/log.%m
>          max log size = 0
>          os level = 0
>          printcap name = /dev/null
>          realm = OPS.GLOBAL.AD
>          security = ADS
>          server signing = required
>          server string = FTP Samba Server
>          show add printer wizard = No
>          template shell = /bin/bash
>          username map = /etc/samba/user.map
>          winbind refresh tickets = Yes
>          winbind separator = +
>          workgroup = OPS
>          idmap config * : rangesize = 1000000
>          idmap config * : range = 1000000-19999999
>          idmap config * : backend = autorid
>          map acl inherit = Yes
>          printing = bsd
>          store dos attributes = Yes
>          vfs objects = acl_xattr full_audit recycle extd_audit>>
> 
> > 
> > You might also want to explain why you are using sssd's cache with
> > winbind.
> 
> We are running a mixed environment and use sssd for authentication 
> against our unix ldap directory on all our unix servers. On a group
> of servers we need to provide smb shares to windows clients/servers
> and dedicated uid/gid mapping for windows users and groups.
> 
> Our default setup in nsswitch.conf regarding passwd/shadow/groups
> looks like:
> passwd:     files sss
> shadow:     files sss
> group:      files sss
> 
> And on the servers running samba:
> passwd:     files sss winbind
> shadow:     files sss winbind
> group:      files sss winbind
> 
> As mentioned it worked till the update from samba 4.7 to 4.8. The
> sssd is used for ldap and not AD authentication.
> 
> Alex
> 
If you do have 'files sss winbind' in /etc/nsswitch.conf' and sssd
is running, then it is highly likely that even if winbind is running,
it will not be used. You also shouldn't use winbind on the shadow line
and you shouldn't run winbind and sssd together, sssd has its own
version of one of the winbind libs, and this will undoubtedly interfere
with the Samba one.

Finally, your smb.conf is borked for winbind.

Rowland

Am 22.02.19 um 17:02 schrieb Rowland Penny via samba:
> If you do have 'files sss winbind' in /etc/nsswitch.conf' and
sssd
> is running, then it is highly likely that even if winbind is running,
> it will not be used. You also shouldn't use winbind on the shadow line
> and you shouldn't run winbind and sssd together, sssd has its own
> version of one of the winbind libs, and this will undoubtedly interfere
> with the Samba one.
On our linux servers ldap should always be used before asking AD via 
samba/winbind (as stated in nsswitch.conf). The sssd and samba libs are 
separated (we are on unix not windows) so there shouldn't be any 
"randomly" usage and if one of them or both have some buggy code maybe
triggered by special config combinations - but really don't want to 
point to one or the other just want to solve an issue we hit in our 
environment.

Anyhow i did some more patch testing and found two more ways to solve 
our issues instead of adding the line "return false;":
1. replace the patched line with this one:
	fstrcpy(domain, namespace);
    so in our setup domain would be set to namespace which is is set to
2. don't use the patch added via Bug 13503 at all

So for me it looks the issue is caused somehwere later in code, when 
function parse_domain_user sets namespace to something different than 
domain - in our case namespace=lp_netbios_name()='HOSTNAME' and
domain=''.
If i would be asked, i would go with solution 1 (patch file attached) to 
keep fix for Bug 13503 and assume not breaking it - maybe Mr. Schneider 
could take a look if it would still fix the reported Bug.
> 
> Finally, your smb.conf is borked for winbind.
> 
We have different setup on some servers using ldap server as idmap 
backend instead using autorid, but those show same issue and so 
shouldn't be related at all.

Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: samba-4.8.9-fix_winbind_empty_domain_2.patch
Type: text/x-patch
Size: 473 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190222/a9853cc5/samba-4.8.9-fix_winbind_empty_domain_2.bin>

On Fri, 22 Feb 2019 22:40:38 +0100
Alexander Spannagel via samba <samba at lists.samba.org> wrote:
> Am 22.02.19 um 17:02 schrieb Rowland Penny via samba:
> > If you do have 'files sss winbind' in /etc/nsswitch.conf'
and sssd
> > is running, then it is highly likely that even if winbind is
> > running, it will not be used. You also shouldn't use winbind on
the
> > shadow line and you shouldn't run winbind and sssd together, sssd
> > has its own version of one of the winbind libs, and this will
> > undoubtedly interfere with the Samba one.
> On our linux servers ldap should always be used before asking AD via 
> samba/winbind (as stated in nsswitch.conf). The sssd and samba libs
> are separated (we are on unix not windows) so there shouldn't be any 
> "randomly" usage and if one of them or both have some buggy code
> maybe triggered by special config combinations - but really don't
> want to point to one or the other just want to solve an issue we hit
> in our environment.
> 
> Anyhow i did some more patch testing and found two more ways to solve 
> our issues instead of adding the line "return false;":
> 1. replace the patched line with this one:
> 	fstrcpy(domain, namespace);
>     so in our setup domain would be set to namespace which is is set
> to 2. don't use the patch added via Bug 13503 at all
> 
> So for me it looks the issue is caused somehwere later in code, when 
> function parse_domain_user sets namespace to something different than 
> domain - in our case namespace=lp_netbios_name()='HOSTNAME' and
> domain=''. If i would be asked, i would go with solution 1 (patch
> file attached) to keep fix for Bug 13503 and assume not breaking it -
> maybe Mr. Schneider could take a look if it would still fix the
> reported Bug.
> 
> > 
> > Finally, your smb.conf is borked for winbind.
> > 
> We have different setup on some servers using ldap server as idmap 
> backend instead using autorid, but those show same issue and so 
> shouldn't be related at all.
> 
> Alex
So, you are trying to use 4 different methods of authentication on the
same Samba server, Unix, sssd, winbind and ldap, and you expect this to
work ?

I repeat, from a Samba point of view, your smb.conf is borked, see here
for more info:

https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member

You do not need winbind and sssd on the same Samba server, they do the
same thing, pick one and delete the other.

Your borked smb.conf is trying to be a Unix domain member, you do not
use ldap in smb.conf

If your smb.conf is set up correctly, your active directory users will
become Unix users as well.

You can if you so wish, go to git-lab and creating a fork and make your
changes there, see here for more info:

https://wiki.samba.org/index.php/Using_Git_for_Samba_Development

Rowland

On Sat, 23 Feb 2019 00:00:32 +0100
Alexander Spannagel <aspannagel at gmx.de> wrote:
> Am 22.02.19 um 23:02 schrieb Rowland Penny via samba:
> > So, you are trying to use 4 different methods of authentication on
> > the same Samba server, Unix, sssd, winbind and ldap, and you expect
> > this to work ?
> > 
> No. we use max. 3 auth providers: (1. and 2. on all unix servers)
> 1. unix (local passwd)
>     for static OS/service accounts across all our env
> 2. sssd (with unix ldap servers as provider)
>     unix experienced user and application related service accounts
> 3. samba/winbind
>     for windows users/services needing access to a group of unix
> servers
> 
> All that worked fine in coexistence since years and just stopped
> working smoothly with update to samba-4.8 and can be fixed with
> provided patches that fixes patch from Bug#13503 from mid of 2018.
> Initial also provided config changes to fix the issues, but they are 
> only workarounds.
> 
> > I repeat, from a Samba point of view, your smb.conf is borked, see
> > here for more info:
> > 
> > https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member
> >
> Will take a look, but not certain what configuration options you want 
> point me too.
> 
> > You do not need winbind and sssd on the same Samba server, they do
> > the same thing, pick one and delete the other.
> They don't - as stated above we use sssd for query/caching entries
> from our ldap directory server and not Windows DomainConmtrollers -
> also this is possible, but makes more trouble and don't provide what
> samba's smb/windbind does.
> 
> > 
> > Your borked smb.conf is trying to be a Unix domain member, you do
> > not use ldap in smb.conf
> > 
> Samba authenticate and caching AD accounts is working as expected and 
> without issues.
> 
> > If your smb.conf is set up correctly, your active directory users
> > will become Unix users as well.
> > 
> Indeed it works just fine.
> 
> > You can if you so wish, go to git-lab and creating a fork and make
> > your changes there, see here for more info:
> > 
> > https://wiki.samba.org/index.php/Using_Git_for_Samba_Development
> > 
> > Rowland
> > 
> Thanks for point me there, will take a look.
> 
> Alex
If you have, as you have, 'files sss winbind' in the the passwd &
group
line in nsswitch.conf, means this:
First /etc/passwd or /etc/group is searched and if the user or group is
found, this info is returned.
Next sssd will be asked, 'do you know this user or group ?' if found,
the info is returned.
Finally winbind will be asked, 'do you know this user or group ?' if
found, the info is returned.

Lets take a user called 'fred', this user is in AD. The first search
will return nothing, so sssd is asked, this 'asks' AD and returns the
users info. Finally, wait that's it, we have the info, there is no need
to ask winbind for anything.

Rowland

> On 23 Feb 2019, at 09:33, Rowland Penny via samba <samba at
lists.samba.org> wrote:
> 
> If you have, as you have, 'files sss winbind' in the the passwd
& group
> line in nsswitch.conf, means this:
> First /etc/passwd or /etc/group is searched and if the user or group is
> found, this info is returned.
> Next sssd will be asked, 'do you know this user or group ?' if
found,
> the info is returned.
> Finally winbind will be asked, 'do you know this user or group ?'
if
> found, the info is returned.
> 
> Lets take a user called 'fred', this user is in AD. The first
search
> will return nothing, so sssd is asked, this 'asks' AD and returns
the
> users info. Finally, wait that's it, we have the info, there is no need
> to ask winbind for anything.
That is incorrect. Alexander stated:
> No. we use max. 3 auth providers: (1. and 2. on all unix servers)
> 1. unix (local passwd)
>   for static OS/service accounts across all our env
> 2. sssd (with unix ldap servers as provider)
>   unix experienced user and application related service accounts
> 3. samba/winbind
>   for windows users/services needing access to a group of unix servers
And:
> They don't - as stated above we use sssd for query/caching entries from
our ldap directory server and not Windows DomainConmtrollers - also this is
possible, but makes more trouble and don't provide what samba's
smb/windbind does.
He clearly writes (in multiple emails) that sssd is configured to use his unix
ldap servers and not AD.

Maybe three sources of user databases is not regular, but I fail to see why this
should be a problem (provided that usernames, uidNumbers and such are unique
across the databases).

-Remy

Hi Rowland,

On Friday, 22 February 2019 08:03:11 PST Rowland Penny via samba
wrote:
> You also shouldn't use winbind on the shadow line
Perhaps a stupid question, but my google-fu doesn't seem to be good enough
to find an answer myself: what is the exact expected mode of failure of
including 'winbind' for 'shadow' in nsswitch.conf?

I understand that Winbind doesn't implement 'shadow' database, and
I've also seen reference somewhere in the docs to 'wbinfo' tool
failing if that line is there, but I personally haven't observed any
(consistent) wbinfo failures. It also appears that 'authconfig' tool in
RHEL/CentOS puts 'winbind' in 'shadow' line in nsswitch.conf if
you pass --enablewinbind flag to it, is my understanding correct that that's
a bug?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190225/169a0ee2/signature.sig>

On Mon, 25 Feb 2019 16:14:53 -0800
Alexey A Nikitin <nikitin at amazon.com> wrote:
> Hi Rowland,
> 
> On Friday, 22 February 2019 08:03:11 PST Rowland Penny via samba
> wrote:
> > You also shouldn't use winbind on the shadow line
> 
> Perhaps a stupid question, but my google-fu doesn't seem to be good
> enough to find an answer myself: what is the exact expected mode of
> failure of including 'winbind' for 'shadow' in
nsswitch.conf?
> 
> I understand that Winbind doesn't implement 'shadow' database,
and
> I've also seen reference somewhere in the docs to 'wbinfo' tool
> failing if that line is there, but I personally haven't observed any
> (consistent) wbinfo failures. It also appears that 'authconfig'
tool
> in RHEL/CentOS puts 'winbind' in 'shadow' line in
nsswitch.conf if
> you pass --enablewinbind flag to it, is my understanding correct that
> that's a bug?
You don't need to add 'winbind' to the shadow line mainly because it
isn't needed and as you have said, there have been reports of strange
things happening in wbinfo if it is added.

Just because red-hat adds it, doesn't me it is required, they also add a
lot of default lines to smb.conf.

Rowland