samba - Apr 2020 - Prevent `wbinfo -u` from making Winbind unresponsive

Hi,

Recently I by mistake ran `wbinfo -u <username>` when I was actually
intending to run `wbinfo -n <username>`. It ignored the <username>
part and proceeded to fetch the usernames. On a small domain this shouldn't
be too much of an issue, but I did it on a domain with thousands upon thousands
of users. The result was that Winbind became for all intents and purposes
unresponsive for about six minutes - I couldn't authenticate me (or anyone
else) for any new sessions, and it wouldn't even acknowledge me as a valid
user in an existing session ('unknown uid: 3234505'). It pretty much
blocked on that user search request for anything else, even things that were
supposed to be cached locally like my UID.

I do have the following lines in smb.conf:

winbind enum users = no
winbind enum groups = no

Is there a way, preferrably without ugly hacks, to prevent this from happening
on accident, by mistake? By this I mean ideally so that Winbind remains
responsive even if someone mistakenly ran `wbinfo -u` or `wbinfo -g`, but
limiting the result sets of these commands or blocking them altogether is
acceptable too.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.samba.org/pipermail/samba/attachments/20200401/94f80d40/signature.sig>

On Wed, Apr 01, 2020 at 02:09:57PM -0700, Alexey A Nikitin via samba
wrote:
> Hi,
> 
> Recently I by mistake ran `wbinfo -u <username>` when I was actually
intending to run `wbinfo -n <username>`. It ignored the <username>
part and proceeded to fetch the usernames. On a small domain this shouldn't
be too much of an issue, but I did it on a domain with thousands upon thousands
of users. The result was that Winbind became for all intents and purposes
unresponsive for about six minutes - I couldn't authenticate me (or anyone
else) for any new sessions, and it wouldn't even acknowledge me as a valid
user in an existing session ('unknown uid: 3234505'). It pretty much
blocked on that user search request for anything else, even things that were
supposed to be cached locally like my UID.
> 
> I do have the following lines in smb.conf:
> 
> winbind enum users = no
> winbind enum groups = no
Ah, the winbindd code only prohibits
enumerating users when requested from
nsswitch lookups.

The code looks like:

        if (request->wb_flags & WBFLAG_FROM_NSS &&
!lp_winbind_enum_users()) {
                tevent_req_done(req);
                return tevent_req_post(req, ev);
        }

so making an explicit request via wbinfo will
still do the enumeration.

On Wed, Apr 01, 2020 at 03:33:00PM -0700, Jeremy Allison via samba
wrote:
> On Wed, Apr 01, 2020 at 02:09:57PM -0700, Alexey A Nikitin via samba wrote:
> > Hi,
> > 
> > Recently I by mistake ran `wbinfo -u <username>` when I was
actually intending to run `wbinfo -n <username>`. It ignored the
<username> part and proceeded to fetch the usernames. On a small domain
this shouldn't be too much of an issue, but I did it on a domain with
thousands upon thousands of users. The result was that Winbind became for all
intents and purposes unresponsive for about six minutes - I couldn't
authenticate me (or anyone else) for any new sessions, and it wouldn't even
acknowledge me as a valid user in an existing session ('unknown uid:
3234505'). It pretty much blocked on that user search request for anything
else, even things that were supposed to be cached locally like my UID.
> > 
> > I do have the following lines in smb.conf:
> > 
> > winbind enum users = no
> > winbind enum groups = no
> 
> Ah, the winbindd code only prohibits
> enumerating users when requested from
> nsswitch lookups.
> 
> The code looks like:
> 
>         if (request->wb_flags & WBFLAG_FROM_NSS &&
!lp_winbind_enum_users()) {
>                 tevent_req_done(req);
>                 return tevent_req_post(req, ev);
>         }
> 
> so making an explicit request via wbinfo will
> still do the enumeration.
The rpc client code uses the dcerpc call_id
field to allow multiple outstanding calls at
once (asynchronously using tevent). It'd be
interesting to know where exactly winbind
is blocking (I think it might be on queuing
calls between master and client) to see
how we can improve the asynchronous performance.

If you're willing to reproduce and investigate,
that is !

Am 4/1/20 um 11:09 PM schrieb Alexey A Nikitin via
samba:
> Is there a way, preferrably without ugly hacks, to prevent this from
happening on accident, by mistake? By this I mean ideally so that Winbind
remains responsive even if someone mistakenly ran `wbinfo -u` or `wbinfo -g`,
but limiting the result sets of these commands or blocking them altogether is
acceptable too.
well, blocking it altogether by means of a new smb.conf option (maybe
wbinfo enum users|groups ?) would be trivial.

It would be interesting to know whether you see the issue with settings
of winbind max domain connections higher then the default of 1. If so,
does increasing it to some sane value eg 10 help?

-slow

-- 
Ralph Boehme, Samba Team                https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/
GPG-Fingerprint   FAE2C6088A24252051C559E4AA1E9B7126399E46

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20200403/01dcfdcd/signature.sig>

On Friday, 3 April 2020 10:46:54 PDT Ralph Boehme wrote:
> Am 4/1/20 um 11:09 PM schrieb Alexey A Nikitin via samba:
> > Is there a way, preferrably without ugly hacks, to prevent this from
happening on accident, by mistake? By this I mean ideally so that Winbind
remains responsive even if someone mistakenly ran `wbinfo -u` or `wbinfo -g`,
but limiting the result sets of these commands or blocking them altogether is
acceptable too.
> 
> well, blocking it altogether by means of a new smb.conf option (maybe
> wbinfo enum users|groups ?) would be trivial.
> 
> It would be interesting to know whether you see the issue with settings
> of winbind max domain connections higher then the default of 1. If so,
> does increasing it to some sane value eg 10 help?
> 
> -slow
> 
> 
Well, looks like setting 'winbindf max domain connections' to a value
above 1 makes 'wbinfo -u' no longer a threat, but it is pretty much
ignored if 'winbind offline logon' is enabled... Can anyone explain why?
Because when auth can be broken so easily--just run 'wbinfo -u', for
which you don't even need elevated privileges--despite offline logon
enabled, that makes one wonder what is even the point of having that option.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.samba.org/pipermail/samba/attachments/20200403/7cc282a7/signature.sig>

On Fri, Apr 03, 2020 at 07:46:54PM +0200, Ralph Boehme via samba
wrote:
> Am 4/1/20 um 11:09 PM schrieb Alexey A Nikitin via samba:
> > Is there a way, preferrably without ugly hacks, to prevent this from
happening on accident, by mistake? By this I mean ideally so that Winbind
remains responsive even if someone mistakenly ran `wbinfo -u` or `wbinfo -g`,
but limiting the result sets of these commands or blocking them altogether is
acceptable too.
> 
> well, blocking it altogether by means of a new smb.conf option (maybe
> wbinfo enum users|groups ?) would be trivial.
> 
> It would be interesting to know whether you see the issue with settings
> of winbind max domain connections higher then the default of 1. If so,
> does increasing it to some sane value eg 10 help?
This came up in different contexts in the past. One question is whether
it is necessary to have "wbinfo -u" and "wbinfo -g" to go
through
winbindd at all. "net ads search -P objectClass=user" does a similar
query and avoids congesting winbindd with those huge queries.

What do you think of deprecating the -u and -g options and possible
provide a wrapper in "net" as an alternative?

Christof