samba - May 2019 - Debugging Samba is a total PITA and this needs to improve

The smb.conf hasn't changed since the last three or four times I've
posted here asking for help:

https://up.tao.at/u/samba/villach-file.txt

Top level error I'm seeing is that since today *some* Windows users are
denied SMB access to this one member server ("Network password is
invalid"), but not all users. Worked fine before today.

wbinfo -p/-P work, wbinfo -a shows the same problem of some users
working, some not: Those that do work, report success with plaintext
auth, and NT_STATUS_WRONG_PASSWORD for challenge/response auth (wtf?).
Those that don't work at all, fail plaintext auth and report
NT_STATUS_INTERNAL_DB_CORRUPTION for challenge/response. Not sure if
that means anything, given that challenge/response seems to always fail
with nonsensical error messages. All the other working member servers
also report NT_STATUS_WRONG_PASSWORD for c/r auth.

15 MB/s error logs were not an exaggeration, BTW, that's what I saw when
I cranked up the logging level to 10, since the default log level didn't
bother even reporting the logon failures at all (which should be
sensible defaults, but oh well). Since I don't know what component of
Samba is responsible here, I don't know for which I should increase
logging and for which I shouldn't.

Now that I'm digging, there also seem to be some generic WERR_BADFILE
DRS replication errors that our automated monitoring somehow didn't
catch; and one DC apparently no longer has the DNS entries it should
have, and samba_dnsupdates alternates between "FORMERR" and
"GSS-TSIG
unsuccessful" which apparently is only supposed to appear with the BIND9
DNS backend, which we aren't using. These are probably related, but
again I have no idea where these come from or how to debug them.


So how was your morning?

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/069c3770/signature.sig>

On 21/05/2019 12:27, Sven Schwedas via samba wrote:
> The smb.conf hasn't changed since the last three or four times I've
> posted here asking for help:
>
> https://up.tao.at/u/samba/villach-file.txt
>
> Top level error I'm seeing is that since today *some* Windows users are
> denied SMB access to this one member server ("Network password is
> invalid"), but not all users. Worked fine before today.
>
> wbinfo -p/-P work, wbinfo -a shows the same problem of some users
> working, some not: Those that do work, report success with plaintext
> auth, and NT_STATUS_WRONG_PASSWORD for challenge/response auth (wtf?).
> Those that don't work at all, fail plaintext auth and report
> NT_STATUS_INTERNAL_DB_CORRUPTION for challenge/response. Not sure if
> that means anything, given that challenge/response seems to always fail
> with nonsensical error messages. All the other working member servers
> also report NT_STATUS_WRONG_PASSWORD for c/r auth.
>
> 15 MB/s error logs were not an exaggeration, BTW, that's what I saw
when
> I cranked up the logging level to 10, since the default log level
didn't
> bother even reporting the logon failures at all (which should be
> sensible defaults, but oh well). Since I don't know what component of
> Samba is responsible here, I don't know for which I should increase
> logging and for which I shouldn't.
>
> Now that I'm digging, there also seem to be some generic WERR_BADFILE
> DRS replication errors that our automated monitoring somehow didn't
> catch; and one DC apparently no longer has the DNS entries it should
> have, and samba_dnsupdates alternates between "FORMERR" and
"GSS-TSIG
> unsuccessful" which apparently is only supposed to appear with the
BIND9
> DNS backend, which we aren't using. These are probably related, but
> again I have no idea where these come from or how to debug them.
>
>
> So how was your morning?
>
Good, so far ;-)

You need to investigate your DB problems, but just a few comments on 
your smb.conf ;-)

I see no reason to have different smb.conf files for different Unix 
domain members, just don't have 'netbios name' in any smb.conf.

You will also be better better off having 'vfs objects = acl_xattr' in 
your smb.conf and setting the permissions from Windows.

What is the point of this:

     winbind max domain connections = 32

If you also have:

     winbind offline logon = yes

Finally and what could be contributing to your problem:

This could be set too high:
     winbind expand groups = 4

See 'man smb.conf' for more info.

Rowland

On 21.05.19 14:16, Rowland penny via samba wrote:> You need to
investigate your DB problems

Great, but how?
> I see no reason to have different smb.conf files for different Unix
> domain members, just don't have 'netbios name' in any smb.conf.
There's also share definitions in the files which I omitted, which are
the actual meat of the config files.
> You will also be better better off having 'vfs objects = acl_xattr'
in
> your smb.conf and setting the permissions from Windows.
Will that work when half the clients aren't Windows to begin with, and
ACLs still need to work when people can SSH into the server?
> What is the point of this:
> 
>     winbind max domain connections = 32
> 
> If you also have:
> 
>     winbind offline logon = yes
Will it hurt?
> Finally and what could be contributing to your problem:
> 
> This could be set too high:
>     winbind expand groups = 4
Why would that suddenly break after working for years, when the deepest
nesting we actually see is 1?

And going by smb.conf, at most it could lead to timeouts, which is not
the problem we're seeing?

This is *exactly* what I meant with bike shedding. "This has nothing to
do with your problem, but let's waste days on this anyway, it's not
*our* prod environment that's offline in the meantime" is really not a
great attitude.

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/a3590862/signature.sig>