samba - May 2019 - Debugging Samba is a total PITA and this needs to improve

Hai Sven, 

And still i see/think you should change some things to get a better base setup. 
And no its not bike shedding....  It is making a standard setup, work from
there.


[libdefaults]
	default_realm = AD.TAO.AT
	dns_lookup_realm = true	< if you have multple REALM, else false.
(default_realm = AD.TAO.AT)
	dns_lookup_kdc = true


Checking file: /etc/nsswitch.conf
passwd:         files winbind
group:          files winbind
shadow:         files 
( removed winbind from shadow) not used. 

winbind enum users = yes
winbind enum groups = yes
Better no, works the same, but your server is faster. 

#### site.conf
netbios name = villach-file		< in CAPS 

For windows/samba netbios resolving: NETBIOSNAME =! netbiosname 
DNS resolving : NETBIOSNAME == netbiosname 
REALM resolving : REALM =! realm
Dnsdomain name  : realm often looks like dnsdomainname but.. 
			dnsdomainname =! REALM 

.. Clean up you site.conf. Make it as little as possible.

You see this note from the script: 
Running as Unix domain member and no user.map detected. 

Where is you user mapping? You dont use SePrivileges? 
Now its not wrong and possible to run it without, but it is much more work to
setup correctly for this.

And.. You still on 4.5.16, yes, possible, but why do you think i make newer
packages.

Windows and it updates are moving fast, so samba is following fast, while debian
is slow.
Not that's wrong, really i preffer myself slow and good updates, but thats
just not the way for samba.
And this is why i build the samba packages. To keep up with samba. 
You cant fix all with 4.5.16, for that you need higher samba versions. 
I've suggested this to Debian, to make a separated line for samba that
follow the main releases of samba.
But, that as a no-no.., so thats why i supply these, with debian's settings.
Thats also why i use distro-sambaVERSION , to keep track with samba AND windows.

Now, last question, on the pc with the "unable to authenticate", any
windows event id's with warning/errors?
You probley looked at that already?? Or not? 
> Top level error I'm seeing is that since today *some* Windows 
> users are denied SMB access to this one member server ("Network
password is
> invalid"), but not all users. Worked fine before today.
If you delayed your windows updates for for example 6 day, then this is logical
to me.
Because MS updates are on Tuesday..  

Now, what if you reinstall SMB1 for these windows pc and disable autoremovement.
Check with Powershel: 	Get-WindowsFeature FS-SMB1 

If thats not the case, then you should check the attributes of the computer in
the AD.
this could be also due to kerberos mismatchings in AD. 

You can check this as followed. 

samba-tool computer show YOUR_COMPUTERNAME_HERE >
/tmp/YOUR_COMPUTERNAME_HERE.txt
egrep
"dn|name|sAMAccountName|dNSHostName|distinguishedName|servicePrincipalName"
< /tmp/YOUR_COMPUTERNAME_HERE.txt
Safe the file or you keep quering your AD. 

servicePrincipalName: HOST/HOSTNAME.dnsdomain.tld is WRONG ! 

servicePrincipalName: HOST/HOSTNAME is correct. 
servicePrincipalName: HOST/hostname.dnsdomain.tld is correct ! 

So correct: 
HOST/NETBIOSNAME ( uppercase) 
HOST/host.fqdn ( lowercase)
sAMAccountName: NETBIOSNAME$  ( uppercase)  

Check this if this is your case also, there are lots of reports if "unable
to authenticate" or lost trust of domain..
Due to above. 
If a name is wrong, Open ADSIEdit.  Go to the computer object.  
Don't hit properties of the object just right click and choose rename.  

More below... 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Sven 
> Schwedas via samba
> Verzonden: dinsdag 21 mei 2019 13:28
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Debugging Samba is a total PITA and 
> this needs to improve
> 
> The smb.conf hasn't changed since the last three or four times I've
> posted here asking for help:
> 
> https://up.tao.at/u/samba/villach-file.txt
> 
> Top level error I'm seeing is that since today *some* Windows 
> users are
> denied SMB access to this one member server ("Network password is
> invalid"), but not all users. Worked fine before today.
> 
> wbinfo -p/-P work, wbinfo -a shows the same problem of some users
> working, some not: Those that do work, report success with plaintext
> auth, and NT_STATUS_WRONG_PASSWORD for challenge/response auth (wtf?).
> Those that don't work at all, fail plaintext auth and report
> NT_STATUS_INTERNAL_DB_CORRUPTION for challenge/response. Not sure if
> that means anything, given that challenge/response seems to 
> always fail
> with nonsensical error messages. All the other working member servers
> also report NT_STATUS_WRONG_PASSWORD for c/r auth.
> 
> 15 MB/s error logs were not an exaggeration, BTW, that's what 
> I saw when
> I cranked up the logging level to 10, since the default log 
> level didn't
> bother even reporting the logon failures at all (which should be
> sensible defaults, but oh well). Since I don't know what component of
> Samba is responsible here, I don't know for which I should increase
> logging and for which I shouldn't.
man smb.conf /log level   ( + hit 5x n ) and your at the log level point.  ;-) 
That shows this example : 
log level = 1 full_audit:1@/var/log/audit.log

> 
> Now that I'm digging, there also seem to be some generic WERR_BADFILE
> DRS replication errors that our automated monitoring somehow didn't
> catch; and one DC apparently no longer has the DNS entries it should
> have, and samba_dnsupdates alternates between "FORMERR" and
"GSS-TSIG
> unsuccessful" which apparently is only supposed to appear with the
BIND9
> DNS backend, which we aren't using. These are probably related, but
> again I have no idea where these come from or how to debug them.
Date: Tue, 22 May 2018 15:44:36 +0000
- Dynamic DNS updates with GSS-TSIG against Microsoft or samba DNS servers are
not working and fails with the following error: ; TSIG error ...
https://bugzilla.samba.org/show_bug.cgi?id=13019  samba 4.7 and lower. 

You really want to try my packages.. ;-) 
And in your case, update steps, 4.8, and stay there if you want to switch to
Buster then 4.9.5
Or move more up to 4.9 or 4.10. 
Or if the server is an samba only, server, upgrade to buster, but ... Prepair
for that, you will hit more then you expecting to hit.
Not advice, just a suggesting, im not you, i can tell whats best for you, i dont
know you complete network.

> 
> 
> So how was your morning?
Good, thanks for asking. 

And in addition to Rowland, you always replies when im still typing :-p 
;-) 
>> You need to investigate your DB problems, but just a few comments on 
No, start with your resolving and hostname. 

This is the base and this has to be correct and having this correct, helps in
reducing problems in you windows clients.
And it helps if finding your problem. 

Greetz, 

Louis

On 21.05.19 14:37, L.P.H. van Belle via samba wrote:
> winbind enum users = yes
> winbind enum groups = yes
> Better no, works the same, but your server is faster. 
Since Cyrus IMAPD cannot query LDAP for group memberships, we need this
to make shared folders work with groups on our mail servers. Useless on
this machine, yes, but w/e, we're not seeing any performance issues.
> You see this note from the script: 
> Running as Unix domain member and no user.map detected. 
> 
> Where is you user mapping? You dont use SePrivileges? 
> Now its not wrong and possible to run it without, but it is much more work
to setup correctly for this.
Where's this documented?
> And.. You still on 4.5.16, yes, possible, but why do you think i make newer
packages.
If updating Samba didn't have a tendency to lead to breakages, I'd just
chuck it into the daily auto updates. But since debugging breakages is
just too painful, I'd rather not touch it.
> Windows and it updates are moving fast
Sure, but not really relevant here, since the member server broke
authentication for all client OSes, not just Windows clients. `smbclient
-L //localhost` and `wbinfo -a` are just as broken on that member server.

Didn't notice that until after my first email, since it just so happened
that the user /accounts/ affected were all using Windows PCs when they
noticed the problem.
> Now, last question, on the pc with the "unable to authenticate",
any windows event id's with warning/errors?
> You probley looked at that already?? Or not? 
No error message other than that. Network logons to DCs work fine too,
as do logons to other member servers.
> man smb.conf /log level   ( + hit 5x n ) and your at the log level point. 
;-)
> That shows this example : 
> log level = 1 full_audit:1@/var/log/audit.log
full_audit doesn't exist for 4.5. ;)
> Date: Tue, 22 May 2018 15:44:36 +0000
> - Dynamic DNS updates with GSS-TSIG against Microsoft or samba DNS servers
are not working and fails with the following error: ; TSIG error ...
> https://bugzilla.samba.org/show_bug.cgi?id=13019  samba 4.7 and lower. 
> 
> You really want to try my packages.. ;-) 
> And in your case, update steps, 4.8, and stay there if you want to switch
to Buster then 4.9.5
Given that DRS replication and DNS are so broken, what'd be the best
approach for that? Nuke all DCs except the FSMO role holder, update that
one, then add new DCs? Or just export all LDAP data and start over from
a clean 4.10 setup?

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/58de100c/signature.sig>

Hai, 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Sven 
> Schwedas via samba
> Verzonden: dinsdag 21 mei 2019 15:04
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Debugging Samba is a total PITA and 
> this needs to improve
> 
> On 21.05.19 14:37, L.P.H. van Belle via samba wrote:
> > winbind enum users = yes
> > winbind enum groups = yes
> > Better no, works the same, but your server is faster. 
> 
> Since Cyrus IMAPD cannot query LDAP for group memberships, we 
> need this to make shared folders work with groups on our mail servers. 
> Useless on this machine, yes, but w/e, we're not seeing any performance
issues.
Huh... Doesn't this work something like : you can put this in idmap.conf 

https://www.cyrusimap.org/imap/reference/manpages/configs/imapd.conf.html

ldap_group_base: dc=example,dc=com
ldap_group_filter:
(&(cn=%u)(objectclass=WhatYOUneed)(objectclass=Someother))
ldap_group_scope: sub
ldap_member_method: attribute
ldap_member_attribute: mail

> 
> > You see this note from the script: 
> > Running as Unix domain member and no user.map detected. 
> > 
> > Where is you user mapping? You dont use SePrivileges? 
> > Now its not wrong and possible to run it without, but it is 
> much more work to setup correctly for this. 
> 
> Where's this documented?
https://wiki.samba.org/index.php/Samba_Member_Server_Troubleshooting

> 
> > And.. You still on 4.5.16, yes, possible, but why do you 
> think i make newer packages. 
> 
> If updating Samba didn't have a tendency to lead to breakages, I'd
just
> chuck it into the daily auto updates. But since debugging breakages is
> just too painful, I'd rather not touch it.
Do you believe if i say that i run unattended upgrades on 90% of my servers
including kernels and automated reboots.
Only 2 servers not both database servers, i do these manualy. 

If you keep you smb.conf clean, autoupgrades are much better, and latest
versions of samba ignore wrong/predicated settings.
Which helps also. Really, once your in samba 4.8 4.9 or 4.10 your life gets
easier.
> 
> > Windows and it updates are moving fast
> 
> Sure, but not really relevant here, since the member server broke
> authentication for all client OSes, not just Windows clients. 
> `smbclient
> -L //localhost` and `wbinfo -a` are just as broken on that 
> member server.
smbclient -L //localhost ????  Come on...  
I'm always amazed how a "localhost" test is compaired with a
client (remote) test.
Again , localhost =! Hostname 

smbclient -L //hostname.fdqn 
smbclient -L //hostname

Thats a test...  Again what did i say in the previous mail. 

It all begins with correct resolving..    
smbclient -L //localhost << works yes..  But correct, in my optinion not. 
> 
> Didn't notice that until after my first email, since it just  so
happened
> that the user /accounts/ affected were all using Windows PCs when they
> noticed the problem.
> 
> > Now, last question, on the pc with the "unable to
authenticate", any windows event id's with warning/errors?
> > You probley looked at that already?? Or not? 
> 
> No error message other than that. Network logons to DCs work fine too,
> as do logons to other member servers.
> 
> > man smb.conf /log level   ( + hit 5x n ) and your at the 
> log level point.  ;-) 
> > That shows this example : 
> > log level = 1 full_audit:1@/var/log/audit.log
> 
> full_audit doesn't exist for 4.5. ;)
Ah yeah.. this also make it harder for us to help. 
Now i suggest, upgrade, your using an "by samba" unsupported version. 
See: https://wiki.samba.org/index.php/Samba_Release_Planning 
> 
> > Date: Tue, 22 May 2018 15:44:36 +0000
> > - Dynamic DNS updates with GSS-TSIG against Microsoft or 
> samba DNS servers are not working and fails with the 
> following error: ; TSIG error ...
> > https://bugzilla.samba.org/show_bug.cgi?id=13019  samba 4.7 
> and lower. 
> > 
> > You really want to try my packages.. ;-) 
> > And in your case, update steps, 4.8, and stay there if you 
> want to switch to Buster then 4.9.5 
> 
> Given that DRS replication and DNS are so broken, what'd be the best
> approach for that? Nuke all DCs except the FSMO role holder, 
> update that
> one, then add new DCs? Or just export all LDAP data and start 
> over from  a clean 4.10 setup?
I dont think its broken, i think its functioning wrong due to wrong settings. 
Yes, clean setup is nice but not needed really. 

Make sure you review and have smb.conf adjusted to the version of samba your
willing to run.
Review: https://wiki.samba.org/index.php/Updating_Samba 


Greetz, 

Louis

On 21.05.19 16:15, L.P.H. van Belle via samba wrote:
>> Since Cyrus IMAPD cannot query LDAP for group memberships, we 
>> need this to make shared folders work with groups on our mail servers. 
>> Useless on this machine, yes, but w/e, we're not seeing any
performance issues.
> Huh... Doesn't this work something like : you can put this in
idmap.conf
It should work that way, but the current release has a few bugs related
to it, and we still need to have working group ACLs until that's working.
>>> You see this note from the script: 
>>> Running as Unix domain member and no user.map detected. 
>>>
>>> Where is you user mapping? You dont use SePrivileges? 
>>> Now its not wrong and possible to run it without, but it is 
>> much more work to setup correctly for this. 
>>
>> Where's this documented?
> https://wiki.samba.org/index.php/Samba_Member_Server_Troubleshooting
No, I mean SePrivileges in general. What would I want them for?
>>> Windows and it updates are moving fast
>>
>> Sure, but not really relevant here, since the member server broke
>> authentication for all client OSes, not just Windows clients. 
>> `smbclient
>> -L //localhost` and `wbinfo -a` are just as broken on that 
>> member server.
> 
> smbclient -L //localhost ????  Come on...  
It has the same results as Windows Explorer and wbinfo -a.
> I'm always amazed how a "localhost" test is compaired with a
client (remote) test.
> Again , localhost =! Hostname 
> 
> smbclient -L //hostname.fdqn 
> smbclient -L //hostname
Same results: Some users work, some don't. Same users affected.
>> Given that DRS replication and DNS are so broken, what'd be the
best
>> approach for that? Nuke all DCs except the FSMO role holder, 
>> update that
>> one, then add new DCs? Or just export all LDAP data and start 
>> over from  a clean 4.10 setup?
> 
> I dont think its broken, i think its functioning wrong due to wrong
settings.
Yes, you always think that. ;)
> Yes, clean setup is nice but not needed really. 
> 
> Make sure you review and have smb.conf adjusted to the version of samba
your willing to run.
> Review: https://wiki.samba.org/index.php/Updating_Samba 
Sure, that says:
>      Verify that the directory replication between all DCs is working
correctly:
That's already broken before the update:

https://up.tao.at/u/samba/graz-dc-sem.txt (FSMO role holder)
https://up.tao.at/u/samba/graz-dc-1b.txt
https://up.tao.at/u/samba/villach-dc-1a.txt
https://up.tao.at/u/samba/villach-dc-bis.txt

Similarly, if I do "samba-tool dbcheck --cross-ncs" without yet
upgrading, to see in what state the DBs are:

https://up.tao.at/u/samba/graz-dc-sem-dbcheck.txt
https://up.tao.at/u/samba/graz-dc-1b-dbcheck.txt
https://up.tao.at/u/samba/villach-dc-1a-dbcheck.txt
https://up.tao.at/u/samba/villach-dc-bis-dbcheck.txt

Doesn't look particularly healthy to me.

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/e0da54d3/signature.sig>

Hai, 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Sven 
> Schwedas via samba
> Verzonden: dinsdag 21 mei 2019 16:44
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Debugging Samba is a total PITA and 
> this needs to improve
> 
> On 21.05.19 16:15, L.P.H. van Belle via samba wrote:
> >> Since Cyrus IMAPD cannot query LDAP for group memberships, we 
> >> need this to make shared folders work with groups on our 
> mail servers. 
> >> Useless on this machine, yes, but w/e, we're not seeing 
> any performance issues.
> > Huh... Doesn't this work something like : you can put this 
> in idmap.conf 
> 
> It should work that way, but the current release has a few 
> bugs related
> to it, and we still need to have working group ACLs until 
> that's working.
Ok, that i dont know. So a good reason to use it. 
> 
> >>> You see this note from the script: 
> >>> Running as Unix domain member and no user.map detected. 
> >>>
> >>> Where is you user mapping? You dont use SePrivileges? 
> >>> Now its not wrong and possible to run it without, but it is 
> >> much more work to setup correctly for this. 
> >>
> >> Where's this documented?
> > https://wiki.samba.org/index.php/Samba_Member_Server_Troubleshooting
> 
> No, I mean SePrivileges in general. What would I want them for?
Old but shows enough:
https://www.samba.org/samba/docs/old/Samba3-HOWTO/rights.html
And  : https://docs.microsoft.com/en-us/windows/desktop/secauthz/privileges 
> 
> >>> Windows and it updates are moving fast
> >>
> >> Sure, but not really relevant here, since the member server broke
> >> authentication for all client OSes, not just Windows clients. 
> >> `smbclient
> >> -L //localhost` and `wbinfo -a` are just as broken on that 
> >> member server.
> > 
> > smbclient -L //localhost ????  Come on...  
> 
> It has the same results as Windows Explorer and wbinfo -a.
Yes, same in what you "see" but not same in how thing go in the
background what you dont see..
> 
> > I'm always amazed how a "localhost" test is compaired
with
> a client (remote) test. 
> > Again , localhost =! Hostname 
> > 
> > smbclient -L //hostname.fdqn 
> > smbclient -L //hostname
> 
> Same results: Some users work, some don't. Same users affected.
Same users are still only windows clients? 
And how are these logging in with : DOM\user or user at REALM ? 
> 
> >> Given that DRS replication and DNS are so broken, what'd 
> be the best
> >> approach for that? Nuke all DCs except the FSMO role holder, 
> >> update that
> >> one, then add new DCs? Or just export all LDAP data and start 
> >> over from  a clean 4.10 setup?
> > 
> > I dont think its broken, i think its functioning wrong due 
> to wrong settings. 
> 
> Yes, you always think that. ;)
And you dont know how often im right here.. 

> 
> > Yes, clean setup is nice but not needed really. 
> > 
> > Make sure you review and have smb.conf adjusted to the 
> version of samba your willing to run. 
> > Review: https://wiki.samba.org/index.php/Updating_Samba 
> 
> Sure, that says:
> 
> >      Verify that the directory replication between all DCs 
> is working correctly:
> 
> That's already broken before the update:
> 
> https://up.tao.at/u/samba/graz-dc-sem.txt (FSMO role holder)
> https://up.tao.at/u/samba/graz-dc-1b.txt
> https://up.tao.at/u/samba/villach-dc-1a.txt
> https://up.tao.at/u/samba/villach-dc-bis.txt
> 
> Similarly, if I do "samba-tool dbcheck --cross-ncs" without yet
> upgrading, to see in what state the DBs are:
> 
> https://up.tao.at/u/samba/graz-dc-sem-dbcheck.txt
> https://up.tao.at/u/samba/graz-dc-1b-dbcheck.txt
> https://up.tao.at/u/samba/villach-dc-1a-dbcheck.txt
> https://up.tao.at/u/samba/villach-dc-bis-dbcheck.txt
> 
> Doesn't look particularly healthy to me.
No, but its not that bad as far i can see. 

Argg. I have to to thing here now, move workspaces.. 

Sync graz-dc-sem to VILLACH-DC-BIS ( full sync ) 
Reboot: VILLACH-DC-BIS 
Wait 5 min, check again. 

Verify this GUID: e70407fd-019e-42f8-a60d-4504d2df230c 
In zone _msdc. Check it compleet. 

<GUID=e1569c90-50f9-4bb5-bd85-79145e3ff6fd>;CN=NTDS
Settings,CN=VILLACH-DC-BIS,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=tao,DC=at
Not fixing old string component	<< old ... ( keyword ) 
Diffent GUIDs 

I expect that your problem for the sinc is in that area..

I have to go. 
I nobody help you out today, i'll help you tomorrow while i'll build new
samba packages..

Sofar, 

Greetzz, 

Louis

Ps. You should have updated/cleanup you config a bit more since nov 2017. i
hardly changed..