samba - May 2019 - Debugging Samba is a total PITA and this needs to improve

Once again, something with Samba thirty bazillion components broke. Once
again, my choices for logging are "nothing" or "15 MB/s spread of
ten
different files, because 'client authentication failed' totally needs to
be lower priority than malloc debug info". Once again, none of these
messages is actually able to convey what broke, where, why. Why is it
impossible for Samba to provide useful error reporting? How is anyone
supposed to use this in production? Why is the only way we have to
troubleshoot is having Rowland and LPH bike shedding over smb.conf
formatting styles for two weeks until the problem randomly solves itself?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/088033ac/signature.sig>

On 21/05/2019 10:44, Sven Schwedas via samba wrote:
> Once again, something with Samba thirty bazillion components broke. Once
> again, my choices for logging are "nothing" or "15 MB/s
spread of ten
> different files, because 'client authentication failed' totally
needs to
> be lower priority than malloc debug info". Once again, none of these
> messages is actually able to convey what broke, where, why. Why is it
> impossible for Samba to provide useful error reporting? How is anyone
> supposed to use this in production? Why is the only way we have to
> troubleshoot is having Rowland and LPH bike shedding over smb.conf
> formatting styles for two weeks until the problem randomly solves itself?
Hi, its the ace bike shedder here ;-)

The main reason why we have to 'bike shed' is the lack of info provided.
Many initial posts basically start with 'help, it doesn't work', no 
mention of what doesn't work or why, no mention of OS or version of 
Samba and sometimes getting information can feel like extracting teeth ;-)

You can get better logging, can I suggest you read the manpage for 
smb.conf: man smb.conf

Pay attention to these parameters: log file, logging, log level, max log 
size, syslog only, syslog

Having said all that, what OS and Samba version? What is in your 
smb.conf and most importantly, what isn't working with authentication ?

Rowland

Hai Sven, 

I do agree on somepoint here, but not all. 

First off all, samba does produce usefull reporting info. 
Depending on log levels you have set. ( man smb.conf /log level ). 

And then again we always have to ask, can you show the smb.conf. 
Not that i dont believe you have wrong setting but because i must see you have
good setttings.

And the problem is not always/only samba, keep in mind that the current windows
updates are not as stable as years ago.
My personal opinion here, current windows updates are really bad in quality, but
why is this...
This is because MS is testing on users computers, that simple. 
You dont believe this.. 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PolicyManager\current\device\System 
The key used: (AllowExperimentation) 

About, client authentication failed... 
Windows 10, and are use useing SMB1 then disable the autoremove of smb1 in
windows10.
Dont forget, i use windows since windows 2.1, i've seen enough with windows.

The current changes pointing at NTLM V1/V2 a lot has been done by MS and Samba.

Simple things changed.. For example 
https://support.microsoft.com/nl-nl/help/3181029/smb-file-server-share-access-is-unsuccessful-through-dns-cname-alias

I would preffer also to see a clear difference in settings in manuals for 
samba-AD-DC
samba-AD-Member-file server
samba-AD-Member-Print server
samba-AD-Member-authonly server 

Samba-NT4-PDC
Samba-NT4-BDC
Samba-NT4-Member

Samba-Standalone

Now, add to this the problem Rowland and i are looking into, found recently. 
A "renamed" computer, might have or is missing or has wrong settings
in AD.
A common problem we found yesterday, that also exists in windows itself and
caused by windows clients.
And these found bugs found in windows are inherreted in samba, not always.. But
in this case it is.

Sorry if you feel bad about this but we are here to help, in such cases. 

So if you have ideas to improve the debugging.

This is on github.
https://github.com/thctlo/samba4/blob/master/samba-collect-debug-info.sh 
Fork/add you parts. 

And yes, the script really helps in quicker finding the problems. 
You can ask that here at list users.. 

So far.. Time to lunch.. 

Greetz, 

Louis






> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Sven 
> Schwedas via samba
> Verzonden: dinsdag 21 mei 2019 11:44
> Aan: samba at lists.samba.org
> Onderwerp: [Samba] Debugging Samba is a total PITA and this 
> needs to improve
> 
> Once again, something with Samba thirty bazillion components broke. Once
> again, my choices for logging are "nothing" or "15 MB/s
spread of ten
> different files, because 'client authentication failed' totally
needs to
> be lower priority than malloc debug info". Once again, none of these
> messages is actually able to convey what broke, where, why. Why is it
> impossible for Samba to provide useful error reporting? How is anyone
> supposed to use this in production? Why is the only way we have to
> troubleshoot is having Rowland and LPH bike shedding over smb.conf
> formatting styles for two weeks until the problem randomly 
> solves itself?
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

On 21/05/2019 11:20, L.P.H. van Belle via samba wrote:
> Now, add to this the problem Rowland and i are looking into, found
recently.
> A "renamed" computer, might have or is missing or has wrong
settings in AD.
> A common problem we found yesterday, that also exists in windows itself and
caused by windows clients.
> And these found bugs found in windows are inherreted in samba, not always..
But in this case it is.
To add to this, if you rename a domain computer on Windows, you do not 
get the following changed in the Samba AD object:

dn, cn, name and distinguishedname

Whilst if you use ldbrename on an AD DC, exactly the opposite happens 
and you cannot tell the difference by examining the computer object.

The main problem I have with this is, is this a windows problem or a 
Samba problem ? I cannot find anything that definitely says that when 
you rename a computer, everything should be changed. Lots of webpages 
that infer this is the case, but nothing that shows a computer object 
before and after a rename, or lists what should be changed.

Rowland

The smb.conf hasn't changed since the last three or four times I've
posted here asking for help:

https://up.tao.at/u/samba/villach-file.txt

Top level error I'm seeing is that since today *some* Windows users are
denied SMB access to this one member server ("Network password is
invalid"), but not all users. Worked fine before today.

wbinfo -p/-P work, wbinfo -a shows the same problem of some users
working, some not: Those that do work, report success with plaintext
auth, and NT_STATUS_WRONG_PASSWORD for challenge/response auth (wtf?).
Those that don't work at all, fail plaintext auth and report
NT_STATUS_INTERNAL_DB_CORRUPTION for challenge/response. Not sure if
that means anything, given that challenge/response seems to always fail
with nonsensical error messages. All the other working member servers
also report NT_STATUS_WRONG_PASSWORD for c/r auth.

15 MB/s error logs were not an exaggeration, BTW, that's what I saw when
I cranked up the logging level to 10, since the default log level didn't
bother even reporting the logon failures at all (which should be
sensible defaults, but oh well). Since I don't know what component of
Samba is responsible here, I don't know for which I should increase
logging and for which I shouldn't.

Now that I'm digging, there also seem to be some generic WERR_BADFILE
DRS replication errors that our automated monitoring somehow didn't
catch; and one DC apparently no longer has the DNS entries it should
have, and samba_dnsupdates alternates between "FORMERR" and
"GSS-TSIG
unsuccessful" which apparently is only supposed to appear with the BIND9
DNS backend, which we aren't using. These are probably related, but
again I have no idea where these come from or how to debug them.


So how was your morning?

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190521/069c3770/signature.sig>

Hai Sven, 

And still i see/think you should change some things to get a better base setup. 
And no its not bike shedding....  It is making a standard setup, work from
there.


[libdefaults]
	default_realm = AD.TAO.AT
	dns_lookup_realm = true	< if you have multple REALM, else false.
(default_realm = AD.TAO.AT)
	dns_lookup_kdc = true


Checking file: /etc/nsswitch.conf
passwd:         files winbind
group:          files winbind
shadow:         files 
( removed winbind from shadow) not used. 

winbind enum users = yes
winbind enum groups = yes
Better no, works the same, but your server is faster. 

#### site.conf
netbios name = villach-file		< in CAPS 

For windows/samba netbios resolving: NETBIOSNAME =! netbiosname 
DNS resolving : NETBIOSNAME == netbiosname 
REALM resolving : REALM =! realm
Dnsdomain name  : realm often looks like dnsdomainname but.. 
			dnsdomainname =! REALM 

.. Clean up you site.conf. Make it as little as possible.

You see this note from the script: 
Running as Unix domain member and no user.map detected. 

Where is you user mapping? You dont use SePrivileges? 
Now its not wrong and possible to run it without, but it is much more work to
setup correctly for this.

And.. You still on 4.5.16, yes, possible, but why do you think i make newer
packages.

Windows and it updates are moving fast, so samba is following fast, while debian
is slow.
Not that's wrong, really i preffer myself slow and good updates, but thats
just not the way for samba.
And this is why i build the samba packages. To keep up with samba. 
You cant fix all with 4.5.16, for that you need higher samba versions. 
I've suggested this to Debian, to make a separated line for samba that
follow the main releases of samba.
But, that as a no-no.., so thats why i supply these, with debian's settings.
Thats also why i use distro-sambaVERSION , to keep track with samba AND windows.

Now, last question, on the pc with the "unable to authenticate", any
windows event id's with warning/errors?
You probley looked at that already?? Or not? 
> Top level error I'm seeing is that since today *some* Windows 
> users are denied SMB access to this one member server ("Network
password is
> invalid"), but not all users. Worked fine before today.
If you delayed your windows updates for for example 6 day, then this is logical
to me.
Because MS updates are on Tuesday..  

Now, what if you reinstall SMB1 for these windows pc and disable autoremovement.
Check with Powershel: 	Get-WindowsFeature FS-SMB1 

If thats not the case, then you should check the attributes of the computer in
the AD.
this could be also due to kerberos mismatchings in AD. 

You can check this as followed. 

samba-tool computer show YOUR_COMPUTERNAME_HERE >
/tmp/YOUR_COMPUTERNAME_HERE.txt
egrep
"dn|name|sAMAccountName|dNSHostName|distinguishedName|servicePrincipalName"
< /tmp/YOUR_COMPUTERNAME_HERE.txt
Safe the file or you keep quering your AD. 

servicePrincipalName: HOST/HOSTNAME.dnsdomain.tld is WRONG ! 

servicePrincipalName: HOST/HOSTNAME is correct. 
servicePrincipalName: HOST/hostname.dnsdomain.tld is correct ! 

So correct: 
HOST/NETBIOSNAME ( uppercase) 
HOST/host.fqdn ( lowercase)
sAMAccountName: NETBIOSNAME$  ( uppercase)  

Check this if this is your case also, there are lots of reports if "unable
to authenticate" or lost trust of domain..
Due to above. 
If a name is wrong, Open ADSIEdit.  Go to the computer object.  
Don't hit properties of the object just right click and choose rename.  

More below... 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Sven 
> Schwedas via samba
> Verzonden: dinsdag 21 mei 2019 13:28
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Debugging Samba is a total PITA and 
> this needs to improve
> 
> The smb.conf hasn't changed since the last three or four times I've
> posted here asking for help:
> 
> https://up.tao.at/u/samba/villach-file.txt
> 
> Top level error I'm seeing is that since today *some* Windows 
> users are
> denied SMB access to this one member server ("Network password is
> invalid"), but not all users. Worked fine before today.
> 
> wbinfo -p/-P work, wbinfo -a shows the same problem of some users
> working, some not: Those that do work, report success with plaintext
> auth, and NT_STATUS_WRONG_PASSWORD for challenge/response auth (wtf?).
> Those that don't work at all, fail plaintext auth and report
> NT_STATUS_INTERNAL_DB_CORRUPTION for challenge/response. Not sure if
> that means anything, given that challenge/response seems to 
> always fail
> with nonsensical error messages. All the other working member servers
> also report NT_STATUS_WRONG_PASSWORD for c/r auth.
> 
> 15 MB/s error logs were not an exaggeration, BTW, that's what 
> I saw when
> I cranked up the logging level to 10, since the default log 
> level didn't
> bother even reporting the logon failures at all (which should be
> sensible defaults, but oh well). Since I don't know what component of
> Samba is responsible here, I don't know for which I should increase
> logging and for which I shouldn't.
man smb.conf /log level   ( + hit 5x n ) and your at the log level point.  ;-) 
That shows this example : 
log level = 1 full_audit:1@/var/log/audit.log

> 
> Now that I'm digging, there also seem to be some generic WERR_BADFILE
> DRS replication errors that our automated monitoring somehow didn't
> catch; and one DC apparently no longer has the DNS entries it should
> have, and samba_dnsupdates alternates between "FORMERR" and
"GSS-TSIG
> unsuccessful" which apparently is only supposed to appear with the
BIND9
> DNS backend, which we aren't using. These are probably related, but
> again I have no idea where these come from or how to debug them.
Date: Tue, 22 May 2018 15:44:36 +0000
- Dynamic DNS updates with GSS-TSIG against Microsoft or samba DNS servers are
not working and fails with the following error: ; TSIG error ...
https://bugzilla.samba.org/show_bug.cgi?id=13019  samba 4.7 and lower. 

You really want to try my packages.. ;-) 
And in your case, update steps, 4.8, and stay there if you want to switch to
Buster then 4.9.5
Or move more up to 4.9 or 4.10. 
Or if the server is an samba only, server, upgrade to buster, but ... Prepair
for that, you will hit more then you expecting to hit.
Not advice, just a suggesting, im not you, i can tell whats best for you, i dont
know you complete network.

> 
> 
> So how was your morning?
Good, thanks for asking. 

And in addition to Rowland, you always replies when im still typing :-p 
;-) 
>> You need to investigate your DB problems, but just a few comments on 
No, start with your resolving and hostname. 

This is the base and this has to be correct and having this correct, helps in
reducing problems in you windows clients.
And it helps if finding your problem. 

Greetz, 

Louis

Sven Schwedas via samba <samba at lists.samba.org>
writes:
> Once again, something with Samba thirty bazillion components broke. Once
> again, my choices for logging are "nothing" or "15 MB/s
spread of ten
> different files, because 'client authentication failed' totally
needs to
> be lower priority than malloc debug info". Once again, none of these
> messages is actually able to convey what broke, where, why. Why is it
Regarding the huge and verbose logs: I can suggest you smblog-mode[1],
it's not really a solution and it's kind of made for emacs users but it
makes sorting thru logs less painful.

It looks like this [2]. You can filter log levels dynamically with + and
-, expand/collapse messages with TAB, highlight all messages printed
between receiving a request and sending its response, etc.

I have another tool smbcmp [3] you can use to compare SMB captures which
can be useful e.g. if something works from a machine and not from
another.

As a side note: I'm doing a talk on smb debugging at next sambaxp to
show how these are used.

1: https://github.com/aaptel/smblog-mode
2:
https://lists.samba.org/archive/samba-technical/attachments/20160315/4a04dddf/smblog3-0001.png
3: https://github.com/aaptel/smbcmp

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Linux GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)

On 5/21/19 3:32 PM, Aurélien Aptel via samba wrote:
> Sven Schwedas via samba <samba at lists.samba.org> writes:
>> Once again, something with Samba thirty bazillion components broke.
Once
>> again, my choices for logging are "nothing" or "15 MB/s
spread of ten
>> different files, because 'client authentication failed' totally
needs to
>> be lower priority than malloc debug info". Once again, none of
these
>> messages is actually able to convey what broke, where, why. Why is it
> 
> Regarding the huge and verbose logs: I can suggest you smblog-mode[1],
> it's not really a solution and it's kind of made for emacs users
but it
> makes sorting thru logs less painful.
...I guess we also need wblog-mode for winbindd. :)

Cheerio!
-slow

Alright, I managed to jury-rig LDAP-authenticated FTP access to the file
shares, which works well enough that I no longer have the whole company
breathing down my neck. Sorry for being so cranky yesterday.



From what we've established, it seems that there are a bunch of
replication issues, maybe others:

• Authentication on one member server fails for some users, but not all.
There doesn't seem to be any pattern to who is and isn't affected.

• Trust between PCs and the domain is randomly lost. That was handled by
L1 support without telling me (they just re-joined the PCs every other
week), so I wasn't aware of that yesterday

• All sorts of replication issues between the various DCs:
> https://up.tao.at/u/samba/graz-dc-sem.txt (FSMO role holder)
> https://up.tao.at/u/samba/graz-dc-1b.txt
> https://up.tao.at/u/samba/villach-dc-1a.txt
> https://up.tao.at/u/samba/villach-dc-bis.txt
(Unchanged from yesterday)

• Some DB issues:
> https://up.tao.at/u/samba/graz-dc-sem-dbcheck.txt
> https://up.tao.at/u/samba/graz-dc-1b-dbcheck.txt
> https://up.tao.at/u/samba/villach-dc-1a-dbcheck.txt
> https://up.tao.at/u/samba/villach-dc-bis-dbcheck.txt
(Unchanged from yesterday)

And since we now actually have time for this, all collected info for all
domain-joined servers:

https://up.tao.at/u/samba/graz-dc-1b.info.txt
https://up.tao.at/u/samba/graz-dc-sem.info.txt
https://up.tao.at/u/samba/graz-file.info.txt
https://up.tao.at/u/samba/graz-mail.info.txt
https://up.tao.at/u/samba/villach-dc-1a.info.txt
https://up.tao.at/u/samba/villach-dc-bis.info.txt
https://up.tao.at/u/samba/villach-file.info.txt
https://up.tao.at/u/samba/villach-mail.info.txt

I have no idea how villach-dc-bis worked so far with that krb5.conf
setup, but I guess that needs fixing.

For general smb.conf stuff, the features we now still need are

• rfc2307-based Unix attributes (it's where we get all UIDs/GIDs from)
• ACLs are set programmatically with a script (management wants that
locked down), so if there's a better way to handle those, we can just
nuke the current setup and reset them all
• ACLs still need to work for people SSHing (or now FTPing) into the
servers with their domain accounts, not just from Windows
• New features (like SePrivileges support) aren't interesting, file
shares will be deprecated in favour of Cloud™ Stuff™ soon-ish anyway
(let's see for how long)
• Mail servers need winbind expansion and enumeration to support legacy
stuff that we won't get around to fix for a while yet (the domain has
like 50 users and a maximum nesting depth of 1, so performance is fine
for now)

With these constraints, what are the minimal smb.conf setups I need?

Other todos for now:

• Fix krb5.conf everywhere (remove dns_lookup_realm?)
• Improve nsswitch setup

I'll get to that when I know what all the smb.confs need to look like.

And going back to Louis' email:

On 22.05.19 10:29, L.P.H. van Belle wrote:
> Yes, then it does take the hostname. Just be warned, that in such
> cases the hostname max lenght is 15 chars.
That used to be a problem a while back when we had longer hostnames, but
shouldn't apply with the current setup. So I guess we can get rid of
that too.
> You need to verify all GUID of the DC's. ( and A/PTR/CNAME records ) 
> ldbsearch -H /var/lib/samba/private/sam.ldb
'(fromServer=*CN=Windows-DC*)' --cross-ncs dn
That gives me no results on all DCs. The wiki suggests using "ldbsearch
-H /var/lib/samba/private/sam.ldb '(invocationId=*)' --cross-ncs
objectguid" for GUIDs, which does give me results.
> Samba INTERNAL_DNS Back End - Troubleshooting
That seems to work fine, there's nothing blocking samba's internal DNS
from listening on :53, and I can query it just fine.
> https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_DNS_Record
villach-dc-bis doesn't have its _msdcs record set on /any/ server. All
other records are fine on all servers.

I set it on graz-dc-sem as detailed in the wiki, and it replicated fine
to all servers except graz-dc-1b. I restarted samba-ad-dc on that to
increase the log level, and that restart alone seems to have done the
trick, it now has the record as well.

I also fixed villach-dc-bis Kerberos to
> [libdefaults]
> 	default_realm = AD.TAO.AT
> 	dns_lookup_kdc = true
> 	default_keytab_name = FILE:/etc/krb5.keytab
> 
> [domain_realm]
> 	.ad.tao.at = AD.TAO.AT
> 	ad.tao.at = AD.TAO.AT
> 	.tao.at = AD.TAO.AT
> 	tao.at = AD.TAO.AT
and restarted samba-ad-dc on it.

Now, trying to full sync replicate works fine for the base partition and
Configuration, but replicating ForestDnsZones or DomainDnsZones both
fail with:
> ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync
failed - drsException: DsReplicaSync failed (8440, 'WERR_DS_DRA_BAD_NC')
>   File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py",
line 368, in run
>     drs_utils.sendDsReplicaSync(server_bind, server_bind_handle,
source_dsa_guid, NC, req_options)
>   File "/usr/lib/python2.7/dist-packages/samba/drs_utils.py",
line 83, in sendDsReplicaSync
>     raise drsException("DsReplicaSync failed %s" % estr)
Replicating Schema still runs to a timeout on the sending server and a
PANIC on villach-dc-bis.

Log file on villach-dc-bis: https://up.tao.at/u/samba/log.samba.txt

Relevant time stamps 2019/05/22 12:26:26.456202 (ForestDnsZones
failure), 2019/05/22 12:26:49.027595 (DomainDnsZones failure) and
2019/05/22 12:27:03.755312 (Schema panic)

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190522/05e97fb7/signature.sig>

Hai, 

Well, good that your more relax and releaved in pressure.. 
Apoligies accepted. ;-), i know the fealing but do understand, we are trying to
help out..

As you know, it very frustration to ask the same things again, and you showing
them again
But now as you showed with all the configs. It is needed.. (sorry) 

So I had a quick look, and you up to some changes. All are fixable, no worries,
but it takes a bit of time and you need to be precise/accurate. 

For example, 

https://up.tao.at/u/samba/graz-dc-1b.info.txt 
Hostname: graz-dc-1b
DNS Domain:	
FQDN: graz-dc-1b
ipaddress: 169.254.93.226 192.168.17.66
- Where is the DNS domain?  ( i'll add warning in the script there if its
missing.)
Cause:  Checking file: /etc/hosts

127.0.0.1 graz-dc-1b localhost
- Which is wrong. 

- I would have expected to see: 
127.0.0.1 localhost

::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

192.168.17.66	graz-dc-1b.ad.tao.at graz-dc-1b

And this one, i should not see that, execpt if its a dhcp thats failing. 
169.254.93.226 NetName: LINKLOCAL-RFC3927-IANA-RESERVED

And if you did disable IPv6, which you attempted, at least looks like it. 
 inet6 fe80::b8de:f4ff:fe1e:11e5/64 scope link is on the interface, but the ipv6
parts where missing in /etc/hosts.

/etc/resolv.conf
nameserver 192.168.17.1
Only one DC? And you have 4.. 

For every DC and per site, you have 2 sites correct? 
I suggest, per site, 2 DC of the same site, 1 DC of the remote as backup. 
Look at this, explanation is below it. The DC1 with fsmo. 
/etc/resolv.conf ( by example )
domain ad.tao.at
nameserver IP_DC1_WITH_FSMO
nameserver IP_DC2
nameserver IP_DC3_S2

DC2. 
/etc/resolv.conf ( by example ) 
domain ad.tao.at
nameserver IP_DC1_WITH_FSMO
nameserver IP_DC2
nameserver IP_DC4_S2
# Before reboot and after reboot and wait time and db replication check. 
#nameserver IP_DC2
#nameserver IP_DC1_WITH_FSMO
#nameserver IP_DC4_S2

# one example site 2.
# Before.. DC2.
/etc/resolv.conf
# depending on how your site is defined in AD. 
# search site2.tao.at ad.tao.at 
# search site2.tao.at
# if all in same domain: 
search ad.tao.at
nameserver IP_DC1_WITH_FSMO
nameserver IP_DC1_SITE2
nameserver IP_DC2_SITE2
# After reboot and after checking the resolving. 
/etc/resolv.conf
nameserver IP_DC1_SITE2
nameserver IP_DC2_SITE2
nameserver IP_DC1_WITH_FSMO


# some options you can play with, put them in the resolv.conf but do not enable
them yet.
# I use bind, you are using internal DNS. 
#options timeout:2
#options attempts:2
#options edns0
#options single-request
#options single-request-reopen

Now the 2 resolv.conf examples and /etc/hosts file, should make sure your
hostname and dnsdomain is correct.
As shown per example this needs to be correct before we do anything else, and
yes, fix this for every server.

For the replication. 
You see, i've kept in both resolv.conf files DC1 as first, you need this for
the first replication.
After a reboot and at least 15 min-30 main waiting, on DC2, switch the DC1 and
DC2 lines and set DC2 first.
And this is for every server needed. I wont hurt if DC1 stays first, but that
could overload DC1 on DNS requests.


For all DC's and members, all you need: ( i install krb5-user, define the
REALM and keep everythings default.
So this is sufficient.  ( yes only that one default_realm line, others are
defaults )

/etc/krb5.conf
[libdefaults]
        default_realm = AD.TAO.AT

Except, i noticed, 
	.tao.at = AD.TAO.AT
	tao.at = AD.TAO.AT
That might need some more explanation why you added it.
That helps me understanding your setup. SSO login with users email adresses
maybe?

/etc/nsswitch.conf
passwd:         files winbind
group:          files winbind
shadow:         files 			< winbind removed here. 

/etc/samba/smb.conf
	ldap ssl = start tls
	ldap ssl ads = yes
These are not supported in the AD setup. These are for an NT4DOM/ PDC/BDC setup
with ldap.

And i like how you did split up the global part and the "real" server
config part.
One im going to use also..  So noo, not only bad thin, also good things.. 

I noticed also: 
	#FIXME: Temporary to fix PHP shit
	ldap server require strong auth = no

# explain this to me, can offlist if needed. 
Because here, i think you forgot to also define /etc/ldap/ldap.conf 
.. And i missed this in the script, i'll add this ..

# /etc/ldap/ldap.conf 
# example, change the hostnames.. ( saved me typing ) ;-) 
BASE    dc=ad,dc=tao,dc=at
URI     ldaps://dc1.ad.tao.at ldaps://rtd-dc2....  ldaps://dc3... 
ldaps://dc4...
# and here again, first the 2 within the same site, then the 2 others. 
# Note, all my servers use ldaps (ssl), you can use ldap with TLS also. 

# TLS certificates (needed for GnuTLS)
TLS_CACERT      /etc/ssl/certs/ca-certificates.crt

# and minimal 
TLS_REQCERT allow

# i did see you added the certificates also with update-ca-certificates correct,
if yes.
# then the company cert is in : /etc/ssl/certs/ca-certificates.crt 
# if no, then you need to define it. 


Last one, 
[homes]
	msdfs root = yes
	msdfs proxy = \\graz-file\homes

	As per MS advice, \\graz-file.ad.tao.at\users


You might hit a bug here because [homes] on the DC's is not really
supported.
Better might be [users] but this one needs planning... before you change it. 
You understand that..

Installed packages:
Missing attr 
Which is a must install for the DC's. 

Now this was only 1 server but this is exact the same for all DC's 
This must be corrected before we can even look at the DB replication problem. 
Almost all server suffer from simular problems in the setup. 

I suggest start with above. That is the base. 
For the members, hosts resolv.conf nsswitch.conf krb5.conf all need to be fixed
first.
Scary, yes, i understand, start with DC1_fsmo DC. 

Do one at the time, check all the above, so make a checklist for this. 

I have more, but start here, doing this will give you the ability to resync the
AD-DB from DC1 to DC2 in site1.
And from there we can work to the member fixes. 
Most probly some problem will disappear after these fixed, and sure you get some
back, but these you get only once.
Because now the base setup is correct, and will never change again. 
My changes on my DC's ( since wheeze and samba 4.1.x ), only 3 lines in
smb.conf due to samba upgrades.
And one in bind configs due to changes in the path of bind_dlz. 

Sorry, its a bit of work, i know, but put in one day now, and your able to set
it and forget it and your future ready..

Now, im back to building the new 4.10.4 packages. 


For sofar, 

Greetz, 

Louis



> -----Oorspronkelijk bericht-----
> Van: Sven Schwedas [mailto:sven.schwedas at tao.at] 
> Verzonden: woensdag 22 mei 2019 12:42
> Aan: samba at lists.samba.org
> CC: L.P.H. van Belle
> Onderwerp: Various AD issues; summary
> 
> Alright, I managed to jury-rig LDAP-authenticated FTP access 
> to the file
> shares, which works well enough that I no longer have the 
> whole company
> breathing down my neck. Sorry for being so cranky yesterday.
> 
> 
> 
> From what we've established, it seems that there are a bunch of
> replication issues, maybe others:
> 
> ? Authentication on one member server fails for some users, 
> but not all.
> There doesn't seem to be any pattern to who is and isn't affected.

> 
> ? Trust between PCs and the domain is randomly lost. That was 
> handled by
> L1 support without telling me (they just re-joined the PCs every other
> week), so I wasn't aware of that yesterday
> 
> ? All sorts of replication issues between the various DCs:
> 
> > https://up.tao.at/u/samba/graz-dc-sem.txt (FSMO role holder)
> > https://up.tao.at/u/samba/graz-dc-1b.txt
> > https://up.tao.at/u/samba/villach-dc-1a.txt
> > https://up.tao.at/u/samba/villach-dc-bis.txt
> 
> (Unchanged from yesterday)
> 
> ? Some DB issues:
> 
> > https://up.tao.at/u/samba/graz-dc-sem-dbcheck.txt
> > https://up.tao.at/u/samba/graz-dc-1b-dbcheck.txt
> > https://up.tao.at/u/samba/villach-dc-1a-dbcheck.txt
> > https://up.tao.at/u/samba/villach-dc-bis-dbcheck.txt
> 
> (Unchanged from yesterday)
> 
> And since we now actually have time for this, all collected 
> info for all
> domain-joined servers:
> 
> https://up.tao.at/u/samba/graz-dc-1b.info.txt
> https://up.tao.at/u/samba/graz-dc-sem.info.txt
> https://up.tao.at/u/samba/graz-file.info.txt
> https://up.tao.at/u/samba/graz-mail.info.txt
> https://up.tao.at/u/samba/villach-dc-1a.info.txt
> https://up.tao.at/u/samba/villach-dc-bis.info.txt
> https://up.tao.at/u/samba/villach-file.info.txt
> https://up.tao.at/u/samba/villach-mail.info.txt
> 
> I have no idea how villach-dc-bis worked so far with that krb5.conf
> setup, but I guess that needs fixing.
> 
> For general smb.conf stuff, the features we now still need are
> 
> ? rfc2307-based Unix attributes (it's where we get all UIDs/GIDs from)
> ? ACLs are set programmatically with a script (management wants that
> locked down), so if there's a better way to handle those, we can just
> nuke the current setup and reset them all
> ? ACLs still need to work for people SSHing (or now FTPing) into the
> servers with their domain accounts, not just from Windows
> ? New features (like SePrivileges support) aren't interesting, file
> shares will be deprecated in favour of Cloud? Stuff? soon-ish anyway
> (let's see for how long)
> ? Mail servers need winbind expansion and enumeration to 
> support legacy
> stuff that we won't get around to fix for a while yet (the domain has
> like 50 users and a maximum nesting depth of 1, so performance is fine
> for now)
> 
> With these constraints, what are the minimal smb.conf setups I need?
> 
> Other todos for now:
> 
> ? Fix krb5.conf everywhere (remove dns_lookup_realm?)
> ? Improve nsswitch setup
> 
> I'll get to that when I know what all the smb.confs need to look like.
> 
> And going back to Louis' email:
> 
> On 22.05.19 10:29, L.P.H. van Belle wrote:
> > Yes, then it does take the hostname. Just be warned, that in such
> > cases the hostname max lenght is 15 chars.
> 
> That used to be a problem a while back when we had longer 
> hostnames, but
> shouldn't apply with the current setup. So I guess we can get rid of
> that too.
> 
> > You need to verify all GUID of the DC's. ( and A/PTR/CNAME 
> records ) 
> > ldbsearch -H /var/lib/samba/private/sam.ldb 
> '(fromServer=*CN=Windows-DC*)' --cross-ncs dn
> 
> That gives me no results on all DCs. The wiki suggests using 
> "ldbsearch
> -H /var/lib/samba/private/sam.ldb '(invocationId=*)' --cross-ncs
> objectguid" for GUIDs, which does give me results.
> 
> > Samba INTERNAL_DNS Back End - Troubleshooting
> 
> That seems to work fine, there's nothing blocking samba's internal
DNS
> from listening on :53, and I can query it just fine.
> 
> > 
> https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_D
> NS_Record
> 
> villach-dc-bis doesn't have its _msdcs record set on /any/ server. All
> other records are fine on all servers.
> 
> I set it on graz-dc-sem as detailed in the wiki, and it 
> replicated fine
> to all servers except graz-dc-1b. I restarted samba-ad-dc on that to
> increase the log level, and that restart alone seems to have done the
> trick, it now has the record as well.
> 
> I also fixed villach-dc-bis Kerberos to
> > [libdefaults]
> > 	default_realm = AD.TAO.AT
> > 	dns_lookup_kdc = true
> > 	default_keytab_name = FILE:/etc/krb5.keytab
> > 
> > [domain_realm]
> > 	.ad.tao.at = AD.TAO.AT
> > 	ad.tao.at = AD.TAO.AT
> > 	.tao.at = AD.TAO.AT
> > 	tao.at = AD.TAO.AT
> 
> and restarted samba-ad-dc on it.
> 
> Now, trying to full sync replicate works fine for the base 
> partition and
> Configuration, but replicating ForestDnsZones or DomainDnsZones both
> fail with:
> 
> > ERROR(<class 'samba.drs_utils.drsException'>): 
> DsReplicaSync failed - drsException: DsReplicaSync failed 
> (8440, 'WERR_DS_DRA_BAD_NC')
> >   File 
> "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line 
> 368, in run
> >     drs_utils.sendDsReplicaSync(server_bind, 
> server_bind_handle, source_dsa_guid, NC, req_options)
> >   File 
> "/usr/lib/python2.7/dist-packages/samba/drs_utils.py", line 
> 83, in sendDsReplicaSync
> >     raise drsException("DsReplicaSync failed %s" % estr)
> 
> Replicating Schema still runs to a timeout on the sending server and a
> PANIC on villach-dc-bis.
> 
> Log file on villach-dc-bis: https://up.tao.at/u/samba/log.samba.txt
> 
> Relevant time stamps 2019/05/22 12:26:26.456202 (ForestDnsZones
> failure), 2019/05/22 12:26:49.027595 (DomainDnsZones failure) and
> 2019/05/22 12:27:03.755312 (Schema panic)
> 
> -- 
> Mit freundlichen Grüßen, / Best Regards,
> Sven Schwedas, Systemadministrator
> ??? sven.schwedas at tao.at | ??? +43 680 301 7167
> TAO Digital   | Teil der TAO Beratungs- & Management GmbH
> Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
> A8020 Graz    | https://www.tao-digital.at
> 
>

On 22.05.19 15:31, L.P.H. van Belle wrote:
> Hai, 
> 
> Well, good that your more relax and releaved in pressure.. 
> Apoligies accepted. ;-), i know the fealing but do understand, we are
trying to help out..
> 
> As you know, it very frustration to ask the same things again, and you
showing them again
> But now as you showed with all the configs. It is needed.. (sorry) 
Yeah, now that I have the time for it. That's why I posted them all.
> So I had a quick look, and you up to some changes. All are fixable, no
worries,
> but it takes a bit of time and you need to be precise/accurate. 
> 
> For example, 
> 
> https://up.tao.at/u/samba/graz-dc-1b.info.txt 
> Hostname: graz-dc-1b
> DNS Domain:	
> FQDN: graz-dc-1b
> ipaddress: 169.254.93.226 192.168.17.66
> - Where is the DNS domain?  ( i'll add warning in the script there if
its missing.)
Fixed.
> Cause:  Checking file: /etc/hosts
> 
> 127.0.0.1 graz-dc-1b localhost
> - Which is wrong. 
> 
> - I would have expected to see: 
> 127.0.0.1 localhost
> 
> ::1     localhost ip6-localhost ip6-loopback
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> 
> 192.168.17.66	graz-dc-1b.ad.tao.at graz-dc-1b
Fixed.
> And this one, i should not see that, execpt if its a dhcp thats failing. 
> 169.254.93.226 NetName: LINKLOCAL-RFC3927-IANA-RESERVED
>
> And if you did disable IPv6, which you attempted, at least looks like it. 
>  inet6 fe80::b8de:f4ff:fe1e:11e5/64 scope link is on the interface, but the
ipv6 parts where missing in /etc/hosts.
No idea what systemd-networkd is even doing there. None of this is
intended… Fixed now.
> /etc/resolv.conf
> nameserver 192.168.17.1
> Only one DC? And you have 4.. 
For the record, that's not a DC, that's a dnsmasq on the firewall.
Requests for ad.tao.at and internal range RDNS requests are forwarded to
the samba servers, the rest goes to external servers.

DNSMasq is there to provide stuff not related to the samba domain, which
is mostly needed for client PCs. So switching over DCs and member
servers to using AD DCs directly is easy, but client PCs will need to
remain on DNSMasq. From what I've seen, all AD related queries resolve
fine with it, too.
> For every DC and per site, you have 2 sites correct? 
2 physical sites (192.168.17/24 and 192.168.16/24) in different subnets,
but they're organized as a single logical site in AD and VPN'd together
via default gateway.
> I suggest, per site, 2 DC of the same site, 1 DC of the remote as backup. 
> Look at this, explanation is below it. The DC1 with fsmo. 
>
> Now the 2 resolv.conf examples and /etc/hosts file, should make sure your
hostname and dnsdomain is correct.
> As shown per example this needs to be correct before we do anything else,
and yes, fix this for every server.
Should be fixed now.
> For the replication. 
> You see, i've kept in both resolv.conf files DC1 as first, you need
this for the first replication.
> After a reboot and at least 15 min-30 main waiting, on DC2, switch the DC1
and DC2 lines and set DC2 first.
> And this is for every server needed. I wont hurt if DC1 stays first, but
that could overload DC1 on DNS requests.
Shouldn't be an issue with how few clients are involved here, but I'll
keep an eye on it.
> For all DC's and members, all you need: ( i install krb5-user, define
the REALM and keep everythings default.
> So this is sufficient.  ( yes only that one default_realm line, others are
defaults )
>
> /etc/krb5.conf
> [libdefaults]
>         default_realm = AD.TAO.AT
> 
> Except, i noticed, 
> 	.tao.at = AD.TAO.AT
> 	tao.at = AD.TAO.AT
> That might need some more explanation why you added it.
> That helps me understanding your setup. SSO login with users email adresses
maybe?
Legacy setup had our domain on tao.at directly, I think we needed that
to smooth over the transition. But that was years ago and shouldn't be
needed any more.
> /etc/nsswitch.conf
> passwd:         files winbind
> group:          files winbind
> shadow:         files 			< winbind removed here. 
Fixed too, probably.
> /etc/samba/smb.conf
> 	ldap ssl = start tls
> 	ldap ssl ads = yes
> These are not supported in the AD setup. These are for an NT4DOM/ PDC/BDC
setup with ldap.
Might want to actually document that somewhere, this isn't clear from
either wiki nor manpage.
> And i like how you did split up the global part and the "real"
server config part.
> One im going to use also..  So noo, not only bad thin, also good things.. 
> 
> I noticed also: 
> 	#FIXME: Temporary to fix PHP shit
> 	ldap server require strong auth = no
> 
> # explain this to me, can offlist if needed. 
Some horribly obsolete PHP5 LDAP library breaks with SSL, as a
workaround we're SSH tunneling it until we can kill it. Which
*hopefully* should be sometimes this month anyway, so no point in fixing
it now.
> Because here, i think you forgot to also define /etc/ldap/ldap.conf 
> .. And i missed this in the script, i'll add this ..
Added as well.
> Last one, 
> [homes]
> 	msdfs root = yes
> 	msdfs proxy = \\graz-file\homes
> 
> 	As per MS advice, \\graz-file.ad.tao.at\users
> 
> 
> You might hit a bug here because [homes] on the DC's is not really
supported.
> Better might be [users] but this one needs planning... before you change
it.
> You understand that..
I don't think we're even using it nowadays, so I've gutted it from
all DCs.
> Installed packages:
> Missing attr 
> Which is a must install for the DC's. 
Should be fixed too.
> Now this was only 1 server but this is exact the same for all DC's 
> This must be corrected before we can even look at the DB replication
problem.
> Almost all server suffer from simular problems in the setup. 
> 
> I suggest start with above. That is the base. 
> For the members, hosts resolv.conf nsswitch.conf krb5.conf all need to be
fixed first.
> Scary, yes, i understand, start with DC1_fsmo DC. 
> 
> Do one at the time, check all the above, so make a checklist for this. 
They should be all modified accordingly now:

https://up.tao.at/u/samba/graz-file.info2.txt
https://up.tao.at/u/samba/graz-mail.info2.txt
https://up.tao.at/u/samba/graz-dc-sem.info2.txt
https://up.tao.at/u/samba/graz-dc-1b.info2.txt
https://up.tao.at/u/samba/villach-file.info2.txt
https://up.tao.at/u/samba/villach-mail.info2.txt
https://up.tao.at/u/samba/villach-dc-1a.info2.txt
https://up.tao.at/u/samba/villach-dc-bis.info2.txt

And now that business hours are over, also all restarted. I'll let them
do their thing overnight and check up on replication status tomorrow
morning.

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
✉ sven.schwedas at tao.at | ☎ +43 680 301 7167
TAO Digital   | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz    | https://www.tao-digital.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20190522/e477c933/signature.sig>

Hi Sven, 

Ok, before i back to my packaging, i reviewed the changes. 

You still missed a bit but it looks much better already, your completely there,
but we will get there.
And you really need to be more precise, why im telling that... 

Well the review. ( thats not all, im still reviewing the base only)  ... 
 
https://up.tao.at/u/samba/graz-file.info2.txt 

       Checking file: /etc/nsswitch.conf

passwd:         files winbind winbind
group:          files winbind winbind
shadow:         files winbind

Uhmm..  If your using puppet or ansible, then i suggest go over these files once
manualy.


https://up.tao.at/u/samba/graz-mail.info2.txt
Missing packages. :  apt install acl samba-dsdb-modules  samba-vfs-modules 
Yes, its possible to run without these, but for a domain member better to
install them.


/etc/hosts
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.0.1 localhost

192.168.17.79 graz-mail.ad.tao.at graz-mail mail.graz.tao.at
192.168.17.58  database.local
192.168.16.209 mail.villach.tao.at

Now the last 2 entries, if you resolving is correct, then the last too lines
should not be needed.
But, its not wrong, as long as the resolves also on the other servers where its
needed.


https://up.tao.at/u/samba/graz-dc-sem.info2.txt
You are probley having a good reason for it. 
	template shell = /bin/zsh 
Your (some) other server have template shell = /bin/bash 


https://up.tao.at/u/samba/graz-dc-1b.info2.txt

Debian 9.8 the others are at 9.9 ( apt dist-upgrade) 

# /etc/nsswitch.conf
passwd:         files winbind
group:          files winbind
shadow:         files winbind		<<  ?? You missed this one. 
gshadow:        files


https://up.tao.at/u/samba/villach-dc-1a.info2.txt
This computer is running Debian 9.0 .. You missed a lot of upgrades here. 
Missing acl package 


Well, it looks much better already. 

Now once all above is done, then next is. 
Compair your installed packages, you can do yourself. 

Now, this is a diff of you packages installed on the DC's. 
What do you see here... I see alot of differences. 


diff graz-dc-1b.info2.txt graz-dc-sem.info2.txt
11,21c11,24
< ii  libnss-winbind:amd64        2:4.5.16+dfsg-1                amd64       
Samba nameservice integration plugins
< ii  libpam-winbind:amd64        2:4.5.16+dfsg-1                amd64       
Windows domain authentication integration plugin
< ii  libwbclient0:amd64          2:4.5.16+dfsg-1                amd64       
Samba winbind client library
< ii  python-samba                2:4.5.16+dfsg-1                amd64       
Python bindings for Samba
< ii  samba                       2:4.5.16+dfsg-1                amd64       
SMB/CIFS file, print, and login server for Unix
< ii  samba-common                2:4.5.16+dfsg-1                all         
common files used by both the Samba server and client
< ii  samba-common-bin            2:4.5.16+dfsg-1                amd64       
Samba common files used by both the server and the client
< ii  samba-dsdb-modules          2:4.5.16+dfsg-1                amd64       
Samba Directory Services Database
< ii  samba-libs:amd64            2:4.5.16+dfsg-1                amd64       
Samba core libraries
< ii  samba-vfs-modules           2:4.5.16+dfsg-1                amd64       
Samba Virtual FileSystem plugins
< ii  winbind                     2:4.5.16+dfsg-1                amd64       
service to resolve user and group information from Windows NT servers
---
> ii  libnss-winbind:amd64        2:4.5.16+dfsg-1+deb9u2         amd64       
Samba nameservice integration plugins
> ii  libpam-krb5:amd64           4.7-4                          amd64       
PAM module for MIT Kerberos
> ii  libpam-winbind:amd64        2:4.5.16+dfsg-1+deb9u2         amd64       
Windows domain authentication integration plugin
> ii  libsmbclient:amd64          2:4.5.16+dfsg-1+deb9u2         amd64       
shared library for communication with SMB/CIFS servers
> ii  libwbclient0:amd64          2:4.5.16+dfsg-1+deb9u2         amd64       
Samba winbind client library
> ii  python-samba                2:4.5.16+dfsg-1+deb9u2         amd64       
Python bindings for Samba
> ii  samba                       2:4.5.16+dfsg-1+deb9u2         amd64       
SMB/CIFS file, print, and login server for Unix
> ii  samba-common                2:4.5.16+dfsg-1+deb9u2         all         
common files used by both the Samba server and client
> ii  samba-common-bin            2:4.5.16+dfsg-1+deb9u2         amd64       
Samba common files used by both the server and the client
> ii  samba-dsdb-modules          2:4.5.16+dfsg-1+deb9u2         amd64       
Samba Directory Services Database
> ii  samba-libs:amd64            2:4.5.16+dfsg-1+deb9u2         amd64       
Samba core libraries
> ii  samba-vfs-modules           2:4.5.16+dfsg-1+deb9u2         amd64       
Samba Virtual FileSystem plugins
> ii  smbclient                   2:4.5.16+dfsg-1+deb9u2         amd64       
command-line SMB/CIFS clients for Unix
> ii  winbind                     2:4.5.16+dfsg-1+deb9u2         amd64       
service to resolve user and group information from Windows NT servers

diff graz-dc-sem.info2.txt villach-dc-1a.info2.txt
2,24c2,17
< ii  acl                         2.2.52-3+b1                    amd64       
Access control list utilities
< ii  attr                        1:2.4.47-2+b2                  amd64       
Utilities for manipulating filesystem extended attributes
< ii  krb5-config                 2.6                            all         
Configuration files for Kerberos Version 5
< ii  krb5-user                   1.15-1+deb9u1                  amd64       
basic programs to authenticate using MIT Kerberos
< ii  libacl1:amd64               2.2.52-3+b1                    amd64       
Access control list shared library
< ii  libattr1:amd64              1:2.4.47-2+b2                  amd64       
Extended attribute shared library
< ii  libgssapi-krb5-2:amd64      1.15-1+deb9u1                  amd64       
MIT Kerberos runtime libraries - krb5 GSS-API Mechanism
< ii  libkrb5-3:amd64             1.15-1+deb9u1                  amd64       
MIT Kerberos runtime libraries
< ii  libkrb5support0:amd64       1.15-1+deb9u1                  amd64       
MIT Kerberos runtime libraries - Support library
< ii  libnss-winbind:amd64        2:4.5.16+dfsg-1+deb9u2         amd64       
Samba nameservice integration plugins
< ii  libpam-krb5:amd64           4.7-4                          amd64       
PAM module for MIT Kerberos
< ii  libpam-winbind:amd64        2:4.5.16+dfsg-1+deb9u2         amd64       
Windows domain authentication integration plugin
< ii  libsmbclient:amd64          2:4.5.16+dfsg-1+deb9u2         amd64       
shared library for communication with SMB/CIFS servers
< ii  libwbclient0:amd64          2:4.5.16+dfsg-1+deb9u2         amd64       
Samba winbind client library
< ii  python-samba                2:4.5.16+dfsg-1+deb9u2         amd64       
Python bindings for Samba
< ii  samba                       2:4.5.16+dfsg-1+deb9u2         amd64       
SMB/CIFS file, print, and login server for Unix
< ii  samba-common                2:4.5.16+dfsg-1+deb9u2         all         
common files used by both the Samba server and client
< ii  samba-common-bin            2:4.5.16+dfsg-1+deb9u2         amd64       
Samba common files used by both the server and the client
< ii  samba-dsdb-modules          2:4.5.16+dfsg-1+deb9u2         amd64       
Samba Directory Services Database
< ii  samba-libs:amd64            2:4.5.16+dfsg-1+deb9u2         amd64       
Samba core libraries
< ii  samba-vfs-modules           2:4.5.16+dfsg-1+deb9u2         amd64       
Samba Virtual FileSystem plugins
< ii  smbclient                   2:4.5.16+dfsg-1+deb9u2         amd64       
command-line SMB/CIFS clients for Unix
< ii  winbind                     2:4.5.16+dfsg-1+deb9u2         amd64       
service to resolve user and group information from Windows NT servers
---
> ii  attr                        1:2.4.47-2+b2                amd64       
Utilities for manipulating filesystem extended attributes
> ii  krb5-config                 2.6                          all         
Configuration files for Kerberos Version 5
> ii  krb5-user                   1.15-1                       amd64       
basic programs to authenticate using MIT Kerberos
> ii  libacl1:amd64               2.2.52-3+b1                  amd64       
Access control list shared library
> ii  libattr1:amd64              1:2.4.47-2+b2                amd64       
Extended attribute shared library
> ii  libgssapi-krb5-2:amd64      1.15-1                       amd64       
MIT Kerberos runtime libraries - krb5 GSS-API Mechanism
> ii  libkrb5-3:amd64             1.15-1                       amd64       
MIT Kerberos runtime libraries
> ii  libkrb5support0:amd64       1.15-1                       amd64       
MIT Kerberos runtime libraries - Support library
> ii  libwbclient0:amd64          2:4.5.6+dfsg-1               amd64       
Samba winbind client library
> ii  python-samba                2:4.5.6+dfsg-1               amd64       
Python bindings for Samba
> ii  samba                       2:4.5.6+dfsg-1               amd64       
SMB/CIFS file, print, and login server for Unix
> ii  samba-common                2:4.5.6+dfsg-1               all         
common files used by both the Samba server and client
> ii  samba-common-bin            2:4.5.6+dfsg-1               amd64       
Samba common files used by both the server and the client
> ii  samba-dsdb-modules          2:4.5.6+dfsg-1               amd64       
Samba Directory Services Database
> ii  samba-libs:amd64            2:4.5.6+dfsg-1               amd64       
Samba core libraries
> ii  winbind                     2:4.5.6+dfsg-1               amd64       
service to resolve user and group information from Windows NT servers
diff villach-dc-1a.info2.txt villach-dc-bis.info2.txt
2,17c2,24
< ii  attr                        1:2.4.47-2+b2                amd64       
Utilities for manipulating filesystem extended attributes
< ii  krb5-config                 2.6                          all         
Configuration files for Kerberos Version 5
< ii  krb5-user                   1.15-1                       amd64       
basic programs to authenticate using MIT Kerberos
< ii  libacl1:amd64               2.2.52-3+b1                  amd64       
Access control list shared library
< ii  libattr1:amd64              1:2.4.47-2+b2                amd64       
Extended attribute shared library
< ii  libgssapi-krb5-2:amd64      1.15-1                       amd64       
MIT Kerberos runtime libraries - krb5 GSS-API Mechanism
< ii  libkrb5-3:amd64             1.15-1                       amd64       
MIT Kerberos runtime libraries
< ii  libkrb5support0:amd64       1.15-1                       amd64       
MIT Kerberos runtime libraries - Support library
< ii  libwbclient0:amd64          2:4.5.6+dfsg-1               amd64       
Samba winbind client library
< ii  python-samba                2:4.5.6+dfsg-1               amd64       
Python bindings for Samba
< ii  samba                       2:4.5.6+dfsg-1               amd64       
SMB/CIFS file, print, and login server for Unix
< ii  samba-common                2:4.5.6+dfsg-1               all         
common files used by both the Samba server and client
< ii  samba-common-bin            2:4.5.6+dfsg-1               amd64       
Samba common files used by both the server and the client
< ii  samba-dsdb-modules          2:4.5.6+dfsg-1               amd64       
Samba Directory Services Database
< ii  samba-libs:amd64            2:4.5.6+dfsg-1               amd64       
Samba core libraries
< ii  winbind                     2:4.5.6+dfsg-1               amd64       
service to resolve user and group information from Windows NT servers
---
> ii  acl                            2.2.52-3+b1                    amd64    
Access control list utilities
> ii  attr                           1:2.4.47-2+b2                  amd64    
Utilities for manipulating filesystem extended attributes
> ii  krb5-config                    2.6                            all      
Configuration files for Kerberos Version 5
> ii  krb5-locales                   1.15-1+deb9u1                  all      
internationalization support for MIT Kerberos
> ii  krb5-user                      1.15-1+deb9u1                  amd64    
basic programs to authenticate using MIT Kerberos
> ii  libacl1:amd64                  2.2.52-3+b1                    amd64    
Access control list shared library
> ii  libattr1:amd64                 1:2.4.47-2+b2                  amd64    
Extended attribute shared library
> ii  libgssapi-krb5-2:amd64         1.15-1+deb9u1                  amd64    
MIT Kerberos runtime libraries - krb5 GSS-API Mechanism
> ii  libkrb5-3:amd64                1.15-1+deb9u1                  amd64    
MIT Kerberos runtime libraries
> ii  libkrb5support0:amd64          1.15-1+deb9u1                  amd64    
MIT Kerberos runtime libraries - Support library
> ii  libnss-winbind:amd64           2:4.5.16+dfsg-1+deb9u2         amd64    
Samba nameservice integration plugins
> ii  libpam-winbind:amd64           2:4.5.16+dfsg-1+deb9u2         amd64    
Windows domain authentication integration plugin
> ii  libsmbclient:amd64             2:4.5.16+dfsg-1+deb9u2         amd64    
shared library for communication with SMB/CIFS servers
> ii  libwbclient0:amd64             2:4.5.16+dfsg-1+deb9u2         amd64    
Samba winbind client library
> ii  python-samba                   2:4.5.16+dfsg-1+deb9u2         amd64    
Python bindings for Samba
> ii  samba                          2:4.5.16+dfsg-1+deb9u2         amd64    
SMB/CIFS file, print, and login server for Unix
> ii  samba-common                   2:4.5.16+dfsg-1+deb9u2         all      
common files used by both the Samba server and client
> ii  samba-common-bin               2:4.5.16+dfsg-1+deb9u2         amd64    
Samba common files used by both the server and the client
> ii  samba-dsdb-modules             2:4.5.16+dfsg-1+deb9u2         amd64    
Samba Directory Services Database
> ii  samba-libs:amd64               2:4.5.16+dfsg-1+deb9u2         amd64    
Samba core libraries
> ii  samba-vfs-modules              2:4.5.16+dfsg-1+deb9u2         amd64    
Samba Virtual FileSystem plugins
> ii  smbclient                      2:4.5.16+dfsg-1+deb9u2         amd64    
command-line SMB/CIFS clients for Unix
> ii  winbind                        2:4.5.16+dfsg-1+deb9u2         amd64    
service to resolve user and group information from Windows NT servers

For all DCs you need to fix/align the packages.
apt-get install samba samba-libs samba-vfs-modules samba-dsdb-modules winbind
ldb-tools acl attr

Optional but adviced: (sso) ssh logins: apt-get install libnss-winbind
libpam-winbind libpam-krb5
Optional but adviced:  Client tools: apt-get install smbclient 

And your probely are missing this file on most of your servers.
one i can add also to my script. 

/usr/share/pam-configs/winbind

echo "Name: Winbind NT/Active Directory authentication
Default: yes
Priority: 192
Auth-Type: Primary
Auth:
        [success=end default=ignore]    pam_winbind.so krb5_auth
krb5_ccache_type=FILE cached_login try_first_pass
Auth-Initial:
        [success=end default=ignore]    pam_winbind.so krb5_auth
krb5_ccache_type=FILE cached_login
Account-Type: Primary
Account:
        [success=end new_authtok_reqd=done default=ignore]      pam_winbind.so
Password-Type: Primary
Password:
        [success=end default=ignore]    pam_winbind.so try_authtok
try_first_pass
Password-Initial:
        [success=end default=ignore]    pam_winbind.so
Session-Type: Additional
Session:
        optional                        pam_winbind.so
" > /usr/share/pam-configs/winbind

And enable/disable it with: pam-auth-update
If needed, and you probely need it for ssh logins, select/enable winbind.

After this is done on all the DC's then i have 2 more things. 

How are you replication sysvol? 

How are you syncing the time between all DC'.s  ( and again need to add that
also to the script. )

Here i have only one advice. 
Make sure the time sources on all DCs are the same. 
And ntp.pool.org is not the same, textwise yes, ip wise no. 
I have seen more then 5 min differnces in the past. 
Here i suggest, use the NTP of you internet provider. 


What i now suggest, you have these files, as in these :XXXXX.info2.txt  for all
servers.
Use these and diff between the servers. 

Now beside the above changes which need to be done first. 
Next will be.

More standardization of you smb.conf for all servers. 
Your setup with the includes are good, but the settings are a bit off. 
And with off i mean, not wrong, but different, but this can cause problems. 
SO the more these are the same the less problems.
And, keep site/server specific differences in these site includes, thats good. 

And some comments below in between. 
> -----Oorspronkelijk bericht-----
> Van: Sven Schwedas [mailto:sven.schwedas at tao.at] 
> Verzonden: woensdag 22 mei 2019 17:53
> Aan: L.P.H. van Belle; samba at lists.samba.org
> Onderwerp: Re: Various AD issues; summary
> 
> 
> 
> On 22.05.19 15:31, L.P.H. van Belle wrote:
> > Hai, 
> > 
> > Well, good that your more relax and releaved in pressure.. 
> > Apoligies accepted. ;-), i know the fealing but do 
> understand, we are trying to help out.. 
> > 
> > As you know, it very frustration to ask the same things 
> again, and you showing them again
> > But now as you showed with all the configs. It is needed.. (sorry) 
> 
> Yeah, now that I have the time for it. That's why I posted them all.
> 
> > So I had a quick look, and you up to some changes. All are 
> fixable, no worries,
> > but it takes a bit of time and you need to be precise/accurate. 
> > 
> > For example, 
> > 
> > https://up.tao.at/u/samba/graz-dc-1b.info.txt 
> > Hostname: graz-dc-1b
> > DNS Domain:	
> > FQDN: graz-dc-1b
> > ipaddress: 169.254.93.226 192.168.17.66
> > - Where is the DNS domain?  ( i'll add warning in the 
> script there if its missing.) 
> 
> Fixed.
> 
> > Cause:  Checking file: /etc/hosts
> > 
> > 127.0.0.1 graz-dc-1b localhost
> > - Which is wrong. 
> > 
> > - I would have expected to see: 
> > 127.0.0.1 localhost
> > 
> > ::1     localhost ip6-localhost ip6-loopback
> > ff02::1 ip6-allnodes
> > ff02::2 ip6-allrouters
> > 
> > 192.168.17.66	graz-dc-1b.ad.tao.at graz-dc-1b
> 
> Fixed.
> 
> > And this one, i should not see that, execpt if its a dhcp 
> thats failing. 
> > 169.254.93.226 NetName: LINKLOCAL-RFC3927-IANA-RESERVED
> >
> > And if you did disable IPv6, which you attempted, at least 
> looks like it. 
> >  inet6 fe80::b8de:f4ff:fe1e:11e5/64 scope link is on the 
> interface, but the ipv6 parts where missing in /etc/hosts. 
> 
> No idea what systemd-networkd is even doing there. None of this is
> intended? Fixed now.
> 
> > /etc/resolv.conf
> > nameserver 192.168.17.1
> > Only one DC? And you have 4.. 
> 
> For the record, that's not a DC, that's a dnsmasq on the firewall.
> Requests for ad.tao.at and internal range RDNS requests are 
> forwarded to
> the samba servers, the rest goes to external servers.
> 
> DNSMasq is there to provide stuff not related to the samba 
> domain, which
> is mostly needed for client PCs. So switching over DCs and member
> servers to using AD DCs directly is easy, but client PCs will need to
> remain on DNSMasq. From what I've seen, all AD related queries resolve
> fine with it, too.
What you her is fine, but resolv.conf on the DCs need to resolve to the
DC's.
Then you can add the ip of the dnsmask forwarder in the smb.conf 
Now, these are different also.

> 
> > For every DC and per site, you have 2 sites correct? 
> 
> 2 physical sites (192.168.17/24 and 192.168.16/24) in 
> different subnets,
> but they're organized as a single logical site in AD and 
> VPN'd together via default gateway.
I've dont same here..  I have one site but 4 subdomains and 6 subnets. 
Now, that VPN part, keep that in mind, after all things are done i have
questions about that one also.
But lets not do that all at once. 
> 
> > I suggest, per site, 2 DC of the same site, 1 DC of the 
> remote as backup. 
> > Look at this, explanation is below it. The DC1 with fsmo. 
> >
> > Now the 2 resolv.conf examples and /etc/hosts file, should 
> make sure your hostname and dnsdomain is correct. 
> > As shown per example this needs to be correct before we do 
> anything else, and yes, fix this for every server. 
> 
> Should be fixed now.
> 
> > For the replication. 
> > You see, i've kept in both resolv.conf files DC1 as first, 
> you need this for the first replication. 
> > After a reboot and at least 15 min-30 main waiting, on DC2, 
> switch the DC1 and DC2 lines and set DC2 first. 
> > And this is for every server needed. I wont hurt if DC1 
> stays first, but that could overload DC1 on DNS requests.
> 
> Shouldn't be an issue with how few clients are involved here, but
I'll
> keep an eye on it.
The dnsmask should help in avoiding that. 
> 
> > For all DC's and members, all you need: ( i install 
> krb5-user, define the REALM and keep everythings default. 
> > So this is sufficient.  ( yes only that one default_realm 
> line, others are defaults ) 
> >
> > /etc/krb5.conf
> > [libdefaults]
> >         default_realm = AD.TAO.AT
> > 
> > Except, i noticed, 
> > 	.tao.at = AD.TAO.AT
> > 	tao.at = AD.TAO.AT
> > That might need some more explanation why you added it.
> > That helps me understanding your setup. SSO login with 
> users email adresses maybe? 
> 
> Legacy setup had our domain on tao.at directly, I think we needed that
> to smooth over the transition. But that was years ago and shouldn't be
> needed any more.
Good, verify it and if its not needed remove it.
> 
> > /etc/nsswitch.conf
> > passwd:         files winbind
> > group:          files winbind
> > shadow:         files 			< winbind removed here. 
> 
> Fixed too, probably.
> 
> > /etc/samba/smb.conf
> > 	ldap ssl = start tls
> > 	ldap ssl ads = yes
> > These are not supported in the AD setup. These are for an 
> NT4DOM/ PDC/BDC setup with ldap. 
> 
> Might want to actually document that somewhere, this isn't clear from
> either wiki nor manpage.
.. I think its "not clear" but work is done/will be done to improve
that.
> 
> > And i like how you did split up the global part and the 
> "real" server config part. 
> > One im going to use also..  So noo, not only bad thin, also 
> good things.. 
> > 
> > I noticed also: 
> > 	#FIXME: Temporary to fix PHP shit
> > 	ldap server require strong auth = no
> > 
> > # explain this to me, can offlist if needed. 
> 
> Some horribly obsolete PHP5 LDAP library breaks with SSL, as a
> workaround we're SSH tunneling it until we can kill it. Which
> *hopefully* should be sometimes this month anyway, so no 
> point in fixing it now.
Hm, your using debian stretch, then why are you useing jessie's php5? 
Old web app that needed php5? 

Stretch its default is php7.0 
One that might help fixing that php5-ldap bug is, use the surg.org packages. 
https://tecadmin.net/install-php-debian-9-stretch/ 
Just a suggestion, you can run php5.6 7.0 7.1 7.2 7.3 on stretch. 
It works the same, but it might fix the bug in 5.6. 
Just... DONT install that now, first fix all samba things. 

> 
> > Because here, i think you forgot to also define /etc/ldap/ldap.conf 
> > .. And i missed this in the script, i'll add this ..
> 
> Added as well.
> 
> > Last one, 
> > [homes]
> > 	msdfs root = yes
> > 	msdfs proxy = \\graz-file\homes
> > 
> > 	As per MS advice, \\graz-file.ad.tao.at\users
> > 
> > 
> > You might hit a bug here because [homes] on the DC's is not 
> really supported. 
> > Better might be [users] but this one needs planning... 
> before you change it. 
> > You understand that..
> 
> I don't think we're even using it nowadays, so I've gutted it 
> from all DCs.
> 
> > Installed packages:
> > Missing attr 
> > Which is a must install for the DC's. 
> 
> Should be fixed too.
> 
> > Now this was only 1 server but this is exact the same for all DC's
> > This must be corrected before we can even look at the DB 
> replication problem. 
> > Almost all server suffer from simular problems in the setup. 
> > 
> > I suggest start with above. That is the base. 
> > For the members, hosts resolv.conf nsswitch.conf krb5.conf 
> all need to be fixed first. 
> > Scary, yes, i understand, start with DC1_fsmo DC. 
> > 
> > Do one at the time, check all the above, so make a 
> checklist for this. 
> 
> They should be all modified accordingly now:
> 
> https://up.tao.at/u/samba/graz-file.info2.txt
> https://up.tao.at/u/samba/graz-mail.info2.txt
> https://up.tao.at/u/samba/graz-dc-sem.info2.txt
> https://up.tao.at/u/samba/graz-dc-1b.info2.txt
> https://up.tao.at/u/samba/villach-file.info2.txt
> https://up.tao.at/u/samba/villach-mail.info2.txt
> https://up.tao.at/u/samba/villach-dc-1a.info2.txt
> https://up.tao.at/u/samba/villach-dc-bis.info2.txt
> 
> And now that business hours are over, also all restarted. 
> I'll let them
> do their thing overnight and check up on replication status tomorrow
> morning.
> 
> -- 
> Mit freundlichen Grüßen, / Best Regards,
> Sven Schwedas, Systemadministrator
> ??? sven.schwedas at tao.at | ??? +43 680 301 7167
> TAO Digital   | Teil der TAO Beratungs- & Management GmbH
> Lendplatz 45  | FN 213999f/Klagenfurt, FB-Gericht Villach
> A8020 Graz    | https://www.tao-digital.at
> 
>