samba - Jul 2018 - Failed to establish your Kerberos Ticket cache due time differences with the domain controller

On Sat, 21 Jul 2018 18:59:08 +0100
Rowland Penny via samba <samba at lists.samba.org> wrote:
> On Sat, 21 Jul 2018 18:30:48 +0100
> Roy Eastwood via samba <samba at lists.samba.org> wrote:
> 
> > Thanks Rowland.
> > 
> > > -----Original Message-----
> > > From: samba [mailto:samba-bounces at lists.samba.org] On Behalf
Of
> > > Rowland Penny via samba
> > > Sent: 21 July 2018 18:04
> > > To: samba at lists.samba.org
> > > Subject: Re: [Samba] Failed to establish your Kerberos Ticket
> > > cache due time differences with the domain controller
> > > 
> > > On Sat, 21 Jul 2018 17:36:14 +0100
> > > Roy Eastwood via samba <samba at lists.samba.org> wrote:
> > > 
> > > > >
> > > > > Try restarting Samba on 'debian-vb'.
> > > > > If this doesn't help, try 'samba-tool
dbcheck' and compare the
> > > > > two DC's with 'samba-tool ldapcmp'
> > > > >
> > > > > Rowland
> > > > >
> > > >
> > > > OK, have tried that but no change.   I used Louis' 
script:
> > > > samba-check-db-repl.sh which includes samba-tool ldapcmp and
> > > > samba-tool drs showrepl it passes both tests.
> > > >
> > > > Roy
> > > >
> > > >
> > > 
> > > Did you run 'samba-tool dbcheck' ? Louis's script
doesn't do this.
> > > 
> > > Was this machine provisioned quite a few versions ago ? and then
> > > updated in place ?
> > > 'samba-tool time' was changed at sometime (cannot just
when) and a
> > > python module was replaced, but 'make install' did not
remove the
> > > old python script. Check if you have:
> > > 
> > >
'/usr/local/samba/lib/python2.7/site-packages/samba/netcmd/time.py'
> > > 
> > > If you have, delete it.
> > > 
> > > Rowland
> > > 
> > Sorry, yes I ,meant to say, I ran samba-tool dbcheck and initially
> > it threw up some errors about deleted items (I had recently demoted
> > a DC after the pi-dc was joined) but these were repaired with the
> > --fix option.   I tried the samba-tool time commands afterwards
> > with the same result.    I also had to clean up all the DNS records
> > as the demote command doesn't tidy things up properly.
> > 
> > Whist this is a new domain provision with v 4.8.3, the machine has
> > had versions going back to 4.7.4 compiled and installed (albeit with
> > different domains).  I used make uninstall on the last version of
> > samba before installing 4.8.3 if that makes any difference.   I
> > checked for that time.py file and it's not in that folder (or
> > anywhere else according to find).
> > 
> > Roy
> > 
> > 
> 
> No, it wouldn't have been there, 4.7.4 isn't old enough.
> 
> When you built Samba, did you have all the correct packages installed,
> see here:
> 
>
https://wiki.samba.org/index.php/Package_Dependencies_Required_to_Build_Samba#Debian_.2F_Ubuntu
> 
> Is Apparmor installed, or a firewall ?
> 
> Rowland
> 
Another thought, could this be an authentication problem ? try adding
'-U Administrator' and see if this helps.

Rowland

> > >
> > > Whist this is a new domain provision with v 4.8.3, the machine
has
> > > had versions going back to 4.7.4 compiled and installed (albeit
with
> > > different domains).  I used make uninstall on the last version of
> > > samba before installing 4.8.3 if that makes any difference.   I
> > > checked for that time.py file and it's not in that folder (or
> > > anywhere else according to find).
> > >
> > > Roy
> > >
> > >
> >
> > No, it wouldn't have been there, 4.7.4 isn't old enough.
> >
> > When you built Samba, did you have all the correct packages installed,
> > see here:
> >
> >
> https://wiki.samba.org/index.php/Package_Dependencies_Required_to_Build_
> Samba#Debian_.2F_Ubuntu
Yes, I copied the list from the WiKi (when I installed 4.7.4) but haven't
reviewed it since - so if there's been additions since, that may be an
issue.
> >
> > Is Apparmor installed, or a firewall ?
> >
No, neither.  Nor SELinux.
> > Rowland
> >
> 
> Another thought, could this be an authentication problem ? try adding
> '-U Administrator' and see if this helps.
> 
> Rowland
I did this and it worked OK.   Then I did it without the -U Administrator and it
also worked!   I have no idea why it now works  as I haven't actually
changed anything, other than issuing net cache flush.   BUT the original problem
remains!   But even more confusing- see the transcript below:

login as: roy
roy at 192.168.2.4's password:
Failed to establish your Kerberos Ticket cache due time differences
with the domain controller.  Please verify the system time.

Linux pi-dc 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jul 21 19:55:43 2018 from 192.168.2.240
MICROLYNX\roy at pi-dc:~ $ samba-tool time
ldb: Unable to open tdb '/usr/local/samba/private/secrets.ldb':
Permission denied
ldb: Failed to connect to '/usr/local/samba/private/secrets.ldb' with
backend 'tdb': Unable to open tdb
'/usr/local/samba/private/secrets.ldb': Permission denied
Could not find machine account in secrets database: Failed to fetch machine
account password from secrets.ldb: Could not open secrets.ldb and failed to open
/usr/local/samba/private/secrets.tdb: NT_STATUS_CANT_ACCESS_DOMAIN_INFO
Thu Nov  3 17:17:15 2016 GMT
MICROLYNX\roy at pi-dc:~ $ sudo samba-tool time
[sudo] password for MICROLYNX\roy:
Failed to establish your Kerberos Ticket cache due time differences
with the domain controller.  Please verify the system time.

Sat Jul 21 20:02:24 2018 BST
MICROLYNX\roy at pi-dc:~ $ sudo samba-tool time
Sat Jul 21 20:03:08 2018 BST
MICROLYNX\roy at pi-dc:~ $

As you can see one time it fails, then it works!

So next I stopped the samba-ad-dc service on Debian-vb.   I then couldn't
log in to pi-dc with my AD user.   Even restarting the service on pi-dc had no
effect.   However, running pam-auth-update again, allowed me to login once more
with Debian-vb off.   As such the time message disappears on login and when
running samba-tool time.

Restarting the samba-ad-dc service on Debian-vb brings the error message back
when logging on to pi-dc.   So I assume it's some kind of interaction
between the two DCs.

I'm getting confused...:-)

Roy

Hai, 

I've reading this thread more closely. 

I suggest you try the followoing.

Check the servers hardware clock in the bios first.
Set these within 5 min, if they are not about the same.

Run :  dpkg-reconfigure tzdata 
Check/set the correct timezones on both servers, and both servers should show
you the same date/time and (optional) zone.

Run : ntpq -p
Check the offset on both servers. 

Add :  winbind refresh tickets = yes to you smb.conf

If these are member servers, make sure you have only the server lines pointed to
you AD DC's.
If these are DC's, them make sure the both point to the same ntp servers. 
Dont use pool servers for the AD DC's, but thats my advice. 

Reboot the servers, first DC with FSMO, if there are DC's involved. 
This wil clear kerberos cache tickets and should make sure the time is really
set ok.

Login again, do have still have the time message, if yes.. 

Check : 
/etc/pam.d/common-auth
You should see a line like : 
auth    [success=1 default=ignore]      pam_winbind.so krb5_auth
krb5_ccache_type=FILE cached_login try_first_pass

Change that one to 
auth    [success=1 default=ignore]      pam_winbind.so krb5_auth try_first_pass
Try again, put it back again after a successull login without messages. 

When this is done. 
Now go clear the kerberos cache. 
Run : klist -ef
Check the ETYPES and Flags. 


Now mail us back with the results. 
Above should determine if its and old kerberos cache problem or ntp problem. 


Greetz, 

Louis



> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens Roy 
> Eastwood via samba
> Verzonden: zaterdag 21 juli 2018 21:28
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Failed to establish your Kerberos 
> Ticket cache due time differences with the domain controller
> 
> > > >
> > > > Whist this is a new domain provision with v 4.8.3, the 
> machine has
> > > > had versions going back to 4.7.4 compiled and installed 
> (albeit with
> > > > different domains).  I used make uninstall on the last 
> version of
> > > > samba before installing 4.8.3 if that makes any difference. 
I
> > > > checked for that time.py file and it's not in that
folder (or
> > > > anywhere else according to find).
> > > >
> > > > Roy
> > > >
> > > >
> > >
> > > No, it wouldn't have been there, 4.7.4 isn't old enough.
> > >
> > > When you built Samba, did you have all the correct 
> packages installed,
> > > see here:
> > >
> > >
> > 
> https://wiki.samba.org/index.php/Package_Dependencies_Required
> _to_Build_
> > Samba#Debian_.2F_Ubuntu
> 
> Yes, I copied the list from the WiKi (when I installed 4.7.4) 
> but haven't reviewed it since - so if there's been additions 
> since, that may be an issue.
> 
> > >
> > > Is Apparmor installed, or a firewall ?
> > >
> No, neither.  Nor SELinux.
> 
> > > Rowland
> > >
> > 
> > Another thought, could this be an authentication problem ? 
> try adding
> > '-U Administrator' and see if this helps.
> > 
> > Rowland
> 
> I did this and it worked OK.   Then I did it without the -U 
> Administrator and it also worked!   I have no idea why it now 
> works  as I haven't actually changed anything, other than 
> issuing net cache flush.   BUT the original problem remains!  
>  But even more confusing- see the transcript below:
> 
> login as: roy
> roy at 192.168.2.4's password:
> Failed to establish your Kerberos Ticket cache due time differences
> with the domain controller.  Please verify the system time.
> 
> Linux pi-dc 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l
> 
> The programs included with the Debian GNU/Linux system are 
> free software;
> the exact distribution terms for each program are described in the
> individual files in /usr/share/doc/*/copyright.
> 
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
> Last login: Sat Jul 21 19:55:43 2018 from 192.168.2.240
> MICROLYNX\roy at pi-dc:~ $ samba-tool time
> ldb: Unable to open tdb 
> '/usr/local/samba/private/secrets.ldb': Permission denied
> ldb: Failed to connect to 
> '/usr/local/samba/private/secrets.ldb' with backend 'tdb': 
> Unable to open tdb '/usr/local/samba/private/secrets.ldb': 
> Permission denied
> Could not find machine account in secrets database: Failed to 
> fetch machine account password from secrets.ldb: Could not 
> open secrets.ldb and failed to open 
> /usr/local/samba/private/secrets.tdb: 
> NT_STATUS_CANT_ACCESS_DOMAIN_INFO
> Thu Nov  3 17:17:15 2016 GMT
> MICROLYNX\roy at pi-dc:~ $ sudo samba-tool time
> [sudo] password for MICROLYNX\roy:
> Failed to establish your Kerberos Ticket cache due time differences
> with the domain controller.  Please verify the system time.
> 
> Sat Jul 21 20:02:24 2018 BST
> MICROLYNX\roy at pi-dc:~ $ sudo samba-tool time
> Sat Jul 21 20:03:08 2018 BST
> MICROLYNX\roy at pi-dc:~ $
> 
> As you can see one time it fails, then it works!
> 
> So next I stopped the samba-ad-dc service on Debian-vb.   I 
> then couldn't log in to pi-dc with my AD user.   Even 
> restarting the service on pi-dc had no effect.   However, 
> running pam-auth-update again, allowed me to login once more 
> with Debian-vb off.   As such the time message disappears on 
> login and when running samba-tool time.
> 
> Restarting the samba-ad-dc service on Debian-vb brings the 
> error message back when logging on to pi-dc.   So I assume 
> it's some kind of interaction between the two DCs.
> 
> I'm getting confused...:-)
> 
> Roy
> 
> 
> 
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
> 
>

On Mon, 23 Jul 2018 09:28:37 +0200
"L.P.H. van Belle via samba" <samba at lists.samba.org> wrote:
> Hai, 
> 
> I've reading this thread more closely. 
> 
> I suggest you try the followoing.
> 
> Check the servers hardware clock in the bios first.
An rpi doesn't have a bios ;-)
> Set these within 5 min, if they are not about the same.
> 
> Run :  dpkg-reconfigure tzdata 
> Check/set the correct timezones on both servers, and both servers
> should show you the same date/time and (optional) zone.
> 
> Run : ntpq -p
> Check the offset on both servers. 
> 
> Add :  winbind refresh tickets = yes to you smb.conf
> 
> If these are member servers, make sure you have only the server lines
> pointed to you AD DC's. If these are DC's, them make sure the both
> point to the same ntp servers. Dont use pool servers for the AD DC's,
> but thats my advice. 
They are both DC's and I use pool servers without any problems, of
course YMMV.
> 
> Reboot the servers, first DC with FSMO, if there are DC's involved. 
> This wil clear kerberos cache tickets and should make sure the time
> is really set ok.
Worth trying
 
Rowland

Thanks Louis.   Results below.
 
> Hai,
> 
> I've reading this thread more closely.
> 
> I suggest you try the followoing.
> 
> Check the servers hardware clock in the bios first.
> Set these within 5 min, if they are not about the same.
> 
There no RTC in the pi;  the other DC is running in a VM with RTC set to UTC.  
I have disabled the guest from getting the time from the host OS.
> Run :  dpkg-reconfigure tzdata
> Check/set the correct timezones on both servers, and both servers should
show
> you the same date/time and (optional) zone.
> 
Done, both show the TZ to be correct ie Europe/London and local and UTC times
are correct and identical on both DCs.
> Run : ntpq -p
> Check the offset on both servers.
Don't have ntpq (part of ntp package?) but ran chronyc sources with the
following results:

root at pi-dc:~# chronyc sources
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample
==============================================================================^+
www.bhay.org                  2   6   377    42  -2411us[-2629us] +/-   14ms
^* 85.199.214.100                1   6   377    41   +673us[ +455us] +/- 6491us
^+ 213.246.159.21                1   6   377    42   +340us[ +123us] +/-   16ms

root at debian-vb:~# chronyc sources
210 Number of sources = 3
MS Name/IP address         Stratum Poll Reach LastRx Last sample
==============================================================================^*
85.199.214.102                1   6   377    42   -357us[ -354us] +/- 7721us
^+ server1.quickdrivingtest>     1   6   377    41   -617us[ -617us] +/-
6199us
^+ time.netweaver.uk             2   6   377    41   +323us[ +323us] +/-   12ms

Both DCs are configured to use the same servers (0.uk.pool.ntp.org,
1.uk.pool.ntp.org and 2.uk.pool.ntp.org)
> 
> Add :  winbind refresh tickets = yes to you smb.conf
Done.
> 
> If these are member servers, make sure you have only the server lines
pointed
> to you AD DC's.
> If these are DC's, them make sure the both point to the same ntp
servers.
yes, see above
> Dont use pool servers for the AD DC's, but thats my advice.
OK, will try Stratum1 and see what happens, but for now here are the results so
far.
> 
> Reboot the servers, first DC with FSMO, if there are DC's involved.
> This wil clear kerberos cache tickets and should make sure the time is
really set
> ok.
> 
> Login again, do have still have the time message, if yes..
No change, message still there.   Even with the other DC switched off (the one
with FSMO roles).
> 
> Check :
> /etc/pam.d/common-auth
> You should see a line like :
> auth    [success=1 default=ignore]      pam_winbind.so krb5_auth
> krb5_ccache_type=FILE cached_login try_first_pass
> 
> Change that one to
> auth    [success=1 default=ignore]      pam_winbind.so krb5_auth
try_first_pass
> Try again, put it back again after a successull login without messages.
> 
Done, that but still get the warning even with the shorter version.    Never
able to logon without the warning, so put it back anyway.
The only way I have found not to get the message is to remove the krb5_auth (and
the other ones) completely.   But then we are not using Kerberos.
> When this is done.
> Now go clear the kerberos cache.
> Run : klist -ef
> Check the ETYPES and Flags.
> 
As roy (after logging in and getting the message:
Failed to establish your Kerberos Ticket cache due time differences
with the domain controller.  Please verify the system time.
MICROLYNX\roy at pi-dc:~ $ klist -ef
klist: No credentials cache found (filename: /tmp/krb5cc_3000022)

So generate a ticket:
MICROLYNX\roy at pi-dc:~ $ kinit roy
Password for roy at MICROLYNX.ORG:
MICROLYNX\roy at pi-dc:~ $ klist -ef
Ticket cache: FILE:/tmp/krb5cc_3000022
Default principal: roy at MICROLYNX.ORG

Valid starting     Expires            Service principal
23/07/18 21:25:51  24/07/18 07:25:51  krbtgt/MICROLYNX.ORG at MICROLYNX.ORG
        renew until 24/07/18 21:25:48, Flags: RIA
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
> 
> Now mail us back with the results.
> Above should determine if its and old kerberos cache problem or ntp
problem.
> 
> 
> Greetz,
> 
> Louis
> 
Cheers,

Roy