samba - Dec 2019 - Failed to find [principal](kvno 4) in keytab MEMORY:cifs_srv_keytab (arcfour-hmac-md5)]

On Wed, Dec 18, 2019 at 9:52 AM Rowland penny via samba <
samba at lists.samba.org> wrote:
> On 18/12/2019 14:34, Jonathon Reinhart wrote:
> > On Wed, Dec 18, 2019 at 9:13 AM Rowland penny via samba
> > <samba at lists.samba.org <mailto:samba at
lists.samba.org>> wrote:
> >
> >     Problem is, and as I said, Samba  4.3.x is EOL as far as Samba is
> >     concerned and if you have found a bug in it, it is very, very
> >     unlikely
> >     to get fixed, unless it is still in a later, supported, Samba
> >     version.
> >
> >
> > Of course; I wouldn't expect any more patches for 4.3.x. I've
dug
> > through a lot of the code and most of it is identical to latest
> > master. So to word it another way: I'm trying to see if this is a
> > known bug that has been fixed since 4.3.x, or if this instance is
> > highlighting a new unknown bug that could still exist in master.
> >
> > I understand your role on the list is a first line of response; keeper
> > of the gates sort of thing. Is there a more appropriate channel where
> > I can get some input from developers familiar with this part of the
code?
>
> Yes and I am try to point you in the same direction that the other Samba
> team members will, upgrade and see if the 'bug' is still there. If
it
> is, then we will need log level 10 output etc and a bug report. Your
> 'bug' may have been fixed since 4.3.x and if it has been then your
> problem will be gone, if it hasn't, then it will never get fixed in
> 4.3.x, but it should be in supported versions.
>
> Rowland
>
I updated to FreeNAS 11.1u7 which shows samba at "Version
4.7.0-GIT-ea139bffada-FreeNAS".

The issue persists just as it did on the old version.

Can anyone answer my questions about the in-memory keytab? How can two
clients both use the same service principal name (and kvno) but one can't
be found in the keytab?

Thanks,
Jonathon

>
> I updated to FreeNAS 11.1u7 which shows samba at "Version
> 4.7.0-GIT-ea139bffada-FreeNAS".
>
> The issue persists just as it did on the old version.
>
> Can anyone answer my questions about the in-memory keytab? How can two
> clients both use the same service principal name (and kvno) but one
can't
> be found in the keytab?
>
I'm starting to suspect this has something to do with kerberos
encryption types.

Most of the errors in the log reference arcfour-hmac-md5, but I see others
(with higher kvno) that reference aes256-cts-hmac-sha1-96.

Some additional information that may be relevant:
- The client failures weren't immediate; clients would slowly drop one by
one
- This domain and forest functional level were previously on Server 2008
R2, and recently upgraded to Server 2016 after the DCs were upgraded to
Windows Server 2016

Just now my Windows 10 machine failed to connect. "klist" showed a
ticket
for "cifs/nas01 @ EXAMPLE.com" that used KerbTicket Encryption Type:
RSADSI
RC4-HMAC(NT).  I issued "klist purge" and successfully reconnected.
The
ticket is now of type AES-256-CTS-HMAC-SMA1-96.

I'm starting to consider the nuclear option:
- Disable directory services on FreeNAS
- Nuke some Samba state (what files? secrets.tdb?)
- Delete the computer account
- Re-join

On 27/12/2019 17:06, Jonathon Reinhart wrote:
> On Wed, Dec 18, 2019 at 9:52 AM Rowland penny via samba 
> <samba at lists.samba.org <mailto:samba at lists.samba.org>>
wrote:
>
>     On 18/12/2019 14:34, Jonathon Reinhart wrote:
>     > On Wed, Dec 18, 2019 at 9:13 AM Rowland penny via samba
>     > <samba at lists.samba.org <mailto:samba at
lists.samba.org>
>     <mailto:samba at lists.samba.org <mailto:samba at
lists.samba.org>>> wrote:
>     >
>     >? ? ?Problem is, and as I said, Samba? 4.3.x is EOL as far as
>     Samba is
>     >? ? ?concerned and if you have found a bug in it, it is very, very
>     >? ? ?unlikely
>     >? ? ?to get fixed, unless it is still in a later, supported, Samba
>     >? ? ?version.
>     >
>     >
>     > Of course; I wouldn't expect any more patches for 4.3.x.
I've dug
>     > through a lot of the code and most of it is identical to latest
>     > master. So to word it another way: I'm trying to see if this
is a
>     > known bug that has been fixed since 4.3.x, or if this instance is
>     > highlighting a new unknown bug that could still exist in master.
>     >
>     > I understand your role on the list is a first line of response;
>     keeper
>     > of the gates sort of thing. Is there a more appropriate channel
>     where
>     > I can get some input from developers familiar with this part of
>     the code?
>
>     Yes and I am try to point you in the same direction that the other
>     Samba
>     team members will, upgrade and see if the 'bug' is still there.
If it
>     is, then we will need log level 10 output etc and a bug report. Your
>     'bug' may have been fixed since 4.3.x and if it has been then
your
>     problem will be gone, if it hasn't, then it will never get fixed in
>     4.3.x, but it should be in supported versions.
>
>     Rowland
>
>
> I updated to FreeNAS 11.1u7 which shows samba at "Version 
> 4.7.0-GIT-ea139bffada-FreeNAS".
>
> The issue persists just as it did on the old version.
>
> Can anyone answer my questions about the in-memory keytab? How can two 
> clients both use the same service principal name (and kvno) but one 
> can't be found in the keytab?
>
> Thanks,
> Jonathon
Not sure what is going on here, that Samba version appears to be a 
Freenas version (and is still EOL), but the release notes here: 
https://www.ixsystems.com/blog/library/freenas-11-2-u7/ clearly states 
that the Samba version is now 4.9.15 (which is still supported by Samba)

There are a few ways to mount a share with kerberos, how are you doing it ?

Whichever way, it usually relies on the server having an SPN in the 
format cifs/fqdn at REALM

If one client works and another doesn't, I would be checking to see if 
there are any differences between the clients.

If after all this, it still doesn't work and you are using a supported 
Samba version, then I would open a bug report, giving as much data as 
possible (log level 10 output, network traces etc)

Rowland

On Fri, Dec 27, 2019 at 1:12 PM Rowland penny via samba <
samba at lists.samba.org> wrote:
> On 27/12/2019 17:06, Jonathon Reinhart wrote:
> > I updated to FreeNAS 11.1u7 which shows samba at "Version
> > 4.7.0-GIT-ea139bffada-FreeNAS".
> >
> > The issue persists just as it did on the old version.
> >
> > Can anyone answer my questions about the in-memory keytab? How can two
> > clients both use the same service principal name (and kvno) but one
> > can't be found in the keytab?
> >
> > Thanks,
> > Jonathon
>
> Not sure what is going on here, that Samba version appears to be a
> Freenas version (and is still EOL), but the release notes here:
> https://www.ixsystems.com/blog/library/freenas-11-2-u7/ clearly states
> that the Samba version is now 4.9.15 (which is still supported by Samba)
>
I was coming from FreeNAS 9.10, so I first updated to FreeNAS 11.1 which
has Samba 4.7 (not 11.2 which has Samba 4.9). I will continue the upgrade
path to 11.2, but wanted to stop here at 11.1 and take a look...

> here are a few ways to mount a share with kerberos, how are you doing it ?
>
Entering \\nas01.example.com into Windows Explorer from a domain-joined
client machine.

> Whichever way, it usually relies on the server having an SPN in the
> format cifs/fqdn at REALM
>
But aside from joining the fileserver, an admin doesn't have to do anything
as long as there are no CNAME aliases being used, right? (IOW clients are
accessing via the "joined" name.)

> If one client works and another doesn't, I would be checking to see if
> there are any differences between the clients.
>
I've been trying unsuccessfully to do that. What's weird is that it also
seemed to sometimes pop-up after users changed their password. But I'll
reiterate that the problem is not tied to any user; a user who can't access
the share from "their" box can logon to a different machine and
successfully access the share.

This is why I've been playing with "klist prune"...

> If after all this, it still doesn't work and you are using a supported
> Samba version, then I would open a bug report, giving as much data as
> possible (log level 10 output, network traces etc)
>
I will update to FreeNAS 11.2 (and Samba 4.9) but I can't imagine providing
any more information than I already have. I have to manually transcribe log
entries from this network. And I've provided the highest log level relevant
log entry and a description of what I'm seeing in Wireshark.  Unfortunately
there are no debug entries in the success paths where the in-memory keytab
is being reconstructed.