samba - Dec 2019 - Failed to find [principal](kvno 4) in keytab MEMORY:cifs_srv_keytab (arcfour-hmac-md5)]

Hello everyone,

I have a FreeNAS server (9.10 running samba 4.3.11-GIT-UNKNOWN) that's
recently
started emitting this error:

gss_accept_sec_context failed with [ Miscellaneous failure (see text):
Failed to find cifs/nas01 at EXAMPLE.COM(kvno 4) in keytab
MEMORY:cifs_srv_keytab (arcfour-hmac-md5)]

I've looked at bug 12262 [1], which is why I've cc'd Stefan
Metzmacher.
I don't think this is is the same as that bug, and I will explain why.

I've looked at a lot of mailing list/forum posts, and the problem either
"goes away", or the conversation ends. [2][3][4][5][6]

Here's my understanding of the problem: The message "Starting GENSEC
submechanism gse_krb5" shows up every time a new connection comes in. That
means this happens each connection:

- gensec_gse_server_start
  - gse_init_server
    - gse_context_init
      - krb5_init_context(&gse_ctx->k5ctx)
    - gse_krb5_get_server_keytab

FreeNAS generates an smb4.conf without 'kerberos method', so it defaults
to
'secrets only'. That means only fill_mem_keytab_from_secrets() is used.
That function gets the (current and previous) machine password from
secrets.tdb, which I can dump with:

tdbdump /var/db/system/samba4/private/secrets.tdb

Troubleshooting:
- I've obviously tried restarting the smb service
- I've tried having users log-off/reboot
- I've performed an un-join and re-join (using FreeNAS interface), and I can
  see the machine password changing, as well as the kvno number in the error
  message.


Here's why I'm particularly confused:
- This is not tied to any particular end user. User A was unable to connect to
  the share from his machine, but was able to from a different machine.
- It seems to be tied to a client machine. Users A and B were unable to connect
  to the share from a problematic machine.
- I took a wireshark capture of a successful "Session Setup Request"
from my
  machine, and an unsuccessful request from User A's machine. In both cases,
  the GSSAPI->SPNEGO->Kerberos AP-REQ ticket looked the same:

  ticket:
    tkt-vno: 5
    realm: EXAMPLE.COM
    sname:
      name-type: kRB5-NT-SRV-INST (2)
      sname-string: 2 items
        SNameString: cifs
        SNameString: nas01.example.com
    enc-part:
      etype: eTYPE-ARCFOUR-HMAC-MD5 (23)cifs
      kvno: 4
      cipher: [random]

Since both "good" and "bad" requests have the same
principal, that implies that
the MEMORY:cifs_srv_keytab keytab ends up with different contents depending on
the client machine that connects O_o  -- This is why I don't think it is
(directly) related to 12262.

Unfortunately because the keytab is in memory, I have no (remotely easy) way of
seeing what's in it. And even with 'log level = 19', there is no
debug output
in the path where the keytab is being built.

So I'm really hoping a developer can step in here and help understand a few
things:

A) Am I correct in my understanding that the in-memory keytab is created on
   each request? How could it possibly be different for different client
   machines?

B) How does fill_mem_keytab_from_secrets() work without ever setting kvno?

      krb5_kvno kvno = 0; /* FIXME: fetch current vno from KDC ? */

  This is particularly interesting because fill_keytab_from_password() is
  called with old(er) machine passwords, passing `kvno - 1` and `kvno - 2`
  which will will be (-1) and (-2) which seem completely invalid.

  There's a lot of confusing 'vno' logic in krb5_kt_get_entry().

C) Is there any other information I'm not looking at that would be helpful
   in understanding this?


Regards,
Jonathon Reinhart


[1] https://bugzilla.samba.org/show_bug.cgi?id=12262
[2]
https://www.google.com/search?q=%22Failed+to+find%22+kvno+%22keytab+MEMORY%3Acifs_srv_keytab%22
[3] https://lists.samba.org/archive/samba/2016-September/203338.html
[4] https://redmine.ixsystems.com/issues/6228
[5] https://www.ixsystems.com/community/threads/36439
[6] https://www.ixsystems.com/community/threads/61835

On 16/12/2019 20:22, Jonathon Reinhart via samba wrote:
> Hello everyone,
>
> I have a FreeNAS server (9.10 running samba 4.3.11-GIT-UNKNOWN) that's
recently
> started emitting this error:
>
If you are running Samba 4.3.11 then you should be aware that it is EOL 
and bug 12262 was fixed in 4.4, so I would suggest you upgrade to a 
supported Samba version. You will probably find that your problem is 
fixed, but if it isn't, you will at least stand a chance of getting it 
fixed, you have little or no chance of getting 4.3.11 fixed.

Rowland

Yes, I am planning on upgrading the FreeNAS which will include a newer
version of samba.
However, I'm quite confident that this is not a duplicate of #12262.

To be clear: I'm offering to leave my production system in a degraded
state to help myself and the Samba developers understand exactly
what's going on here, to determine if this is a new bug, or an
existing one. I'm worried that upgrading will make the problem "go
away" but not actually be resolved.

I mainly don't understand how two clients can be presenting tickets
for the exact same principal (and kvno), but Samba says that one of
them can't be found, but the other is acceptable.


On Mon, Dec 16, 2019 at 3:46 PM Rowland penny via samba
<samba at lists.samba.org> wrote:
> > I have a FreeNAS server (9.10 running samba 4.3.11-GIT-UNKNOWN)
that's recently
> > started emitting this error:
> >
> If you are running Samba 4.3.11 then you should be aware that it is EOL
> and bug 12262 was fixed in 4.4, so I would suggest you upgrade to a
> supported Samba version. You will probably find that your problem is
> fixed, but if it isn't, you will at least stand a chance of getting it
> fixed, you have little or no chance of getting 4.3.11 fixed.
>
> Rowland