samba - Oct 2019 - AD domain member cannot authenticate user in remote forest unless smbclient uses "localhost"

Hi folks,

I'm trying to support a customer with multiple AD forests, and during my
research, I've observed some odd behavior. In my lab tests, it seems like
authentication works for users in all trusted forests, but only if NTLMSSP
is used. When Kerberos ends up being used, authentication only seems to
work for users in the local domain.

Here's the test setup:
- Two Active Directory forests, tc83.local and tc84.local, with a forest
trust between them.
- The Linux server is a member of domain tc83.local.
- Samba built from git master this afternoon (commit 2669cecc51f) on Ubuntu
19.10. (I first reproduced this on CentOS 7, but wanted to test against
latest code before asking this list.)

ubuntu at kvm7246-vm022:~/samba$ sudo realm join --client-software=winbind
tc83.local
Password for Administrator:

ubuntu at kvm7246-vm022:~/samba$ realm list
tc83.local
  type: kerberos
  realm-name: TC83.LOCAL
  domain-name: tc83.local
  configured: kerberos-member
  server-software: active-directory
  client-software: winbind
  required-package: winbind
  required-package: libpam-winbind
  required-package: samba-common-bin
  login-formats: TC83\%U
  login-policy: allow-any-login

ubuntu at kvm7246-vm022:~/samba$ testparm
Load smb config files from //etc/samba/smb.conf
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER

Press enter to see a dump of your service definitions

# Global parameters
[global]
kerberos method = system keytab
logging = systemd
realm = TC83.LOCAL
security = ADS
template homedir = /home/%U@%D
template shell = /bin/bash
winbind offline logon = Yes
winbind refresh tickets = Yes
workgroup = TC83
idmap config * : range = 10000-999999
idmap config * : backend = tdb


[test]
path = /srv/test
valid users = "@tc83.local\domain users" "@tc84.local\domain
users"

Authentication works for a user in either forest when accessing the server
as "localhost", but fails for user in the remote forest when the real
hostname is used:

ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
administrator at tc83.local
Enter administrator at tc83.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
administrator at tc84.local
Enter administrator at tc84.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
administrator at tc83.local
Enter administrator at tc83.local's password:
Try "help" to get a list of possible commands.
smb: \> exit
ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
administrator at tc84.local
Enter administrator at tc84.local's password:
session setup failed: NT_STATUS_LOGON_FAILURE
ubuntu at kvm7246-vm022:~/samba$

(Logs from each smbclient attempt are at
https://drive.google.com/open?id=1_355NuN1L9BW5JvtP9WG-dEGkaQqNT3Y)

The logs seem to show that in the "localhost" cases, the final
authentication step uses "GENSEC submechanism gse_krb5", while in the
cases
where the actual hostname is specified, the final authentication step uses
"GENSEC submechanism ntlmssp". The Kerberos auth seems only to work if
the
authenticating user is in the local domain; if the user is in the other
domain, it fails looking for a keytab entry that does not exist:

Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.429043,  5]
../../auth/gensec/gensec_start.c:737(gensec_start_mech)
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   Starting GENSEC submechanism
gse_krb5
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.430349,  1]
../../source3/librpc/crypto/gse.c:659(gse_get_server_auth_token)
Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   gss_accept_sec_context failed
with [ Miscellaneous failure (see text): Failed to find
cifs/kvm7246-vm022 at TC84.LOCAL(kvno 10) in keytab MEMORY:cifs_srv_keytab
(aes256-cts-hmac-sha1-96)]

Is this expected behavior? A known issue? Am I doing something silly?

It's probably obvious, but I made a typo, swapping gse_krb5 and ntlmssp ---
what I meant to write near the end of that was this:

The logs seem to show that in the "localhost" cases, the final
authentication step uses "GENSEC submechanism ntlmssp", while in the
cases
where the actual hostname is specified, the final authentication step uses
"GENSEC submechanism gse_krb5".

On Mon, Oct 28, 2019 at 5:53 PM Nathaniel W. Turner <
nathanielwyliet at gmail.com> wrote:
> Hi folks,
>
> I'm trying to support a customer with multiple AD forests, and during
my
> research, I've observed some odd behavior. In my lab tests, it seems
like
> authentication works for users in all trusted forests, but only if NTLMSSP
> is used. When Kerberos ends up being used, authentication only seems to
> work for users in the local domain.
>
> Here's the test setup:
> - Two Active Directory forests, tc83.local and tc84.local, with a forest
> trust between them.
> - The Linux server is a member of domain tc83.local.
> - Samba built from git master this afternoon (commit 2669cecc51f) on
> Ubuntu 19.10. (I first reproduced this on CentOS 7, but wanted to test
> against latest code before asking this list.)
>
> ubuntu at kvm7246-vm022:~/samba$ sudo realm join --client-software=winbind
> tc83.local
> Password for Administrator:
>
> ubuntu at kvm7246-vm022:~/samba$ realm list
> tc83.local
>   type: kerberos
>   realm-name: TC83.LOCAL
>   domain-name: tc83.local
>   configured: kerberos-member
>   server-software: active-directory
>   client-software: winbind
>   required-package: winbind
>   required-package: libpam-winbind
>   required-package: samba-common-bin
>   login-formats: TC83\%U
>   login-policy: allow-any-login
>
> ubuntu at kvm7246-vm022:~/samba$ testparm
> Load smb config files from //etc/samba/smb.conf
> Loaded services file OK.
> Server role: ROLE_DOMAIN_MEMBER
>
> Press enter to see a dump of your service definitions
>
> # Global parameters
> [global]
> kerberos method = system keytab
> logging = systemd
> realm = TC83.LOCAL
> security = ADS
> template homedir = /home/%U@%D
> template shell = /bin/bash
> winbind offline logon = Yes
> winbind refresh tickets = Yes
> workgroup = TC83
> idmap config * : range = 10000-999999
> idmap config * : backend = tdb
>
>
> [test]
> path = /srv/test
> valid users = "@tc83.local\domain users" "@tc84.local\domain
users"
>
> Authentication works for a user in either forest when accessing the server
> as "localhost", but fails for user in the remote forest when the
real
> hostname is used:
>
> ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
> administrator at tc83.local
> Enter administrator at tc83.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
> administrator at tc84.local
> Enter administrator at tc84.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
> administrator at tc83.local
> Enter administrator at tc83.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
> administrator at tc84.local
> Enter administrator at tc84.local's password:
> session setup failed: NT_STATUS_LOGON_FAILURE
> ubuntu at kvm7246-vm022:~/samba$
>
> (Logs from each smbclient attempt are at
> https://drive.google.com/open?id=1_355NuN1L9BW5JvtP9WG-dEGkaQqNT3Y)
>
> The logs seem to show that in the "localhost" cases, the final
> authentication step uses "GENSEC submechanism gse_krb5", while in
the cases
> where the actual hostname is specified, the final authentication step uses
> "GENSEC submechanism ntlmssp". The Kerberos auth seems only to
work if the
> authenticating user is in the local domain; if the user is in the other
> domain, it fails looking for a keytab entry that does not exist:
>
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.429043,
>  5] ../../auth/gensec/gensec_start.c:737(gensec_start_mech)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   Starting GENSEC submechanism
> gse_krb5
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.430349,
>  1] ../../source3/librpc/crypto/gse.c:659(gse_get_server_auth_token)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   gss_accept_sec_context failed
> with [ Miscellaneous failure (see text): Failed to find
> cifs/kvm7246-vm022 at TC84.LOCAL(kvno 10) in keytab MEMORY:cifs_srv_keytab
> (aes256-cts-hmac-sha1-96)]
>
> Is this expected behavior? A known issue? Am I doing something silly?
>

Hi Nathaniel,
> ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
could you use your fqdn instead of hostname command, just to be sure 
that your hostname is properly configured (seen that in the past).
> administrator at tc84.local
You are using a domain in .local, be sure that avahi-daemon is not 
running otherwise you might get strange DNS resolution.
> Enter administrator at tc84.local's password:
> session setup failed: NT_STATUS_LOGON_FAILURE
> ubuntu at kvm7246-vm022:~/samba$
Just to be sure, did you do a kinit before hand? What do you have in 
your klist after smbclient command? Can you resolv DNS of both domains?

Cheers,

Denis
>
> (Logs from each smbclient attempt are at
> https://drive.google.com/open?id=1_355NuN1L9BW5JvtP9WG-dEGkaQqNT3Y)
>
> The logs seem to show that in the "localhost" cases, the final
> authentication step uses "GENSEC submechanism gse_krb5", while in
the cases
> where the actual hostname is specified, the final authentication step uses
> "GENSEC submechanism ntlmssp". The Kerberos auth seems only to
work if the
> authenticating user is in the local domain; if the user is in the other
> domain, it fails looking for a keytab entry that does not exist:
>
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.429043,  5]
> ../../auth/gensec/gensec_start.c:737(gensec_start_mech)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   Starting GENSEC submechanism
> gse_krb5
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.430349,  1]
> ../../source3/librpc/crypto/gse.c:659(gse_get_server_auth_token)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   gss_accept_sec_context failed
> with [ Miscellaneous failure (see text): Failed to find
> cifs/kvm7246-vm022 at TC84.LOCAL(kvno 10) in keytab MEMORY:cifs_srv_keytab
> (aes256-cts-hmac-sha1-96)]
>
> Is this expected behavior? A known issue? Am I doing something silly?
>
-- 
Denis Cardon
Tranquil IT
12 avenue Jules Verne (Bat. A)
44230 Saint S?bastien sur Loire (FRANCE)
tel : +33 (0) 240 975 755
http://www.tranquil.it

Tranquil IT recrute! https://www.tranquil.it/nous-rejoindre/
Samba install wiki for Frenchies : https://dev.tranquil.it
WAPT, software deployment made easy : https://wapt.fr

On 28/10/2019 21:53, Nathaniel W. Turner via samba
wrote:
> Hi folks,
>
> I'm trying to support a customer with multiple AD forests, and during
my
> research, I've observed some odd behavior. In my lab tests, it seems
like
> authentication works for users in all trusted forests, but only if NTLMSSP
> is used. When Kerberos ends up being used, authentication only seems to
> work for users in the local domain.
>
> Here's the test setup:
> - Two Active Directory forests, tc83.local and tc84.local, with a forest
> trust between them.
> - The Linux server is a member of domain tc83.local.
> - Samba built from git master this afternoon (commit 2669cecc51f) on Ubuntu
> 19.10. (I first reproduced this on CentOS 7, but wanted to test against
> latest code before asking this list.)
>
> ubuntu at kvm7246-vm022:~/samba$ sudo realm join --client-software=winbind
> tc83.local
> Password for Administrator:
>
> ubuntu at kvm7246-vm022:~/samba$ realm list
> tc83.local
>    type: kerberos
>    realm-name: TC83.LOCAL
>    domain-name: tc83.local
>    configured: kerberos-member
>    server-software: active-directory
>    client-software: winbind
>    required-package: winbind
>    required-package: libpam-winbind
>    required-package: samba-common-bin
>    login-formats: TC83\%U
>    login-policy: allow-any-login
>
> ubuntu at kvm7246-vm022:~/samba$ testparm
> Load smb config files from //etc/samba/smb.conf
> Loaded services file OK.
> Server role: ROLE_DOMAIN_MEMBER
>
> Press enter to see a dump of your service definitions
>
> # Global parameters
> [global]
> kerberos method = system keytab
> logging = systemd
> realm = TC83.LOCAL
> security = ADS
> template homedir = /home/%U@%D
> template shell = /bin/bash
> winbind offline logon = Yes
> winbind refresh tickets = Yes
> workgroup = TC83
> idmap config * : range = 10000-999999
> idmap config * : backend = tdb
>
>
> [test]
> path = /srv/test
> valid users = "@tc83.local\domain users" "@tc84.local\domain
users"
>
> Authentication works for a user in either forest when accessing the server
> as "localhost", but fails for user in the remote forest when the
real
> hostname is used:
>
> ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
> administrator at tc83.local
> Enter administrator at tc83.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //localhost/test -U
> administrator at tc84.local
> Enter administrator at tc84.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
> administrator at tc83.local
> Enter administrator at tc83.local's password:
> Try "help" to get a list of possible commands.
> smb: \> exit
> ubuntu at kvm7246-vm022:~/samba$ smbclient //`hostname`/test -U
> administrator at tc84.local
> Enter administrator at tc84.local's password:
> session setup failed: NT_STATUS_LOGON_FAILURE
> ubuntu at kvm7246-vm022:~/samba$
>
> (Logs from each smbclient attempt are at
> https://drive.google.com/open?id=1_355NuN1L9BW5JvtP9WG-dEGkaQqNT3Y)
>
> The logs seem to show that in the "localhost" cases, the final
> authentication step uses "GENSEC submechanism gse_krb5", while in
the cases
> where the actual hostname is specified, the final authentication step uses
> "GENSEC submechanism ntlmssp". The Kerberos auth seems only to
work if the
> authenticating user is in the local domain; if the user is in the other
> domain, it fails looking for a keytab entry that does not exist:
>
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.429043,  5]
> ../../auth/gensec/gensec_start.c:737(gensec_start_mech)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   Starting GENSEC submechanism
> gse_krb5
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]: [2019/10/28 20:02:26.430349,  1]
> ../../source3/librpc/crypto/gse.c:659(gse_get_server_auth_token)
> Oct 28 20:02:26 kvm7246-vm022 smbd[30735]:   gss_accept_sec_context failed
> with [ Miscellaneous failure (see text): Failed to find
> cifs/kvm7246-vm022 at TC84.LOCAL(kvno 10) in keytab MEMORY:cifs_srv_keytab
> (aes256-cts-hmac-sha1-96)]
>
> Is this expected behavior? A known issue? Am I doing something silly?
I am sorry but you seem to be asking on the wrong list, you appear to be 
using sssd (which isn't supported with Samba from 4.8.0), Samba isn't 
doing the authentication.

Samba does not produce sssd, so we know little or nothing about it, for 
help with it, you should contact the sssd-users mailing list.

If you require help setting up Unix domain members with winbind, can I 
suggest you read this:

https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member

Rowland

Hi Rowland,

On Tue, Oct 29, 2019 at 5:37 AM Rowland penny via samba <
samba at lists.samba.org> wrote:
>
> I am sorry but you seem to be asking on the wrong list, you appear to be
> using sssd (which isn't supported with Samba from 4.8.0), Samba
isn't
> doing the authentication.
>
What part of my problem description, or which log entries make you think I
am using sssd?
n

On Tue, Oct 29, 2019 at 5:37 AM Rowland penny via samba <
samba at lists.samba.org> wrote:
>
> If you require help setting up Unix domain members with winbind, can I
> suggest you read this:
>
> https://wiki.samba.org/index.php/Setting_up_Samba_as_a_Domain_Member

Well, I was not having problems with the actual process of joining using
"realm join --client-software=winbind". The resulting membership also
appears be mostly functional, as I was able to authenticate users in all
trusted domains (via the localhost samba test I described in my first
message, or directly, using "wbinfo -a" or "ntml_auth").

However, since it seemed important, I tried starting from scratch using the
wiki instructions you linked to above. These seem to be incomplete, as with
that approach, I am unable to join at all. The document first says to
delete your smb.conf, and then only discusses setting up the id mapping
section. I assume there are other things that need to go in there, but I
was attempting to follow the instructions exactly, so that folks would
consider my test to be valid. Have you tried following those instructions
recently?

(Btw, I should mention that yes, forward and reverse DNS is configured
correctly here.)

n