Short story: cannot get Kerberized NFSv4 to work. I've googled a great
deal and cannot find where I have goofed (and there sure is a lot of
misleading and just plain incorrect information out there), so would
appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as
does ID mapping. We're using NFSv4 in production with sec=sys, but I'm
not
happy with that. My kerberized NFSv4 attempts are on a separate test
cluster.
Longer story (sorry for the length):
All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and
nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64,
clients are both x86_64 and i686. Two DC's are both i686, running Samba
4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems
use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc
all good). Samba is at 3.6.9 on all systems except for the DC's.
Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.
I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:
# net ads join -U Administrator ...
# net ads testjoin
and created the nfs service principals (on the client and NFSv4 server)
with:
# net ads keytab add nfs -U Administrator
This all works. I can see that the nfs service principals have been added;
on the client abbott.test.cornell.edu, for example:
# net ads keytab list | grep -i nfs
2 DES cbc mode with CRC-32 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 DES cbc mode with CRC-32 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at
TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at
TITAN.TEST.CORNELL.EDU
and on one of the DC's:
# ldbsearch cn=abbott | grep -i nfs
servicePrincipalName: NFS/abbott
servicePrincipalName: NFS/abbott.test.cornell.edu
and on the client "net ads search '(sAMAccountName=abbott$)'"
also works,
as does klist:
# klist -ke | grep -i nfs
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
(aes128-cts-hmac-sha1-96)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
(aes256-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and
rpc.gssd and rpc.svcgssd are running (although no need for the latter on
the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said,
NFSv4 works fine without krb5, so I believe the exports file to be
correct).
But when I try to mount, I get the catch-all error:
# mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp
mount.nfs4: access denied by server while mounting
costello.test.cornell.edu:/data
and in /var/log/messages on the same client:
Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU'
Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in
Kerberos
database while getting initial ticket for principal
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab
'FILE:/etc/krb5.keytab'
Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for
connection to server costello.test.cornell.edu
With tcpdump I can see that the DC is contacted during the mount, but the
NFSv4 server is not. The log files on the NFSv4 server are silent.
I have tried (everything was restarted between tests); "no change"
means
that it still does not work and gives the same exact errors:
- verified that /etc/idmapd.conf on all systems has the same domains and
realms. This works anyway with sec=sys.
- reduced the keytab to the DES enctypes for nfs/... on all systems; no
change.
- used "allow_weak_crypto=true" in /etc/krb5.conf; no change.
- set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5
des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.
- tried adding the service principals on the DC with "samba-tool spn
add"
instead of "net ads keytab add" on the client, and then exporting
the
keytab to the client; no change.
- add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU"
and
"nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm
was a
factor; no change.
- started rpc.gssd on the clients with the -l switch; no change.
- started rpc.svcgssd with the -n switch so that nfs/ service principals
are in theory not required; no change.
I do not understand why rpc.gssd canot find the service principal,
when it does exist in the database. I've probably missed out some
other things that I have tried to no effect. Help!
Steve
--
----------------------------------------------------------------------------
Steve Thompson E-mail: smt AT vgersoft DOT com
Voyager Software LLC Web: http://www DOT vgersoft DOT com
39 Smugglers Path VSW Support: support AT vgersoft DOT com
Ithaca, NY 14850
"186,282 miles per second: it's not just a good idea, it's the
law"
----------------------------------------------------------------------------
On Wed, Jun 5, 2013 at 5:58 PM, Steve Thompson <smt at vgersoft.com> wrote:> Short story: cannot get Kerberized NFSv4 to work. I've googled a great > deal and cannot find where I have goofed (and there sure is a lot of > misleading and just plain incorrect information out there), so would > appreciate another pair of eyes. NFSv4 without Kerberos does work fine, as > does ID mapping. We're using NFSv4 in production with sec=sys, but I'm not > happy with that. My kerberized NFSv4 attempts are on a separate test > cluster. > > Longer story (sorry for the length): > > All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and > nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64, > clients are both x86_64 and i686. Two DC's are both i686, running Samba > 4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems > use sssd only and no winbind; this is also working (kinit, sudo, ssh, etc > all good). Samba is at 3.6.9 on all systems except for the DC's. > > Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys. > > I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with: > > # net ads join -U Administrator ... > # net ads testjoin > > and created the nfs service principals (on the client and NFSv4 server) > with: > > # net ads keytab add nfs -U Administrator > > This all works. I can see that the nfs service principals have been added; > on the client abbott.test.cornell.edu, for example: > > # net ads keytab list | grep -i nfs > 2 DES cbc mode with CRC-32 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with RSA-MD5 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 ArcFour with HMAC/md5 nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with CRC-32 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 ArcFour with HMAC/md5 nfs/abbott at TITAN.TEST.CORNELL.EDU > 2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott at TITAN.TEST.CORNELL.EDU > 2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/ > abbott at TITAN.TEST.CORNELL.EDU > > and on one of the DC's: > > # ldbsearch cn=abbott | grep -i nfs > servicePrincipalName: NFS/abbott > servicePrincipalName: NFS/abbott.test.cornell.edu > > and on the client "net ads search '(sAMAccountName=abbott$)'" also works, > as does klist: > > # klist -ke | grep -i nfs > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes128-cts-hmac-sha1-96) > 2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU(aes256-cts-hmac-sha1-96) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96) > 2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96) > > In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and > rpc.gssd and rpc.svcgssd are running (although no need for the latter on > the clients). The NFSv4 server exports with sec=sys:krb5 (and as I said, > NFSv4 works fine without krb5, so I believe the exports file to be > correct). > > But when I try to mount, I get the catch-all error: > > # mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp > mount.nfs4: access denied by server while mounting > costello.test.cornell.edu:/data > > and in /var/log/messages on the same client: > > Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' > Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in > Kerberos > database while getting initial ticket for principal > 'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab > 'FILE:/etc/krb5.keytab' > Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for > connection to server costello.test.cornell.edu > > With tcpdump I can see that the DC is contacted during the mount, but the > NFSv4 server is not. The log files on the NFSv4 server are silent. > > I have tried (everything was restarted between tests); "no change" means > that it still does not work and gives the same exact errors: > > - verified that /etc/idmapd.conf on all systems has the same domains and > realms. This works anyway with sec=sys. > > - reduced the keytab to the DES enctypes for nfs/... on all systems; no > change. > > - used "allow_weak_crypto=true" in /etc/krb5.conf; no change. > > - set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5 > des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change. > > - tried adding the service principals on the DC with "samba-tool spn add" > instead of "net ads keytab add" on the client, and then exporting the > keytab to the client; no change. > > - add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU" and > "nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm was a > factor; no change. > > - started rpc.gssd on the clients with the -l switch; no change. > > - started rpc.svcgssd with the -n switch so that nfs/ service principals > are in theory not required; no change. > > I do not understand why rpc.gssd canot find the service principal, > when it does exist in the database. I've probably missed out some > other things that I have tried to no effect. Help! > > Steve > -- > > ---------------------------------------------------------------------------- > Steve Thompson E-mail: smt AT vgersoft DOT com >What is in your nsswitch.conf? I only ask because you mention it never goes to the nfs server on the same client. Can you confirm that the rpc.svcgssd and rpc.gssd are not crashing after starting?
Hi Steve,
I guess you're running into the same problems I encountered:
Active Directory (and therefore Samba4) differs in some fields from basic
MIT/heimdal Kerberos.
The only Service I ran into problems with this difference by now is NFS :-(
One reason for this is the different handling of ServicePrincipals and
UserPrincipals.
If I recall things correctly the NFS server itself should be fine with a
servicePrincipal, the NFS clients however refuse to work with SPNs and require
separate UPNs (and therefore user objects) in AD/Samba4.
"net ads keytab add ..." only adds new SPNs to existing objects and
does not create a new object/UPN (this could be done using windows'
ktpass.exe or msktutil on linux).
Newer versions of nfs-utils (>= 1.2.4) support the HOSTNAME$ format (treated
like a UPN) used by Samba/Windows, which makes things easier (and could/should
work out of the box with a keytab created by samba itself).
Let me know, if you need further help to track this issue down.
Bye,
Marcel
-----Urspr?ngliche Nachricht-----
Von: samba-bounces at lists.samba.org [mailto:samba-bounces at lists.samba.org]
Im Auftrag von Steve Thompson
Gesendet: Mittwoch, 5. Juni 2013 23:58
An: samba at lists.samba.org; centos at centos.org
Betreff: [Samba] Samba4 and NVSv4
Short story: cannot get Kerberized NFSv4 to work. I've googled a great deal
and cannot find where I have goofed (and there sure is a lot of misleading and
just plain incorrect information out there), so would appreciate another pair of
eyes. NFSv4 without Kerberos does work fine, as does ID mapping. We're using
NFSv4 in production with sec=sys, but I'm not happy with that. My kerberized
NFSv4 attempts are on a separate test cluster.
Longer story (sorry for the length):
All servers and clients are CentOS 6.4 with kernel 2.6.32-358.6.2.el6 and
nfs-utils 1.2.3-36.el6; patches are up to date. NFSv4 servers are x86_64,
clients are both x86_64 and i686. Two DC's are both i686, running Samba
4.0.5 with bind 9.91 + bind_dlz. Replication is good. All CentOS systems use
sssd only and no winbind; this is also working (kinit, sudo, ssh, etc all good).
Samba is at 3.6.9 on all systems except for the DC's.
Samba4 works; DNS works; Kerberos works; NFSv4 works with sec=sys.
I joined the clients to the domain ("TITAN.TEST.CORNELL.EDU") with:
# net ads join -U Administrator ...
# net ads testjoin
and created the nfs service principals (on the client and NFSv4 server)
with:
# net ads keytab add nfs -U Administrator
This all works. I can see that the nfs service principals have been added; on
the client abbott.test.cornell.edu, for example:
# net ads keytab list | grep -i nfs
2 DES cbc mode with CRC-32 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott.test.cornell.edu at
TITAN.TEST.CORNELL.EDU
2 DES cbc mode with CRC-32 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 DES cbc mode with RSA-MD5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 ArcFour with HMAC/md5 nfs/abbott at TITAN.TEST.CORNELL.EDU
2 AES-128 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at
TITAN.TEST.CORNELL.EDU
2 AES-256 CTS mode with 96-bit SHA-1 HMAC nfs/abbott at
TITAN.TEST.CORNELL.EDU
and on one of the DC's:
# ldbsearch cn=abbott | grep -i nfs
servicePrincipalName: NFS/abbott
servicePrincipalName: NFS/abbott.test.cornell.edu
and on the client "net ads search '(sAMAccountName=abbott$)'"
also works, as does klist:
# klist -ke | grep -i nfs
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
(aes128-cts-hmac-sha1-96)
2 nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU
(aes256-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-crc)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (des-cbc-md5)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (arcfour-hmac)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes128-cts-hmac-sha1-96)
2 nfs/abbott at TITAN.TEST.CORNELL.EDU (aes256-cts-hmac-sha1-96)
In /etc/sysconfig/nfs, SECURE_NFS=yes on all clients and servers, and rpc.gssd
and rpc.svcgssd are running (although no need for the latter on the clients).
The NFSv4 server exports with sec=sys:krb5 (and as I said,
NFSv4 works fine without krb5, so I believe the exports file to be correct).
But when I try to mount, I get the catch-all error:
# mount -t nfs4 -o sec=krb5 costello.test.cornell.edu:/data /mnt/tmp
mount.nfs4: access denied by server while mounting
costello.test.cornell.edu:/data
and in /var/log/messages on the same client:
Jun 5 17:11:13 abbott rpc.gssd[1439]: Success getting keytab entry for
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU'
Jun 5 17:11:13 abbott rpc.gssd[1439]: WARNING: Client
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' not found in
Kerberos
database while getting initial ticket for principal
'nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU' using keytab
'FILE:/etc/krb5.keytab'
Jun 5 17:11:13 abbott rpc.gssd[1439]: ERROR: No credentials found for
connection to server costello.test.cornell.edu
With tcpdump I can see that the DC is contacted during the mount, but the
NFSv4 server is not. The log files on the NFSv4 server are silent.
I have tried (everything was restarted between tests); "no change"
means that it still does not work and gives the same exact errors:
- verified that /etc/idmapd.conf on all systems has the same domains and
realms. This works anyway with sec=sys.
- reduced the keytab to the DES enctypes for nfs/... on all systems; no
change.
- used "allow_weak_crypto=true" in /etc/krb5.conf; no change.
- set default_tgs_enctypes and default_tkt_enctypes to "des-cbc-md5
des-cbc-md4 des-cbc-crc" in /etc/krb5.conf; no change.
- tried adding the service principals on the DC with "samba-tool spn
add"
instead of "net ads keytab add" on the client, and then exporting
the
keytab to the client; no change.
- add the SPN "nfs/abbott.test.cornell.edu at TITAN.TEST.CORNELL.EDU"
and
"nfs/abbott at TITAN.TEST.CORNELL.EDU" in case the missing realm
was a
factor; no change.
- started rpc.gssd on the clients with the -l switch; no change.
- started rpc.svcgssd with the -n switch so that nfs/ service principals
are in theory not required; no change.
I do not understand why rpc.gssd canot find the service principal, when it does
exist in the database. I've probably missed out some other things that I
have tried to no effect. Help!
Steve
--
----------------------------------------------------------------------------
Steve Thompson E-mail: smt AT vgersoft DOT com
Voyager Software LLC Web: http://www DOT vgersoft DOT com
39 Smugglers Path VSW Support: support AT vgersoft DOT com
Ithaca, NY 14850
"186,282 miles per second: it's not just a good idea, it's the
law"
----------------------------------------------------------------------------
--
To unsubscribe from this list go to the following URL and read the
instructions: https://lists.samba.org/mailman/options/samba
On Thu, 6 Jun 2013, Ritter, Marcel wrote:> Newer versions of nfs-utils (>= 1.2.4) support the HOSTNAME$ format > (treated like a UPN) used by Samba/Windows, which makes things easier > (and could/should work out of the box with a keytab created by samba > itself).I have tried creating a Samba4 user object with a suitable UPN using msktutil on the DC (this is successfully entered into the database): # OU=Computers # HOST=<short hostname> # msktutil -c -b CN=$OU \ -k nfs-$HOST.keytab \ --computer-name nfs-$HOST \ --upn nfs/$HOST.test.cornell.edu \ --service nfs/$HOST.test.cornell.edu \ --server `hostname` \ --dont-expire-password \ --hostname $HOST.test.cornell.edu \ --enctypes 0x3 and then importing this keytab into the host's keytab with ktutil (so, not using "net ads keytab add"). Verified the keytab with klist. Get permission denied when trying to mount with sec=krb5. Various different enctypes all get the same result. I tried also building nfs-utils 1.2.8 from source and installing that on the NFSv4 server (using the NFSv4 server as a client for this test). All I get then, no matter what I put in the keytab, is: rpc.gssd[1679]: ERROR: GSS-API: error in gss_set_allowable_enctypes(): GSS_S_BAD_MECH (An unsupported mechanism was requested) - Unknown error The build of nfs-utils (via rpmbuild) appeared clean but I suspect that there may be something wrong with it. Still using the 2.6.32-358.6.2.el6 kernel. I tried the workaround suggested in: https://bugzilla.redhat.com/show_bug.cgi?id=720479 just in case, but it made no difference. Running out of ideas! -Steve