samba - Oct 2018 - Again NFSv4 and Kerberos at the 'samba way'...

Hai Marco, 
 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Marco Gaiarin via samba
> Verzonden: vrijdag 26 oktober 2018 11:23
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Again NFSv4 and Kerberos at the 'samba
way'...
> 
> Mandi! L.P.H. van Belle via samba
>   In chel di` si favelave...
> 
> > >  root at vdcsv1:~# samba-tool spn list vdmpp1$
> > Hmm, 
> > > 	 nfs/vdmpp1.ad.fvg.lnf.it   << correct 
> > And these are wrong. 
> > > 	 nfs/vdmpp1.ad.fvg.lnf.it/vdmpp1
> > > 	 nfs/vdmpp1.ad.fvg.lnf.it/vdmpp1.ad.fvg.lnf.it
> > Remove these 2. 
> 
> Removed, both on server and client. But, really, i've only do:
> 
> 	samba-tool spn add nfs/vdmpp1.ad.fvg.lnf.it vdmpp1$
> 
> strange.
Yes, it is, what is the DC's samba version? Same as the members?
> 
> 
> > What is the output of : 
> > dig -x $(hostname -i)
> 
> Still i'm using the old domain DNS for (back)resolving, so reverse
> point to old address (vdmpp2.pp.lnf.it).
> Clearly, i've addedd in /etc/hosts relevant record, and added to
> svcgssd the option '-p nfs/vdmpp1.ad.fvg.lnf.it' thatm, AFAI've
> understood, fix that.
Fixed? Yes and no, this is (still) one of you problems. 
All servers, in this case the DCs and vdmpp1 vdmpp2 need to know the correct
hostnames and ip.
And the members must have the resolving correctly to the DC's to be able to
lookup the SPN's

if you cant setup in the dns correct and you need the hosts files for both
server and client.

And on both servers add in /etc/krb5.conf  in libdefaults part. 
rdns = no
# no PTR lookups are done now. 

Reboot boot servers to make sure these settings are correctly applied. 
When thats done recheck then resolving on both these servers. 
hostname -f 
hostname -s
hostname -i 
These must be correct. 

> 
> 
> > exportfs
> > getfacl /home
> 
>  root at vdmpp1:~# exportfs
>  /home         	10.27.0.0/21
>  root at vdmpp1:~# getfacl /home
>  getfacl: Removing leading '/' from absolute path names
>  # file: home
>  # owner: root
>  # group: root
>  user::rwx
>  group::r-x
>  other::r-x
Ok this part, check again after the reboot, i forget the -v for the exportfs... 
( sorry )
exportfs -v 

Set chmod 1777 /home on both servers ( but leave this for the last. ), i suggest
read the complete mail first.
Test with sec=sys, and when that works we test with kerberos. 
Then we might need to look at the rights of /home 
> 
> 
> > And if you test with 
> > mount -t nfs4 -o sec=sys vdmpp1.ad.fvg.lnf.it:/home /home
> > Or 
> > mount -t nfs4 -o sec=krb5,vers=4.1 vdmpp1.ad.fvg.lnf.it:/home /home
> > Does that work or one of these work? If sys works then its 
> not firewalling. 
> 
> No, both does not work, same error.
Expected when i see above problem points. 
> 
> 
> > Have you set the encryption types i suggested in /etc/krb5.conf ?
> > The one i posted support CIFS and NFS both. 
> 
> I have on both server and client:
> 
>  ; for Windows 2008 with AES
>     default_tgs_enctypes =  aes256-cts-hmac-sha1-96 
> aes128-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
>     default_tkt_enctypes = aes256-cts-hmac-sha1-96 
> aes128-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
>     permitted_enctypes = aes256-cts-hmac-sha1-96 
> aes128-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
> 
> 
> > ? No key table table entry??  Hmm.. 
> > Check this with : klist -ke | grep "vdmpp2\\$"
> 
> Return empty.
This is a mayor error in you keytab file. 

When you join as domain member you should have these 
klist -k| egrep -i "host|$(hostname -s)\\$" | sort
   2 host/HOSTNAME at INTERNAL.DOMAIN.TLD (aes128-cts-hmac-sha1-96)
   2 host/HOSTNAME at INTERNAL.DOMAIN.TLD (aes256-cts-hmac-sha1-96)
   2 host/HOSTNAME at INTERNAL.DOMAIN.TLD (arcfour-hmac)
   2 host/HOSTNAME at INTERNAL.DOMAIN.TLD (des-cbc-crc)
   2 host/HOSTNAME at INTERNAL.DOMAIN.TLD (des-cbc-md5)
   2 host/hostname.internal.domain.tld at INTERNAL.DOMAIN.TLD
(aes128-cts-hmac-sha1-96)
   2 host/hostname.internal.domain.tld at INTERNAL.DOMAIN.TLD
(aes256-cts-hmac-sha1-96)
   2 host/hostname.internal.domain.tld at INTERNAL.DOMAIN.TLD (arcfour-hmac)
   2 host/hostname.internal.domain.tld at INTERNAL.DOMAIN.TLD (des-cbc-crc)
   2 host/hostname.internal.domain.tld at INTERNAL.DOMAIN.TLD (des-cbc-md5)
   2 HOSTNAME$@INTERNAL.DOMAIN.TLD (aes128-cts-hmac-sha1-96)
   2 HOSTNAME$@INTERNAL.DOMAIN.TLD (aes256-cts-hmac-sha1-96)
   2 HOSTNAME$@INTERNAL.DOMAIN.TLD (arcfour-hmac)
   2 HOSTNAME$@INTERNAL.DOMAIN.TLD (des-cbc-crc)
   2 HOSTNAME$@INTERNAL.DOMAIN.TLD (des-cbc-md5)

and if these are also in the AD? ( you should only see 2 in the AD )
HOST/hostname.internal.domain.tld
HOST/HOSTNAME


The part below here  
NFS/vdmpp2. ... < wrong 
nfs/vdmpp2..... < correct 

Remove the one with NFS. 
You want : 
nfs/HOSTNAME$@INTERNAL.DOMAIN.TLD  ( per cipher ) 
nfs/HOSTNAME.internal.domain.tld at INTERNAL.DOMAIN.TLD

And remember, dont add the @REALM when adding this. 
If you see in the ad also the part @INTERNAL.DOMAIN.TLD and the result wil be
@INTERNAL.DOMAIN.TLD at INTERNAL.DOMAIN.TLD  when resolved on the client.
> 
> > Looks like the local keytab is having problems. 
> > Run  on vdmpp2 :
> > klist -ke
> > kinit nfs/$(hostname -f) -kt /etc/krb5.keytab
> > klist | grep "Default principal"
> > That should show :
> > Default principal: nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT
> 
>  root at vdmpp2:~# klist -ke | grep "vdmpp2\\$"
>  root at vdmpp2:~# klist -ke
>  Keytab name: FILE:/etc/krb5.keytab
>  KVNO Principal
>  ---- 
> --------------------------------------------------------------
> ------------
>     2 NFS/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-crc) 	< wron
>     2 NFS/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-md5) 
>     2 NFS/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT 
> (aes128-cts-hmac-sha1-96) 
>     2 NFS/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT 
> (aes256-cts-hmac-sha1-96) 
>     2 NFS/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (arcfour-hmac) 
>     2 nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-crc) 
>     2 nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-md5) 
>     2 nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT 
> (aes128-cts-hmac-sha1-96) 
>     2 nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT 
> (aes256-cts-hmac-sha1-96) 
>     2 nfs/vdmpp2.ad.fvg.lnf.it at AD.FVG.LNF.IT (arcfour-hmac) 
>     2 nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-crc) 
>     2 nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT (des-cbc-md5) 
>     2 nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT 
> (aes128-cts-hmac-sha1-96) 
>     2 nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT 

If this server is not in production. 

Then remove it from the AD, clear the DNS, check the AD objects and remove
these.
Backup you old keytab, ( ! Tip never trow the away until your 1000% sure ) it
did bite me also once..
Remove the old ketab. 
Remove nfs-*  ( apt-get remove --purge --auto-remove nfs-* libnfsidmap2 ) 
Remove /var/lib/nfs 
install mlocate to create a db of you file system entries. 
apt-get install mlocate && updatedb && locate nfs
Remove any leftovers in /etc/systemd/. 

Clear you logs, reboot the server, check you logs, should be error free now. 

And re-add the server to the samba domain. 
Make sure you have the idmap config settings correct, you have the already. 
Make sure you have the resolving setup ok, minimal a correct A in the DNS.
  kerberos method = secrets and keytab
    dedicated keytab file = /etc/krb5.keytab

    # Renew the kerberos ticket
    winbind refresh tickets = yes

Make sure you have these in smb.conf

And now re-add you server. 

systemctl mask nmbd samba-ad-dc
systemctl stop nmbd samba-ad-dc

When your at this point, reboot then check the keytab file again, above shows
what you need.

Then add the nfs/SPN  then reinstall nfs- .. Again.  

If you use ktutil 
Use the write command to /etc/krb5.keytab-NEW 
Or your adding to the existing keytab, but that also add the part you already
had.
Thats not what you want. 
Stop samba backup the old keytab and place the new one. 

See how far you get, if needed, you know where to find me..
User vers=4.1 for the mounts  


And on the nfs server you can also check this. 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867067 

mkdir /var/lib/nfs/nfsdcltrack 
nfsdcltrack init
Check if the .sqlite is created : ls /var/lib/nfs/nfsdcltrack/*.sqlite
systemctl restart nfs-server



Greetz, 

Louis

Mandi! L.P.H. van Belle via samba
  In chel di` si favelave...
> > 	samba-tool spn add nfs/vdmpp1.ad.fvg.lnf.it vdmpp1$
> > strange.
> Yes, it is, what is the DC's samba version? Same as the members? 
No. DS are still on 4.5.


> if you cant setup in the dns correct and you need the hosts files for both
server and client.
> And on both servers add in /etc/krb5.conf  in libdefaults part. 
> rdns = no
> # no PTR lookups are done now. 
Ok, done that seems that at least 'sec=sys' mount now work. WOW!
Probably is not due to rdns, but by the fact that:

> Ok this part, check again after the reboot, i forget the -v for the
exportfs...  ( sorry )
> exportfs -v 
Now i've:

 root at vdmpp1:~# exportfs -v
 /home         
10.27.0.0/21(rw,wdelay,root_squash,no_subtree_check,sec=sys:krb5,rw,secure,root_squash,no_all_squash)

but i've had sec=krb5 only, so... O;-)))

> Remove the one with NFS. 
OK. But server is in production, so... how can i do that, without
deinstalling and reinstalling all the stuff?


I've stopped and run by hand /usr/sbin/rpc.gssd with '-vvv' and
/usr/sbin/rpc.svcgssd
with '-vvv -p nfs/vdmpp1.ad.fvg.lnf.it' (/etc/default/nfs-* parameters
variables seems are ignored) and still /usr/sbin/rpc.svcgssd write no
log, and thsi seeems strage o me...

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''         
http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

Hai, 

A quick reply,
Since there is a major traffic jam here, still at the office, but its resolving
now..
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Marco Gaiarin via samba
> Verzonden: maandag 29 oktober 2018 17:33
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Again NFSv4 and Kerberos at the 'samba
way'...
> 
> Mandi! L.P.H. van Belle via samba
>   In chel di` si favelave...
> 
> > > 	samba-tool spn add nfs/vdmpp1.ad.fvg.lnf.it vdmpp1$
> > > strange.
> > Yes, it is, what is the DC's samba version? Same as the members? 
> 
> No. DS are still on 4.5.
Hm, ok, i would preffer 4.8, but it should work also. 
I think the wrong spn is coming from the 4.5 line, but not 100% sure.
> 
> 
> 
> > if you cant setup in the dns correct and you need the hosts 
> files for both server and client.
> > And on both servers add in /etc/krb5.conf  in libdefaults part. 
> > rdns = no
> > # no PTR lookups are done now. 
> 
> Ok, done that seems that at least 'sec=sys' mount now work. WOW!
> Probably is not due to rdns, but by the fact that:
> 
> 
> > Ok this part, check again after the reboot, i forget the -v 
> for the exportfs...  ( sorry ) 
> > exportfs -v 
> 
> Now i've:
> 
>  root at vdmpp1:~# exportfs -v
>  /home         	
> 10.27.0.0/21(rw,wdelay,root_squash,no_subtree_check,sec=sys:kr
> b5,rw,secure,root_squash,no_all_squash)
> 
> but i've had sec=krb5 only, so... O;-)))
Ok, so sys works, this confirms a problem with detecting the nfs spns. 
> 
> 
> > Remove the one with NFS. 
> 
> OK. But server is in production, so... how can i do that, without
> deinstalling and reinstalling all the stuff?
I'll think a bit about this for you so you can fix it without removeing it
all.
I'll re-read the thread again tomorrow and let you know. 
> 
> 
> I've stopped and run by hand /usr/sbin/rpc.gssd with '-vvv' 
> and /usr/sbin/rpc.svcgssd
> with '-vvv -p nfs/vdmpp1.ad.fvg.lnf.it' (/etc/default/nfs-*
parameters
> variables seems are ignored) and still /usr/sbin/rpc.svcgssd write no
> log, and thsi seeems strage o me...
Wel, the sys option is not kerberize so seems logical to me you dont see thing
in the log now.

> 
> -- 
> dott. Marco Gaiarin				        GNUPG 
> 
> 
Sofar, until tomorrow, 

Greetz, 

Louis

Hai Marco, 
> 
> Mandi! L.P.H. van Belle via samba
>   In chel di` si favelave...
> 
> > Sofar, until tomorrow, 
> 
> Done some tests, metoo.
> 
> 1) seems that nfs-common is disabled 'by design'. Looking at debian
> changelog:
> 
>  nfs-utils (1:1.2.8-9.1) unstable; urgency=medium
> 
>   Partial sync from ubuntu, included changes:
> 
>   [ Martin Pitt ]
>   [...]
>   * 27-systemd-enable-with-systemctl-statd.patch: let the admin
>     enable/disable statd via systemd tools. (LP: #1428486)
> 
>   [...]
>   [ Andreas Henriksson ]
>   * Restore anything related to nfs-common.init and nfs-common.default
>   * debian/nfs-common.links: Mask nfs-common init script with 
> a symlink
>     to /dev/null to avoid using it under systemd.
> 
> so seems you have to enable/disable/mask single services. Note that
> still there are some troubles, eg on client:
> 
> 	root at vdmpp2:~# systemctl start nfs-idmapd
> 	Failed to start nfs-idmapd.service: Unit 
> nfs-server.service not found.
> 
> (but probably idmap is a server-only service, so it is normal?)
> and also seems that /etc/default/nfs-common are *totally* ignored (eg,
> there's no way to pass options to services).
> 
> Anyway, now i'm able to restart nfs/rpc services. ;-)
Ok, thats at least better. 

And no, /etc/default/nfs-common is not ignored. Its just harder to see it. 

systemctl cat nfs-config  
contains :  ExecStart=/usr/lib/systemd/scripts/nfs-utils_env.sh
And the nfs-utils_env.sh contains : 
[ -r /etc/default/nfs-common ] && . /etc/default/nfs-common
[ -r /etc/default/nfs-kernel-server ] && .
/etc/default/nfs-kernel-server

;-) 

And 
/lib/systemd/system/rpc-svcgssd.service
Contains:  ConditionPathExists=/etc/krb5.keytab

Thats all ok. 

All i did for the server was systemctl enable nfs-server
And for the client systemctl enable nfs-client
After the setup, all other servers start if needed based on the settings in 
/etc/default/nfs-common and/or /etc/default/nfs-kernel-server 

> 
> 
> 2) doing some mounts on the same host, with verbose output, i get:
> 
>  Oct 30 15:13:33 vdmpp1 rpc.gssd[6448]: Success getting 
> keytab entry for 'nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT'
>  Oct 30 15:13:33 vdmpp1 rpc.gssd[6448]: WARNING: 
> Preauthentication failed while getting initial ticket for 
> principal 'nfs/vdmpp1.ad.fvg.lnf.it at AD.FVG.LNF.IT' using 
> keytab 'FILE:/etc/krb5.keytab'
>  Oct 30 15:13:33 vdmpp1 rpc.gssd[6448]: ERROR: No credentials 
> found for connection to server vdmpp1.ad.fvg.lnf.it
> 
> 'Preauthentication'?
Hmm, that is strange, it looks like this computer account is acting like a real
user.
If i look in ADUC, Tab Account, only a user has the option to "disable
preauthentication"
So this might help in solving the problem. 
Can you check in ADUC of you see the Account tab or not. 
If its really a computer, you should not see the Account tab. 


Your are getting closer at least to what is causing this problem. 

Greetz, 

Louis