zfs discuss - Feb 2012 - Problem with ESX NFS store on ZFS

Running Solaris 11 with ZFS and the VM''s on this storage can only be
opened and run on 1 ESX host, if I move the files to another host I get access
denied, even though root has full permissions to the files.

Any ideas or does it ring any bells for anyone before I contact VMware or
something?

Thanks
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20120229/65e8b187/attachment.html>

2012-02-29 21:15, Mark Wolek wrote:
> Running Solaris 11 with ZFS and the VM?s on this storage can only be
> opened and run on 1 ESX host, if I move the files to another host I get
> access denied, even though root has full permissions to the files.
>
> Any ideas or does it ring any bells for anyone before I contact VMware
> or something?


Probably, NFS UID mappng is faulty, or the NFS server ACL
does not allow for another server.

For UID mapping, in particular see the domain name settings:
/etc/defaultdomain
/etc/resolv.conf (search, domain lines)
/etc/default/nfs or appropriate SMF settings (NFSMAPID_DOMAIN)

For NFS ACL see the sharenfs property:

# zfs set 
sharenfs=''rw=esx:cvs:.domain.com:.jumbo.domain.com:@192.168.127.0/24,root=esx:cvs:192.168.127.99''
pool/esxfiles

Critical fields are ''rw'', ''ro'' and
''root'' lists of hosts or
subnets of clients which have appropriate types of access.
For hosts not in ''root'' list, their allowed
''ro'' or ''rw''
access as "root" user will be remapped to "nobody".
You might also want ''anon=0,sec=sys'' which seem to be
appended by default on my installations of Solaris, not
sure if it is the default in Sol11.

Note that clients'' hostnames can be resolved via /etc/hosts,
DNS or LDAP, as configured in your /etc/nsswitch.conf, and
sometimes via /etc/inet/ipnodes as well as a fallback mechanism.
Your server only gets one shot at resolving the client''s
name, and if it is not literally the same as in NFS ACL,
access is denied. You might want to fall back to domain-based
or subnet-based ACLs (may require the @ character).

For pointers to server-side ACL denials see the server''s
dmesg with entries resembling this:

Feb 29 19:35:01 thumper mountd[10782]: [ID 770583 daemon.error] 
esx.demo.domain.com denied access to /esxfiles/vm5

In particular, the entry produces the client''s hostname
as the server resolved it, so you can see if your ACL
(or naming service) was misconfigured.

HTH,
//Jim

Thank you, it was the NFS ACL I had wrong!  Fixed now and working on all 3
nodes.  I changed below and it works now, very simple can''t believe I
missed that

zfs get sharenfs
pool1/nas/vol1 sharenfs  rw,nosuid,root=192.168.1.52  local

zfs get sharenfs
pool1/nas/vol1 sharenfs  rw,nosuid,root=192.168.1.52:192.168.1.51:192.168.1.53 
local

-----Original Message-----
From: Jim Klimov [mailto:jimklimov at cos.ru] 
Sent: Wednesday, February 29, 2012 1:44 PM
To: Mark Wolek
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] Problem with ESX NFS store on ZFS

2012-02-29 21:15, Mark Wolek wrote:
> Running Solaris 11 with ZFS and the VM''s on this storage can only
be
> opened and run on 1 ESX host, if I move the files to another host I 
> get access denied, even though root has full permissions to the files.
>
> Any ideas or does it ring any bells for anyone before I contact VMware 
> or something?


Probably, NFS UID mappng is faulty, or the NFS server ACL does not allow for
another server.

For UID mapping, in particular see the domain name settings:
/etc/defaultdomain
/etc/resolv.conf (search, domain lines)
/etc/default/nfs or appropriate SMF settings (NFSMAPID_DOMAIN)

For NFS ACL see the sharenfs property:

# zfs set
sharenfs=''rw=esx:cvs:.domain.com:.jumbo.domain.com:@192.168.127.0/24,root=esx:cvs:192.168.127.99''
pool/esxfiles

Critical fields are ''rw'', ''ro'' and
''root'' lists of hosts or subnets of clients which have
appropriate types of access.
For hosts not in ''root'' list, their allowed
''ro'' or ''rw''
access as "root" user will be remapped to "nobody".
You might also want ''anon=0,sec=sys'' which seem to be appended
by default on my installations of Solaris, not sure if it is the default in
Sol11.

Note that clients'' hostnames can be resolved via /etc/hosts, DNS or
LDAP, as configured in your /etc/nsswitch.conf, and sometimes via
/etc/inet/ipnodes as well as a fallback mechanism.
Your server only gets one shot at resolving the client''s name, and if
it is not literally the same as in NFS ACL, access is denied. You might want to
fall back to domain-based or subnet-based ACLs (may require the @ character).

For pointers to server-side ACL denials see the server''s dmesg with
entries resembling this:

Feb 29 19:35:01 thumper mountd[10782]: [ID 770583 daemon.error]
esx.demo.domain.com denied access to /esxfiles/vm5

In particular, the entry produces the client''s hostname as the server
resolved it, so you can see if your ACL (or naming service) was misconfigured.

HTH,
//Jim