zfs discuss - Nov 2007 - Problem with sharing multiple zfs file systems

Here is one issue I am running into when setting up a new NFS server to share
several zfs file systems.

I created following zfs file system from a zfs pool called bigpool. The bigpool
is the top level file system and mounted as /export/bigpool.

file system                   mount point

bigpool                         /export/bigpool
bigpool/zfs1                  /export/bigpool/zfs1
bigpool/zfs2                  /export/bigpool/zfs2

All directories under /export are owned by a group called users. Also group
users have write access to them.

Next,  I exported bigpool (zfs1 and zfs2 inherited from bigpool) as NFS share.

zfs set sharenfs=on bigpool

On a Linux client, I can mounte all shares directly without problem. If I
mounted /export/bigpool to /mnt/nfs_share on the Linux client. The ownership and
permissions
on /mnt/nfs_share match to /export/bigpool on the nfs server.

However, permissions on /mnt/nfs_share/zfs1 or /mnt/nfs_share/zfs2 are not
inherited correctly. The group ownership is switched to root on
/mnt/nfs_share/zfs1,zfs2 and write permission is removed. I expect
/mnt/nfs_share/zfs1 should match /export/bigpool/zfs1, so does for zfs2. Why
ownership and permissions do not get inherited?

When I directly mount /export/bigpool/zfs1 to /mnt/nfs_share/zfs1, then
ownership and permissions match again.

Since with ZFS, creating and using multiple file systems are recommended
practice, does it mean that it will be lots of more trouble in managing NFS
shares on the system? Is there a way to only export top level file system and
let all permissions and ownership flow down correctly on client side? Or maybe
there are some special settings out there to solve my problem?

Any help is appreciated.

Simon
 
 
This message posted from opensolaris.org

On Nov 21, 2007, at 2:11 PM, Simon Gao wrote:
> Here is one issue I am running into when setting up a new NFS  
> server to share several zfs file systems.
>
> I created following zfs file system from a zfs pool called bigpool.  
> The bigpool is the top level file system and mounted as /export/ 
> bigpool.
>
> file system                   mount point
>
> bigpool                         /export/bigpool
> bigpool/zfs1                  /export/bigpool/zfs1
> bigpool/zfs2                  /export/bigpool/zfs2
>
> All directories under /export are owned by a group called users.  
> Also group users have write access to them.
>
> Next,  I exported bigpool (zfs1 and zfs2 inherited from bigpool) as  
> NFS share.
>
> zfs set sharenfs=on bigpool
>
> On a Linux client, I can mounte all shares directly without  
> problem. If I mounted /export/bigpool to /mnt/nfs_share on the  
> Linux client. The ownership and permissions
> on /mnt/nfs_share match to /export/bigpool on the nfs server.
>
> However, permissions on /mnt/nfs_share/zfs1 or /mnt/nfs_share/zfs2  
> are not inherited correctly. The group ownership is switched to  
> root on /mnt/nfs_share/zfs1,zfs2 and write permission is removed. I  
> expect /mnt/nfs_share/zfs1 should match /export/bigpool/zfs1, so  
> does for zfs2. Why ownership and permissions do not get inherited?
>
> When I directly mount /export/bigpool/zfs1 to /mnt/nfs_share/zfs1,  
> then ownership and permissions match again.
>
> Since with ZFS, creating and using multiple file systems are  
> recommended practice, does it mean that it will be lots of more  
> trouble in managing NFS shares on the system? Is there a way to  
> only export top level file system and let all permissions and  
> ownership flow down correctly on client side? Or maybe there are  
> some special settings out there to solve my problem?
>
> Any help is appreciated.
What you are describing is general NFS behavior.  Nothing special
about ZFS usage here.  When mounting /export/bigpool at the client,
the client observes the underlying directory /export/bigpool/zfs1 and
hence the change in ownership and permissions.  When the client
mounts the path /export/bigpool/zfs1, it is accessing that
filesystems directory and has the ownership and other attributes
that are expected of that filesystem.

With an NFSv4 client that provides ''mirror mounts'', the client
will be able to mount /export/bigpool and have the underlying  
filesystems
automatically mounted when accessed and the behavior you describe
will be alleviated by the access of the desired filesystem.

Spencer

Given that it will be some time before NFSv4 support, let alone NFSv4 support
for mount point crossing, in most client operating systems ... what obstacles
are in the way of constructing an NFSv3 server which would ''do the
right thing'' transparently to clients so long as the file systems
involved were within a single ZFS pool?

So far I can think of (a) clients expect inode numbers to be unique -- this
could be solved by making them (optionally) unique within a pool; (b) rename and
link semantics depend on the file system -- for rename this is easy, for link it
might require a cross-file-system hard link object, which is certainly doable.

This would go a long way towards making ZFS-with-many-filesystems approaches
more palatable. (Hmmm, how does CIFS support deal with the many-filesystems
problem today?)
 
 
This message posted from opensolaris.org

Anton B. Rang wrote:
> Given that it will be some time before NFSv4 support, let alone NFSv4
support for mount point crossing, in most client operating systems ... what
obstacles are in the way of constructing an NFSv3 server which would
''do the right thing'' transparently to clients so long as the
file systems involved were within a single ZFS pool?
>
> So far I can think of (a) clients expect inode numbers to be unique -- this
could be solved by making them (optionally) unique within a pool; (b) rename and
link semantics depend on the file system -- for rename this is easy, for link it
might require a cross-file-system hard link object, which is certainly doable.
>
> This would go a long way towards making ZFS-with-many-filesystems
approaches more palatable. (Hmmm, how does CIFS support deal with the
many-filesystems problem today?)
>  
>   
One solution, which is only about 18 years old, is automount.
Does anyone know how ubiquitous automount clients are?
 -- richard

On Nov 27, 2007, at 1:36 AM, Anton B. Rang wrote:
> Given that it will be some time before NFSv4 support, let alone  
> NFSv4 support for mount point crossing, in most client operating  
> systems ... what obstacles are in the way of constructing an NFSv3  
> server which would ''do the right thing'' transparently to
clients so
> long as the file systems involved were within a single ZFS pool?
>
> So far I can think of (a) clients expect inode numbers to be unique  
> -- this could be solved by making them (optionally) unique within a  
> pool; (b) rename and link semantics depend on the file system --  
> for rename this is easy, for link it might require a cross-file- 
> system hard link object, which is certainly doable.
>
> This would go a long way towards making ZFS-with-many-filesystems  
> approaches more palatable. (Hmmm, how does CIFS support deal with  
> the many-filesystems problem today?)
>
What you describe is the "nohide" option that was first
introduced in Irix and picked up in the Linux NFS server implementation.

As you say, inode number uniqueness would be one key issue along
with the others you mention.

I have a bias but I would rather see effort placed into dealing
with issues that stand in the way of effective deployment and
use of NFSv4; it will be more effective at CIFS/NFS co-existence
and deals with a number of other issues that can not be
easily solved with NFSv3.

Spencer

On Nov 27, 2007, at 9:48 AM, Richard Elling wrote:
> Anton B. Rang wrote:
>> Given that it will be some time before NFSv4 support, let alone  
>> NFSv4 support for mount point crossing, in most client operating  
>> systems ... what obstacles are in the way of constructing an NFSv3  
>> server which would ''do the right thing'' transparently
to clients
>> so long as the file systems involved were within a single ZFS pool?
>>
>> So far I can think of (a) clients expect inode numbers to be  
>> unique -- this could be solved by making them (optionally) unique  
>> within a pool; (b) rename and link semantics depend on the file  
>> system -- for rename this is easy, for link it might require a  
>> cross-file-system hard link object, which is certainly doable.
>>
>> This would go a long way towards making ZFS-with-many-filesystems  
>> approaches more palatable. (Hmmm, how does CIFS support deal with  
>> the many-filesystems problem today?)
>>
>>
> One solution, which is only about 18 years old, is automount.
> Does anyone know how ubiquitous automount clients are?
I think you have answered your own question.  18 years and
we have to ask the question? It must not be very ubiquitous. :-)

The fact is that "automount" has ended up meaning different
things to the various operating environments.  The maps and options
are slightly different in implementations and therefore it leads
to not being an effective method of managing a namespace (along with
a host of other issues that stand in the way).  We need to move
away from automount usage and on to namespaces that are managed
at the server side of things and integrated in a way to allow
admins to effectively manage large environments.  NFSv4 and
the client''s ability to move from filesystem to filesystem at
the same server and then to be "referred" to another server
gives us a reasonable base to start from.  There is an effort
that is going on to define the back end server namespace management
and it turns out that it will be discussed at the next IETF meeting.
If you are interested in the ideas to date, you can check out the
following internet draft:

http://www.ietf.org/internet-drafts/draft-tewari-federated-fs- 
protocol-00.txt

Spencer