zfs discuss - Nov 2005 - NFS question (and Best Practices)

I saw in another post that a best practices doc will be coming, but I figured I
would try to get this working.

I''m trying to understand why zfs uses so many "zfs create" so
I can use it better.  What makes sense is that each zfs fs can have
it''s own options (compression, nfs, atime, quota, etc).  I really love
this because it is so tuneable -- compression on these dirs, no atime on those
-- so flexible.  (But please let me know if I''m not on the right track)

But here is the problem I''ve run into -- nfs exporting.  In the
examples they make tank/home/{ann,bob,etc}.  Now if I "zfs set sharenfs=on
tank/home", that option gets inherited down and "showmount -e"
gives:

export list for solaristest:
/tank/home              (everyone)
/tank/home/ann       (everyone)
etc...

So now if I have another server mount solaristest:/tank/home, the {ann,bob,etc}
dirs are there, but writes do not go to /tank/home/{ann,bob,etc}.  Just so I am
clear, mounting solaristest:/tank/home to /home and cd''ing to ann IS
NOT the same as mounting solaristest:/tank/home/ann to /home/ann.  Note that I
understand this is not a nfs bug -- the *BSD''s are very similar in this
regard while linux couldn''t care less.  (client was linux2.6 and I
assume nfs v3)

So I guess I''m asking for best practices if we want to export
"tank/home" and be able set compression, snapshots, quotas, etc on the
zfs''s under that?

Oh, and sorry if this is a dumb question -- just installed solaris and I
didn''t see any solutions in share_nfs(1M) and I''m looking to
buy a good book now.
This message posted from opensolaris.org

Hi

On 11/20/05 04:45, Jeb Campbell wrote:
> I saw in another post that a best practices doc will be coming, but I
figured I would try to get this working.
> 
> I''m trying to understand why zfs uses so many "zfs
create" so I can use it better.  What makes sense is that each zfs fs can
have it''s own options (compression, nfs, atime, quota, etc).  I really
love this because it is so tuneable -- compression on these dirs, no atime on
those -- so flexible.  (But please let me know if I''m not on the right
track)
There are also advantages of separation - eg if you have a great big home pool
with each user
having their own filesystem from it then it is trivial to see who is using what.
> But here is the problem I''ve run into -- nfs exporting.  In the
examples they make tank/home/{ann,bob,etc}.  Now if I "zfs set sharenfs=on
tank/home", that option gets inherited down and "showmount -e"
gives:
> 
> export list for solaristest:
> /tank/home              (everyone)
> /tank/home/ann       (everyone)
> etc...
> 
> So now if I have another server mount solaristest:/tank/home, the
{ann,bob,etc} dirs are there, but writes do not go to /tank/home/{ann,bob,etc}. 
Just so I am clear, mounting solaristest:/tank/home to /home and cd''ing
to ann IS NOT the same as mounting solaristest:/tank/home/ann to /home/ann. 
Note that I understand this is not a nfs bug -- the *BSD''s are very
similar in this regard while linux couldn''t care less.  (client was
linux2.6 and I assume nfs v3)
You won''t even be *able* to mount solaristest:/tank/home from an nfs
client.  The subdirectories
/tank/home/* can''t be shared if their parent directory /tank/home is
already shared.

You''re setting the sharenfs property on the tank/home container.  The
container itself
does not get shared (ie /tank/home) but as you''ve said this becomes the
default share option for filesystem within this container.  So each
individual filesystem is shared - you''ll see this in share(1m) on the
nfs server.
> So I guess I''m asking for best practices if we want to export
"tank/home" and be able set compression, snapshots, quotas, etc on the
zfs''s under that?
You''re halfway there.  Create separate filesystems as you have in the
container using the
default or individual properties for compression etc, and share them (using the
container default or any individual special ones if requried).

Now you need to tell the nfs client where to get the filesystems from.  The
easiest and best way
is to use the automounter.  For solaris clients the automounter is exceptionally
flexible;  others
are less so but you should be able to find the common denominator for your
client mix.  At
it''s simplest you''ll want to appoint /home as an indirect
automount mount in auto_master
(this is the default in Solaris) and in auto_home (in nis/nis+, or
/etc/auto_home)
you''d have

* solaristest:/tank/home/&

The ''*'' matches all, and the ''&'' expands
to the match.  So when somebody cd''s to /home/ann
this entry will tell the automounter to mount solaristest:/tank/home/ann

Hope that helps.  automount(1m) has much info on automounting in Solaris.

Gavin

Thanks for your response.  I imagined I would have to go to automount, but
wanted to see if I was missing something.
> There are also advantages of separation - eg if you
> have a great big home pool with each user
> having their own filesystem from it then it is
> trivial to see who is using what.
Ok, that is what I thought.  (lol -- could you imagine keeping /etc/fstab up to
date?).  Just from 2 days of use, ZFS has changed what I expect from a
filesystem . . .
> You won''t even be *able* to mount
> solaristest:/tank/home from an nfs client.  The
> subdirectories
> /tank/home/* can''t be shared if their parent
> directory /tank/home is already shared.
Shouldn''t or Can''t be shared.  I did mount /tank/home and that
is where I noticed this (but I was trying to find out if solaris is like freebsd
or linux).  Hmm, I guess I need to read more.

Thanks again for your time and help.
This message posted from opensolaris.org

On 11/21/05 17:36, Jeb Campbell wrote:
>>You won''t even be *able* to mount
>>solaristest:/tank/home from an nfs client.  The
>>subdirectories
>>/tank/home/* can''t be shared if their parent
>>directory /tank/home is already shared.
> 
> 
> Shouldn''t or Can''t be shared.  I did mount /tank/home and
that is where I noticed this (but I was trying to find out if solaris is like
freebsd or linux).  Hmm, I guess I need to read more.
Can''t (on Solaris).

# mkdir -p /tmp/t/a /tmp/t/b
# share -F nfs /tmp/t
# share -F nfs /tmp/t/a
share_nfs: /tmp/t/a: parent-directory (/tmp/t) already shared

If you do the equivalent thing from zfs:

# zfs create -c tank/test
# zfs set sharenfs=on tank/test
# zfs create tank/test/a
# zfs create tank/test/b

and run ''share'' to see what is shared you see it is the
subdirectories (mountpoints):

-               /zfs/tank/test/a   rw   ""
-               /zfs/tank/test/b   rw   ""

 From a Solaris client if you try to mount the parent:

# mount -F nfs tb2:/zfs/tank/test /mnt
nfs mount: tb2:/zfs/tank/test: Permission denied

If you go via the special /net automounter point which exists
by default you appear to be able to mount the parent directory:

$ cd /net/tb2/zfs/tank/test
$ ls
test

But ... this is actually an autofs-magic mount point created on the fly for
you:

$ df -h .
Filesystem             size   used  avail capacity  Mounted on
-hosts                   0K     0K     0K     0%    /net/tb2/zfs/tank

$ grep tank /etc/mnttab
-hosts  /net/tb2/zfs/tank/test  autofs  nosuid,ignore,nest,nobrowse,dev=4c055aa
1132607830

Cheers

Gavin

Thanks for the great tips in there!

I guess the confusion is coming from the fact that my full install of Solaris
Express b27a is not "pruning" those top shares:

bash-3.00# share
-               /test/jeb   anon=0   ""
-               /test/jeb/nocomp   anon=0   ""

(and yeah, anon=0 is bad, but this testing @ home... )

I just created test/jeb/again to verify:

bash-3.00# share
-               /test/jeb   anon=0   ""
-               /test/jeb/nocomp   anon=0   ""
-               /test/jeb/again   anon=0   ""

So is this a bug?  I haven''t installed anything (don''t know
how yet).

Just to make sure I''m not crazy, I blew everything away and started
again:

bash-3.00# share
-               /test/nfs   rw   ""
bash-3.00# zfs create test/nfs/jeb
bash-3.00# zfs create test/nfs/another
bash-3.00# share
-               /test/nfs   rw   ""
-               /test/nfs/jeb   rw   ""
-               /test/nfs/another   rw   ""
This message posted from opensolaris.org

Hi again,

On 11/21/05 21:39, Jeb Campbell wrote:
> Just to make sure I''m not crazy, I blew everything away and
started again:
> 
> bash-3.00# share
> -               /test/nfs   rw   ""
If you blew everything away why is this share still present?
> bash-3.00# zfs create test/nfs/jeb
> bash-3.00# zfs create test/nfs/another
> bash-3.00# share
> -               /test/nfs   rw   ""
> -               /test/nfs/jeb   rw   ""
> -               /test/nfs/another   rw   ""
Hmmm, I think ''nfs'' in the above is not a zfs container.  Can
you mail the
output of ''zfs list -o name,type,mountpoint,sharenfs''

I can recreate
your situation with:

# zfs create tank/test
# zfs create tank/test/a
# zfs create tank/test/b
# zfs set sharenfs=on tank/test
# zfs set sharenfs=on tank/test/a
# zfs set sharenfs=on tank/test/b

after which

# share | grep test
-               /zfs/tank/test   rw   ""
-               /zfs/tank/test/a   rw   ""
-               /zfs/tank/test/b   rw   ""

That, I believe, is because we''re sharing whole filesystems.  What
I said about not being able to share subdirs applies within a
filesystem.

If instead we use the zfs container concept:

# zfs create -c tank/test
# zfs set sharenfs=on tank/test
# zfs create tank/test/a
# zfs create tank/test/b

# share | grep test
-               /zfs/tank/test/a   rw   ""
-               /zfs/tank/test/b   rw   ""

... just the filesystems themselves shared.

Cheers

Gavin

On Mon, Nov 21, 2005 at 11:03:57PM +0000, Gavin Maltby
wrote:
> 
> If instead we use the zfs container concept:
> 
> # zfs create -c tank/test
> # zfs set sharenfs=on tank/test
> # zfs create tank/test/a
> # zfs create tank/test/b
> 
> # share | grep test
> -               /zfs/tank/test/a   rw   ""
> -               /zfs/tank/test/b   rw   ""
> 
> ... just the filesystems themselves shared.
ZFS containers in this form no longer exist.  They were removed a few
weeks prior to integration.  Similar functionality may be added at some
point in the form of a property.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Mon, Nov 21, 2005 at 11:03:57PM +0000, Gavin Maltby
wrote:
> Hi again,
> If instead we use the zfs container concept:
> 
> # zfs create -c tank/test
> # zfs set sharenfs=on tank/test
> # zfs create tank/test/a
> # zfs create tank/test/b
> 
> # share | grep test
> -               /zfs/tank/test/a   rw   ""
> -               /zfs/tank/test/b   rw   ""
This was only in the pre-putback ZFS;  there is no longer anything called
a "container" in ZFS.

Cheers,
- jonathan

-- 
Jonathan Adams, Solaris Kernel Development

On 11/21/05 23:13, Eric Schrock wrote:
> ZFS containers in this form no longer exist.  They were removed a few
> weeks prior to integration.  Similar functionality may be added at some
> point in the form of a property.
Awww nuts - I''m still running bits from the last internal packages.
Looks like I have some relearning to do - I really liked the old
container concept.

Gavin

To add a little background info here...

If your server has one pool/filesystem (FSA) and one filesystem under it (FSB)
and everything''s shared:
fsh-suzuki# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
FSA                   67.5K  67.5G   9.5K  /FSA
FSA/FSB                  8K  67.5G     8K  /FSA/FSB
server# zfs get -o name,value,property sharenfs FSA FSA/FSB
NAME             VALUE                      PROPERTY       
FSA              on                         sharenfs
FSA/FSB          on                         sharenfs
server# 

Then if the client manually mounts FSA, it won''t be able to see the
contents of FSB due to FSID crossing:
server#ls /FSA/FSB                        
this_is_subfs.txt
server# 

client# mount -o vers=4 fsh-suzuki:/FSA /mnt
client# ls /mnt
FSB               this_is_root.txt
client# ls /mnt/FSB
client# 

The NFSv4 protocol allows for "mirror mounting" (as we internally call
it), that would allow the client to detect a FSID change and then automatically
mount the new filesystem.  We want this, just haven''t coded it yet (any
volunteers? ).  I talk a little about this here:
http://blogs.sun.com/roller/page/erickustarz?entry=bigger_filehandles_for_nfsv4_die

Getting access to both filesystems *can* work with the automounter - if the
timing of created fileystems is right with when you access the share
filesystems.  Basically, the automounter will do ''showmount
-e'' the first time you access that machine''s exports, and
everything that''s listed will become "trigger" nodes.  So in
the above case:

server# showmount -e
export list for server:
/FSA     (everyone)
/FSA/FSB (everyone)
server# 

Now if a client access''s a trigger node (/FSA or /FSA/FSB), it knows to
do a mount on that trigger, and you''ll get the contents you were
looking for:
client# cd /net/server
client# ls
FSA
client# cd FSA
client# ls
FSB               this_is_root.txt
client# cd FSB
client# ls
this_is_subdir.txt
client# 

If however, you just had ''FSA'' created and shared (FSB
doesn''t exist yet):
server# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
FSA                   62.0K  67.5G   9.5K  /FSA
server# 
server# showmount -e
export list for server:
/FSA     (everyone)
server# 

Then have the client access the server''s exports via the autmounter,
the client sets its "triggers" and only sees one trigger.  If the FSB
is then created after this point, the client will *not* update its
"trigger"s, and thusly won''t know to auto-mount FSB, and
won''t see its real contents:

client# cd /net/server
client# ls
FSA
client# cd FSA
client# ls
this_is_root.txt
client#

At this point, the "triggers" from the client''s point of view
have already settled, now if the server exports more, the client won''t
see it:

server# zfs create FSA/FSB 
server# echo 1 > /FSA/FSB/this_is_subfs.txt
server# ls /FSA/FSB
this_is_subfs.txt
server# showmount -e
export list for server:
/FSA     (everyone)
/FSA/FSB (everyone)
server# 

client# ls
FSB               this_is_root.txt
client# ls FSB
client#

Note: FSB appears to be empty - though locally on ''server'' it
is not.

Yeah, this is less than desireable, but it isn''t easy to fix, and i
won''t go into the details and keep this a zfs thread :)

eric
This message posted from opensolaris.org

Ok, for completeness and searching I''ll attempt to summarize:
(notice the attempt...)

1.  Use automount (and have everything created if I read that last one right) or
just don''t "zfs create" below what you want to export... --
Choose your poison.

2.  ZFS had, can, and might later have containers, an object that can only hold
other zfs filesystems for the purpose of setting default options of
"contained" zfs fs''s.

Thanks again folks!  And I can''t say it enough.
This message posted from opensolaris.org

Jeb Campbell wrote:
>Ok, for completeness and searching I''ll attempt to summarize:
>(notice the attempt...)
>
>1.  Use automount (and have everything created if I read that last one
right) or just don''t "zfs create" below what you want to
export... -- Choose your poison.
>  
>
Sorta...  if you''ve already accessed the server''s exported
filesystems,
and a new ''zfs create'' comes along, just know that either you
won''t see
inside the new filesystem, or you can umount all auto-mounted mounts 
from that server and everything will be cool - the "triggers" will get
updated the next time you access that server.
>2.  ZFS had, can, and might later have containers, an object that can only
hold other zfs filesystems for the purpose of setting default options of
"contained" zfs fs''s.
>  
>
ZFS (before it integrated into nevada) had containers and they aren''t 
coming back.

eric
>Thanks again folks!  And I can''t say it enough.
>This message posted from opensolaris.org
>_______________________________________________
>zfs-discuss mailing list
>zfs-discuss at opensolaris.org
>http://opensolaris.org/mailman/listinfo/zfs-discuss
>  
>