ZFSfolk, Pardon the slightly offtopic post, but I figured this would be a good forum to get some feedback. I am looking at implementing zfs group quotas on some X4540s and X4140/J4400s, 64GB of RAM per server, running Solaris 10 Update 8 servers with IDR143158-06. There is one large filesystem per server that is served via NFSv3 to linux based clients for web and email loads. There will be at least a few thousand group quotas per filesystem. Are there any scaling/performance issues with group based quotas? For the filesystems that are already populated with thousands of groups and terabytes of data in relatively small files, will there be any performance impacts as the quotas are created? Also for the pre-populated filesystems will "zfs get groupspace@$gid $zpool/$fs" return the total usage for the group? Thanks for any feedback, Jordan Schwartz
Jordan Schwartz wrote:> There is one large filesystem per server that is served via NFSv3 toI''ve noticed that everytime someone mentions using NFS with ZFS here, they always seem to be using NFSv3. Is there a reason for this that I just don''t know about? To me, using NFSv4 is a no-brainer. ZFS supports it natively, it supports all the wonderful extra capabilities that the ZFS ACLs allow, has stronger security, stateful protocol, and all kinds of other nifty stuff. Why do people seem to be clinging so rabidly to the old version? Is there some technical reason I''m missing? Doug Linder I apologize for all the stupid cruft below that my company''s mail server adds. ---------- Learn more about Merchant Link at www.merchantlink.com. THIS MESSAGE IS CONFIDENTIAL. This e-mail message and any attachments are proprietary and confidential information intended only for the use of the recipient(s) named above. If you are not the intended recipient, you may not print, distribute, or copy this message or any attachments. If you have received this communication in error, please notify the sender by return e-mail and delete this message and any attachments from your computer.
On Wed, August 18, 2010 15:14, Linder, Doug wrote:> I''ve noticed that everytime someone mentions using NFS with ZFS here, they > always seem to be using NFSv3. Is there a reason for this that I just > don''t know about? To me, using NFSv4 is a no-brainer. ZFS supports it > natively, it supports all the wonderful extra capabilities that the ZFS > ACLs allow, has stronger security, stateful protocol, and all kinds of > other nifty stuff. Why do people seem to be clinging so rabidly to the > old version? Is there some technical reason I''m missing?At $WORK it''s generally namespace issues: http://blogs.sun.com/tdh/entry/linux_nfsv4_namespace_implementation_fools Haven''t really found a use for the "extras" that NFSv4 adds, so it''s not worth the effort.
Il giorno 18/ago/2010, alle ore 21.24, David Magda ha scritto:> On Wed, August 18, 2010 15:14, Linder, Doug wrote: >> I''ve noticed that everytime someone mentions using NFS with ZFS here, they >> always seem to be using NFSv3. Is there a reason for this that I just >> don''t know about? > At $WORK it''s generally namespace issues: > > http://blogs.sun.com/tdh/entry/linux_nfsv4_namespace_implementation_foolsAlso the linux NFSv4 client is bugged (as in hang-the-whole-machine bugged). I am deploying a new osol fileserver for home directories and I''m using NFSv3 + automounter (because I am also using one dataset per user, and thus I have to mount each home dir separately). -- Simone Caldana
> > Also the linux NFSv4 client is bugged (as in hang-the-whole-machine bugged). > I am deploying a new osol fileserver for home directories and I''m using NFSv3 + automounter (because I am also using one dataset per user, and thus I have to mount each home dir separately).We are also in the same boat here. I have about 125TB of ZFS storage in production currently, running OSOL, across 5 X4540s. We tried the NFSv4 route, and crawled back to NFSv3 and the linux automounter because NFSv4 on Linux is *that* broken. As in hung-disk-io-that-wedges-the-whole-box broken. We know that NFSv3 was never meant for the scale we''re using it at, but we have no choice in the matter. On the topic of Linux clients, NFS and ZFS: We''ve also found that Linux is bad at handling lots of mounts/umounts. We will occasionally find a client where the automounter requested a mount, but it never actually completed. It''ll show as mounted in /proc/mounts, but won''t *actually* be mounted. A umount -f for the affected filesystem fixes this. On ~250 clients in an HPC environment, we''ll see such an error every week or so. I''m hoping that recent versions of linux (i.e. RHEL 6) are a bit better at NFSv4, but i''m not holding my breath. -- Greg Mason HPC Administrator Michigan State University Institute for Cyber Enabled Research High Performance Computing Center web: www.icer.msu.edu email: gmason at msu.edu
Well I seemed to have hit on that hot button topic of NFSv4, (good thing I didn''t mention that we are running IPv4). To get back to the topic, is anyone running ZFS group quota on large filesystem with lots of smaller files and thousands for groups per filesystem, or have any quota related experinces to share? Thanks, Jordan On Tue, Aug 17, 2010 at 5:20 PM, Jordan Schwartz <jordan247 at gmail.com> wrote:> ZFSfolk, > > Pardon the slightly offtopic post, but I figured this would be a good > forum to get some feedback. > > I am looking at implementing zfs group quotas on some X4540s and > X4140/J4400s, 64GB of RAM per server, running Solaris 10 Update 8 > servers with IDR143158-06. > > There is one large filesystem per server that is served via NFSv3 to > linux based clients for web and email loads. There will be at least a > few thousand group quotas per filesystem. > > Are there any scaling/performance issues with group based quotas? > > For the filesystems that ?are already populated with thousands of > groups and terabytes of data in relatively small files, will there be > any performance impacts as the quotas are created? > > Also for the pre-populated filesystems will "zfs get groupspace@$gid > $zpool/$fs" return the total usage for the group? > > Thanks for any feedback, > > Jordan Schwartz >
Not sure if there was similar threads in this list before. Three scenarios: 1): df cannot count snapshot space in a file system with quota set. 2): df cannot count sub-filesystem space in a file system with quota set. 3): df cannot count space saved by de-dup in a file system with quota set. Are they possible? Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? Thanks. Fred
df serves a purpose though. There are other commands which output that information.. On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com> wrote:> Not sure if there was similar threads in this list before. > Three scenarios: > 1): df cannot count snapshot space in a file system with quota set. > 2): df cannot count sub-filesystem space in a file system with quota set. > 3): df cannot count space saved by de-dup in a file system with quota set. > > Are they possible? > > Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? > > Thanks. > > Fred > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/b7a660d9/attachment.html>
Can you shed more lights on *other commands* which output that information? Appreciations. Fred From: Thomas Burgess [mailto:wonslung at gmail.com] Sent: ???, ?? 20, 2010 17:34 To: Fred Liu Cc: ZFS Discuss Subject: Re: [zfs-discuss] make df have accurate out upon zfs? df serves a purpose though. There are other commands which output that information.. On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com<mailto:Fred_Liu at issi.com>> wrote: Not sure if there was similar threads in this list before. Three scenarios: 1): df cannot count snapshot space in a file system with quota set. 2): df cannot count sub-filesystem space in a file system with quota set. 3): df cannot count space saved by de-dup in a file system with quota set. Are they possible? Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? Thanks. Fred _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org<mailto:zfs-discuss at opensolaris.org> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/03454ded/attachment.html>
can''t the "zfs" command provide that information? 2010/8/20 Fred Liu <Fred_Liu at issi.com>> Can you shed more lights on **other commands** which output that > information? > > Appreciations. > > > > Fred > > > > *From:* Thomas Burgess [mailto:wonslung at gmail.com] > *Sent:* ???, ?? 20, 2010 17:34 > *To:* Fred Liu > *Cc:* ZFS Discuss > *Subject:* Re: [zfs-discuss] make df have accurate out upon zfs? > > > > df serves a purpose though. > > > > There are other commands which output that information.. > > On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com> wrote: > > Not sure if there was similar threads in this list before. > Three scenarios: > 1): df cannot count snapshot space in a file system with quota set. > 2): df cannot count sub-filesystem space in a file system with quota set. > 3): df cannot count space saved by de-dup in a file system with quota set. > > Are they possible? > > Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? > > Thanks. > > Fred > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/8d93a4fe/attachment.html>
as for the difference between the two df''s, one is the gnu df (liek you''d have on linux) and the other is the solaris df. 2010/8/20 Thomas Burgess <wonslung at gmail.com>> can''t the "zfs" command provide that information? > > > 2010/8/20 Fred Liu <Fred_Liu at issi.com> > > Can you shed more lights on **other commands** which output that >> information? >> >> Appreciations. >> >> >> >> Fred >> >> >> >> *From:* Thomas Burgess [mailto:wonslung at gmail.com] >> *Sent:* ???, ?? 20, 2010 17:34 >> *To:* Fred Liu >> *Cc:* ZFS Discuss >> *Subject:* Re: [zfs-discuss] make df have accurate out upon zfs? >> >> >> >> df serves a purpose though. >> >> >> >> There are other commands which output that information.. >> >> On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com> wrote: >> >> Not sure if there was similar threads in this list before. >> Three scenarios: >> 1): df cannot count snapshot space in a file system with quota set. >> 2): df cannot count sub-filesystem space in a file system with quota set. >> 3): df cannot count space saved by de-dup in a file system with quota set. >> >> Are they possible? >> >> Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? >> >> Thanks. >> >> Fred >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/84721f38/attachment.html>
Sure, I know this. What I want to say is following: root at cn03:~# /usr/gnu/bin/df -h /cn03/3 Filesystem Size Used Avail Use% Mounted on cn03/3 298G 154K 298G 1% /cn03/3 root at cn03:~# /bin/df -h /cn03/3 Filesystem size used avail capacity Mounted on cn03/3 800G 154K 297G 1% /cn03/3 root at cn03:~# zfs get all cn03/3 NAME PROPERTY VALUE SOURCE cn03/3 type filesystem - cn03/3 creation Sat Jul 10 9:35 2010 - cn03/3 used 503G - cn03/3 available 297G - cn03/3 referenced 154K - cn03/3 compressratio 1.00x - cn03/3 mounted yes - cn03/3 quota 800G local cn03/3 reservation none default cn03/3 recordsize 128K default cn03/3 mountpoint /cn03/3 default cn03/3 sharenfs rw,root=nfsroot local cn03/3 checksum on default cn03/3 compression off default cn03/3 atime on default cn03/3 devices on default cn03/3 exec on default cn03/3 setuid on default cn03/3 readonly off default cn03/3 zoned off default cn03/3 snapdir hidden default cn03/3 aclmode groupmask default cn03/3 aclinherit restricted default cn03/3 canmount on default cn03/3 shareiscsi off default cn03/3 xattr on default cn03/3 copies 1 default cn03/3 version 4 - cn03/3 utf8only off - cn03/3 normalization none - cn03/3 casesensitivity sensitive - cn03/3 vscan off default cn03/3 nbmand off default cn03/3 sharesmb off default cn03/3 refquota none default cn03/3 refreservation none default cn03/3 primarycache all default cn03/3 secondarycache all default cn03/3 usedbysnapshots 46.8G - cn03/3 usedbydataset 154K - cn03/3 usedbychildren 456G - cn03/3 usedbyrefreservation 0 - cn03/3 logbias latency default cn03/3 dedup off default cn03/3 mlslabel none default cn03/3 com.sun:auto-snapshot true inherited from cn03 Thanks. Fred From: Thomas Burgess [mailto:wonslung at gmail.com] Sent: ???, ?? 20, 2010 18:44 To: Fred Liu Cc: ZFS Discuss Subject: Re: [zfs-discuss] make df have accurate out upon zfs? as for the difference between the two df''s, one is the gnu df (liek you''d have on linux) and the other is the solaris df. 2010/8/20 Thomas Burgess <wonslung at gmail.com<mailto:wonslung at gmail.com>> can''t the "zfs" command provide that information? 2010/8/20 Fred Liu <Fred_Liu at issi.com<mailto:Fred_Liu at issi.com>> Can you shed more lights on *other commands* which output that information? Appreciations. Fred From: Thomas Burgess [mailto:wonslung at gmail.com<mailto:wonslung at gmail.com>] Sent: ???, ?? 20, 2010 17:34 To: Fred Liu Cc: ZFS Discuss Subject: Re: [zfs-discuss] make df have accurate out upon zfs? df serves a purpose though. There are other commands which output that information.. On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com<mailto:Fred_Liu at issi.com>> wrote: Not sure if there was similar threads in this list before. Three scenarios: 1): df cannot count snapshot space in a file system with quota set. 2): df cannot count sub-filesystem space in a file system with quota set. 3): df cannot count space saved by de-dup in a file system with quota set. Are they possible? Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? Thanks. Fred _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org<mailto:zfs-discuss at opensolaris.org> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/494b36bc/attachment.html>
try something like zfs list -o space zfs -t snapshot stuff like that 2010/8/20 Fred Liu <Fred_Liu at issi.com>> Sure, I know this. > > What I want to say is following: > > root at cn03:~# /usr/gnu/bin/df -h /cn03/3 > > Filesystem Size Used Avail Use% Mounted on > > cn03/3 298G 154K 298G 1% /cn03/3 > > root at cn03:~# /bin/df -h /cn03/3 > > Filesystem size used avail capacity Mounted on > > cn03/3 800G 154K 297G 1% /cn03/3 > > > > root at cn03:~# zfs get all cn03/3 > > NAME PROPERTY VALUE SOURCE > > cn03/3 type filesystem - > > cn03/3 creation Sat Jul 10 9:35 2010 - > > cn03/3 used 503G - > > cn03/3 available 297G - > > cn03/3 referenced 154K - > > cn03/3 compressratio 1.00x - > > cn03/3 mounted yes - > > cn03/3 quota 800G local > > cn03/3 reservation none default > > cn03/3 recordsize 128K default > > cn03/3 mountpoint /cn03/3 default > > cn03/3 sharenfs rw,root=nfsroot local > > cn03/3 checksum on default > > cn03/3 compression off default > > cn03/3 atime on default > > cn03/3 devices on default > > cn03/3 exec on default > > cn03/3 setuid on default > > cn03/3 readonly off default > > cn03/3 zoned off default > > cn03/3 snapdir hidden default > > cn03/3 aclmode groupmask default > > cn03/3 aclinherit restricted default > > cn03/3 canmount on default > > cn03/3 shareiscsi off default > > cn03/3 xattr on default > > cn03/3 copies 1 default > > cn03/3 version 4 - > > cn03/3 utf8only off - > > cn03/3 normalization none - > > cn03/3 casesensitivity sensitive - > > cn03/3 vscan off default > > cn03/3 nbmand off default > > cn03/3 sharesmb off default > > cn03/3 refquota none default > > cn03/3 refreservation none default > > cn03/3 primarycache all default > > cn03/3 secondarycache all default > > cn03/3 usedbysnapshots 46.8G - > > cn03/3 usedbydataset 154K - > > cn03/3 usedbychildren 456G - > > cn03/3 usedbyrefreservation 0 - > > cn03/3 logbias latency default > > cn03/3 dedup off default > > cn03/3 mlslabel none default > > cn03/3 com.sun:auto-snapshot true inherited from cn03 > > > > Thanks. > > > > Fred > > > > *From:* Thomas Burgess [mailto:wonslung at gmail.com] > *Sent:* ???, ?? 20, 2010 18:44 > > *To:* Fred Liu > *Cc:* ZFS Discuss > *Subject:* Re: [zfs-discuss] make df have accurate out upon zfs? > > > > as for the difference between the two df''s, one is the gnu df (liek you''d > have on linux) and the other is the solaris df. > > > > > > 2010/8/20 Thomas Burgess <wonslung at gmail.com> > > can''t the "zfs" command provide that information? > > > > 2010/8/20 Fred Liu <Fred_Liu at issi.com> > > > > Can you shed more lights on **other commands** which output that > information? > > Appreciations. > > > > Fred > > > > *From:* Thomas Burgess [mailto:wonslung at gmail.com] > *Sent:* ???, ?? 20, 2010 17:34 > *To:* Fred Liu > *Cc:* ZFS Discuss > *Subject:* Re: [zfs-discuss] make df have accurate out upon zfs? > > > > df serves a purpose though. > > > > There are other commands which output that information.. > > On Thu, Aug 19, 2010 at 3:01 PM, Fred Liu <Fred_Liu at issi.com> wrote: > > Not sure if there was similar threads in this list before. > Three scenarios: > 1): df cannot count snapshot space in a file system with quota set. > 2): df cannot count sub-filesystem space in a file system with quota set. > 3): df cannot count space saved by de-dup in a file system with quota set. > > Are they possible? > > Btw, what is the difference between " /usr/gnu/bin/df" and "/bin/df"? > > Thanks. > > Fred > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100820/63d9808c/attachment.html>
2010/8/20 Fred Liu <Fred_Liu at issi.com>:> Sure, I know this. > > What I want to say is following: > > root at cn03:~# /usr/gnu/bin/df -h /cn03/3 > > Filesystem??????????? Size? Used Avail Use% Mounted on > > cn03/3??????????????? 298G? 154K? 298G?? 1% /cn03/3 > > root at cn03:~# /bin/df -h /cn03/3 > > Filesystem???????????? size?? used? avail capacity? Mounted on > > cn03/3??????? ?????????800G?? 154K?? 297G???? 1%??? /cn03/3 >The Sun df (/usr/bin/df) is zfs aware, the GNU df is not. There are other areas where it matters, although I am blanking on where right now. I ran into it a couple years ago, where the GNU version of a command under Solaris 10 was just broken, while the Sun version was fine (I think the problem was with su). -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, RPI Players
Jordan Schwartz wrote:> ZFSfolk, > > Pardon the slightly offtopic post, but I figured this would be a good > forum to get some feedback. > > I am looking at implementing zfs group quotas on some X4540s and > X4140/J4400s, 64GB of RAM per server, running Solaris 10 Update 8 > servers with IDR143158-06. > > There is one large filesystem per server that is served via NFSv3 to > linux based clients for web and email loads. There will be at least a > few thousand group quotas per filesystem. > > Are there any scaling/performance issues with group based quotas? >No, this should work just fine.> For the filesystems that are already populated with thousands of > groups and terabytes of data in relatively small files, will there be > any performance impacts as the quotas are created? >If the filesystem is old (created on pool version < 15), then the first time you do "zfs userspace <fs>" or "zfs groupspace <fs>", it will take some time to do an initial gather of the accounting information. After that happens, there should be no noticeable performance impact of setting or using user or group quotas.> Also for the pre-populated filesystems will "zfs get groupspace@$gid > $zpool/$fs" return the total usage for the group?Yes. (Though if the filesystem is old you may need to do "zfs groupspace" once to gather the data.) --matt
Matt,> If the filesystem is old (created on pool version < 15), then the first time > you do "zfs userspace <fs>" or "zfs groupspace <fs>", it will take some time > to do an initial gather of the accounting information.This is good to know since we do have some upgraded filesystems / pools, thanks! I noticed you mentioned the pool version, I assume this is because a zfs version 4 filesystem can only exist on a zpool version 15 or later. Does that first run of zfs userspace or zfs groupspace gather all accounting information whether or not quotas have been set for all users or groups, or will there be an "full" accounting scan everytime a new quota is set and zfs groupspace is run? Thanks, Jordan