It seems by design that ZFS counts the space used by snapshots towards the filesystem quotas. I can see many cases where this would be the desired behavior, but for the common case of using one filesystem per user home directory in combination with quotas and snapshots, this doesn''t seem to work too well. If the user hits their quota, there seems to be no way for the user to recover space assuming the fairly common case where they have one or more snapshots from while their filesystem was fairly full. Instead they''d need to have an administrator either increase their quota (which seems to defeat the purpose of the quotas), or destroy all of their snapshots (which seems to defeat the purpose of using the snapshots for any sort of backup mechanism). Is there any way to work around this with the quota and snapshot mechanisms built into ZFS? Thanks, Ed Plese
Ed Plese wrote:> It seems by design that ZFS counts the space used by snapshots towards > the filesystem quotas. I can see many cases where this would be the > desired behavior, but for the common case of using one filesystem per > user home directory in combination with quotas and snapshots, this > doesn''t seem to work too well. > > If the user hits their quota, there seems to be no way for the user to > recover space assuming the fairly common case where they have one or > more snapshots from while their filesystem was fairly full. Instead > they''d need to have an administrator either increase their quota (which > seems to defeat the purpose of the quotas), or destroy all of their > snapshots (which seems to defeat the purpose of using the snapshots for > any sort of backup mechanism).I believe this problem will go away once we have the ability to delegate the file system to the end user. That way they would have the ability to create and delete either own snapshots below their part of the namespace. I think even with that though you have a good point, why should the users qutoa be reduced because the admin is taking snapshots. I can see that it would be valuable to indicate if snapshots count against the quota. However once you start using the snapshots for clones you need to do the correct space accounting to the user. -- Darren J Moffat
No, but this is a known issue. See: 6431277 want filesystem-only quotas - Eric On Wed, Jun 14, 2006 at 11:58:25AM -0500, Ed Plese wrote:> It seems by design that ZFS counts the space used by snapshots towards > the filesystem quotas. I can see many cases where this would be the > desired behavior, but for the common case of using one filesystem per > user home directory in combination with quotas and snapshots, this > doesn''t seem to work too well. > > If the user hits their quota, there seems to be no way for the user to > recover space assuming the fairly common case where they have one or > more snapshots from while their filesystem was fairly full. Instead > they''d need to have an administrator either increase their quota (which > seems to defeat the purpose of the quotas), or destroy all of their > snapshots (which seems to defeat the purpose of using the snapshots for > any sort of backup mechanism). > > Is there any way to work around this with the quota and snapshot > mechanisms built into ZFS? > > Thanks, > > Ed Plese > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
Thanks, that''s exactly what I was looking for. Ed Plese On Wed, Jun 14, 2006 at 10:09:35AM -0700, Eric Schrock wrote:> No, but this is a known issue. See: > > 6431277 want filesystem-only quotas > > - Eric > > On Wed, Jun 14, 2006 at 11:58:25AM -0500, Ed Plese wrote: > > It seems by design that ZFS counts the space used by snapshots towards > > the filesystem quotas. I can see many cases where this would be the > > desired behavior, but for the common case of using one filesystem per > > user home directory in combination with quotas and snapshots, this > > doesn''t seem to work too well. > > > > If the user hits their quota, there seems to be no way for the user to > > recover space assuming the fairly common case where they have one or > > more snapshots from while their filesystem was fairly full. Instead > > they''d need to have an administrator either increase their quota (which > > seems to defeat the purpose of the quotas), or destroy all of their > > snapshots (which seems to defeat the purpose of using the snapshots for > > any sort of backup mechanism). > > > > Is there any way to work around this with the quota and snapshot > > mechanisms built into ZFS? > > > > Thanks, > > > > Ed Plese > > > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > -- > Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
I''ve run into this myself. (I am in a university setting). after reading bug ID 6431277 (URL below for noobs like myself who didn''t know what "see 6431277" meant): http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6431277 ...it''s not clear to me how this will be resolved. What I''d really like to see is the ability to specify where the snapshot backing store will be. (more specifically, the ability for the snapshot space to *not* impact the filesystem space). We have a network Appliance box, whose snapshots are very popular for their value as online backups. Netapp charges snapshots to the storage pool, so they don''t cost the filesystem anything. I''m drooling over ZFS as an alternative to the expensive netapp hardware/software, but since we sell RAID space but perform backups administratively, I can''t have the snapshots consuming people''s space. I could increase the filesystem quota to accomodate the snapshots, but since the snapshot size is dynamic, I would have to increase it well beyond the current snapshot size. Once I do that, users *will* fill the space (that they have not paid for). I could tune the size of the filesystem to match the snapshot + filesystem data, but since snapshot size is dynamic, this is impractical. I also have some very small quotas (50 MB) for users, and would like to be able to create snapshots of them going back 30 days or so without it costing the user anything. The snapshots save us tons of time and effort, but they''re not worth it to the user to pay for double or triple the space they''re currently using, and I don''t want the users going over the original quota of 50 MB, so I can''t make enough space in the filesystem to make snapshots of their data... it''s maddening. If we must contain snapshots inside a filesystem, perhaps it''s possible to set a distinct quota for snapshot space vs. live data space? I could then set snapshot quotas for my filesystems arbitrarily large for my administrative backups, or down to the filesystem size or some other value if there has been delegated authority for the filesystem. It would also be nice to be able to make snapshots of parent filesystems that include their descendants. Then, for example, I could create zfspool/grandparent/parent/child ...and set a filesystem quotas on parent, a "user" quota on child, and a "snapshot" quota on grandparent, and this solves most of my problems. in fact, I think a lot of ZFS''s hierarchical features would be more valuable if parent filesystems included their descendants (backups and NFS sharing, for example), but I''m sure there are just as many arguments against that as for it. This message posted from opensolaris.org
On Tue, 25 Jul 2006, Brad Plecs wrote:> I''ve run into this myself. (I am in a university setting). after reading bug ID 6431277 (URL below for noobs like myself who didn''t know what "see 6431277" meant): > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6431277 > > ...it''s not clear to me how this will be resolved. What I''d really like to see is > the ability to specify where the snapshot backing store will be. (more specifically, the > ability for the snapshot space to *not* impact the filesystem space). > > We have a network Appliance box, whose snapshots are very popular for their value as online > backups. Netapp charges snapshots to the storage pool, so they don''t cost the filesystem anything. I''m drooling over ZFS as an alternative to the expensive netapp hardware/software, > but since we sell RAID space but perform backups administratively, I can''t have the snapshots > consuming people''s space. > > I could increase the filesystem quota to accomodate the snapshots, but since the snapshot > size is dynamic, I would have to increase it well beyond the current snapshot size. Once I > do that, users *will* fill the space (that they have not paid for). > > I could tune the size of the filesystem to match the snapshot + filesystem data, but since > snapshot size is dynamic, this is impractical. > > I also have some very small quotas (50 MB) for users, and would like to be able to > create snapshots of them going back 30 days or so without it costing the user anything. > The snapshots save us tons of time and effort, but they''re not worth it to the user to pay > for double or triple the space they''re currently using, and I don''t want the users going > over the original quota of 50 MB, so I can''t make enough space in the filesystem > to make snapshots of their data... it''s maddening.I''ll play devils advocate here - because I don''t see this as a ZFS related issue or even one that is even in the ZFS domain - in terms of resolving the issue at hand. First, ZFS allows one to take advantage of large, inexpensive Serial ATA disk drives. Paraphrased: "ZFS loves large, cheap SATA disk drives". So the first part of the solution looks (to me) as simple as adding some cheap SATA disk drives. Next, after extra storage space has been added to the pool, it''s a simple matter of accounting to subtract the size of the ZFS snapshots from the users'' disk space to calculate their actual live storage and bill for it! Next, periodic snapshots can be made and older snapshots either deleted or moved to even lower cost storage media (e.g., tape, CDROMs, DVDs etc). Next - 50Mb quotas!?? You''ve got to be kidding. Let me check my calendar; yep - it''s 2006. You are kidding ... right? If you''re not kidding, then you''ve got a business/management issue and not a technical issue to resolve.> If we must contain snapshots inside a filesystem, perhaps it''s possible to set a distinct > quota for snapshot space vs. live data space? I could then set snapshot quotas for > my filesystems arbitrarily large for my administrative backups, or down to the filesystem size > or some other value if there has been delegated authority for the filesystem. > > It would also be nice to be able to make snapshots of parent filesystems that include their > descendants. Then, for example, I could create > > zfspool/grandparent/parent/child > > ...and set a filesystem quotas on parent, a "user" quota on child, and a "snapshot" quota on > grandparent, and this solves most of my problems. > > in fact, I think a lot of ZFS''s hierarchical features would be more valuable if parent filesystems > included their descendants (backups and NFS sharing, for example), but I''m sure > there are just as many arguments against that as for it. >Regards, Al Hopper Logical Approach Inc, Plano, TX. al at logical-approach.com Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005 OpenSolaris Governing Board (OGB) Member - Feb 2006
> First, ZFS allows one to take advantage of large, inexpensive Serial ATA > disk drives. Paraphrased: "ZFS loves large, cheap SATA disk drives". So > the first part of the solution looks (to me) as simple as adding some > cheap SATA disk drives.I hope not. We have quotas available for a reason. There are legitimate reasons for putting an administrative upper bound on storage space. It''s not always about disk acquisition costs. The ability to have the user own their space and the administrator own the snapshot would look good to me. If the user owns both, I expect them to trade snapshots for space. I would prefer to be able to guarantee snapshots as part of a defined recovery system without limiting access to their quota.> Next, after extra storage space has been added to the pool, it''s a simple > matter of accounting to subtract the size of the ZFS snapshots from the > users'' disk space to calculate their actual live storage and bill for it! > > Next, periodic snapshots can be made and older snapshots either deleted or > moved to even lower cost storage media (e.g., tape, CDROMs, DVDs etc).I don''t care how much bigger you make it. At some point the space will be used. Then the user deletes a subtree and doesn''t get any of the space back because of snapshots.... and I don''t want the snapshots deleted.> Next - 50Mb quotas!?? You''ve got to be kidding. Let me check my > calendar; yep - it''s 2006. You are kidding ... right? If you''re not > kidding, then you''ve got a business/management issue and not a technical > issue to resolve.So pretend it''s 500G. The suggestions still seem very valid to me. -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
> First, ZFS allows one to take advantage of large, inexpensive Serial ATA > disk drives. Paraphrased: "ZFS loves large, cheap SATA disk drives". So > the first part of the solution looks (to me) as simple as adding some > cheap SATA disk drives. > > Next, after extra storage space has been added to the pool, it''s a simple > matter of accounting to subtract the size of the ZFS snapshots from the > users'' disk space to calculate their actual live storage and bill for it! > > Next, periodic snapshots can be made and older snapshots either deleted or > moved to even lower cost storage media (e.g., tape, CDROMs, DVDs etc). > > Next - 50Mb quotas!?? You''ve got to be kidding. Let me check my > calendar; yep - it''s 2006. You are kidding ... right? If you''re not > kidding, then you''ve got a business/management issue and not a technical > issue to resolve.Our problem isn''t "we don''t have enough storage space". Our problem is that the snapshots reduce the filesystem space available to users. Simply creating the snapshot begins squeezing them out of their own space. Just billing people for it doesn''t solve the problem. Simply giving people more space doesn''t solve the problem, either. I did think of another option -- allow filesystems to dynamically grow their quotas themselves by the size of the snapshots. This might be easiest to implement -- just add a toggle in the zfs set/get parameters. If it''s integrated into the filesystem itself, keeping up with the changing size of the data should be a non-issue. Actually, I guess that''s fairly similar to the separate quota for snapshot data idea. BP -- bplecs at cs.umd.edu
On Tue, Jul 25, 2006 at 11:13:16AM -0700, Brad Plecs wrote:> > If we must contain snapshots inside a filesystem, perhaps it''s > possible to set a distinct quota for snapshot space vs. live data > space? I could then set snapshot quotas for my filesystems > arbitrarily large for my administrative backups, or down to the > filesystem size or some other value if there has been delegated > authority for the filesystem.This is the purpose of the above RFE. I''m not sure how what you''re asking for is any different.> in fact, I think a lot of ZFS''s hierarchical features would be more > valuable if parent filesystems included their descendants (backups and > NFS sharing, for example), but I''m sure there are just as many > arguments against that as for it.They do include their children for NFS shares[1], and recursive send/receive is in the works. - Eric [1] Unless you want only a single NFS-visible mount. This requires a client-side NFSv4 feature known as ''mirror mounts'', but there is no way in the (current) NFS spec to detect whether the client supports the notion. -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On Tue, Jul 25, 2006 at 11:13:16AM -0700, Brad Plecs wrote:> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6431277 > > What I''d really like to see is ... the ability for the snapshot space > to *not* impact the filesystem space).Yep, as Eric mentioned, that is the purpose of this RFE ("want filesystem-only quotas"). I imagine that this would be implemented as a quota against the space "referenced" (as currently reported by ''zfs list'', ''zfs get refer'', ''df'', etc; see the zfs(1m) manpage for details).> in fact, I think a lot of ZFS''s hierarchical features would be more > valuable if parent filesystems included their descendants (backups and > NFS sharing, for example), but I''m sure there are just as many > arguments against that as for it.Yep, we''re working on making more features work on "this and all descendents". For example, the recently implemented ''zfs snapshot -r'' can create snapshots of a filesystem and all its descendents. This feature will be part of Solaris 10 update 3. We''re also working on ''zfs send -r'' (RFE 6421958). --matt
On 7/25/06, Brad Plecs <bplecs at cs.umd.edu> wrote:> I''ve run into this myself. (I am in a university setting). after reading bug ID 6431277 (URL below for noobs like myself who didn''t know what "see 6431277" meant): > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6431277 > > ...it''s not clear to me how this will be resolved. What I''d really like to see is > the ability to specify where the snapshot backing store will be. (more specifically, the > ability for the snapshot space to *not* impact the filesystem space).Given the fact that you probably want this functionality quicker than it is likely to be delivered in a supported version of Solaris, perhaps you would be better off with two copies of the data. This may or may not have a cost advantage over NetApp. The idea is that you have two storage pools - one for live data, one for backup data. Your live data is *probably* on faster disks than your backup data. The live data and backup data may or may not be on the same server. Whenever you need to perform backups you do something along the lines of: yesterday=$1 today=$2 for user in $allusers ; do zfs snapshot users/$user@$today zfs snapshot backup/$user/$yesterday@$today zfs clone backup/$user/$yesterday@$today backup/$user/$today rsync -axuv /users/$user/.zfs/snapshot/$today /backup/$user/$today zfs destroy users/$user@$today zfs destroy backup/$user/$lastweek done It''s an ugly hack in that it doesn''t use a lot of ZFS'' goodness on the live file system. However, it would provide many days (weeks?) of space-efficient backups on the backup server. Obviously there is more logic required for ensuring that the right number of backups are kept, new and deleted users are dealt with appropriately, etc. The block-level rsync mechanism may be useful, but it is unlikely unless your users are using a lot of database files (oracle, dbm, etc.). This hack does have the benefit that you get a copy of the data stored safely somewhere else to give better disaster protection of the data, assuminng the backup is on a different server somewhere else on (or off) campus. Assuming you go with something like this and the students have read access to the backups, how long will it be until the students start ciculating the script to automatically rotate their extra data in and out of their home directory and backup directories so that they get 50 MB * "number of snapshots" of space? Back in the day, I had pitched battles with a few people over print quotas and customized postscript to lie about the number of pages. I won once I found a way to get to the page counter on JetDirect cards. :) Mike -- Mike Gerdts http://mgerdts.blogspot.com/
On Tue, Jul 25, 2006 at 07:24:51PM -0500, Mike Gerdts wrote:> On 7/25/06, Brad Plecs <bplecs at cs.umd.edu> wrote: > >What I''d really like to see is ... the ability for the snapshot space > >to *not* impact the filesystem space). > > The idea is that you have two storage pools - one for live data, one > for backup data. Your live data is *probably* on faster disks than > your backup data. The live data and backup data may or may not be on > the same server. Whenever you need to perform backups you do > something along the lines of: > > yesterday=$1 > today=$2 > for user in $allusers ; do > zfs snapshot users/$user@$today > zfs snapshot backup/$user/$yesterday@$today > zfs clone backup/$user/$yesterday@$today backup/$user/$today > rsync -axuv /users/$user/.zfs/snapshot/$today /backup/$user/$today > zfs destroy users/$user@$today > zfs destroy backup/$user/$lastweek > doneYou can simplify and improve the performance of this considerably by using ''zfs send'': for user in $allusers ; do zfs snapshot users/$user@$today zfs send -i $yesterday users/$user@$today | \ ssh $host zfs recv -d $backpath ssh $host zfs destroy $backpath/$user/$lastweek done You can send the backup to the same or different host, and the same or different pool, as your hardware needs dictate. ''zfs send'' will be much faster than rsync because we can use ZFS metadata to determine which blocks were changed without traversing all files & directories. --matt
On 7/25/06, Matthew Ahrens <ahrens at eng.sun.com> wrote:> You can simplify and improve the performance of this considerably by > using ''zfs send'': > > for user in $allusers ; do > zfs snapshot users/$user@$today > zfs send -i $yesterday users/$user@$today | \ > ssh $host zfs recv -d $backpath > ssh $host zfs destroy $backpath/$user/$lastweek > done > > You can send the backup to the same or different host, and the same or > different pool, as your hardware needs dictate. ''zfs send'' will be much > faster than rsync because we can use ZFS metadata to determine which > blocks were changed without traversing all files & directories. >This is what I had originally intended to say, but it seems with this approach the yesterday snapshot has to stick around in order to do incrementals. The fact that snapshots counted against current quota was part of the problem statement. My approach with rsync avoids this but, as I said before, is an ugly hack because it doesn''t use the features of zfs. Mike -- Mike Gerdts http://mgerdts.blogspot.com/
On Tue, Jul 25, 2006 at 08:37:28PM -0500, Mike Gerdts wrote:> On 7/25/06, Matthew Ahrens <ahrens at eng.sun.com> wrote: > >You can simplify and improve the performance of this considerably by > >using ''zfs send'': > > > > for user in $allusers ; do > > zfs snapshot users/$user@$today > > zfs send -i $yesterday users/$user@$today | \ > > ssh $host zfs recv -d $backpath > > ssh $host zfs destroy $backpath/$user/$lastweek > > done > > > >You can send the backup to the same or different host, and the same or > >different pool, as your hardware needs dictate. ''zfs send'' will be much > >faster than rsync because we can use ZFS metadata to determine which > >blocks were changed without traversing all files & directories. > > > > This is what I had originally intended to say, but it seems with this > approach the yesterday snapshot has to stick around in order to do > incrementals. The fact that snapshots counted against current quota > was part of the problem statement. My approach with rsync avoids this > but, as I said before, is an ugly hack because it doesn''t use the > features of zfs.Ah yes, I overlooked that. It''s a pity though, since implementation-wise, we don''t really *need* the old snapshot, we just need to be sure that when we do the incremental receive, we''re starting where we left off. Though potentially more complicated to administer, it should be possible to ensure that even without keeping the old snapshot around. I''ve filed RFE 6452850 "want incremental zfs send without initial snapshot" to track this enhancement. --matt