freebsd stable - May 2010 - Make ZFS auto-destroy snapshots when the out of space?

I found some nice scripts to regularly snapshot all the filesystems in 
my ZFS pool at 
http://www.neces.com/blog/technology/integrating-freebsd-zfs-and-periodic-snapshots-and-scrubs
. One thing bothers me, though: I have to intentionally set how many 
months' worth of snapshots I want to keep. Too many and I run out of 
room. Too few and I lose some of the benefits of easy recovery of 
deleted data. My computer is better at bookkeeping than I am, so why not 
let it?

I'd propose standardizing on an attribute like 
org.freebsd:allowautodestroy. Modify ZFS's disk full behavior to scan 
for snapshots with that attribute set and destroy the oldest one, and 
continue until there's enough free space to complete a write requests or 
until out of "expendable" snapshots to destroy (at which time the
normal
disk full handler would run). Also run a daily periodic script to ensure 
that the free space stays below a configurable threshold each day so 
that ZFS isn't constantly butting up against completely full drives.

This would take all configuration guesswork out of the equation and 
would let me keep as many snapshots as I have space to maintain. If I 
want to extend my reach back in time, I can add another drive to the 
pool and the rest is handled automatically. At the same time, should I 
suddenly *want* to store massive amounts of new data, the snapshots can 
be easily and automatically cleared out to make room for the stuff I 
want to hold.

What do you think? It seems like this should be pretty easy to implement 
without requiring any upstream changes or new FreeBSD-only data 
structures. The whole thing could possibly be implemented in userspace, 
but I don't know that ZFS has any exception handling callbacks that 
would make it easy.

An unused resource is a wasted resource, right?

In the last episode (May 29), Kirk Strauser said:
> I found some nice scripts to regularly snapshot all the filesystems in my
> ZFS pool at
>
http://www.neces.com/blog/technology/integrating-freebsd-zfs-and-periodic-snapshots-and-scrubs
> . One thing bothers me, though: I have to intentionally set how many 
> months' worth of snapshots I want to keep. Too many and I run out of 
> room. Too few and I lose some of the benefits of easy recovery of 
> deleted data. My computer is better at bookkeeping than I am, so why not 
> let it?
> 
> I'd propose standardizing on an attribute like
> org.freebsd:allowautodestroy.  Modify ZFS's disk full behavior to scan
for
> snapshots with that attribute set and destroy the oldest one, and continue
> until there's enough free space to complete a write requests or until
out
> of "expendable" snapshots to destroy (at which time the normal
disk full
> handler would run).  Also run a daily periodic script to ensure that the
> free space stays below a configurable threshold each day so that ZFS
isn't
> constantly butting up against completely full drives.
If the kernel does the snapshot deleting itself, why not add a pool-level
property that sets the amount of free space at which the deletion starts? 
That way you don't need the cleanup script.  Alternatively, make the
org.freebsd:allowautodestroy property hold the trigger freespace amount. 
That way you can have monthly/daily/hourly snapshots but set it so the
hourly ones disappear first, then the dailies (by setting the destroy
trigger slightly higher for the ones you want to expire first).

-- 
	Dan Nelson
	dnelson@allantgroup.com

On Sat, May 29, 2010 at 1:07 PM, Kirk Strauser <kirk@strauser.com>
wrote:
> I found some nice scripts to regularly snapshot all the filesystems in my
> ZFS pool at
>
http://www.neces.com/blog/technology/integrating-freebsd-zfs-and-periodic-snapshots-and-scrubs
> . One thing bothers me, though: I have to intentionally set how many
months'
> worth of snapshots I want to keep. Too many and I run out of room. Too few
> and I lose some of the benefits of easy recovery of deleted data. My
> computer is better at bookkeeping than I am, so why not let it?
>
> I'd propose standardizing on an attribute like
org.freebsd:allowautodestroy.
> Modify ZFS's disk full behavior to scan for snapshots with that
attribute
> set and destroy the oldest one, and continue until there's enough free
space
> to complete a write requests or until out of "expendable"
snapshots to
> destroy (at which time the normal disk full handler would run). Also run a
> daily periodic script to ensure that the free space stays below a
> configurable threshold each day so that ZFS isn't constantly butting up
> against completely full drives.
>
> This would take all configuration guesswork out of the equation and would
> let me keep as many snapshots as I have space to maintain. If I want to
> extend my reach back in time, I can add another drive to the pool and the
> rest is handled automatically. At the same time, should I suddenly *want*
to
> store massive amounts of new data, the snapshots can be easily and
> automatically cleared out to make room for the stuff I want to hold.
>
> What do you think? It seems like this should be pretty easy to implement
> without requiring any upstream changes or new FreeBSD-only data structures.
> The whole thing could possibly be implemented in userspace, but I don't
know
> that ZFS has any exception handling callbacks that would make it easy.
>
> An unused resource is a wasted resource, right?
So basically you're saying deal with an LRU snapshot deletion when you
reach a certain threshold of free space, type scheme? This might get
tricky, but it does lend itself to other systems I suppose (I hate to
mention it, but the best one I can think of is Windows' system
restore... there might be something else available with OSX's Time
Machine).

What would be more tricky is when the automated system is filling in a
bunch of useless snapshots unnecessarily, but as you'd be providing
the snapshot criteria, I suppose that you would know what snapshots
you want to save and what ones you want to toss...

It's an interesting thought though -- just increases the overall
complexity of the system and may only meet one need.

Cheers,
-Garrett

Hi, Kirk,

On Sat, May 29, 2010 at 1:07 PM, Kirk Strauser <kirk@strauser.com>
wrote:
> I found some nice scripts to regularly snapshot all the filesystems in my
> ZFS pool at
>
http://www.neces.com/blog/technology/integrating-freebsd-zfs-and-periodic-snapshots-and-scrubs
> . One thing bothers me, though: I have to intentionally set how many
months'
> worth of snapshots I want to keep. Too many and I run out of room. Too few
> and I lose some of the benefits of easy recovery of deleted data. My
> computer is better at bookkeeping than I am, so why not let it?
>
> I'd propose standardizing on an attribute like
org.freebsd:allowautodestroy.
> Modify ZFS's disk full behavior to scan for snapshots with that
attribute
> set and destroy the oldest one, and continue until there's enough free
space
> to complete a write requests or until out of "expendable"
snapshots to
> destroy (at which time the normal disk full handler would run). Also run a
> daily periodic script to ensure that the free space stays below a
> configurable threshold each day so that ZFS isn't constantly butting up
> against completely full drives.
>
> This would take all configuration guesswork out of the equation and would
> let me keep as many snapshots as I have space to maintain. If I want to
> extend my reach back in time, I can add another drive to the pool and the
> rest is handled automatically. At the same time, should I suddenly *want*
to
> store massive amounts of new data, the snapshots can be easily and
> automatically cleared out to make room for the stuff I want to hold.
>
> What do you think? It seems like this should be pretty easy to implement
> without requiring any upstream changes or new FreeBSD-only data structures.
> The whole thing could possibly be implemented in userspace, but I don't
know
> that ZFS has any exception handling callbacks that would make it easy.
>
> An unused resource is a wasted resource, right?
I think this sounds like a good idea but I think we may probably want
to explore a more general mechanism, e.g. a daemon that "listen"s
system events like file system full, etc. and execute some user
defined actions.

One thing that we want to avoid is that by making the "automatic
recycle" we would open a new race between system and user backup
programs, i.e. if you remove an intermediate snapshot, 'zfs send' may
fail at receiving side, if incremental send is being used.  We would
need a way to "notify" that a 'zfs send' is underway.

Cheers,
-- 
Xin LI <delphij@delphij.net> http://www.delphij.net

On May 29, 2010, at 6:45 PM, Denny Lin wrote:
> How about writing a shell script with this functionality? Get the
> available disk space using:
> $ zfs list -H -o avail tank
>
> And then compare the figures against a limit. Then delete the oldest
> snapshots when the limit is exceeded.

That's certainly an option, and easy to implement in a shell script.  
It just wouldn't work in the case where a bunch of data comes in  
between runs of that script, and that's why I wanted something lower- 
level.
-- 
Kirk Strauser

On May 29, 2010, at 16:07, Kirk Strauser wrote:
> I'd propose standardizing on an attribute like  
> org.freebsd:allowautodestroy. Modify ZFS's disk full behavior [...]   
> Also run a daily periodic script to ensure that the free space stays  
> below a configurable threshold each day so that ZFS isn't constantly  
> butting up against completely full drives.

Why not simply have a script that runs and checks for pool usage and  
then deletes snapshots with that attribute if necessary? Why do you  
need to have have it built into ZFS?
> What do you think? It seems like this should be pretty easy to  
> implement without requiring any upstream changes or new FreeBSD-only  
> data structures. The whole thing could possibly be implemented in  
> userspace, but I don't know that ZFS has any exception handling  
> callbacks that would make it easy.
IMHO this shouldn't be built into the file system. You have one script  
to automatically generate snapshots, and another to monitor usage and  
delete old ones.

This idea was talked about on zfs-discuss in 2006:

	http://mail.opensolaris.org/pipermail/zfs-discuss/2006-May/thread.html#2266

Good summary in this post:

	http://mail.opensolaris.org/pipermail/zfs-discuss/2006-May/002313.html

Generally I don't think this is the "Unix Way". I don't want
my kernel
doing stuff behind my back. If I want snapshots I'll create them  
(scripted or manual); if I want to get rid of them for whatever  
reason, I'll destroy them (scripted or manual). Either of these  
behaviours can then be controlled by an rc.conf(5) variable perhaps.

There's already an useful creation tool for OpenSolaris:

	http://src.opensolaris.org/source/xref/jds/zfs-snapshot/

There's also an auto-scrub script:

	http://blogs.sun.com/constantin/entry/new_opensolaris_zfs_auto_scrub