zfs discuss - Sep 2008 - Automatic removal of old snapshots

Hi,

I''m building a new ZFS fileserver for our lab and I''d like to
have these features:

- take a snapshot of users'' home directories every N minutes (N is 5 or
10)
- remove all old snapshots, keep just these:
   - all snapshots made during last H hours (H=24)
   - keep one snapshot per day (e.g. made at 16:00 hrs) for the last D days
(D=7)
   - keep one snapshot per week for the last W weeks (W=4)
   - keep the last Y "monthly" snapshots (Y=12)

In other words, every user would have "(H * 60/N) + D + W + Y"
snapshots in
their home directories (N=5 => 311 snapshots).

Before re-inventing the wheel, does anyone have any nice shell script to do this
kind of thing (to be executed from cron)?

We have only few users and quite static home directories so this kind of 
snapshot amount should be fine...

Martti

> Before re-inventing the wheel, does anyone have any nice shell script to do
this
> kind of thing (to be executed from cron)?
http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_10
http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_11

On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll
wrote:
> > Before re-inventing the wheel, does anyone have any nice shell script
to do this
> > kind of thing (to be executed from cron)?
> 
> http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_10
> http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_11
Storage Checkpoints in Veritas software has this feature (removing
the oldest checkpoint in case of 100% filesystem usage) by default.
Why not add such option to ZFS ?

Regards
Przemyslaw Bak (przemol)
--
http://przemol.blogspot.com/





















----------------------------------------------------------------------
Znajd? mieszkanie w Twoim regionie!
kliknij >>> http://link.interia.pl/f1f19

On Thu, 2008-09-25 at 12:07 +0200, przemolicc at poczta.fm
wrote:
> On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote:
> Storage Checkpoints in Veritas software has this feature (removing
> the oldest checkpoint in case of 100% filesystem usage) by default.
> Why not add such option to ZFS ?
 - because we shouldn''t assume a user''s intentions - some
would consider
automatic destruction of data (even historical data) a p1 bug.

nv_100 will have the ZFS Automatic Snapshot service in it, disabled by
default[1]

The GUI side of this, ("Time Slider") has exactly this functionality -
it''ll start deleting snapshots that it has taken when the filesystem
reaches a certain threshold.

More details at:
http://opensolaris.org/os/community/arc/caselog/2008/571/mail

	cheers,
			tim

[1] actually while I''m here, quick poll - the schedules for retaining
snapshots currently are:

- Frequent snapshots, taken every 15 minutes, keeping the 4 most recent
- Hourly snapshots taken once every hour, keeping 24
- Daily snapshots taken once every 24 hours, keeping 7
- Weekly snapshots taken once every 7 days, keeping 4
- Monthly snapshots taken on the first day of every month, keeping 12

I''m arguing that we should be keeping 31 daily snapshots, and not just
7. There''s some overlap with the weekly ones yes, but the extra
granularity over the past month could be useful, anyone else have an
opinion?

On Thu, Sep 25, 2008 at 11:30:04AM +0100, Tim Foster
wrote:
> On Thu, 2008-09-25 at 12:07 +0200, przemolicc at poczta.fm wrote:
> > On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote:
> 
> > Storage Checkpoints in Veritas software has this feature (removing
> > the oldest checkpoint in case of 100% filesystem usage) by default.
> > Why not add such option to ZFS ?
> 
>  - because we shouldn''t assume a user''s intentions - some
would consider
> automatic destruction of data (even historical data) a p1 bug.
Tim,

nobody is going to assume user''s intentions. Just give us
snapshot-related property which we can set to on/off and everybody
can setup zfs according to his/her needs.

Regards
Przemyslaw Bak (przemol)
--
http://przemol.blogspot.com/





















----------------------------------------------------------------------
Znajd? mieszkanie w Twoim regionie!
kliknij >>> http://link.interia.pl/f1f19

On Thu, 2008-09-25 at 12:52 +0200, przemolicc at poczta.fm
wrote:
> nobody is going to assume user''s intentions. Just give us
> snapshot-related property which we can set to on/off and everybody
> can setup zfs according to his/her needs.
Then that''ll be there in nv_100. Enjoy!

	cheers,
			tim

Tim,

 > - Frequent snapshots, taken every 15 minutes, keeping the 4 most recent
 > - Hourly snapshots taken once every hour, keeping 24
 > - Daily snapshots taken once every 24 hours, keeping 7
 > - Weekly snapshots taken once every 7 days, keeping 4
 > - Monthly snapshots taken on the first day of every month, keeping 12

I prefer the default number of snapshots kept to span twice the interval of the 
next period (? right wording?).

to illustrate:

frequent: keep 8
hourly: keep 48
daily: keep 14
weekly: keep 8
monthy: dont care, I use 48

I believe it is important keep as many snapshots as necessary to give the user a
change to move a snapshot from a "finer class" to a "coarser
class" to prevent
snapshotted data from the time frame in question from expiring prematurely in 
case a "coarser" snapshot was not be taken.

Nils

Almost exactly what I was planning to configure here Nils, with a couple of
minor changes.  I was planning on taking 10 weekly backups since you
occasionally get 5 week months, and depending on storage capacity,
we''re also considering annual snapshots.

I quite like Tim''s idea of having 31 daily snapshots too.  We already
have that for some of our data, so it might be easier to just use that globaly.

Our policy is likely to be:

15 mins:  keep 8
hourly:  keep 48
daily:  keep 14 (or 31)
weekly:  keep 10
monthly:  keep 24
yearly:  keep 10

For a default setup, I would have thought a years worth of data would be enough,
something like:

15 mins:  keep 8
hourly:  keep 48
daily:  keep 31
weekly:  keep 10
monthly:  keep 12

Ross
--
This message posted from opensolaris.org

On 25 Sep 2008, at 14:40, Ross wrote:
> For a default setup, I would have thought a years worth of data  
> would be enough, something like:
Given that this can presumably be configured to suit everyone''s  
particular data retention plan, for a default setup, what was  
originally proposed seems obvious and sensible to me.

Going slightly off-topic:

All this auto-snapshot stuff is ace, but what''s really missing, in my  
view, is some easy way to actually determine where the version of the  
file you want is. I typically find myself futzing about with diff  
across a dozen mounted snapshots trying to figure out when the last  
good version is.

It would be great if there was some way to know if a snapshot contains  
blocks for a particular file, i.e., that snapshot contains an earlier  
version of the file than the next snapshot / now. If you could do that  
and make ls support it with an additional flag/column, it''d be a real  
time-saver.

The current mechanism is especially hard as the auto-mount dirs can  
only be found at the top of the filesystem so you have to work with  
long path names. An fs trick to make .snapshot dirs of symbolic links  
appear automagically would rock, i.e.,

% cd /foo/bar/baz
% ls -l .snapshot
[...] nightly.0 -> /foo/.zfs/snapshot/nightly.0/bar/baz
% diff {,.snapshot/nightly.0/}importantfile

Yes, I know this last command can just be written as:

% diff /foo/{,.zfs/snapshot/nightly.0}/bar/baz/importantfile

but this requires me to a) type more; and b) remember where the top of  
the filesystem is in order to split the path. This is obviously more  
of a pain if the path is 7 items deep, and the split means you can''t  
just use $PWD.

[My choice of .snapshot/nightly.0 is a deliberate nod to the  
competition ;-)]

Jonathan

I asked Tim something like that when he posted his last update and from his
reply it looks like something is in the works:
http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_0_11

He was also blogging about this stuff in 2006 :)
http://blogs.sun.com/timf/entry/zfs_on_your_desktop

I''ve also been pestering the CIFS guys to get VSS support working,
which then lets you use Microsoft''s Shadow Copy Client from windows:
http://mswhs.files.wordpress.com/2007/12/previous-versions.png.  I''m
told there''s an internal build of that working now, but they have no
idea when it will even be ready for inclusion in ON, so while it''s in
the works it''s likely to be some time before we see it.

Ross
--
This message posted from opensolaris.org

zfs-discuss-bounces at opensolaris.org wrote on 09/25/2008 05:30:04 AM:
> On Thu, 2008-09-25 at 12:07 +0200, przemolicc at poczta.fm wrote:
> > On Thu, Sep 25, 2008 at 11:43:51AM +0200, Nils Goroll wrote:
>
> > Storage Checkpoints in Veritas software has this feature (removing
> > the oldest checkpoint in case of 100% filesystem usage) by default.
> > Why not add such option to ZFS ?
>
>  - because we shouldn''t assume a user''s intentions - some
would consider
> automatic destruction of data (even historical data) a p1 bug.
>
> nv_100 will have the ZFS Automatic Snapshot service in it, disabled by
> default[1]
>
> The GUI side of this, ("Time Slider") has exactly this
functionality -
> it''ll start deleting snapshots that it has taken when the
filesystem
> reaches a certain threshold.
>
> More details at:
> http://opensolaris.org/os/community/arc/caselog/2008/571/mail
>
>    cheers,
>          tim
>
> [1] actually while I''m here, quick poll - the schedules for
retaining
> snapshots currently are:
>
> - Frequent snapshots, taken every 15 minutes, keeping the 4 most recent
> - Hourly snapshots taken once every hour, keeping 24
> - Daily snapshots taken once every 24 hours, keeping 7
> - Weekly snapshots taken once every 7 days, keeping 4
> - Monthly snapshots taken on the first day of every month, keeping 12
>
> I''m arguing that we should be keeping 31 daily snapshots, and not
just
> 7. There''s some overlap with the weekly ones yes, but the extra
> granularity over the past month could be useful, anyone else have an
> opinion?
Tim,



      That snap schedule seems reasonable to me. Relate to the cleanup part
of the doc linked, do you know the rational for killing off the most recent
(15 minute and hourly) snaps vs the oldest (monthly) first?

To me, it seems like the biggest bang for the buck (snapshot held data)
would be killing off (oldest) monthly, weekly, hourly, daily, hourly, 15 in
that order.  Also I guess user case in my mind would leave a desktop user
more likely to need access to a few minutes, hours or days ago then 12
months ago.

-Wade
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

zfs-discuss-bounces at opensolaris.org wrote on 09/25/2008 09:16:48 AM:
> On 25 Sep 2008, at 14:40, Ross wrote:
>
> > For a default setup, I would have thought a years worth of data
> > would be enough, something like:
>
> Given that this can presumably be configured to suit everyone''s
> particular data retention plan, for a default setup, what was
> originally proposed seems obvious and sensible to me.
>
> Going slightly off-topic:
>
> All this auto-snapshot stuff is ace, but what''s really missing, in
my
> view, is some easy way to actually determine where the version of the
> file you want is. I typically find myself futzing about with diff
> across a dozen mounted snapshots trying to figure out when the last
> good version is.
>
> It would be great if there was some way to know if a snapshot contains
> blocks for a particular file, i.e., that snapshot contains an earlier
> version of the file than the next snapshot / now. If you could do that
> and make ls support it with an additional flag/column, it''d be a
real
> time-saver.
>
> The current mechanism is especially hard as the auto-mount dirs can
> only be found at the top of the filesystem so you have to work with
> long path names. An fs trick to make .snapshot dirs of symbolic links
> appear automagically would rock, i.e.,
>
> % cd /foo/bar/baz
> % ls -l .snapshot
> [...] nightly.0 -> /foo/.zfs/snapshot/nightly.0/bar/baz
> % diff {,.snapshot/nightly.0/}importantfile
>
> Yes, I know this last command can just be written as:
>
> % diff /foo/{,.zfs/snapshot/nightly.0}/bar/baz/importantfile
>
> but this requires me to a) type more; and b) remember where the top of
> the filesystem is in order to split the path. This is obviously more
> of a pain if the path is 7 items deep, and the split means you
can''t
> just use $PWD.
>
I use a perl script that emulates ls,  but in addition of listing the files
in the current directory looks at the timestamps of each zfs snap to see if
there are different versions and shows the FQ path and timestmp info for
each unique version.  files that exist in the snapshot, but not in the
current list are shown last (with their timetamps).  I suppose a
backup/restore like gui interface would be relatively simple to build with
the same principle.

My code is in horrible shape (wrote it as a one off, but it works).  If
there is interest I can cleanup and post.


> [My choice of .snapshot/nightly.0 is a deliberate nod to the
> competition ;-)]
>
> Jonathan
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Thu, 2008-09-25 at 10:19 -0500, Wade.Stuart at fallon.com
wrote:
>       That snap schedule seems reasonable to me. Relate to the cleanup part
> of the doc linked, do you know the rational for killing off the most recent
> (15 minute and hourly) snaps vs the oldest (monthly) first?
It''s a tough call (which thankfully I didn''t have to make).
I''m not sure
there was a rationale, other than the guys who were implementing it
messing about on their desktops finding out where /they/ tended to get
the most savings.

Really, it all depends on how volatile your filesystems are, where the
best place to retrieve data from. I suspect it''ll take more real-user
testing to determine what''s the best balance between data availability
and disk space.

	cheers,
			tim

Tim.Foster at Sun.COM wrote on 09/25/2008 10:34:41 AM:
> On Thu, 2008-09-25 at 10:19 -0500, Wade.Stuart at fallon.com wrote:
> >       That snap schedule seems reasonable to me. Relate to the cleanup
part
> > of the doc linked, do you know the rational for killing off the most
recent
> > (15 minute and hourly) snaps vs the oldest (monthly) first?
>
> It''s a tough call (which thankfully I didn''t have to
make). I''m not sure
> there was a rationale, other than the guys who were implementing it
> messing about on their desktops finding out where /they/ tended to get
> the most savings.
>
> Really, it all depends on how volatile your filesystems are, where the
> best place to retrieve data from. I suspect it''ll take more
real-user
> testing to determine what''s the best balance between data
availability
> and disk space.
Thanks Tim.  It seems like a very cool project.  Maybe there will be more
push to show more block born/dead/held stats from a public ZFS API when
stuff like this gets rolling.  That should allow more of an educated guess
as to what snapshots are holding what data.

-Wade

Wade,
> that order.  Also I guess user case in my mind would leave a desktop user
> more likely to need access to a few minutes, hours or days ago then 12
> months ago.
You are guessing that, but I am a desktop user who''d rather like the
contrary.

I think Tim has already stated that he would not want to hardcode any 
assumptions as to which snapshots one would want to delete first, and I think he
is very right in doing so.

Nils

Hi Wade,

We considered a number of approaches including just deleting oldest snapshots
first
and progressing through to the newest snapshots.
When  you consider the default snapshot schedules we are going to use, the
model is that snapshots get thinned out over time. So in situations were disk
space runs low, we stay consistent with this model but accelerate the aging
process
and thin out snapshots quicker, instead of just chopping the end off.
The exception to this is that frequent snapshots will only get deleted as a very
last
resort. The reason for this is that  frequent snapshots typically consume very
little
space, and they will get deleted within an hour or so anyway. And more
importantly,
the most common type of error is human error rather than hardware failure, such
as
accidentally deleting or overwriting a file or corrupting it. These kind of
errors are usually
noticed instantly or very soon after the event so retaining the frequent
snapshots will
hopefully provide some reasonable protection against this most common type of
error.

Cheers,
Niall.
--
This message posted from opensolaris.org

Jonathan Hogg wrote:
> It would be great if there was some way to know if a snapshot contains  
> blocks for a particular file, i.e., that snapshot contains an earlier  
> version of the file than the next snapshot / now. If you could do that  
> and make ls support it with an additional flag/column, it''d be a
real
> time-saver.
> 
> The current mechanism is especially hard as the auto-mount dirs can  
> only be found at the top of the filesystem so you have to work with  
> long path names. An fs trick to make .snapshot dirs of symbolic links  
> appear automagically would rock, i.e.,
> 
> % cd /foo/bar/baz
> % ls -l .snapshot
> [...] nightly.0 -> /foo/.zfs/snapshot/nightly.0/bar/baz
> % diff {,.snapshot/nightly.0/}importantfile
> 
> Yes, I know this last command can just be written as:
> 
> % diff /foo/{,.zfs/snapshot/nightly.0}/bar/baz/importantfile
> 
> but this requires me to a) type more; and b) remember where the top of  
> the filesystem is in order to split the path. This is obviously more  
> of a pain if the path is 7 items deep, and the split means you
can''t
> just use $PWD.
Chris Gerhard has a zfs_versions script that might help: 
http://blogs.sun.com/chrisg/entry/that_there_is

--
Darren J Moffat

On 25 Sep 2008, at 17:14, Darren J Moffat wrote:
> Chris Gerhard has a zfs_versions script that might help:
http://blogs.sun.com/chrisg/entry/that_there_is
Ah. Cool. I will have to try this out.

Jonathan

>>>>> "tf" == Tim Foster <Tim.Foster at Sun.COM>
writes:
    tf> anyone else have an opinion?

keep the number of snapshots small until the performacne problems with
booting/importing/scrubbing while having lots of snapshots are
resolved.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/2d3c8737/attachment.bin>