zfs discuss - May 2006 - Properties of ZFS snapshots I''d like to see...

Hi,

(apologies if this has been discussed before, I hope not)

while setting up a script at home to do automatic snapshots, a number of
wishes popped into my mind:

The basic problem with regular snapshotting is that you end up managing
so many of them. Wouldn''t it be nice if you could assign an expiration
date
to a snapshot?

For instance:

   zfs snapshot -e 3d tank/home at 20060505

would create a regular snapshot with an expiration date of 3 days from the
date it was created.

You could then change the expiration date with zfs set if you want to keep
it longer. "0" would mean no expiration date and so on.

Then, ZFS would be free to destroy the snapshot to free up space, but only if
it must: Just like the yogurt in your fridge, you may or may not be able to eat
it after the best before date, but you are guaranteed to be able to eat it
(or sue the yogurt company) if it''s inside the best before date.

Another property could control the rigidness of this policy: Hard expiration
would destroy the snapshot as soon as the expiry time arrives, soft
expiration would work like the yogurt example above.

The benefits of this approach would be ease of complexity: Imagine you do
a snapshot every week, then you''ll have 52 snapshots by the end of one
year.
This means that sysadmins will start writing scripts to automatically
delete snapshots they don''t need (I''m about to do just that)
at the risk
of deleting the wrong snapshot. Or, they won''t because it takes too
much
thinking (you really want to make that script really robust).

Another set of expiration related properties could allow for more complex
snapshot management:

- Multiple layers of snapshots: Keep one Yearly, one monthly, one weekly
   and the snapshot from yesterday always available.

- Multiple priorities: Assign priorities to snapshots so less important
   ones get destroyed first.

- Specify date ranges to destroy/modify attributes on multiple snapshots at
   once.

Is this something we''re already looking at or should we start looking
at
this as an RFE?

Thinking further, ZFS could start doing automatic snapshots (invisible from
the user) by just keeping every uber-block at each interval. Then, when the
admin panics, ZFS could say "hmm, here''s a couple of leftover
snapshots
that happen to still exist because you had a lot space left on the disks
that you may find useful".

The basic idea behind this whole thinking is to maximize utilization of free
blocks. If your disk utilization is only 50%, why not use the other 50% for
snapshots by default, that could save your life?

Best regards,
    Constantin

-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

It would be also nice that if you have many snapshoots and you do run out of 
space that the oldest snapshoot would be automatically removed untill space is 
freed up. I did setup this snashoot that is beiing made every minute, then every
hour, day and a month, and I finally got to the point where I did run out of 
space, olders snashoots were not removed space was gone and server could not be 
used. I was just testing this backup and no harm was done but I think haveing 
space is the most important, snapshoots are not as critical as beeing able to 
write to a disk... but hey. just a suggestion.

Chris


On Fri, 5 May 2006, Constantin Gonzalez wrote:
> Hi,
>
> (apologies if this has been discussed before, I hope not)
>
> while setting up a script at home to do automatic snapshots, a number of
> wishes popped into my mind:
>
> The basic problem with regular snapshotting is that you end up managing
> so many of them. Wouldn''t it be nice if you could assign an
expiration date
> to a snapshot?
>
> For instance:
>
>  zfs snapshot -e 3d tank/home at 20060505
>
> would create a regular snapshot with an expiration date of 3 days from the
> date it was created.
>
> You could then change the expiration date with zfs set if you want to keep
> it longer. "0" would mean no expiration date and so on.
>
> Then, ZFS would be free to destroy the snapshot to free up space, but only
if
> it must: Just like the yogurt in your fridge, you may or may not be able to
> eat
> it after the best before date, but you are guaranteed to be able to eat it
> (or sue the yogurt company) if it''s inside the best before date.
>
> Another property could control the rigidness of this policy: Hard
expiration
> would destroy the snapshot as soon as the expiry time arrives, soft
> expiration would work like the yogurt example above.
>
> The benefits of this approach would be ease of complexity: Imagine you do
> a snapshot every week, then you''ll have 52 snapshots by the end of
one year.
> This means that sysadmins will start writing scripts to automatically
> delete snapshots they don''t need (I''m about to do just
that) at the risk
> of deleting the wrong snapshot. Or, they won''t because it takes
too much
> thinking (you really want to make that script really robust).
>
> Another set of expiration related properties could allow for more complex
> snapshot management:
>
> - Multiple layers of snapshots: Keep one Yearly, one monthly, one weekly
>  and the snapshot from yesterday always available.
>
> - Multiple priorities: Assign priorities to snapshots so less important
>  ones get destroyed first.
>
> - Specify date ranges to destroy/modify attributes on multiple snapshots at
>  once.
>
> Is this something we''re already looking at or should we start
looking at
> this as an RFE?
>
> Thinking further, ZFS could start doing automatic snapshots (invisible from
> the user) by just keeping every uber-block at each interval. Then, when the
> admin panics, ZFS could say "hmm, here''s a couple of leftover
snapshots
> that happen to still exist because you had a lot space left on the disks
> that you may find useful".
>
> The basic idea behind this whole thinking is to maximize utilization of
free
> blocks. If your disk utilization is only 50%, why not use the other 50% for
> snapshots by default, that could save your life?
>
> Best regards,
>   Constantin
>
> -- 
> Constantin Gonzalez                            Sun Microsystems GmbH,
Germany
> Platform Technology Group, Client Solutions               
http://www.sun.de/
> Tel.: +49 89/4 60 08-25 91                  
http://blogs.sun.com/constantin/
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
> !DSPAM:122,445b13fd55112750596!
>

Interesting idea Constantin.

However, perhaps instead of or in addition to your idea, I''d like to
have a mechanism or script that would overwrite the older snapshots [u]only
if[/u] some more current snapshot were created.  Ideally this mechanism would
prevent your idea of expired snapshots being removed in some case where the
creation of new snapshots somehow failed.

Additionally, by only removing the snapshots after the creation of their
replacement is successful, this should prevent the possibility of data loss if
there were a major system problem during the creation of a new snapshot as well.
 
 
This message posted from opensolaris.org

On Fri, 5 May 2006, Constantin Gonzalez wrote:
> Hi,
>
> (apologies if this has been discussed before, I hope not)
>
> while setting up a script at home to do automatic snapshots, a number of
> wishes popped into my mind:
>
> The basic problem with regular snapshotting is that you end up managing
> so many of them. Wouldn''t it be nice if you could assign an
expiration date
> to a snapshot?
>
> For instance:
>
>    zfs snapshot -e 3d tank/home at 20060505
>
> would create a regular snapshot with an expiration date of 3 days from the
> date it was created.
1) But is this something that belongs in ZFS or is this a backup/restore
type tool that is simply a "user" of zfs?
> You could then change the expiration date with zfs set if you want to keep
> it longer. "0" would mean no expiration date and so on.
With zfs or with your backup/restore tool?
> Then, ZFS would be free to destroy the snapshot to free up space, but only
if
> it must: Just like the yogurt in your fridge, you may or may not be able to
eat
> it after the best before date, but you are guaranteed to be able to eat it
> (or sue the yogurt company) if it''s inside the best before date.
See 1) above
> Another property could control the rigidness of this policy: Hard
expiration
> would destroy the snapshot as soon as the expiry time arrives, soft
> expiration would work like the yogurt example above.
See 1) above
> The benefits of this approach would be ease of complexity: Imagine you do
> a snapshot every week, then you''ll have 52 snapshots by the end of
one year.
> This means that sysadmins will start writing scripts to automatically
> delete snapshots they don''t need (I''m about to do just
that) at the risk
> of deleting the wrong snapshot. Or, they won''t because it takes
too much
> thinking (you really want to make that script really robust).
>
> Another set of expiration related properties could allow for more complex
> snapshot management:
>
> - Multiple layers of snapshots: Keep one Yearly, one monthly, one weekly
>    and the snapshot from yesterday always available.
>
> - Multiple priorities: Assign priorities to snapshots so less important
>    ones get destroyed first.
>
> - Specify date ranges to destroy/modify attributes on multiple snapshots at
>    once.
Again - this looks like an operational backup/restore policy.  Not a ZFS
function.
> Is this something we''re already looking at or should we start
looking at
> this as an RFE?
>
> Thinking further, ZFS could start doing automatic snapshots (invisible from
> the user) by just keeping every uber-block at each interval. Then, when the
> admin panics, ZFS could say "hmm, here''s a couple of leftover
snapshots
> that happen to still exist because you had a lot space left on the disks
> that you may find useful".
Now you''re describing a form of filesystem snapshotting function that
might have to be closely integrated with zfs.  This is in addition to the
other data replication features that are already in the pipeline for zfs.
> The basic idea behind this whole thinking is to maximize utilization of
free
> blocks. If your disk utilization is only 50%, why not use the other 50% for
> snapshots by default, that could save your life?
IMHO the majority of the functionality you''re describing belongs in a
backup/restore tool that is simply a consumer of zfs functionality.  And
this functionality could be easily scripted using your scripting tool of
choice.

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005

Hi Al,
> 1) But is this something that belongs in ZFS or is this a backup/restore
> type tool that is simply a "user" of zfs?
...
> Again - this looks like an operational backup/restore policy.  Not a ZFS
> function.
So the question is: Is advanced management of snapshots (aging, expiring,
etc.) something left to the domain of a ZFS user (backup/restore application,
administrator, script) or should these concepts be adopted by ZFS as a
filesystem (which BTW is already much more).

IMHO, backup/restore is much more than playing with snapshots. The dividing
line starts when you copy your data to a different medium. As soon as the
data stays on the disk, I wouldn''t say it''s backup/restore
related. As long
as it''s just snapshots, it should be definitely not be called
"backup/restore".

But you''re right in that my desired functionality can
"easily" be implemented
with scripts. Then I would still argue for including this functionality as
part of the ZFS user interface, because of ease of use and minimization of
possible errors for the administrator. If it ain''t simple to use,
chances
are that people won''t. Same goes for snapshots: If admins
don''t have a really
easy way to get rid of them, chances are they will use them less.

Another point of view might be ease of implementation. A few person-months spent
at Sun (or the OpenSolaris developer community) might come up with a more
robust, clean, efficient, bug-free, elegant way of achieving the task of
snapshot management than millions of person-months in many admins creating
scripts and re-inventing wheels that may be half-baked.

But yes, it is a matter of interpretation who should take care of managing
snapshots after they''ve been created, ZFS or some
application/script/user
action.
>> Thinking further, ZFS could start doing automatic snapshots (invisible
from
>> the user) by just keeping every uber-block at each interval. Then, when
the
>> admin panics, ZFS could say "hmm, here''s a couple of
leftover snapshots
>> that happen to still exist because you had a lot space left on the
disks
>> that you may find useful".
> 
> Now you''re describing a form of filesystem snapshotting function
that
> might have to be closely integrated with zfs.  This is in addition to the
> other data replication features that are already in the pipeline for zfs.
Yes, this is when the above discussed features definitively cross the line
towards ZFS'' responsibilities.

Actually, it would be cool if ZFS took a hidden snapshot each time a zfs or
zpool command is issued. Then an admin could say "zfs undo" after
she/he
discovered that she/he just did a horrible mistake.
>> The basic idea behind this whole thinking is to maximize utilization of
free
>> blocks. If your disk utilization is only 50%, why not use the other 50%
for
>> snapshots by default, that could save your life?
> 
> IMHO the majority of the functionality you''re describing belongs
in a
> backup/restore tool that is simply a consumer of zfs functionality.  And
> this functionality could be easily scripted using your scripting tool of
> choice.
yes and no, depending on the interpretation. The potential of having a
"zfs undo" subcommand and the automatic exploitation of free space on
disk
for keeping snapshots as part of overall snapshot management are definitely
something that ZFS can do much better internally, as opposed to having to
implement it with some other app.

Best regards,
    Constantin

-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

Hi Wes,

Wes Williams wrote:
> Interesting idea Constantin.
> 
> However, perhaps instead of or in addition to your idea, I''d like
to have a
> mechanism or script that would overwrite the older snapshots [u]only if[/u]
> some more current snapshot were created.  Ideally this mechanism would
> prevent your idea of expired snapshots being removed in some case where the
> creation of new snapshots somehow failed.
yeah, that could be another pair of snapshot properties: number of snapshots to
minimally/maximally keep.
> Additionally, by only removing the snapshots after the creation of their
> replacement is successful, this should prevent the possibility of data loss
> if there were a major system problem during the creation of a new snapshot
as
> well.
Yes, snapshot replacement should always be split up into creating a successor,
see if it was successful, then delete the old one.

Best regards,
    Constantin

-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

On Fri, May 05, 2006 at 05:17:43PM +0200, Constantin Gonzalez
wrote:
> But you''re right in that my desired functionality can
"easily" be
> implemented
> with scripts. Then I would still argue for including this functionality as
> part of the ZFS user interface, because of ease of use and minimization of
> possible errors for the administrator. If it ain''t simple to use,
chances
> are that people won''t. Same goes for snapshots: If admins
don''t have a
> really
> easy way to get rid of them, chances are they will use them less.
I''ve some scripts that do something like this (snapshot current, delete
all older snapshots, without forcing it, thus clones keep snapshots
alive).

Now, the thing about implementing expiration dates is that dealing with
time in shell scripts can be pretty painful -- it''d be nice if zfs get
had better time formatting options.

Nico
--

On Fri, 5 May 2006, Nicolas Williams wrote:
> On Fri, May 05, 2006 at 05:17:43PM +0200, Constantin Gonzalez wrote:
> > But you''re right in that my desired functionality can
"easily" be
> > implemented
> > with scripts. Then I would still argue for including this
functionality as
> > part of the ZFS user interface, because of ease of use and
minimization of
> > possible errors for the administrator. If it ain''t simple to
use, chances
> > are that people won''t. Same goes for snapshots: If admins
don''t have a
> > really
> > easy way to get rid of them, chances are they will use them less.
>
> I''ve some scripts that do something like this (snapshot current,
delete
> all older snapshots, without forcing it, thus clones keep snapshots
> alive).
>
> Now, the thing about implementing expiration dates is that dealing with
> time in shell scripts can be pretty painful -- it''d be nice if zfs
get
> had better time formatting options.
Agreed.

A couple of questions:
- Do you think that zfs (and other subsystems) should use the ISO8601 time
formatting standards?

- I''ve been writing a lot more scripts in python for about a year now.
The resulting scripts are easy to read/maintain and runtime performance is
not an issue.  Well written python is portable and re-usable.  Are others
writing scripts in other than ksh/csh these days?

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005

Krzys wrote:
> It would be also nice that if you have many snapshoots and you do run 
> out of space that the oldest snapshoot would be automatically removed 
> untill space is freed up. I did setup this snashoot that is beiing made 
> every minute, then every hour, day and a month, and I finally got to the 
> point where I did run out of space, olders snashoots were not removed 
> space was gone and server could not be used. I was just testing this 
> backup and no harm was done but I think haveing space is the most 
> important, snapshoots are not as critical as beeing able to write to a 
> disk... but hey. just a suggestion.
I really wouldn''t want that policy.  Sometimes the oldest snapshot is 
the most valuable one of all because they are the "golden egg" that 
started everything.  Though it really depends on what you are using 
snapshots for.  I certainly wouldn''t support this policy being the
default.

-- 
Darren J Moffat

I did not think of it this way and it is a very valid point, but I still think 
that most likely you would have a backup already on tape if need be and haveing 
space available for writing rhather than having no disk space for live data is 
much more important than a snap, but thats my opinion. I think it certainly 
should be an option.

:)

Chris


On Fri, 5 May 2006, Darren J Moffat wrote:
> Krzys wrote:
>> It would be also nice that if you have many snapshoots and you do run
out
>> of space that the oldest snapshoot would be automatically removed
untill
>> space is freed up. I did setup this snashoot that is beiing made every 
>> minute, then every hour, day and a month, and I finally got to the
point
>> where I did run out of space, olders snashoots were not removed space
was
>> gone and server could not be used. I was just testing this backup and
no
>> harm was done but I think haveing space is the most important,
snapshoots
>> are not as critical as beeing able to write to a disk... but hey. just
a
>> suggestion.
>
> I really wouldn''t want that policy.  Sometimes the oldest snapshot
is the
> most valuable one of all because they are the "golden egg" that
started
> everything.  Though it really depends on what you are using snapshots for. 
I
> certainly wouldn''t support this policy being the default.
>
> -- 
> Darren J Moffat
>
>
> !DSPAM:122,445b7898169711123715201!
>

Krzys wrote:
> I did not think of it this way and it is a very valid point, but I still 
> think that most likely you would have a backup already on tape if need 
> be and haveing space available for writing rhather than having no disk 
> space for live data is much more important than a snap, but thats my 
> opinion. I think it certainly should be an option.
What if my first snapshot is the golden image of my zones or diskless 
clients ?  I need that online not on a tape.

-- 
Darren J Moffat

Maybe there could be a flag for certain snaps where it could be made read 
only?!? But I dont know how this could be implemented and I do not think that 
would be possible... Anyway I still think that if I had a production system with
those snaps I would rather remove that "golden image" and continue
with
operations rather than have no space and put my system to a halt.

:)

On Fri, 5 May 2006, Darren J Moffat wrote:
> Krzys wrote:
>> I did not think of it this way and it is a very valid point, but I
still
>> think that most likely you would have a backup already on tape if need
be
>> and haveing space available for writing rhather than having no disk
space
>> for live data is much more important than a snap, but thats my opinion.
I
>> think it certainly should be an option.
>
> What if my first snapshot is the golden image of my zones or diskless
clients
> ?  I need that online not on a tape.
>
> -- 
> Darren J Moffat
>
>
> !DSPAM:122,445b7eed184751123715201!
>

Interesting discussion.  I''ve often been impressed at how NetApp-like
the overal ZFS feature-set is (implies that I like NetApp''s).  Is it
verboten to compare ZFS to NetApp?  I hope not....

NetApp has two ways of making snapshots.  There is a set of automatic
snapshots, which are created, rotate and expire on their own (i.e. the
filer does all of this).  Often you''ll have a number of hourly, daily,
weekly, etc. snapshots in this category.  These are the ones that users
can count on seeing when they seek to perform a self-recovery of a
mistakenly damaged file.

Then you have the ones you create manually, or which are created by
backup software.  The filer itself will never delete these, it''s up
to the external creator to manage them.

This has proven to be a fantastic model for the usage patterns that I have
experienced (over probably 6+ years of NetApp use), and I would like to
see something similar available for ZFS.

Personally, I think that having an expiration time (and creation) be
associated with the snapshot/pool itself is a good thing.  What happens
if one exports said filesystem/pool (with snapshots) to another system,
if such creation/expiration is handled by some outside utility?

Hmm, I''m not sure if the NetApp auto-snapshot schedule follows a disk
volume if it''s exported to a different filer.  I think it
doesn''t.

Regards,

Marion

Krzys wrote:
> Maybe there could be a flag for certain snaps where it could be made 
> read only?!? But I dont know how this could be implemented and I do not 
> think that would be possible... Anyway I still think that if I had a 
> production system with those snaps I would rather remove that "golden 
> image" and continue with operations rather than have no space and put
my
> system to a halt.
That may work for you but it might not work for everyone and it 
certainly doesn''t work for my use case.

-- 
Darren J Moffat

I realy do like the way NetApp is handling snaps :) that would be an excelent 
thing in ZFS :)

On Fri, 5 May 2006, Marion Hakanson wrote:
> Interesting discussion.  I''ve often been impressed at how
NetApp-like
> the overal ZFS feature-set is (implies that I like NetApp''s).  Is
it
> verboten to compare ZFS to NetApp?  I hope not....
>
> NetApp has two ways of making snapshots.  There is a set of automatic
> snapshots, which are created, rotate and expire on their own (i.e. the
> filer does all of this).  Often you''ll have a number of hourly,
daily,
> weekly, etc. snapshots in this category.  These are the ones that users
> can count on seeing when they seek to perform a self-recovery of a
> mistakenly damaged file.
>
> Then you have the ones you create manually, or which are created by
> backup software.  The filer itself will never delete these, it''s
up
> to the external creator to manage them.
>
> This has proven to be a fantastic model for the usage patterns that I have
> experienced (over probably 6+ years of NetApp use), and I would like to
> see something similar available for ZFS.
>
> Personally, I think that having an expiration time (and creation) be
> associated with the snapshot/pool itself is a good thing.  What happens
> if one exports said filesystem/pool (with snapshots) to another system,
> if such creation/expiration is handled by some outside utility?
>
> Hmm, I''m not sure if the NetApp auto-snapshot schedule follows a
disk
> volume if it''s exported to a different filer.  I think it
doesn''t.
>
> Regards,
>
> Marion
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
> !DSPAM:122,445b80b818937266247132!
>

I agree and I do not say what you do is wrong, I was just expressing my opinion 
after what happend today on my system that there would be an excelent idea to 
have such extra feature in ZFS...

Regards,

Chris


On Fri, 5 May 2006, Darren J Moffat wrote:
> Krzys wrote:
>> Maybe there could be a flag for certain snaps where it could be made
read
>> only?!? But I dont know how this could be implemented and I do not
think
>> that would be possible... Anyway I still think that if I had a
production
>> system with those snaps I would rather remove that "golden
image" and
>> continue with operations rather than have no space and put my system to
a
>> halt.
>
> That may work for you but it might not work for everyone and it certainly 
> doesn''t work for my use case.
>
> -- 
> Darren J Moffat
>
>
> !DSPAM:122,445b810319014137527416!
>

On Fri, May 05, 2006 at 09:43:05AM -0700, Marion Hakanson
wrote:
> Interesting discussion.  I''ve often been impressed at how
NetApp-like
> the overal ZFS feature-set is (implies that I like NetApp''s).  Is
it
> verboten to compare ZFS to NetApp?  I hope not....
It''s a public list, you can do the comparison if you like :)

Here''s what I think: Solaris is a general purpose OS, as such it has
scheduling facilities (cron(1M), at(1)) and what not, so it is tempting
to say that we need only make it easy to build this kind of facility and
much more.  And for me better time formatting is a huge part of this
(and, to answer the question, like what the date(1) supports, but I''d
also like raw time in seconds/microseconds since the epoch).

Now, it may be that we really should offer this functinality out of the
box, with no scripting being required.  Being neither in the ZFS team
nor in Marketing I can''t say what is planned, if anything.  I imagine
that Marketing would have some input on this, if they haven''t already;
and your input may impact what Marketing has to say on this, so keep it
coming.

Nico
--

On 5/5/06, Constantin Gonzalez <Constantin.Gonzalez at sun.com>
wrote:
> Hi,
>
> (apologies if this has been discussed before, I hope not)
>
> while setting up a script at home to do automatic snapshots, a number of
> wishes popped into my mind:
>
> The basic problem with regular snapshotting is that you end up managing
> so many of them. Wouldn''t it be nice if you could assign an
expiration date
> to a snapshot?
>
while reading this thread i did come up with an enhancement that could
make dealing with snapshots better, perhaps we can create a directory
structure under .zfs/snapshot so the user can organise the snapshots
for instance.

cd .zfs/snapshot
mkdir daily
mkdir monthly
mkdir important

then the user could use unix commands like mv  to put snapshots where
he needs them it would also make snapshot management easier if i want
to delete all daily snapshots older than a week.

find .zfs/snapshot/daily -ctime +7d

and to create snapshots and place in the special directories

zfs snapshot  data/myfilesystem at daily/05-05-2006

all snapshot directories would start under .zfs/snapshot

James Dickens
uadmin.blogspot.com


> For instance:
>
>    zfs snapshot -e 3d tank/home at 20060505
>
> would create a regular snapshot with an expiration date of 3 days from the
> date it was created.
>
> You could then change the expiration date with zfs set if you want to keep
> it longer. "0" would mean no expiration date and so on.
>
> Then, ZFS would be free to destroy the snapshot to free up space, but only
if
> it must: Just like the yogurt in your fridge, you may or may not be able to
eat
> it after the best before date, but you are guaranteed to be able to eat it
> (or sue the yogurt company) if it''s inside the best before date.
>
> Another property could control the rigidness of this policy: Hard
expiration
> would destroy the snapshot as soon as the expiry time arrives, soft
> expiration would work like the yogurt example above.
>
> The benefits of this approach would be ease of complexity: Imagine you do
> a snapshot every week, then you''ll have 52 snapshots by the end of
one year.
> This means that sysadmins will start writing scripts to automatically
> delete snapshots they don''t need (I''m about to do just
that) at the risk
> of deleting the wrong snapshot. Or, they won''t because it takes
too much
> thinking (you really want to make that script really robust).
>
> Another set of expiration related properties could allow for more complex
> snapshot management:
>
> - Multiple layers of snapshots: Keep one Yearly, one monthly, one weekly
>    and the snapshot from yesterday always available.
>
> - Multiple priorities: Assign priorities to snapshots so less important
>    ones get destroyed first.
>
> - Specify date ranges to destroy/modify attributes on multiple snapshots at
>    once.
>
> Is this something we''re already looking at or should we start
looking at
> this as an RFE?
>
> Thinking further, ZFS could start doing automatic snapshots (invisible from
> the user) by just keeping every uber-block at each interval. Then, when the
> admin panics, ZFS could say "hmm, here''s a couple of leftover
snapshots
> that happen to still exist because you had a lot space left on the disks
> that you may find useful".
>
> The basic idea behind this whole thinking is to maximize utilization of
free
> blocks. If your disk utilization is only 50%, why not use the other 50% for
> snapshots by default, that could save your life?
>
> Best regards,
>     Constantin
>
> --
> Constantin Gonzalez                            Sun Microsystems GmbH,
Germany
> Platform Technology Group, Client Solutions               
http://www.sun.de/
> Tel.: +49 89/4 60 08-25 91                  
http://blogs.sun.com/constantin/
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Fri, 5 May 2006, Marion Hakanson wrote:
> Interesting discussion.  I''ve often been impressed at how
NetApp-like
> the overal ZFS feature-set is (implies that I like NetApp''s).  Is
it
> verboten to compare ZFS to NetApp?  I hope not....
Of course not.  And if "Thumper" is similar to the rumoured
description
published by theregister.com, then it will be widely recognized as a
strong NetApp competitor product and invite direct comparison.
> NetApp has two ways of making snapshots.  There is a set of automatic
> snapshots, which are created, rotate and expire on their own (i.e. the
> filer does all of this).  Often you''ll have a number of hourly,
daily,
> weekly, etc. snapshots in this category.  These are the ones that users
> can count on seeing when they seek to perform a self-recovery of a
> mistakenly damaged file.
>
> Then you have the ones you create manually, or which are created by
> backup software.  The filer itself will never delete these, it''s
up
> to the external creator to manage them.
>
> This has proven to be a fantastic model for the usage patterns that I have
> experienced (over probably 6+ years of NetApp use), and I would like to
> see something similar available for ZFS.
>
> Personally, I think that having an expiration time (and creation) be
> associated with the snapshot/pool itself is a good thing.  What happens
> if one exports said filesystem/pool (with snapshots) to another system,
> if such creation/expiration is handled by some outside utility?
Playing devils advocate and assuming that utility is not a part of zfs
[0]: As long as you can access the "serial numbers"[1] that zfs
associates
with a pool and with pool member disks, an external utility could still
maintain the relationship between the snapshot and the pool that it was
created from.
> Hmm, I''m not sure if the NetApp auto-snapshot schedule follows a
disk
> volume if it''s exported to a different filer.  I think it
doesn''t.
>
[0] it could be in a zfs contributed software repository
[1] really Unique Identifiers

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005

> A couple of questions:
> - Do you think that zfs (and other subsystems) should
> use the ISO8601 time
> formatting standards?
>  
> Regards,
> 
> Al Hopper  Logical Approach Inc, Plano, TX.
I wish that were the only time standard!  

No matter what country you live in, the ISO 8601 time/date standard is by far
the most logical by relevance ranking, especially when sorted by automated
systems.
 
 
This message posted from opensolaris.org

Hi,

thank you for the excellent comments, thoughts and ideas on the
"Properties of ZFS snapshots I''d like to see..." thread.

I took the liberty of renaming the thread to $SUBJECT because I think that what
we really are looking for is an ability for ZFS to automatically manage snapshot
after they have been created.

To summarize:

- "Snapshot management in ZFS" can be defined as an automatic way of:

   - Using free space on disk for automatically or manually generated snapshots
     but giving priority to new data on the disk at the expense of destroying
     old snapshots that are considered less useful.

   - Implementing policies that decide which snapshot to keep even at the
     cost of not having enough space for new data. Possible policies include:

     - Prioritize recent snapshots over older one.
       (Assuming that the older the snapshot, the less the user cares)

     - Prioritize older snapshots over recent ones.
       (Assuming, the older the snapshots, the more errors can be corrected)

     - Any combination of the above. (I.e. keep at least one yearly, one
       monthly, one weekly and one daily snapshot).

   - Giving users the possibility to decide what policies to apply to snapshots
     when created.

   - Giving users the possibility to configure automatic snapshots at regular
     intervals (similar to NetApps).

   - Automatically snapshotting a pool or a filesystem before any administrative
     action in order to facilitate a "zfs undo" functionality.

- Much or all of the above can be implemented today with user- or admin-level
   scripts. The question therefore is whether this should be incorporated into
   ZFS or not. Here are pros and cons:

   Pros:
   - Make it easier for users and admins to enjoy the benefits of snapshots
     without having to write scripts.

   - Make advanced functionality available to users and admins that would take
     a lot of complex scripting and therefore can be implemented more elegantly
     inside zfs than outside zfs. ("zfs undo" and free space
management for
     instance).

   - Reduce the risk of user and admin errors when scripting by providing a
     single point of development for a critical functionality.
     (Example: dealing with different time standards is non-trivial; scripts
     may be less robust than OS-level code, etc.).

   Cons:
   - ZFS is a file system, not a backup management system. Leave that to the
     application and 3rd party vendors.

   - Deleting snapshots is a difficult question and each user/admin/site may
     have very different policies about when to delete them and when not. This
     makes a one-size-fits all approach either insufficient or not generic
     enough for all users to be really useful.

Feel free to add to the lists so we can make up our minds. Maybe this can
evolve into something the ZFS team may be interested in.

Best regards,
    Constantin

-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

I think there may be a middle ground here -- provide some relatively
simple policy-independent mechanisms in the filesystem to allow admins
to implement the snapshot/backup management policy of their choice.

I can think of a couple mechanisms which would help:

 - additional admin-defined filesystem properties, giving the
administrator a place to store properties such as snapshot frequency,
lifetime, and backup policy/parameters; these could be  maintained in a
separate database or file, but that is likely to get out of synch with
the pool so better that they''re kept in the pool itself..

 - notifications when pool free space crosses low and high water marks
so a daemon wouldn''t have to keep polling free space to decide when to
reap expired snapshots.

An optional daemon process which does time-based snapshots and snapshot
expiration based on the above mechanisms could then be used at many
sites as-is, while sites with more complicated requirements could modify
or replace that daemon to implement their site''s policy.

							- Bill

On Mon, 8 May 2006, Bill Sommerfeld wrote:
> I think there may be a middle ground here -- provide some relatively
> simple policy-independent mechanisms in the filesystem to allow admins
> to implement the snapshot/backup management policy of their choice.
>
> I can think of a couple mechanisms which would help:
>
>  - additional admin-defined filesystem properties, giving the
> administrator a place to store properties such as snapshot frequency,
> lifetime, and backup policy/parameters; these could be  maintained in a
> separate database or file, but that is likely to get out of synch with
> the pool so better that they''re kept in the pool itself..
The more general case would provide a generic way to store and retrieve
user defined tag/value pairs that are associated (and stored) with the
underlying zfs pool.
>  - notifications when pool free space crosses low and high water marks
> so a daemon wouldn''t have to keep polling free space to decide
when to
> reap expired snapshots.
>
> An optional daemon process which does time-based snapshots and snapshot
> expiration based on the above mechanisms could then be used at many
> sites as-is, while sites with more complicated requirements could modify
> or replace that daemon to implement their site''s policy.
+1

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005

Hey Constantin,

On Mon, 2006-05-08 at 11:37 +0200, Constantin Gonzalez wrote:
> I took the liberty of renaming the thread to $SUBJECT because I think that
what
> we really are looking for is an ability for ZFS to automatically manage
snapshot
> after they have been created.
Wow, nice summary of the problem!

I thought I''d add a few ideas into the fray.

Here''s what I was thinking could be fairly quick to implement, which
administrators could build upon if necessary.

I believe we certainly should provide a way for users to schedule
automatic snapshots, but probably not build it into the filesystem
itself (imho - ZFS is a filesystem dammit Jim, not a backup solution!)

This could be easily implemented via a set of SMF instances which
create/destroy cron jobs which would themselves call a simple script
responsible for taking the snapshots.

Of course, this isn''t as flexible as an administrator writing their own
scripts, but it could be enough for most users, with those that want
more functionality being able to build on this functionality.

So, it''s not as intelligent as the daemon Bill was suggesting, we
wouldn''t poll the FS to reap snapshots when space is limited. For that
functionality, I''d hope for an as-yet-nonexistent ZFS FMA event to
report that some pools are getting short on space, which could be the
trigger for deleting these auto-snapshots if necessary  (I''d also
imagine lots of other things would be interested in keying off such an
event as well...)


The service that we could have for taking auto snapshots could be called

/system/filesystem/zfs/auto-snapshot

We''d have one instance per set of automatic snapshots taken. Which
isn''t
to say we need one instance per filesystem, as we could define instances
that snapshot all child filesystems contained this top level filesystem.

  /system/filesystem/zfs/auto-snapshot:[fs-name]

The properties we''d have for each instance are:

- interval = minutes | hours | days | months | years
- period = take snapshots every how many [interval]s
- keep = number of snapshots to keep before rolling over 
         (delete the oldest when we hit the limit)
- offset = # seconds into the start of the period
            at which we take the snapshot ( < period * interval)
- snapshot-children = true | false

Here''s some examples of SMF instances that would implement
auto-snapshots.

The following instance takes a snapshot every 4 days, at 01:00, keeping
30 snapshots into the past :

  /system/filesystem/zfs/auto-snapshot:tank
           interval = days
             period = 4
               keep = 30
             offset = (60 * 60)
  snapshot-children = false
  

This instance takes a weekly snapshot, keeping the last two, and will
snapshot all children of export/home/timf[1] :

  /system/filesystem/zfs/auto-snapshot:export/home/timf
           interval = days
             period = 7
               keep = 2
             offset = 0
  snapshot-children = true


Essentially, I''m really just suggesting a glorified interface to cron,
so why not just use cron ? Well, I suspect having a service like this
would be easier to manage than a heap of cron jobs : at a glance, I can
tell which auto-snapshots are being taken, when and how. I also like the
idea of tieing into SMF, since that means other options, like the GUI
interfaces in the Visual Panels project, may become available in the
future.

Anyway, that''s what I was thinking of (and it wouldn''t be too
hard to
implement)  I''ve no doubt this could be refined - but does anyone think
this is the right direction to go in ?

	cheers,
			tim

[1] I''m not yet sure if SMF instance names are allowed
''/'' chars, sorry
[2] http://blogs.sun.com/roller/page/dep?entry=visual_panels_debut

Tim Foster stated:
< Hey Constantin,
< 
< On Mon, 2006-05-08 at 11:37 +0200, Constantin Gonzalez wrote: 
< > I took the liberty of renaming the thread to $SUBJECT because I think
that what
< > we really are looking for is an ability for ZFS to automatically
manage snapshot
< > after they have been created.
< 
< Wow, nice summary of the problem!
< 
< I thought I''d add a few ideas into the fray.
< 
< Here''s what I was thinking could be fairly quick to implement,
which
< administrators could build upon if necessary.
< 
< I believe we certainly should provide a way for users to schedule
< automatic snapshots, but probably not build it into the filesystem
< itself (imho - ZFS is a filesystem dammit Jim, not a backup solution!)
< 
< This could be easily implemented via a set of SMF instances which
< create/destroy cron jobs which would themselves call a simple script
< responsible for taking the snapshots.
< 
< Of course, this isn''t as flexible as an administrator writing
their own
< scripts, but it could be enough for most users, with those that want
< more functionality being able to build on this functionality.
< 
< So, it''s not as intelligent as the daemon Bill was suggesting, we
< wouldn''t poll the FS to reap snapshots when space is limited. For
that
< functionality, I''d hope for an as-yet-nonexistent ZFS FMA event to
< report that some pools are getting short on space, which could be the
< trigger for deleting these auto-snapshots if necessary  (I''d also
< imagine lots of other things would be interested in keying off such an
< event as well...)
< 
< 
< The service that we could have for taking auto snapshots could be called
< 
< /system/filesystem/zfs/auto-snapshot
< 
< We''d have one instance per set of automatic snapshots taken. Which
isn''t
< to say we need one instance per filesystem, as we could define instances
< that snapshot all child filesystems contained this top level filesystem.
< 
<   /system/filesystem/zfs/auto-snapshot:[fs-name]
< 
< The properties we''d have for each instance are:
< 
< - interval = minutes | hours | days | months | years
< - period = take snapshots every how many [interval]s
< - keep = number of snapshots to keep before rolling over 
<          (delete the oldest when we hit the limit)
< - offset = # seconds into the start of the period
<             at which we take the snapshot ( < period * interval)
< - snapshot-children = true | false
< 
< Here''s some examples of SMF instances that would implement
< auto-snapshots.
< 
< The following instance takes a snapshot every 4 days, at 01:00, keeping
< 30 snapshots into the past :
< 
<   /system/filesystem/zfs/auto-snapshot:tank
<            interval = days
<              period = 4
<                keep = 30
<              offset = (60 * 60)
<   snapshot-children = false
<   
< 
< This instance takes a weekly snapshot, keeping the last two, and will
< snapshot all children of export/home/timf[1] :
< 
<   /system/filesystem/zfs/auto-snapshot:export/home/timf
<            interval = days
<              period = 7
<                keep = 2
<              offset = 0
<   snapshot-children = true
< 
< 
< Essentially, I''m really just suggesting a glorified interface to
cron,
< so why not just use cron ? Well, I suspect having a service like this
< would be easier to manage than a heap of cron jobs : at a glance, I can
< tell which auto-snapshots are being taken, when and how. I also like the
< idea of tieing into SMF, since that means other options, like the GUI
< interfaces in the Visual Panels project, may become available in the
< future.

   Thats just so sweet (as in cool, but cute)..

   This can also be easily be extended to include Bill and Al''s
properties and
   user defined tag/values pairs.  Just store them within the service instance.

   SMF can then be called upon to react to any errors or the like from these
   snapshot instances.  That or FMA.

   This could tie a good few solaris features all together in a really nice way.

< 
< Anyway, that''s what I was thinking of (and it wouldn''t be
too hard to
< implement)  I''ve no doubt this could be refined - but does anyone
think
< this is the right direction to go in ?

  Theres no new API to be written, its easily extended and customisable, builds
  on features already there..

< 
< 	cheers,
< 			tim
< 
< [1] I''m not yet sure if SMF instance names are allowed
''/'' chars, sorry
< [2] http://blogs.sun.com/roller/page/dep?entry=visual_panels_debut
< 
< _______________________________________________
< zfs-discuss mailing list
< zfs-discuss at opensolaris.org
< http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

-- 
Sean.
.

On Mon, May 08, 2006 at 10:23:44PM +0100, Tim Foster
wrote:
> This could be easily implemented via a set of SMF instances which
> create/destroy cron jobs which would themselves call a simple script
> responsible for taking the snapshots.
There was talk a while back about an extended cron service with support
for XML job manifests that would get us way beyond current crontab
time specs and which would understand dependencies on SMF services and
on other extended cron jobs.  A tie in with FMA would also be desirable,
I imagine.

I think this is, pretty much, what you want here.

At the ZFS level I think all we need is the right basic building block
functionality: user-defined pool and FS properties, better date
formatting, querying by date, including relative formats.

A "demo" facility that layers atop
ZFS+SMF+FMA+''cronX'' to do all that
has been proposed here and more would rock.

Nico
--

Hi Tim,

thank you for your comments.

Bringing in SMF is an excellent idea and should make what admins like to
do much more elegant.

I guess the question here is to find out:

- What degree of canned functionality is needed to address 80% of every
admin''s
   need.

- Who should provide the functionality: The ZFS core team, a community of
   people (inside OpenSolaris or outside) or some person/group who take this
   as their personal project and maybe publishes it.

We''re probably more than half way through the first. Maybe
it''s time to create
a project under the opensolaris community to nail down the second and start
thinking of what functionality is really needed to be inside ZFS to make our
life with integrating SMF support for managing snapshots easier.

I''m not sure user-defined properties in ZFS is the solution. Depending
on the
apps, scripts and users using such properties, pools, filesystems and snapshots
can easily be polluted with lots of properties that someone thought might be
useful. Perhaps it may be more beneficial to go through the process of 
discussing and deciding which set of properties would make snapshot management
much easier, then go with it.

Best regards,
    Constantin

Tim Foster wrote:
> Hey Constantin,
> 
> On Mon, 2006-05-08 at 11:37 +0200, Constantin Gonzalez wrote: 
>> I took the liberty of renaming the thread to $SUBJECT because I think
that what
>> we really are looking for is an ability for ZFS to automatically manage
snapshot
>> after they have been created.
> 
> Wow, nice summary of the problem!
> 
> I thought I''d add a few ideas into the fray.
> 
> Here''s what I was thinking could be fairly quick to implement,
which
> administrators could build upon if necessary.
> 
> I believe we certainly should provide a way for users to schedule
> automatic snapshots, but probably not build it into the filesystem
> itself (imho - ZFS is a filesystem dammit Jim, not a backup solution!)
> 
> This could be easily implemented via a set of SMF instances which
> create/destroy cron jobs which would themselves call a simple script
> responsible for taking the snapshots.
> 
> Of course, this isn''t as flexible as an administrator writing
their own
> scripts, but it could be enough for most users, with those that want
> more functionality being able to build on this functionality.
> 
> So, it''s not as intelligent as the daemon Bill was suggesting, we
> wouldn''t poll the FS to reap snapshots when space is limited. For
that
> functionality, I''d hope for an as-yet-nonexistent ZFS FMA event to
> report that some pools are getting short on space, which could be the
> trigger for deleting these auto-snapshots if necessary  (I''d also
> imagine lots of other things would be interested in keying off such an
> event as well...)
> 
> 
> The service that we could have for taking auto snapshots could be called
> 
> /system/filesystem/zfs/auto-snapshot
> 
> We''d have one instance per set of automatic snapshots taken. Which
isn''t
> to say we need one instance per filesystem, as we could define instances
> that snapshot all child filesystems contained this top level filesystem.
> 
>   /system/filesystem/zfs/auto-snapshot:[fs-name]
> 
> The properties we''d have for each instance are:
> 
> - interval = minutes | hours | days | months | years
> - period = take snapshots every how many [interval]s
> - keep = number of snapshots to keep before rolling over 
>          (delete the oldest when we hit the limit)
> - offset = # seconds into the start of the period
>             at which we take the snapshot ( < period * interval)
> - snapshot-children = true | false
> 
> Here''s some examples of SMF instances that would implement
> auto-snapshots.
> 
> The following instance takes a snapshot every 4 days, at 01:00, keeping
> 30 snapshots into the past :
> 
>   /system/filesystem/zfs/auto-snapshot:tank
>            interval = days
>              period = 4
>                keep = 30
>              offset = (60 * 60)
>   snapshot-children = false
>   
> 
> This instance takes a weekly snapshot, keeping the last two, and will
> snapshot all children of export/home/timf[1] :
> 
>   /system/filesystem/zfs/auto-snapshot:export/home/timf
>            interval = days
>              period = 7
>                keep = 2
>              offset = 0
>   snapshot-children = true
> 
> 
> Essentially, I''m really just suggesting a glorified interface to
cron,
> so why not just use cron ? Well, I suspect having a service like this
> would be easier to manage than a heap of cron jobs : at a glance, I can
> tell which auto-snapshots are being taken, when and how. I also like the
> idea of tieing into SMF, since that means other options, like the GUI
> interfaces in the Visual Panels project, may become available in the
> future.
> 
> Anyway, that''s what I was thinking of (and it wouldn''t be
too hard to
> implement)  I''ve no doubt this could be refined - but does anyone
think
> this is the right direction to go in ?
> 
> 	cheers,
> 			tim
> 
> [1] I''m not yet sure if SMF instance names are allowed
''/'' chars, sorry
> [2] http://blogs.sun.com/roller/page/dep?entry=visual_panels_debut
> 
-- 
Constantin Gonzalez                            Sun Microsystems GmbH, Germany
Platform Technology Group, Client Solutions                http://www.sun.de/
Tel.: +49 89/4 60 08-25 91                   http://blogs.sun.com/constantin/

Of interest to this thread, Tim Foster has just created an interesting blog
entry at:
http://blogs.sun.com/roller/page/timf#zfs_automatic_snapshots_prototype_1
 
 
This message posted from opensolaris.org

Hi Constantin,
> The basic problem with regular snapshotting is that you end up managing so
many of them. Wouldn''t it be nice if you could assign an expiration
date to a snapshot?
The only reason you want the snapshot removed is because you don''t want
your pool to become full. IIRC VxFS has a feature to automatically delete a
snapshot if a write would return ENOSPC, than the snaphot is deleted and the
write is retried. This might be considered as an additional feature to your
automatic expiry.

Best regards

  -- Dago
 
 
This message posted from opensolaris.org