This may be a bit dimwitted since I don''t really understand how snapshots work. I mean the part concerning COW (copy on right) and how it takes so little room. But here I''m not asking about that. It appears to me that the default snapshot setup shares some aspects of a vcs (version control system) tool. I wonder if any of you use it that way. Here is one thing I''ve considered but not done yet. When I do video projects or any projects for that matter. I sometimes want backups every 10 minutes or so, so as not to loose some piece of script that isn''t finished or the like. Or with something like a flash project, you might want to make sure you will be able to recover a version from a while back. So doing the project on zfs filesystem (maybe as nfs or cifs mount) would offer a way to do that. I wondered if it would be possible to run a snapshot system independent of the default one. I mean so a default setup of auto snapshotting would continue unaffected. I''m thinking of scripting something like 10 minute snapshots during the time I''m working on a project, then just turn it off when not working on it. When project is done... zap all those snapshots. Am I missing something basic that make this a poor use of zfs? Oh, something I meant to ask... is there some standard way to tell before calling for a snapshot, if the directory structure has changed at all, other than aging I mean. Is there something better than running `diff -r [...]'' between existing structure and last snapshot.
This is totally doable, and a reasonable use of zfs snapshots - we do some similar things. You can easily determine if the snapshot has changed by checking the output of zfs list for the snapshot. --M -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Harry Putnam Sent: Monday, March 22, 2010 1:34 PM To: zfs-discuss at opensolaris.org Subject: [zfs-discuss] snapshots as versioning tool This may be a bit dimwitted since I don''t really understand how snapshots work. I mean the part concerning COW (copy on right) and how it takes so little room. But here I''m not asking about that. It appears to me that the default snapshot setup shares some aspects of a vcs (version control system) tool. I wonder if any of you use it that way. Here is one thing I''ve considered but not done yet. When I do video projects or any projects for that matter. I sometimes want backups every 10 minutes or so, so as not to loose some piece of script that isn''t finished or the like. Or with something like a flash project, you might want to make sure you will be able to recover a version from a while back. So doing the project on zfs filesystem (maybe as nfs or cifs mount) would offer a way to do that. I wondered if it would be possible to run a snapshot system independent of the default one. I mean so a default setup of auto snapshotting would continue unaffected. I''m thinking of scripting something like 10 minute snapshots during the time I''m working on a project, then just turn it off when not working on it. When project is done... zap all those snapshots. Am I missing something basic that make this a poor use of zfs? Oh, something I meant to ask... is there some standard way to tell before calling for a snapshot, if the directory structure has changed at all, other than aging I mean. Is there something better than running `diff -r [...]'' between existing structure and last snapshot. _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 03/23/10 09:34 AM, Harry Putnam wrote:> This may be a bit dimwitted since I don''t really understand how > snapshots work. I mean the part concerning COW (copy on right) and > how it takes so little room. > > But here I''m not asking about that. > > It appears to me that the default snapshot setup shares some aspects > of a vcs (version control system) tool. > >It does, but on a filesystem rather than file level. Or to put it another way, less fine grained than a traditional VCS.> I wonder if any of you use it that way. > >I do for things I don''t change very often, such us system configuration files. I always snapshot my root pool before making any changes to files under /etc for example.> Here is one thing I''ve considered but not done yet. > > When I do video projects or any projects for that matter. > I sometimes want backups every 10 minutes or so, so as not to loose > some piece of script that isn''t finished or the like. > > Or with something like a flash project, you might want to make sure > you will be able to recover a version from a while back. > > So doing the project on zfs filesystem (maybe as nfs or cifs mount) > would offer a way to do that. > > I wondered if it would be possible to run a snapshot system > independent of the default one. I mean so a default setup of auto > snapshotting would continue unaffected. > >You can, but I think you would be better off using a traditional VCS (such as Subversion) that works well with binary files. If you have to work in windows, this is your best option (Tortoise SVN is the only reason I know to use windows!).> I''m thinking of scripting something like 10 minute snapshots during > the time I''m working on a project, then just turn it off when not > working on it. When project is done... zap all those snapshots. > > Am I missing something basic that make this a poor use of zfs? > >You don''t really get to track version of a file. I find I commit very frequently (as soon as a new test passes) and use SVN as an "undo" if I mess up a change. Tying commits to changes is different form tying them to time.> Oh, something I meant to ask... is there some standard way to tell > before calling for a snapshot, if the directory structure has changed > at all, other than aging I mean. Is there something better than > running `diff -r [...]'' between existing structure and last snapshot. > >Not really, there is ZFS diff is in the woks, but not here yet. -- Ian.
> This may be a bit dimwitted since I don''t really understand how > snapshots work. I mean the part concerning COW (copy on right) and > how it takes so little room.COW and snapshots are very simple to explain. Suppose you''re chugging along using your filesystem, and then one moment, you tell the filesystem to "freeze." Well, suppose a minute later you tell the FS to overwrite some block that''s in use already. Instead of overwriting the actual block on disk, the FS will overwrite some unused space, and report back to you that the operation is completed. So now there''s a "copy" of the block as it was at the moment of the "freeze," and there''s another "copy" of the block as it looks later in time. The FS only needs to freeze the FS tables, to remember which blocks belonged to which files in each of the snapshots. Hence, Copy On Write. That being said, it''s an inaccurate description to say "COW takes so little room." If anything, it takes more room than a filesystem which can''t do COW, because the FS must not delete any of the old blocks belonging to any of the old "snapshots" of the filesystem. The more frequently you take snapshots, and the older your oldest snap is, and the more volatile your data is, changing large sequences of blocks rapidly ... The more disk space will be consumed. No block is free, as long as any one of the snaps references it. But suppose you have n snapshots. In a non-COW filesystem, you would have n-times the data. While in COW, you still have 1x the total used data size, plus the byte differentials necessary to resurrect any/all of the old snapshots.> I''m thinking of scripting something like 10 minute snapshots during > the time I''m working on a project, then just turn it off when not > working on it. When project is done... zap all those snapshots.Yup, that''s absolutely easy. Just set up a cron job to snap every 10 minutes, using a unique string in the snapname, like "@myprojectsnap" ... and when you''re all done, you "zfs destroy" anything which matches "@myprojectsnap"
Matt Cowger <mcowger at salesforce.com> writes:> This is totally doable, and a reasonable use of zfs snapshots - we > do some similar things.Good, thanks for the input.> You can easily determine if the snapshot has changed by checking the > output of zfs list for the snapshot.Do you mean to just grep it out of the output of zfs list -t snapshot Or is there some finer grained way to get it? (I mean barring feeding the exact snapshot name to zfs list [ which would mean finding the name first, of course] ) Here, it appears adding anything more to that command line causes it to fail. zfs list -t snapshot z3/projects cannot open ''z3/projects'': operation not applicable to datasets of this type An example command line from your usage might be handy.
zfs list | grep ''@'' zpool/fs1 at 1154758 324G - 461G - zpool/fs1 at 1208482 6.94G - 338G - zpool/fs1 at daily.netbackup 1.07G - 344G - zpool/fs2 at 1154758 1.77G - 242G - zpool/fs2 at 1208482 2.26G - 261G - zpool/fs2 at daily.netbackup 323M - 266G - First column there shows the size of the snapshot (e.g. how much has changed). -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Harry Putnam Sent: Monday, March 22, 2010 2:23 PM To: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] snapshots as versioning tool Matt Cowger <mcowger at salesforce.com> writes:> This is totally doable, and a reasonable use of zfs snapshots - we > do some similar things.Good, thanks for the input.> You can easily determine if the snapshot has changed by checking the > output of zfs list for the snapshot.Do you mean to just grep it out of the output of zfs list -t snapshot Or is there some finer grained way to get it? (I mean barring feeding the exact snapshot name to zfs list [ which would mean finding the name first, of course] ) Here, it appears adding anything more to that command line causes it to fail. zfs list -t snapshot z3/projects cannot open ''z3/projects'': operation not applicable to datasets of this type An example command line from your usage might be handy. _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Mon, Mar 22, 2010 at 1:58 PM, Ian Collins <ian at ianshome.com> wrote:> On 03/23/10 09:34 AM, Harry Putnam wrote: > >> Oh, something I meant to ask... is there some standard way to tell >> before calling for a snapshot, if the directory structure has changed >> at all, other than aging I mean. Is there something better than >> running `diff -r [...]'' between existing structure and last snapshot. >> >> >> > Not really, there is ZFS diff is in the woks, but not here yet. >Someone pointed out that you can use bart, but that also scans the directories. It might do what you want, but it doesn''t work at the zpool / zfs level, just at the file level layer. -B -- Brandon High : bhigh at freaks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100322/a9ce18b1/attachment.html>
> > You can easily determine if the snapshot has changed by checking the > > output of zfs list for the snapshot. > > Do you mean to just grep it out of the output of > > zfs list -t snapshotI think the point is: You can easily tell how many MB changed in a snapshot, and therefore you can easily tell "yes the snapshot changed." But unfortunately, no you can''t easily tell which files changed. Yet.
Matt Cowger <mcowger at salesforce.com> writes:> zfs list | grep ''@'' > > zpool/fs1 at 1154758 324G - 461G - > zpool/fs1 at 1208482 6.94G - 338G - > zpool/fs1 at daily.netbackup 1.07G - 344G - > zpool/fs2 at 1154758 1.77G - 242G - > zpool/fs2 at 1208482 2.26G - 261G - > zpool/fs2 at daily.netbackup 323M - 266G - > > First column there shows the size of the snapshot (e.g. how much has changed).I''m clearly missing something here. Is that a typo? (your command line) I can''t get results like that without `zfs list -t snapshot''
+------------------------------------------------------------------------------ | On 2010-03-23 16:09:05, Harry Putnam wrote: | | Date: Tue, 23 Mar 2010 16:09:05 -0500 | From: Harry Putnam <reader at newsguy.com> | To: zfs-discuss at opensolaris.org | Subject: Re: [zfs-discuss] snapshots as versioning tool | | Matt Cowger <mcowger at salesforce.com> writes: | | > zfs list | grep ''@'' | > | > zpool/fs1 at 1154758 324G - 461G - | > zpool/fs1 at 1208482 6.94G - 338G - | > zpool/fs1 at daily.netbackup 1.07G - 344G - | > zpool/fs2 at 1154758 1.77G - 242G - | > zpool/fs2 at 1208482 2.26G - 261G - | > zpool/fs2 at daily.netbackup 323M - 266G - | > | > First column there shows the size of the snapshot (e.g. how much has changed). | | I''m clearly missing something here. Is that a typo? (your command | line) | | I can''t get results like that without `zfs list -t snapshot'' The syntax for `list` changed at some point, to not list everything by default. Use `-t all` or `-t snapshot`. Presumably Matt is using an older version, or an alias? -- bda cyberpunk is dead. long live cyberpunk.
I''m running s10u8, not opensolaris, so I could be a bit behind. --M -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Bryan Allen Sent: Tuesday, March 23, 2010 2:14 PM To: Harry Putnam Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] snapshots as versioning tool +------------------------------------------------------------------------------ | On 2010-03-23 16:09:05, Harry Putnam wrote: | | Date: Tue, 23 Mar 2010 16:09:05 -0500 | From: Harry Putnam <reader at newsguy.com> | To: zfs-discuss at opensolaris.org | Subject: Re: [zfs-discuss] snapshots as versioning tool | | Matt Cowger <mcowger at salesforce.com> writes: | | > zfs list | grep ''@'' | > | > zpool/fs1 at 1154758 324G - 461G - | > zpool/fs1 at 1208482 6.94G - 338G - | > zpool/fs1 at daily.netbackup 1.07G - 344G - | > zpool/fs2 at 1154758 1.77G - 242G - | > zpool/fs2 at 1208482 2.26G - 261G - | > zpool/fs2 at daily.netbackup 323M - 266G - | > | > First column there shows the size of the snapshot (e.g. how much has changed). | | I''m clearly missing something here. Is that a typo? (your command | line) | | I can''t get results like that without `zfs list -t snapshot'' The syntax for `list` changed at some point, to not list everything by default. Use `-t all` or `-t snapshot`. Presumably Matt is using an older version, or an alias? -- bda cyberpunk is dead. long live cyberpunk. _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Tue, Mar 23, 2010 at 11:09 PM, Harry Putnam <reader at newsguy.com> wrote:> Matt Cowger <mcowger at salesforce.com> writes: > >> zfs list | grep ''@'' >> >> zpool/fs1 at 1154758 ? ? ? ? ? ? ? ? ? ? ? ? ? ?324G ? ? ?- ? 461G ?- >> zpool/fs1 at 1208482 ? ? ? ? ? ? ? ? ? ? ? ? ? 6.94G ? ? ?- ? 338G ?- >> zpool/fs1 at daily.netbackup ? ? ? ? ? ? ? ? ? 1.07G ? ? ?- ? 344G ?- >> zpool/fs2 at 1154758 ? ? ? ? ? ? ? ? ? ? ? ?1.77G ? ? ?- ? 242G ?- >> zpool/fs2 at 1208482 ? ? ? ? ? ? ? ? ? ? ? ?2.26G ? ? ?- ? 261G ?- >> zpool/fs2 at daily.netbackup ? ? ? ? ? ? ? ? 323M ? ? ?- ? 266G ?- >> >> First column there shows the size of the snapshot (e.g. how much has changed). > > I''m clearly missing something here. ? Is that a typo? (your command > line) > > I can''t get results like that without ?`zfs list -t snapshot''It depends on listsnapshots _pool_ property being on. (It is off by default) hellride:~$ zpool get listsnapshots rpool NAME PROPERTY VALUE SOURCE rpool listsnapshots on local -- Regards, Cyril
Brandon High <bhigh at freaks.com> writes:> Someone pointed out that you can use bart, but that also scans the > directories. It might do what you want, but it doesn''t work at the zpool / > zfs level, just at the file level layer.Apparently I missed any suggestion about bart, but looking it up just now, I guess maybe in what they call `safe mode'' where changed files aren''t deleted, it might be useful as a versioning tool. Sounds like it could be targeted with a little finer granularity than snapshots generally can be. At just a quick read, it really just sounds like rsync, after its been in a severe wreck and was badly crippled. Maybe `bart'' handles windows files better than rsync? I''m just curious why `bart'' would be recommended over rsync? Are there abilities that make it more attractive?
Harry Putnam <reader at newsguy.com> writes:> At just a quick read, it really just sounds like rsync, after its been in a > severe wreck and was badly crippled.OOps, I may have looked at the wrong bart. One of the first hits google turned up was: http://www.zhornsoftware.co.uk/bart/index.html But I think maybe this `bart'' is what was suggested: http://www.unisol.com/papers/bart_paper.html This looks to be a comprehensive network backup system. But would be way overdone for what I talked about.
> OOps, I may have looked at the wrong bart.I think he meant this BART: http://blogs.sun.com/gbrunett/entry/automating_solaris_10_file_integrity I''m going to make one quick comment about this, despite better judgment to probably keep quiet. I don''t think anyone should use ZFS as a VCS like Subversion ... that''s nuts! How many developers on your project? How many sub projects, how many commits a day? I just started a new repo and I''m up in the hundreds in a few weeks. Do you want to keep that many snapshots around? Someone is going to get the idea to use ZFS like this, and 8 months from now, get bitter and heart-broken and dump on ZFS for not behaving like a VCS, which it is not. VCS has logs, easy diffs, easy rollback, merge, branch, feeds into build automation software, allows IDEs to be fed into it, integrates with tracking tools ... none of which ZFS does (and I''m not saying this like it''s a bad thing.) If you want to do a code release, say, 1.0, put that on a ZFS filesystem, snapshot it, keep developing until you get to somewhere you want to call a 1.1, snapshot that ... that is a wonderful thing to do. You can clone and make active an entire bundle of stuff. But please, use versioning software (good ones are free, even) for versioning and don''t shoehorn ZFS. CT
Christine Tran <christine.tran at gmail.com> writes:>> OOps, I may have looked at the wrong bart. > > I think he meant this BART: > http://blogs.sun.com/gbrunett/entry/automating_solaris_10_file_integrity > > I''m going to make one quick comment about this, despite better > judgment to probably keep quiet. I don''t think anyone should use ZFS > as a VCS like Subversion ... that''s nuts! How many developers on your > project? How many sub projects, how many commits a day? I just > started a new repo and I''m up in the hundreds in a few weeks. Do you > want to keep that many snapshots around? Someone is going to get the > idea to use ZFS like this, and 8 months from now, get bitter and > heart-broken and dump on ZFS for not behaving like a VCS, which it is > not.I can''t tell if you''ve read the thread or not. I''d guess not since the OP was about a temporary usage for specific single user projects. (This was not spelled out) Probably several days of coding or experimenting at most and always by a single user. The OP stated:> When I do video projects or any projects for that matter. > I sometimes want backups every 10 minutes or so, so as not to loose > some piece of script that isn''t finished or the like.So, the usage is as much backup as it is vcs...more really. The special 10 minute or so snapshotting would be a cron run scripted setup that would be turned on during coding and experimentation, and off after wards. The `versions'' (snapshots) would be of use only during that time, and would be completely deleted after. Then possibly some resulting script or etc, would get committed in a real vcs. The snapshots themselves would be separate from any default snapshots that might be setup. I''m liking this notion more and more as I''ve talked about it in this thread.> Someone is going to get the > idea to use ZFS like this, and 8 months from now, get bitter and > heart-broken and dump on ZFS for not behaving like a VCS, which it is > not.That''s pretty far fetched but I suppose possible. However with the vast amount of information online about what zfs is, and how to use it, I don''t see it being a problem.