Hi, I was wondering if there have been any conversations with backup vendors like Veritas or EMC regarding better integration with ZFS. While I understand they can use the "native" mode of reading files from the filesystem, it would be great if there were agents that had options like making a snapshot and storing a "zfs backup" datastream that could be used for zfs restore. This message posted from opensolaris.org
I was thinking about this too the other day. It would also be really neat to put some kind of scheduling method into zfs''s tools as well. I had to write a few scripts to handle snapshots (both taking them and rotating/expunging them as necessary). I imagine 99% of people who use snapshots will be doing the same thing, and dumping it all into cron. Something that would be done by so many people would seem like a good thing to integrate. If it also had the ability to work directly with the backup hardware, that would be a huge boon. Cheers, David> Hi, > I was wondering if there have been any conversations with backup vendors > like Veritas or EMC regarding better integration with ZFS. While I > understand they can use the "native" mode of reading files from the > filesystem, it would be great if there were agents that had options like > making a snapshot and storing a "zfs backup" datastream that could be > used for zfs restore. > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
In netbackup, you can write pre-backup and post-backup scripts to be called when the backup runs. This is commonly used to put databases in backup mode prior to snapshots (pre) and take them out of backup mode (post). With this functionality, you don''t really need a lot of understanding of the filesystem. I''m not saying it shouldn''t understand the filesystem (after all, on a full restore, it has to put it all back together), but at least with netbackup, the functionality already exists. On Mar 28, 2006, at 12:17 PM, David J. Orman wrote:> I was thinking about this too the other day. It would also be > really neat > to put some kind of scheduling method into zfs''s tools as well. I > had to > write a few scripts to handle snapshots (both taking them and > rotating/expunging them as necessary). I imagine 99% of people who use > snapshots will be doing the same thing, and dumping it all into cron. > Something that would be done by so many people would seem like a good > thing to integrate. If it also had the ability to work directly > with the > backup hardware, that would be a huge boon. > > Cheers, > David > >> Hi, >> I was wondering if there have been any conversations with backup >> vendors >> like Veritas or EMC regarding better integration with ZFS. While I >> understand they can use the "native" mode of reading files from the >> filesystem, it would be great if there were agents that had >> options like >> making a snapshot and storing a "zfs backup" datastream that could be >> used for zfs restore. >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss----- Gregory Shaw, IT Architect Phone: (303) 673-8273 Fax: (303) 673-8273 ITCTO Group, Sun Microsystems Inc. 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) Louisville, CO 80028-4382 shaw at fmsoft.com (home) "When Microsoft writes an application for Linux, I''ve Won." - Linus Torvalds
Hi Greg,
What I''d like to see out of an agent is that people
wouldn''t need to
worry about writing pre and post scripts to have a lot of the power of
ZFS natively harnessed by Netbackup.
I understand you could have a pre-backup script create a snapshot and
run zfs backup, but then wouldn''t you need to have it be written to
disk
first? If there was a custom agent, I think it would be cool if it was
able to basically do:
1) create a snapshot
2) zfs backup $dataset@$snap (and reading the data from stdout and
sending it straight back to Netbackup without needing an intermediary
file and thus 2x the space temporarily)
3) (optionally) delete the snapshot
Given that most enterprises I''m aware of use Netbackup I think such
tight integration would help increase ZFS adoption once it is released.
If there is a way to do the above without a custom agent I think that
would make for a great writeup which I would be happy to help with.
Gregory Shaw wrote:> In netbackup, you can write pre-backup and post-backup scripts to be
> called when the backup runs. This is commonly used to put databases in
> backup mode prior to snapshots (pre) and take them out of backup mode
> (post).
>
> With this functionality, you don''t really need a lot of
understanding of
> the filesystem. I''m not saying it shouldn''t understand
the filesystem
> (after all, on a full restore, it has to put it all back together), but
> at least with netbackup, the functionality already exists.
>
> On Mar 28, 2006, at 12:17 PM, David J. Orman wrote:
>
>> I was thinking about this too the other day. It would also be really
neat
>> to put some kind of scheduling method into zfs''s tools as
well. I had to
>> write a few scripts to handle snapshots (both taking them and
>> rotating/expunging them as necessary). I imagine 99% of people who use
>> snapshots will be doing the same thing, and dumping it all into cron.
>> Something that would be done by so many people would seem like a good
>> thing to integrate. If it also had the ability to work directly with
the
>> backup hardware, that would be a huge boon.
>>
>> Cheers,
>> David
>>
>>> Hi,
>>> I was wondering if there have been any conversations with backup
>>> vendors
>>> like Veritas or EMC regarding better integration with ZFS. While I
>>> understand they can use the "native" mode of reading
files from the
>>> filesystem, it would be great if there were agents that had options
like
>>> making a snapshot and storing a "zfs backup" datastream
that could be
>>> used for zfs restore.
>>> This message posted from opensolaris.org
>>> _______________________________________________
>>> zfs-discuss mailing list
>>> zfs-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>>
>>
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
> -----
> Gregory Shaw, IT Architect
> Phone: (303) 673-8273 Fax: (303) 673-8273
> ITCTO Group, Sun Microsystems Inc.
> 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work)
> Louisville, CO 80028-4382 shaw at fmsoft.com (home)
> "When Microsoft writes an application for Linux, I''ve
Won." - Linus
> Torvalds
>
>
--
William D. Hathaway email: william.hathaway at versatile.com
Solutions Architect aim: wdhPO
Versatile, Inc. cell: 717-314-5461
I don''t disagree. However, I think the push-back will come from symantec/veritas (still can''t get used to typing symantec with regard to anything unix): There are a myriad of solutions out there that will request the same sort of activity. Be it database snapsnots, FS snapshots, hardware based snapshots, volume manager''isms, etc, everybody is trying to something similar. Which do you built into the backup product? You''re talking about many operating systems with hundreds of products used in thousands of scenarios. Automatic is good. However flexible is better as it works in more circumstances. I''ve have a lot of experience with customers that want to do things in similar but slightly different ways. Rather than building in a complex operation into a product, it''s much easier to build in scripting interfaces that can be employed to ''do the right thing''. It''s much easier to support. On Mar 29, 2006, at 10:31 AM, William D. Hathaway wrote:> Hi Greg, > What I''d like to see out of an agent is that people wouldn''t > need to worry about writing pre and post scripts to have a lot of > the power of ZFS natively harnessed by Netbackup. > > I understand you could have a pre-backup script create a snapshot > and run zfs backup, but then wouldn''t you need to have it be > written to disk first? If there was a custom agent, I think it > would be cool if it was able to basically do: > > 1) create a snapshot > 2) zfs backup $dataset@$snap (and reading the data from stdout and > sending it straight back to Netbackup without needing an > intermediary file and thus 2x the space temporarily) > 3) (optionally) delete the snapshot > > Given that most enterprises I''m aware of use Netbackup I think > such tight integration would help increase ZFS adoption once it is > released. > > If there is a way to do the above without a custom agent I think > that would make for a great writeup which I would be happy to help > with. > > > > > > Gregory Shaw wrote: >> In netbackup, you can write pre-backup and post-backup scripts to >> be called when the backup runs. This is commonly used to put >> databases in backup mode prior to snapshots (pre) and take them >> out of backup mode (post). >> With this functionality, you don''t really need a lot of >> understanding of the filesystem. I''m not saying it shouldn''t >> understand the filesystem (after all, on a full restore, it has to >> put it all back together), but at least with netbackup, the >> functionality already exists. >> On Mar 28, 2006, at 12:17 PM, David J. Orman wrote: >>> I was thinking about this too the other day. It would also be >>> really neat >>> to put some kind of scheduling method into zfs''s tools as well. I >>> had to >>> write a few scripts to handle snapshots (both taking them and >>> rotating/expunging them as necessary). I imagine 99% of people >>> who use >>> snapshots will be doing the same thing, and dumping it all into >>> cron. >>> Something that would be done by so many people would seem like a >>> good >>> thing to integrate. If it also had the ability to work directly >>> with the >>> backup hardware, that would be a huge boon. >>> >>> Cheers, >>> David >>> >>>> Hi, >>>> I was wondering if there have been any conversations with >>>> backup vendors >>>> like Veritas or EMC regarding better integration with ZFS. While I >>>> understand they can use the "native" mode of reading files from the >>>> filesystem, it would be great if there were agents that had >>>> options like >>>> making a snapshot and storing a "zfs backup" datastream that >>>> could be >>>> used for zfs restore. >>>> This message posted from opensolaris.org >>>> _______________________________________________ >>>> zfs-discuss mailing list >>>> zfs-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>> >>> >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> ----- >> Gregory Shaw, IT Architect >> Phone: (303) 673-8273 Fax: (303) 673-8273 >> ITCTO Group, Sun Microsystems Inc. >> 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) >> Louisville, CO 80028-4382 shaw at fmsoft.com (home) >> "When Microsoft writes an application for Linux, I''ve Won." - >> Linus Torvalds > > > -- > William D. Hathaway email: william.hathaway at versatile.com > Solutions Architect aim: wdhPO > Versatile, Inc. cell: 717-314-5461----- Gregory Shaw, IT Architect Phone: (303) 673-8273 Fax: (303) 673-8273 ITCTO Group, Sun Microsystems Inc. 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) Louisville, CO 80028-4382 shaw at fmsoft.com (home) "When Microsoft writes an application for Linux, I''ve Won." - Linus Torvalds
I think the best scenario would be if there was someone with the Netbackup SDK license that would be interested in doing an open source XBSA agent. It doesn''t have to be Veritas doing the work (although if they did, they could charge for it separately like they do for Flashbackup, Oracle and other agents they offer) I agree on the flexibility of scripting but how do you efficiently backup a ZFS file system that doesn''t have enough free space in the pool to store a copy of the largest file system? Gregory Shaw wrote:> I don''t disagree. However, I think the push-back will come from > symantec/veritas (still can''t get used to typing symantec with regard to > anything unix): There are a myriad of solutions out there that will > request the same sort of activity. Be it database snapsnots, FS > snapshots, hardware based snapshots, volume manager''isms, etc, everybody > is trying to something similar. > > Which do you built into the backup product? You''re talking about many > operating systems with hundreds of products used in thousands of scenarios. > > Automatic is good. However flexible is better as it works in more > circumstances. > > I''ve have a lot of experience with customers that want to do things in > similar but slightly different ways. Rather than building in a complex > operation into a product, it''s much easier to build in scripting > interfaces that can be employed to ''do the right thing''. > > It''s much easier to support. > > On Mar 29, 2006, at 10:31 AM, William D. Hathaway wrote: > >> Hi Greg, >> What I''d like to see out of an agent is that people wouldn''t need >> to worry about writing pre and post scripts to have a lot of the power >> of ZFS natively harnessed by Netbackup. >> >> I understand you could have a pre-backup script create a snapshot >> and run zfs backup, but then wouldn''t you need to have it be written >> to disk first? If there was a custom agent, I think it would be cool >> if it was able to basically do: >> >> 1) create a snapshot >> 2) zfs backup $dataset@$snap (and reading the data from stdout and >> sending it straight back to Netbackup without needing an intermediary >> file and thus 2x the space temporarily) >> 3) (optionally) delete the snapshot >> >> Given that most enterprises I''m aware of use Netbackup I think such >> tight integration would help increase ZFS adoption once it is released. >> >> If there is a way to do the above without a custom agent I think >> that would make for a great writeup which I would be happy to help with. >> >> >> >> >> >> Gregory Shaw wrote: >>> In netbackup, you can write pre-backup and post-backup scripts to be >>> called when the backup runs. This is commonly used to put databases >>> in backup mode prior to snapshots (pre) and take them out of backup >>> mode (post). >>> With this functionality, you don''t really need a lot of understanding >>> of the filesystem. I''m not saying it shouldn''t understand the >>> filesystem (after all, on a full restore, it has to put it all back >>> together), but at least with netbackup, the functionality already >>> exists. >>> On Mar 28, 2006, at 12:17 PM, David J. Orman wrote: >>>> I was thinking about this too the other day. It would also be really >>>> neat >>>> to put some kind of scheduling method into zfs''s tools as well. I >>>> had to >>>> write a few scripts to handle snapshots (both taking them and >>>> rotating/expunging them as necessary). I imagine 99% of people who use >>>> snapshots will be doing the same thing, and dumping it all into cron. >>>> Something that would be done by so many people would seem like a good >>>> thing to integrate. If it also had the ability to work directly with >>>> the >>>> backup hardware, that would be a huge boon. >>>> >>>> Cheers, >>>> David >>>> >>>>> Hi, >>>>> I was wondering if there have been any conversations with backup >>>>> vendors >>>>> like Veritas or EMC regarding better integration with ZFS. While I >>>>> understand they can use the "native" mode of reading files from the >>>>> filesystem, it would be great if there were agents that had options >>>>> like >>>>> making a snapshot and storing a "zfs backup" datastream that could be >>>>> used for zfs restore. >>>>> This message posted from opensolaris.org >>>>> _______________________________________________ >>>>> zfs-discuss mailing list >>>>> zfs-discuss at opensolaris.org >>>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>>> >>>> >>>> >>>> _______________________________________________ >>>> zfs-discuss mailing list >>>> zfs-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> ----- >>> Gregory Shaw, IT Architect >>> Phone: (303) 673-8273 Fax: (303) 673-8273 >>> ITCTO Group, Sun Microsystems Inc. >>> 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) >>> Louisville, CO 80028-4382 shaw at fmsoft.com (home) >>> "When Microsoft writes an application for Linux, I''ve Won." - Linus >>> Torvalds >> >> >> --William D. Hathaway email: william.hathaway at versatile.com >> Solutions Architect aim: wdhPO >> Versatile, Inc. cell: 717-314-5461 > > ----- > Gregory Shaw, IT Architect > Phone: (303) 673-8273 Fax: (303) 673-8273 > ITCTO Group, Sun Microsystems Inc. > 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) > Louisville, CO 80028-4382 shaw at fmsoft.com (home) > "When Microsoft writes an application for Linux, I''ve Won." - Linus > Torvalds > >-- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
On Wed, Mar 29, 2006 at 12:31:25PM -0500, William D. Hathaway wrote:> Hi Greg, > What I''d like to see out of an agent is that people wouldn''t need to > worry about writing pre and post scripts to have a lot of the power of > ZFS natively harnessed by Netbackup. > > I understand you could have a pre-backup script create a snapshot and > run zfs backup, but then wouldn''t you need to have it be written to disk > first? If there was a custom agent, I think it would be cool if it was > able to basically do: > > 1) create a snapshot > 2) zfs backup $dataset@$snap (and reading the data from stdout and > sending it straight back to Netbackup without needing an intermediary > file and thus 2x the space temporarily) > 3) (optionally) delete the snapshotOne thing that may help here is an open RFE (I don''t have the bugID handy) to do a ''diff'' between two snapshots. You could quickly mount a filesystem that only contained the file/directories which had changed between two snapshots. This would make it easy for backup programs to quickly find the necessary files to backup, while maintaining their own backup format. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On a completely separate tack, a few questions/points: It appears that you''re looking to something similar to the database backup modules (like oracle) for ZFS. That''s fine for application backups, but is it practical to use zfs backup for storing data on tape within netbackup? Netbackup stores it''s files on tape in gnu tar format. As such, it''s an open standard that hasn''t changed in a long time. Based on that, it''s reasonable that in 7-20 years that you can restore an offsite backup. zfs backup, by comparison, is a vendor-specific format (currently) that is in flux. I think of it similarly to ufsbackup. If you''ve got the ability to snapshot, why would you need to do a zfs backup at that point? Why not just backup the snapshot with the backup tool (netbackup in this case)? You wouldn''t need to store the backup separately at that point. Perhaps I''m missing something in the below? On Mar 29, 2006, at 10:31 AM, William D. Hathaway wrote:> Hi Greg, > What I''d like to see out of an agent is that people wouldn''t > need to worry about writing pre and post scripts to have a lot of > the power of ZFS natively harnessed by Netbackup. > > I understand you could have a pre-backup script create a snapshot > and run zfs backup, but then wouldn''t you need to have it be > written to disk first? If there was a custom agent, I think it > would be cool if it was able to basically do: > > 1) create a snapshot > 2) zfs backup $dataset@$snap (and reading the data from stdout and > sending it straight back to Netbackup without needing an > intermediary file and thus 2x the space temporarily) > 3) (optionally) delete the snapshot > > Given that most enterprises I''m aware of use Netbackup I think > such tight integration would help increase ZFS adoption once it is > released. > > If there is a way to do the above without a custom agent I think > that would make for a great writeup which I would be happy to help > with. > > > > > > Gregory Shaw wrote: >> In netbackup, you can write pre-backup and post-backup scripts to >> be called when the backup runs. This is commonly used to put >> databases in backup mode prior to snapshots (pre) and take them >> out of backup mode (post). >> With this functionality, you don''t really need a lot of >> understanding of the filesystem. I''m not saying it shouldn''t >> understand the filesystem (after all, on a full restore, it has to >> put it all back together), but at least with netbackup, the >> functionality already exists. >> On Mar 28, 2006, at 12:17 PM, David J. Orman wrote: >>> I was thinking about this too the other day. It would also be >>> really neat >>> to put some kind of scheduling method into zfs''s tools as well. I >>> had to >>> write a few scripts to handle snapshots (both taking them and >>> rotating/expunging them as necessary). I imagine 99% of people >>> who use >>> snapshots will be doing the same thing, and dumping it all into >>> cron. >>> Something that would be done by so many people would seem like a >>> good >>> thing to integrate. If it also had the ability to work directly >>> with the >>> backup hardware, that would be a huge boon. >>> >>> Cheers, >>> David >>> >>>> Hi, >>>> I was wondering if there have been any conversations with >>>> backup vendors >>>> like Veritas or EMC regarding better integration with ZFS. While I >>>> understand they can use the "native" mode of reading files from the >>>> filesystem, it would be great if there were agents that had >>>> options like >>>> making a snapshot and storing a "zfs backup" datastream that >>>> could be >>>> used for zfs restore. >>>> This message posted from opensolaris.org >>>> _______________________________________________ >>>> zfs-discuss mailing list >>>> zfs-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>> >>> >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> ----- >> Gregory Shaw, IT Architect >> Phone: (303) 673-8273 Fax: (303) 673-8273 >> ITCTO Group, Sun Microsystems Inc. >> 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) >> Louisville, CO 80028-4382 shaw at fmsoft.com (home) >> "When Microsoft writes an application for Linux, I''ve Won." - >> Linus Torvalds > > > -- > William D. Hathaway email: william.hathaway at versatile.com > Solutions Architect aim: wdhPO > Versatile, Inc. cell: 717-314-5461----- Gregory Shaw, IT Architect Phone: (303) 673-8273 Fax: (303) 673-8273 ITCTO Group, Sun Microsystems Inc. 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) Louisville, CO 80028-4382 shaw at fmsoft.com (home) "When Microsoft writes an application for Linux, I''ve Won." - Linus Torvalds
Gregory Shaw wrote:> On a completely separate tack, a few questions/points: > > It appears that you''re looking to something similar to the database > backup modules (like oracle) for ZFS. That''s fine for application > backups, but is it practical to use zfs backup for storing data on tape > within netbackup?> > Netbackup stores it''s files on tape in gnu tar format. As such, it''s an > open standard that hasn''t changed in a long time. Based on that, it''s > reasonable that in 7-20 years that you can restore an offsite backup.I agree and if I needed data to be restored in 7+ years, zfs internal backup format certainly wouldn''t be what I would propose. I think there is generally very specific data that needs that level of retention (such as data mandated for archiving by SEC).> > zfs backup, by comparison, is a vendor-specific format (currently) that > is in flux. I think of it similarly to ufsbackup. >I agree completely.> If you''ve got the ability to snapshot, why would you need to do a zfs > backup at that point? Why not just backup the snapshot with the backup > tool (netbackup in this case)? You wouldn''t need to store the backup > separately at that point.A specific example I''m thinking of is large mail servers. In the JES messaging server, each email message is a separate file. On a large system you may end up having 10s of millions of files. Backing those up by reading the file system (or a snapshot) file by file is a losing proposition. That is an example of where there is a huge advantage of zfs backup over using traditional "file at a time" methods.> Perhaps I''m missing something in the below? > > On Mar 29, 2006, at 10:31 AM, William D. Hathaway wrote: > >> Hi Greg, >> What I''d like to see out of an agent is that people wouldn''t need >> to worry about writing pre and post scripts to have a lot of the power >> of ZFS natively harnessed by Netbackup. >> >> I understand you could have a pre-backup script create a snapshot >> and run zfs backup, but then wouldn''t you need to have it be written >> to disk first? If there was a custom agent, I think it would be cool >> if it was able to basically do: >> >> 1) create a snapshot >> 2) zfs backup $dataset@$snap (and reading the data from stdout and >> sending it straight back to Netbackup without needing an intermediary >> file and thus 2x the space temporarily) >> 3) (optionally) delete the snapshot >> >> Given that most enterprises I''m aware of use Netbackup I think such >> tight integration would help increase ZFS adoption once it is released. >> >> If there is a way to do the above without a custom agent I think >> that would make for a great writeup which I would be happy to help with. >> >> >> >> >> >> Gregory Shaw wrote: >>> In netbackup, you can write pre-backup and post-backup scripts to be >>> called when the backup runs. This is commonly used to put databases >>> in backup mode prior to snapshots (pre) and take them out of backup >>> mode (post). >>> With this functionality, you don''t really need a lot of understanding >>> of the filesystem. I''m not saying it shouldn''t understand the >>> filesystem (after all, on a full restore, it has to put it all back >>> together), but at least with netbackup, the functionality already >>> exists. >>> On Mar 28, 2006, at 12:17 PM, David J. Orman wrote: >>>> I was thinking about this too the other day. It would also be really >>>> neat >>>> to put some kind of scheduling method into zfs''s tools as well. I >>>> had to >>>> write a few scripts to handle snapshots (both taking them and >>>> rotating/expunging them as necessary). I imagine 99% of people who use >>>> snapshots will be doing the same thing, and dumping it all into cron. >>>> Something that would be done by so many people would seem like a good >>>> thing to integrate. If it also had the ability to work directly with >>>> the >>>> backup hardware, that would be a huge boon. >>>> >>>> Cheers, >>>> David >>>> >>>>> Hi, >>>>> I was wondering if there have been any conversations with backup >>>>> vendors >>>>> like Veritas or EMC regarding better integration with ZFS. While I >>>>> understand they can use the "native" mode of reading files from the >>>>> filesystem, it would be great if there were agents that had options >>>>> like >>>>> making a snapshot and storing a "zfs backup" datastream that could be >>>>> used for zfs restore. >>>>> This message posted from opensolaris.org >>>>> _______________________________________________ >>>>> zfs-discuss mailing list >>>>> zfs-discuss at opensolaris.org >>>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>>>> >>>> >>>> >>>> _______________________________________________ >>>> zfs-discuss mailing list >>>> zfs-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> ----- >>> Gregory Shaw, IT Architect >>> Phone: (303) 673-8273 Fax: (303) 673-8273 >>> ITCTO Group, Sun Microsystems Inc. >>> 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) >>> Louisville, CO 80028-4382 shaw at fmsoft.com (home) >>> "When Microsoft writes an application for Linux, I''ve Won." - Linus >>> Torvalds >> >> >> --William D. Hathaway email: william.hathaway at versatile.com >> Solutions Architect aim: wdhPO >> Versatile, Inc. cell: 717-314-5461 > > ----- > Gregory Shaw, IT Architect > Phone: (303) 673-8273 Fax: (303) 673-8273 > ITCTO Group, Sun Microsystems Inc. > 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) > Louisville, CO 80028-4382 shaw at fmsoft.com (home) > "When Microsoft writes an application for Linux, I''ve Won." - Linus > Torvalds > >-- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
On Mar 29, 2006, at 12:34 PM, William D. Hathaway wrote:> Gregory Shaw wrote: >> On a completely separate tack, a few questions/points: >> It appears that you''re looking to something similar to the >> database backup modules (like oracle) for ZFS. That''s fine for >> application backups, but is it practical to use zfs backup for >> storing data on tape within netbackup? > >> Netbackup stores it''s files on tape in gnu tar format. As such, >> it''s an open standard that hasn''t changed in a long time. Based >> on that, it''s reasonable that in 7-20 years that you can restore >> an offsite backup. > I agree and if I needed data to be restored in 7+ years, zfs > internal backup format certainly wouldn''t be what I would propose. > I think there is generally very specific data that needs that level > of retention (such as data mandated for archiving by SEC). >In recent history, I''m finding that Sarbanes-Oxley is causing most business people to go to a 7-30 year retention time for most of their data. As time goes on, it''s becoming more and more common. Most engineering groups want a 7 year retention.>> zfs backup, by comparison, is a vendor-specific format (currently) >> that is in flux. I think of it similarly to ufsbackup. > I agree completely. > >> If you''ve got the ability to snapshot, why would you need to do a >> zfs backup at that point? Why not just backup the snapshot with >> the backup tool (netbackup in this case)? You wouldn''t need to >> store the backup separately at that point. > > A specific example I''m thinking of is large mail servers. In the > JES messaging server, each email message is a separate file. On a > large system you may end up having 10s of millions of files. > Backing those up by reading the file system (or a snapshot) file by > file is a losing proposition. That is an example of where there is > a huge advantage of zfs backup over using traditional "file at a > time" methods. >There are other ways of doing this. I don''t see your point with zfs backup, however. zfs backup still has to save a reference to every file. As such, you''ve got to make an entry somewhere or otherwise process the directory information. Also, if you use zfs backup instead of the backup product, you''ve lost the ability to restore a single file or search for a file in a range of dates. That''s a huge loss in flexibility, especially when talking about data that has been gone for a while. Regarding other ways, synthetic fulls and block-level incrementals can help with this sort of work. [ ... deleted for brevity ] ----- Gregory Shaw, IT Architect Phone: (303) 673-8273 Fax: (303) 673-8273 ITCTO Group, Sun Microsystems Inc. 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) Louisville, CO 80028-4382 shaw at fmsoft.com (home) "When Microsoft writes an application for Linux, I''ve Won." - Linus Torvalds
Gregory Shaw wrote:> > On Mar 29, 2006, at 12:34 PM, William D. Hathaway wrote: > >> Gregory Shaw wrote: >>> On a completely separate tack, a few questions/points: >>> It appears that you''re looking to something similar to the database >>> backup modules (like oracle) for ZFS. That''s fine for application >>> backups, but is it practical to use zfs backup for storing data on >>> tape within netbackup? >> >>> Netbackup stores it''s files on tape in gnu tar format. As such, it''s >>> an open standard that hasn''t changed in a long time. Based on that, >>> it''s reasonable that in 7-20 years that you can restore an offsite >>> backup. >> I agree and if I needed data to be restored in 7+ years, zfs internal >> backup format certainly wouldn''t be what I would propose. I think >> there is generally very specific data that needs that level of >> retention (such as data mandated for archiving by SEC). >> > > In recent history, I''m finding that Sarbanes-Oxley is causing most > business people to go to a 7-30 year retention time for most of their data. > > As time goes on, it''s becoming more and more common. Most engineering > groups want a 7 year retention. > >>> zfs backup, by comparison, is a vendor-specific format (currently) >>> that is in flux. I think of it similarly to ufsbackup. >> I agree completely. >> >>> If you''ve got the ability to snapshot, why would you need to do a zfs >>> backup at that point? Why not just backup the snapshot with the >>> backup tool (netbackup in this case)? You wouldn''t need to store the >>> backup separately at that point. >> >> A specific example I''m thinking of is large mail servers. In the JES >> messaging server, each email message is a separate file. On a large >> system you may end up having 10s of millions of files. Backing those >> up by reading the file system (or a snapshot) file by file is a losing >> proposition. That is an example of where there is a huge advantage of >> zfs backup over using traditional "file at a time" methods. >> > > There are other ways of doing this. I don''t see your point with zfs > backup, however. zfs backup still has to save a reference to every > file. As such, you''ve got to make an entry somewhere or otherwise > process the directory information.I''ve done some timing of using zfs backup vs gtar for a case of 1M files with an average size of 13k and I see zfs backup run about 20x faster than gtar. My impression is that gtar has to use tons of cycles doing all the system calls to open/read/close on each of the files and that zfs backup has a huge advantage by just being able to internally iterate and write out data structures. This was on a a machine with 1GB RAM so file system caching should have been negligible.> > Also, if you use zfs backup instead of the backup product, you''ve lost > the ability to restore a single file or search for a file in a range of > dates. That''s a huge loss in flexibility, especially when talking about > data that has been gone for a while. > > Regarding other ways, synthetic fulls and block-level incrementals can > help with this sort of work. > > [ ... deleted for brevity ] > > ----- > Gregory Shaw, IT Architect > Phone: (303) 673-8273 Fax: (303) 673-8273 > ITCTO Group, Sun Microsystems Inc. > 1 StorageTek Drive MS 4382 greg.shaw at sun.com (work) > Louisville, CO 80028-4382 shaw at fmsoft.com (home) > "When Microsoft writes an application for Linux, I''ve Won." - Linus > Torvalds > >-- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
On Wed, Mar 29, 2006 at 01:16:06PM -0500, William D. Hathaway wrote:> I agree on the flexibility of scripting but how do you efficiently > backup a ZFS file system that doesn''t have enough free space in the pool > to store a copy of the largest file system?Since ZFS snapshot are COW, you don''t need enough free space for the largest filesystem. You only need enough free space to absorb the churn for the duration of the backup operation. --Bill
Bill Moore wrote:> On Wed, Mar 29, 2006 at 01:16:06PM -0500, William D. Hathaway wrote: >> I agree on the flexibility of scripting but how do you efficiently >> backup a ZFS file system that doesn''t have enough free space in the pool >> to store a copy of the largest file system? > > Since ZFS snapshot are COW, you don''t need enough free space for the > largest filesystem. You only need enough free space to absorb the churn > for the duration of the backup operation.Yeah, my head is still stuck on the millions of files case where I believe you want to use zfs backup (and optimal to have it go straight into the enterprise backup system). I agree that with snapshots being CoW doing a "normal" backup of a snapshot isn''t typically space constrained unless you have insane change rates. I spend a fair amount of time working with big messaging systems which is why this corner case is of interest to me (although I''m sure there are other applications that hit the "millions of files" problem). I just finished a run of gtar vs zfs backup for 500k small files and it took 2hr 39min for gtar and 7 min for zfs backup.> > > --Bill-- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
Gregory Shaw wrote:> On a completely separate tack, a few questions/points: > > It appears that you''re looking to something similar to the database > backup modules (like oracle) for ZFS. That''s fine for application > backups, but is it practical to use zfs backup for storing data on > tape within netbackup? > > Netbackup stores it''s files on tape in gnu tar format. As such, it''s > an open standard that hasn''t changed in a long time. Based on that, > it''s reasonable that in 7-20 years that you can restore an offsite > backup.Is GNU tar''s file format an _open standard_ or open source product that is a defacto standard?> zfs backup, by comparison, is a vendor-specific format (currently) > that is in flux. I think of it similarly to ufsbackup.The only benefit gtar has is that by virtue of the source being open, the tape format is also open. This doesn''t mean the tape format won''t change or can''t change. And to answer that challenge, I saw this: OpenSolaris!> If you''ve got the ability to snapshot, why would you need to do a zfs > backup at that point? Why not just backup the snapshot with the > backup tool (netbackup in this case)? You wouldn''t need to store the > backup separately at that point. > > Perhaps I''m missing something in the below?In another email you say: "Also, if you use zfs backup instead of the backup product, you''ve lost the ability to restore a single file or search for a file in a range of dates. That''s a huge loss in flexibility, especially when talking about data that has been gone for a while." ufsbackup/ufsrestore does give you the ability to restore a single file but perhaps not within a given range of dates, so maybe you are missing something :) I can still read my ufsbackup tapes from over 10 years ago, so there''s no advantage here to using gtar. The very first "home brew" Unix backup system I used was a collection of perl scripts wrapped around dump/restore on SunOS4 which I was easily able to make function with Solaris, however there was no database of what was on each tape. The best in-house backup program I''ve seen ship with a Unix is HP-UX''s fsbackup (I think that''s the name.) It combines the ease of use/functionality of tar with a dump/restore-like back end. It''s big win over using either of tar/dump was that it could create an index file as the backup progressed. If I recall correctly, fsbackup had its own proprietary format. I don''t see that creating a tool like that to ship with Solaris would necessarily be something for the ZFS team to do, except maybe for the backend that grovels the ZFS partition itself. Darren
Some would argue (and academia may very well) that amanda (www.amanda.org) is the gold standard of open source backup tools. Although it can use the FS dump/restore for xfs/ext3/ufs/etc. many people use gtar (version 1.15) as its platform agnostic backup client of choice. Furthermore, and maybe I am wrong about zfsbackup, but one tends to need to split up volumes into manageable chunks (say 400GB each) that may not map well to a multi terabyte ZFS filesystem. The logical separation vs the disk sizing one uses for backups does not always map to one backup bin to one filesystem. Gnu tar provides this. For our organization, we will not be migrating away from amanda, and our use of ZFS is predicated on using tar. So, at least addressing how such third party utilities interface with ZFS is critical for a lot of organizations, especially heterogeneous ones. On 3/29/06, Darren Reed <Darren.Reed at sun.com> wrote:> Gregory Shaw wrote: > > > On a completely separate tack, a few questions/points: > > > > It appears that you''re looking to something similar to the database > > backup modules (like oracle) for ZFS. That''s fine for application > > backups, but is it practical to use zfs backup for storing data on > > tape within netbackup? > > > > Netbackup stores it''s files on tape in gnu tar format. As such, it''s > > an open standard that hasn''t changed in a long time. Based on that, > > it''s reasonable that in 7-20 years that you can restore an offsite > > backup. > > > Is GNU tar''s file format an _open standard_ or open source product that > is a defacto standard? > > > zfs backup, by comparison, is a vendor-specific format (currently) > > that is in flux. I think of it similarly to ufsbackup. > > > The only benefit gtar has is that by virtue of the source being open, > the tape format is also open. This doesn''t mean the tape format won''t > change or can''t change. And to answer that challenge, I saw this: > OpenSolaris! > > > If you''ve got the ability to snapshot, why would you need to do a zfs > > backup at that point? Why not just backup the snapshot with the > > backup tool (netbackup in this case)? You wouldn''t need to store the > > backup separately at that point. > > > > Perhaps I''m missing something in the below? > > > In another email you say: > > "Also, if you use zfs backup instead of the backup product, you''ve lost > the ability to restore a single file or search for a file in a range of > dates. That''s a huge loss in flexibility, especially when talking > about data that has been gone for a while." > > ufsbackup/ufsrestore does give you the ability to restore a single file > but perhaps not within a given range of dates, so maybe you are missing > something :) I can still read my ufsbackup tapes from over 10 years > ago, so there''s no advantage here to using gtar. > > The very first "home brew" Unix backup system I used was a collection of > perl scripts wrapped around dump/restore on SunOS4 which I was easily > able to make function with Solaris, however there was no database of > what was on each tape. > > The best in-house backup program I''ve seen ship with a Unix is HP-UX''s > fsbackup (I think that''s the name.) It combines the ease of > use/functionality of tar with a dump/restore-like back end. It''s big > win over using either of tar/dump was that it could create an index file > as the backup progressed. If I recall correctly, fsbackup had its own > proprietary format. > > I don''t see that creating a tool like that to ship with Solaris would > necessarily be something for the ZFS team to do, except maybe for the > backend that grovels the ZFS partition itself. > > > Darren > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Darren Reed <Darren.Reed at Sun.COM> wrote:> > On a completely separate tack, a few questions/points: > > > > It appears that you''re looking to something similar to the database > > backup modules (like oracle) for ZFS. That''s fine for application > > backups, but is it practical to use zfs backup for storing data on > > tape within netbackup? > > > > Netbackup stores it''s files on tape in gnu tar format. As such, it''s > > an open standard that hasn''t changed in a long time. Based on that, > > it''s reasonable that in 7-20 years that you can restore an offsite > > backup. > > > Is GNU tar''s file format an _open standard_ or open source product that > is a defacto standard?GNU tar''s file format is definitely not an _open standard_. The only documentation I am aware of, of (~95% of) the format can be found at: http://cdrecord.berlios.de/old/private/man/star.4.html or in the _star_ man pages...... GNU tar itself does not document it''s own archive format and as a result of this, the GNU tar format violates general tar archive structuring rules. This is the reason why other tar programs (except star) get out of sync (and as a result pring print "directory checksum error" messages) in many cases. The GNU tar format violates even POSIX.1-1988 and - more important - GNU tar is unable to correctly restore incremental backups made with GNU tar. It may be that the Veritas versions of GNU tar did fiy thix bug but the official GNU tar did still not fix this problem that is present since 1992 (when this feature was added to GNU tar). If you like to archive files, you should not rely on a program that does not documents it''s own format. Better use star. Star is POSIX compliant and implements working incremental backups/restores.> > zfs backup, by comparison, is a vendor-specific format (currently) > > that is in flux. I think of it similarly to ufsbackup. > > > The only benefit gtar has is that by virtue of the source being open, > the tape format is also open. This doesn''t mean the tape format won''t > change or can''t change. And to answer that challenge, I saw this: > OpenSolaris!See above. If you like to have a OS/FS independent replacement for ufsdump, the best choice is star. Star documents it''s archive format, makes sure that the archive format is compatible to later versions and implements working multi volume support. Note that star uses the same basic idea for creating incremental backups as ufsdump. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
On Mar 29, 2006, at 17:59, Joerg Schilling wrote:> If you like to have a OS/FS independent replacement for ufsdump, > the best > choice is star. Star documents it''s archive format, makes sure that > the archive > format is compatible to later versions and implements working multi > volume > support. Note that star uses the same basic idea for creating > incremental > backups as ufsdump.For the few people that may not know this (and in the interest of "full disclosure"), Joerg Schilling is the author of star. :)
Joe Little wrote:> Some would argue (and academia may very well) that amanda > (www.amanda.org) is the gold standard of open source backup tools. > Although it can use the FS dump/restore for xfs/ext3/ufs/etc. many > people use gtar (version 1.15) as its platform agnostic backup client > of choice. Furthermore, and maybe I am wrong about zfsbackup, but one > tends to need to split up volumes into manageable chunks (say 400GB > each) that may not map well to a multi terabyte ZFS filesystem. The > logical separation vs the disk sizing one uses for backups does not > always map to one backup bin to one filesystem. Gnu tar provides this. > > For our organization, we will not be migrating away from amanda, and > our use of ZFS is predicated on using tar. So, at least addressing how > such third party utilities interface with ZFS is critical for a lot of > organizations, especially heterogeneous ones.My understanding is that the only problem between tar and ZFS is that tar won''t understand ZFS ACLs. If you don''t use ACLs you should be fine and not have any material differences between amanda using tar to backup ZFS vs UFS.> > On 3/29/06, Darren Reed <Darren.Reed at sun.com> wrote: >> Gregory Shaw wrote: >> >>> On a completely separate tack, a few questions/points: >>> >>> It appears that you''re looking to something similar to the database >>> backup modules (like oracle) for ZFS. That''s fine for application >>> backups, but is it practical to use zfs backup for storing data on >>> tape within netbackup? >>> >>> Netbackup stores it''s files on tape in gnu tar format. As such, it''s >>> an open standard that hasn''t changed in a long time. Based on that, >>> it''s reasonable that in 7-20 years that you can restore an offsite >>> backup. >> >> Is GNU tar''s file format an _open standard_ or open source product that >> is a defacto standard? >> >>> zfs backup, by comparison, is a vendor-specific format (currently) >>> that is in flux. I think of it similarly to ufsbackup. >> >> The only benefit gtar has is that by virtue of the source being open, >> the tape format is also open. This doesn''t mean the tape format won''t >> change or can''t change. And to answer that challenge, I saw this: >> OpenSolaris! >> >>> If you''ve got the ability to snapshot, why would you need to do a zfs >>> backup at that point? Why not just backup the snapshot with the >>> backup tool (netbackup in this case)? You wouldn''t need to store the >>> backup separately at that point. >>> >>> Perhaps I''m missing something in the below? >> >> In another email you say: >> >> "Also, if you use zfs backup instead of the backup product, you''ve lost >> the ability to restore a single file or search for a file in a range of >> dates. That''s a huge loss in flexibility, especially when talking >> about data that has been gone for a while." >> >> ufsbackup/ufsrestore does give you the ability to restore a single file >> but perhaps not within a given range of dates, so maybe you are missing >> something :) I can still read my ufsbackup tapes from over 10 years >> ago, so there''s no advantage here to using gtar. >> >> The very first "home brew" Unix backup system I used was a collection of >> perl scripts wrapped around dump/restore on SunOS4 which I was easily >> able to make function with Solaris, however there was no database of >> what was on each tape. >> >> The best in-house backup program I''ve seen ship with a Unix is HP-UX''s >> fsbackup (I think that''s the name.) It combines the ease of >> use/functionality of tar with a dump/restore-like back end. It''s big >> win over using either of tar/dump was that it could create an index file >> as the backup progressed. If I recall correctly, fsbackup had its own >> proprietary format. >> >> I don''t see that creating a tool like that to ship with Solaris would >> necessarily be something for the ZFS team to do, except maybe for the >> backend that grovels the ZFS partition itself. >> >> >> Darren >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>-- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
David Magda wrote:> > On Mar 29, 2006, at 17:59, Joerg Schilling wrote: > >> If you like to have a OS/FS independent replacement for ufsdump, the >> best >> choice is star. Star documents it''s archive format, makes sure that >> the archive >> format is compatible to later versions and implements working multi >> volume >> support. Note that star uses the same basic idea for creating >> incremental >> backups as ufsdump. > > > For the few people that may not know this (and in the interest of > "full disclosure"), Joerg Schilling is the author of star. :)For this kind of tool, my benchmark in its utility is that of fbackup from HP-UX (not fsbackup). http://www.cs.bgu.ac.il/~arik/usail/man/hpux/fbackup.1.html Darren
william.hathaway at versatile.com said:> My understanding is that the only problem between tar and ZFS is that tar > won''t understand ZFS ACLs. If you don''t use ACLs you should be fine and not > have any material differences between amanda using tar to backup ZFS vs UFS.Hmm, perhaps it''s obvious to others, but I''ll ask anyway: What would be wrong/hard/unwise about making "zfs backup" produce output in the format of POSIX tar, the way a number of other backup utilities do? Other than tar and gtar, I also know of various flavors of pax, NetBackup, NetWorker, SAM-FS archiver (another "star"), Reliaty Backup (a.k.a. qtar/rbtar, now owned by Oracle as "Oracle Backup"), and so on. Some of these have "extensions" which cannot be extracted by "regular tar", but the data itself can still be extracted by non-extended tar''s. It would seem to be a great format to offer, at the least as an option. Regards, Marion
My concern was the stated poor performance of gtar on zfs vs gtar on ufs. Looks like there is perchance a bug there. On 3/29/06, William D. Hathaway <william.hathaway at versatile.com> wrote:> Joe Little wrote: > > Some would argue (and academia may very well) that amanda > > (www.amanda.org) is the gold standard of open source backup tools. > > Although it can use the FS dump/restore for xfs/ext3/ufs/etc. many > > people use gtar (version 1.15) as its platform agnostic backup client > > of choice. Furthermore, and maybe I am wrong about zfsbackup, but one > > tends to need to split up volumes into manageable chunks (say 400GB > > each) that may not map well to a multi terabyte ZFS filesystem. The > > logical separation vs the disk sizing one uses for backups does not > > always map to one backup bin to one filesystem. Gnu tar provides this. > > > > For our organization, we will not be migrating away from amanda, and > > our use of ZFS is predicated on using tar. So, at least addressing how > > such third party utilities interface with ZFS is critical for a lot of > > organizations, especially heterogeneous ones. > > My understanding is that the only problem between tar and ZFS is that > tar won''t understand ZFS ACLs. If you don''t use ACLs you should be fine > and not have any material differences between amanda using tar to backup > ZFS vs UFS. > > > > > On 3/29/06, Darren Reed <Darren.Reed at sun.com> wrote: > >> Gregory Shaw wrote: > >> > >>> On a completely separate tack, a few questions/points: > >>> > >>> It appears that you''re looking to something similar to the database > >>> backup modules (like oracle) for ZFS. That''s fine for application > >>> backups, but is it practical to use zfs backup for storing data on > >>> tape within netbackup? > >>> > >>> Netbackup stores it''s files on tape in gnu tar format. As such, it''s > >>> an open standard that hasn''t changed in a long time. Based on that, > >>> it''s reasonable that in 7-20 years that you can restore an offsite > >>> backup. > >> > >> Is GNU tar''s file format an _open standard_ or open source product that > >> is a defacto standard? > >> > >>> zfs backup, by comparison, is a vendor-specific format (currently) > >>> that is in flux. I think of it similarly to ufsbackup. > >> > >> The only benefit gtar has is that by virtue of the source being open, > >> the tape format is also open. This doesn''t mean the tape format won''t > >> change or can''t change. And to answer that challenge, I saw this: > >> OpenSolaris! > >> > >>> If you''ve got the ability to snapshot, why would you need to do a zfs > >>> backup at that point? Why not just backup the snapshot with the > >>> backup tool (netbackup in this case)? You wouldn''t need to store the > >>> backup separately at that point. > >>> > >>> Perhaps I''m missing something in the below? > >> > >> In another email you say: > >> > >> "Also, if you use zfs backup instead of the backup product, you''ve lost > >> the ability to restore a single file or search for a file in a range of > >> dates. That''s a huge loss in flexibility, especially when talking > >> about data that has been gone for a while." > >> > >> ufsbackup/ufsrestore does give you the ability to restore a single file > >> but perhaps not within a given range of dates, so maybe you are missing > >> something :) I can still read my ufsbackup tapes from over 10 years > >> ago, so there''s no advantage here to using gtar. > >> > >> The very first "home brew" Unix backup system I used was a collection of > >> perl scripts wrapped around dump/restore on SunOS4 which I was easily > >> able to make function with Solaris, however there was no database of > >> what was on each tape. > >> > >> The best in-house backup program I''ve seen ship with a Unix is HP-UX''s > >> fsbackup (I think that''s the name.) It combines the ease of > >> use/functionality of tar with a dump/restore-like back end. It''s big > >> win over using either of tar/dump was that it could create an index file > >> as the backup progressed. If I recall correctly, fsbackup had its own > >> proprietary format. > >> > >> I don''t see that creating a tool like that to ship with Solaris would > >> necessarily be something for the ZFS team to do, except maybe for the > >> backend that grovels the ZFS partition itself. > >> > >> > >> Darren > >> > >> _______________________________________________ > >> zfs-discuss mailing list > >> zfs-discuss at opensolaris.org > >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >> > > > -- > William D. Hathaway email: william.hathaway at versatile.com > Solutions Architect aim: wdhPO > Versatile, Inc. cell: 717-314-5461 >
On Wed, Mar 29, 2006 at 07:42:46PM -0800, Marion Hakanson wrote:> Hmm, perhaps it''s obvious to others, but I''ll ask anyway: What would be > wrong/hard/unwise about making "zfs backup" produce output in the format > of POSIX tar, the way a number of other backup utilities do?The problem is that the ZFS backups are done at the block level, below the POSIX layer, where we only know about objects and blocks, not files and directories. Due to the ZFS on-disk structures, such a backup is very fast, and incremental backups (using the block''s birth time) are extremely simple and fast. The backups are O(blocks changed) rather than O(files). It wouldn''t be too hard to do what you suggest by backing up the block level info then writing out a table to map object number to filename. Of course, the thing that makes this difficult is hard links. Because of that, there is not necessarily a one-to-one mapping between object number and filename. If you want to handle hard links correctly, you''re back to O(files). The frustrating part about all of this is that hard links are, by far, not the common case, yet they mess up a lot of otherwise good ideas. --Bill
William D. Hathaway wrote:> I think the best scenario would be if there was someone with the > Netbackup SDK license that would be interested in doing an open source > XBSA agent. It doesn''t have to be Veritas doing the work (although if > they did, they could charge for it separately like they do for > Flashbackup, Oracle and other agents they offer)Hi William, I''m a little confused by your reference to the XBSA api. We''re still talking about ZFS, not Informix, aren''t we? Regarding the suggestion about creating an agent or plugin for ZFS and NetBackup, I really don''t see why you''d need Veritas to do that for you - the general form of a pre-/post-backup script is well known and good examples are provided. Every customer I''ve worked with who used NetBackup had their own scripts because the ones out of the box didn''t address local management requirements. That''s not a problem with the supplied scripts! And if somebody within Sun was willing to get at least a basic script working, we''ve got a good relationship with NetBackup engineering such that I doubt it would be difficult to get such a thing integrated. (speaking as somebody who has had NetBackup RFEs integrated before now). best regards, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
"William D. Hathaway" <william.hathaway at versatile.com> wrote:> > For our organization, we will not be migrating away from amanda, and > > our use of ZFS is predicated on using tar. So, at least addressing how > > such third party utilities interface with ZFS is critical for a lot of > > organizations, especially heterogeneous ones. > > My understanding is that the only problem between tar and ZFS is that > tar won''t understand ZFS ACLs. If you don''t use ACLs you should be fine > and not have any material differences between amanda using tar to backup > ZFS vs UFS.I am not sure if you mean the right thing here...... Note that amanda does not support tar, Amanda obnly supports GNU tar and GNU tar has many pitfalls. One is not beina ablte to deal with ACLs in general. Star on the other side does support ACLs and star will support add suppot for ZFS (and NTFS) ACLs soon. Star does not yet support ZFS ACLs because the ASCII format used by libsec from Sun''s first ZFS ACL implementation was broken and did not support to restore the ZFS ACLs in all cases. The responsible person for libsec did recently send me a mail that this has been changed. When I have a bit free time (most likely before Whitsun), I will add support. BTW: I hope that Amanda will soon support star (The amanda people promised this many years ago....) as star (in contrary to GNU tar) implements a working incremental backup/restore. GNU tar does not deal with renamed directories correctly. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
"Joe Little" <jmlittle at gmail.com> wrote:> Some would argue (and academia may very well) that amanda > (www.amanda.org) is the gold standard of open source backup tools. > Although it can use the FS dump/restore for xfs/ext3/ufs/etc. many > people use gtar (version 1.15) as its platform agnostic backup client > of choice. Furthermore, and maybe I am wrong about zfsbackup, but one > tends to need to split up volumes into manageable chunks (say 400GB > each) that may not map well to a multi terabyte ZFS filesystem. The > logical separation vs the disk sizing one uses for backups does not > always map to one backup bin to one filesystem. Gnu tar provides this.Amanda does not use the multi volume features of GNU tar. Amanda uses it''s own way to split the output. In case that GNU tar would correctly support multi volume archives, this would be a problem as it is most unlikely that you will be able to manually restore a single archive. Note that if you use GNU tar''s multi volume feature, there is a 5% chance that GNU tar is unwilling to accept a multi volume continuation volume. This happens everytime when an actual disk file is _not_ split amongst two multi volume volumes or when the split happens "in the middle" of a GNU tar meta file (e.g. a long name or long link header). J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
>> >> My understanding is that the only problem between tar and ZFS is that >> tar won''t understand ZFS ACLs. If you don''t use ACLs you should >> be fine >> and not have any material differences between amanda using tar to >> backup >> ZFS vs UFS. > > I am not sure if you mean the right thing here...... > > Note that amanda does not support tar, Amanda obnly supports GNU > tar and GNU > tar has many pitfalls. One is not beina ablte to deal with ACLs in > general.That''s the same issue he mentioned. You say GNU tar has many pitfalls (this was not previously mentioned), could you please elaborate?> Star on the other side does support ACLs and star will support add > suppot for > ZFS (and NTFS) ACLs soon. Star does not yet support ZFS ACLs > because the ASCII > format used by libsec from Sun''s first ZFS ACL implementation was > broken and > did not support to restore the ZFS ACLs in all cases.Ok, so star is (basically) in the same position as GNU tar when working with ZFS currently... (barring whatever the many pitfalls you mentioned earlier are). Again, could you fill me in on those pitfalls? I don''t want to fall into one!> The responsible person for libsec did recently send me a mail that > this has > been changed. When I have a bit free time (most likely before > Whitsun), I will > add support.Cool! Can''t wait it in action. :)> BTW: I hope that Amanda will soon support star (The amanda people > promised this > many years ago....) as star (in contrary to GNU tar) implements a > working > incremental backup/restore. GNU tar does not deal with renamed > directories > correctly.What does GNU tar do wrong with renamed directories?> J?rgYou seem to be a pretty smart guy, so I''d really love your input on the above questions. The reason I''m interested, I use GNU tar profusely right now, but you mention "many pitfalls" and it''s got me worried I''ve overlooked something. I just didn''t see any details except the ACL thing, which was already mentioned by the poster you were responding to. I just want to make sure I''m not going to end up finding out the hard way I''m missing something. :) ACLs aren''t an issue to me, but the data is... I suppose what I''m getting at is, I didn''t really get any information from your post that hadn''t been said in the post you replied to, and I''d really appreciate it if you elaborated more clearly why I should migrate to "star" from GNU tar. :) I know you''re quite technically capable and I''m not, so I really value your input. I just need some "meat" behind the "star is better" comments to justify the migration to all of the people I deal with! Cheers, David
Darren Reed <Darren.Reed at Sun.COM> wrote:> For this kind of tool, my benchmark in its utility is that of fbackup > from HP-UX (not fsbackup). > http://www.cs.bgu.ac.il/~arik/usail/man/hpux/fbackup.1.htmlI don''t know the features if this program..... star has been designed to be able to pass the "Zwicky test". ftp://ftp.berlios.de/pub/star/testscripts/zwicky/ Star is currently the only program besides ufsdump that passed this test. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
"David J. Orman" <ormandj at corenode.com> wrote:> That''s the same issue he mentioned. You say GNU tar has many pitfalls > (this was not previously mentioned), could you please elaborate?As I am not sure what you are interestied in, I yould recommend you to first read: ftp://ftp.berlios.de/pub/star/README.otherbugs and ftp://ftp.berlios.de/pub/star/STARvsGNUTAR These files are old and outdated but they should help you to get an overview.> Ok, so star is (basically) in the same position as GNU tar when > working with ZFS currently... (barring whatever the many pitfalls youNot really true: star truely supports sparse files, GNU tar only gives a rough aproximation.> mentioned earlier are). Again, could you fill me in on those > pitfalls? I don''t want to fall into one!The biggest problem may be that GNU tar is virtually unmaintained since 1993. There are Bugs that I reported in 1993 that are still not fixed. An important bug (GNU Tar prints "Skipping to next header" while ignioring files) may have been fixed a few weeks ago, but 13 years to fix such an important bug looks bad.> What does GNU tar do wrong with renamed directories?I am sorry, I do not remember. I did run a simple test in August 2004 when I was adding support for incemental restores to star. While star (after the first code to support incemental restores withing 2 hours) did pass this test, GNU tar failed so miserably that I did not run any other test. Note that while star has true support for renamed directories by looking at inode numbners and the database: "star-symtable", GNU tar needs do recursively remove the old tree and to copy over a complete new tree from the archive. This seems to be implemented incorrectly and I remember that while looking at the problem I was not even sure whether the problem may be dealt with at all by GNU tar because GNU tar does not archive enough file meta data. Note that star archives all file meta data (only extended attribute files are currently still missing).> You seem to be a pretty smart guy, so I''d really love your input on > the above questions. The reason I''m interested, I use GNU tar > profusely right now, but you mention "many pitfalls" and it''s got me > worried I''ve overlooked something. I just didn''t see any details > except the ACL thing, which was already mentioned by the poster you > were responding to. I just want to make sure I''m not going to end up > finding out the hard way I''m missing something. :) ACLs aren''t an > issue to me, but the data is...I am using star on a dayly base since 1982 and I know that it is mature. GNU tar''s main problem is not being POSIX compliant (this change to: not being POSIX compliant by default a few monhthsd ago). This causes many problem when exchanging archives with other people that use tar (not star that kmnows about the standard deviations from GNU tar).> I suppose what I''m getting at is, I didn''t really get any information > from your post that hadn''t been said in the post you replied to, and > I''d really appreciate it if you elaborated more clearly why I should > migrate to "star" from GNU tar. :) I know you''re quite technicallyJust have a look star star. Everybody who did use star for at least 1-2 day and did read the manual page did stay with star. Star has more than twice as many features than GNU tar, star supports many more archive formats (e.g. cpio) than GNU tar. Star includes a build in "find" code that gives you many features you did dream off before ;-) Star is much faster than GNU tar... Ask for more ;-) J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
"William D. Hathaway" <william.hathaway at versatile.com> wrote:> I''ve done some timing of using zfs backup vs gtar for a case of 1M files > with an average size of 13k and I see zfs backup run about 20x faster > than gtar. My impression is that gtar has to use tons of cycles doing > all the system calls to open/read/close on each of the files and that > zfs backup has a huge advantage by just being able to internally iterate > and write out data structures.Did you try star in incremental mode? GNU tar is slow...... J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
"William D. Hathaway" <william.hathaway at versatile.com> wrote:> I spend a fair amount of time working with big messaging systems which > is why this corner case is of interest to me (although I''m sure there > are other applications that hit the "millions of files" problem). > > > I just finished a run of gtar vs zfs backup for 500k small files and it > took 2hr 39min for gtar and 7 min for zfs backup.Did you try to do the same with star? J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Joerg Schilling wrote:> "William D. Hathaway" <william.hathaway at versatile.com> wrote: > >> I spend a fair amount of time working with big messaging systems which >> is why this corner case is of interest to me (although I''m sure there >> are other applications that hit the "millions of files" problem). >> >> >> I just finished a run of gtar vs zfs backup for 500k small files and it >> took 2hr 39min for gtar and 7 min for zfs backup. > > Did you try to do the same with star? > > J?rg >I haven''t tried star for my test yet, I''ll certainly add it to the mix. (I have used it in the past, thanks for all the Solaris and Linux utilities you have provided!) While star might be the best *tar like program out there, I still have trouble thinking that any are going to come within an order of magnitude of zfs backup performance for the millions of small files test case. (Given the overhead involved in all the (stat,open,read,close) calls needed). NOTE TO THREAD: I''m not trying to bash the use of *tar/amanda/Netbackup/etc for use on ZFS. I think in most cases those are the right solution. In this *particular*[1] use case I''m trying to determine if one of the new features of ZFS offers a better solution given my requirements than existing tools. [1] Use case: Millions to tens of millions of small files averaging 10-20k All user-request based restores can be done via restoring a few files and must be requested within a few days of being deleted. Data needs to be backed up to disk so that the file system can be restored in the event of file system corruption, site disaster, or multiple media failures. Restore time is crucial -- William D. Hathaway email: william.hathaway at versatile.com Solutions Architect aim: wdhPO Versatile, Inc. cell: 717-314-5461
amanda 2.5 requires tar 1.15 which changed many things, including how it estimates backup sizing, sparse file handling, etc. I just want to make sure that we are comparing amanda w/ 1.15 (a known workable solution, with 1.15 being a requirement for amanda) vs older information that may no longer be as accurate. Also, 2.5.1 I believe will support star. On 3/30/06, Joerg Schilling <schilling at fokus.fraunhofer.de> wrote:> "David J. Orman" <ormandj at corenode.com> wrote: > > > That''s the same issue he mentioned. You say GNU tar has many pitfalls > > (this was not previously mentioned), could you please elaborate? > > As I am not sure what you are interestied in, I yould recommend you to first > read: > > ftp://ftp.berlios.de/pub/star/README.otherbugs > > and > > ftp://ftp.berlios.de/pub/star/STARvsGNUTAR > > These files are old and outdated but they should help you to get an overview. > > > > Ok, so star is (basically) in the same position as GNU tar when > > working with ZFS currently... (barring whatever the many pitfalls you > > Not really true: star truely supports sparse files, GNU tar only gives a rough > aproximation. > > > mentioned earlier are). Again, could you fill me in on those > > pitfalls? I don''t want to fall into one! > > The biggest problem may be that GNU tar is virtually unmaintained since 1993. > There are Bugs that I reported in 1993 that are still not fixed. An important > bug (GNU Tar prints "Skipping to next header" while ignioring files) may have > been fixed a few weeks ago, but 13 years to fix such an important bug looks > bad. > > > > What does GNU tar do wrong with renamed directories? > > I am sorry, I do not remember. I did run a simple test in August 2004 when I > was adding support for incemental restores to star. While star (after the > first code to support incemental restores withing 2 hours) did pass this test, > GNU tar failed so miserably that I did not run any other test. > > Note that while star has true support for renamed directories by looking at > inode numbners and the database: "star-symtable", GNU tar needs do recursively > remove the old tree and to copy over a complete new tree from the archive. > This seems to be implemented incorrectly and I remember that while looking at > the problem I was not even sure whether the problem may be dealt with at all > by GNU tar because GNU tar does not archive enough file meta data. Note that > star archives all file meta data (only extended attribute files are currently > still missing). > > > > You seem to be a pretty smart guy, so I''d really love your input on > > the above questions. The reason I''m interested, I use GNU tar > > profusely right now, but you mention "many pitfalls" and it''s got me > > worried I''ve overlooked something. I just didn''t see any details > > except the ACL thing, which was already mentioned by the poster you > > were responding to. I just want to make sure I''m not going to end up > > finding out the hard way I''m missing something. :) ACLs aren''t an > > issue to me, but the data is... > > I am using star on a dayly base since 1982 and I know that it is mature. > GNU tar''s main problem is not being POSIX compliant (this change to: > not being POSIX compliant by default a few monhthsd ago). This causes many > problem when exchanging archives with other people that use tar (not star that > kmnows about the standard deviations from GNU tar). > > > I suppose what I''m getting at is, I didn''t really get any information > > from your post that hadn''t been said in the post you replied to, and > > I''d really appreciate it if you elaborated more clearly why I should > > migrate to "star" from GNU tar. :) I know you''re quite technically > > Just have a look star star. Everybody who did use star for at least 1-2 day > and did read the manual page did stay with star. > > Star has more than twice as many features than GNU tar, star supports many > more archive formats (e.g. cpio) than GNU tar. Star includes a build in "find" > code that gives you many features you did dream off before ;-) > > Star is much faster than GNU tar... > > Ask for more ;-) > > > J?rg > > -- > EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin > js at cs.tu-berlin.de (uni) > schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ > URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily >
On Wed, 2006-03-29 at 13:54 -0800, Darren Reed wrote:> Gregory Shaw wrote: > > > On a completely separate tack, a few questions/points: > > > > It appears that you''re looking to something similar to the database > > backup modules (like oracle) for ZFS. That''s fine for application > > backups, but is it practical to use zfs backup for storing data on > > tape within netbackup? > > > > Netbackup stores it''s files on tape in gnu tar format. As such, it''s > > an open standard that hasn''t changed in a long time. Based on that, > > it''s reasonable that in 7-20 years that you can restore an offsite > > backup. > > > Is GNU tar''s file format an _open standard_ or open source product that > is a defacto standard? >An open standard. After all, it *is* gnu.> > zfs backup, by comparison, is a vendor-specific format (currently) > > that is in flux. I think of it similarly to ufsbackup. >> The only benefit gtar has is that by virtue of the source being open, > the tape format is also open. This doesn''t mean the tape format won''t > change or can''t change. And to answer that challenge, I saw this: > OpenSolaris! >Tape format won''t change? The backup vendor relies on this for future interoperability. They keep a copy of the source so that should that happen, you''re not at risk. I don''t expect it to happen, or that it could happen.> > If you''ve got the ability to snapshot, why would you need to do a zfs > > backup at that point? Why not just backup the snapshot with the > > backup tool (netbackup in this case)? You wouldn''t need to store the > > backup separately at that point. > > > > Perhaps I''m missing something in the below? > > > In another email you say: > > "Also, if you use zfs backup instead of the backup product, you''ve lost > the ability to restore a single file or search for a file in a range of > dates. That''s a huge loss in flexibility, especially when talking > about data that has been gone for a while." > > ufsbackup/ufsrestore does give you the ability to restore a single file > but perhaps not within a given range of dates, so maybe you are missing > something :) I can still read my ufsbackup tapes from over 10 years > ago, so there''s no advantage here to using gtar. > > The very first "home brew" Unix backup system I used was a collection of > perl scripts wrapped around dump/restore on SunOS4 which I was easily > able to make function with Solaris, however there was no database of > what was on each tape. > > The best in-house backup program I''ve seen ship with a Unix is HP-UX''s > fsbackup (I think that''s the name.) It combines the ease of > use/functionality of tar with a dump/restore-like back end. It''s big > win over using either of tar/dump was that it could create an index file > as the backup progressed. If I recall correctly, fsbackup had its own > proprietary format. > > I don''t see that creating a tool like that to ship with Solaris would > necessarily be something for the ZFS team to do, except maybe for the > backend that grovels the ZFS partition itself. > > > Darren >
Gregory Shaw <greg.shaw at sun.com> wrote:> > Is GNU tar''s file format an _open standard_ or open source product that > > is a defacto standard? > > > > An open standard. After all, it *is* gnu.Did you forget a smiley? I would at least expect some kind of documentation to make a standard. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Joe Little wrote:>My concern was the stated poor performance of gtar on zfs vs gtar on >ufs. Looks like there is perchance a bug there. > > >What build did you see a negative performance difference between zfs and ufs for *tar? This was addressed in build 35 with: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6350001 eric
Perhaps this has already been addressed. I jumped into the conversation when I saw the difference between tar and zfsdump and assumed that this was still not resolved. On 3/30/06, eric kustarz <eric.kustarz at sun.com> wrote:> Joe Little wrote: > > >My concern was the stated poor performance of gtar on zfs vs gtar on > >ufs. Looks like there is perchance a bug there. > > > > > > > What build did you see a negative performance difference between zfs and > ufs for *tar? This was addressed in build 35 with: > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6350001 > > eric >
Joe Little wrote:>Perhaps this has already been addressed. I jumped into the >conversation when I saw the difference between tar and zfsdump and >assumed that this was still not resolved. > > > >ah, cool - yeah, the performance difference mentioned in this thread was *not* between zfs and ufs, it was between using tar or zfs backup. tar is going to be slower since it operates on all files (not blocks changed) and has to do several syscalls per-file. eric>On 3/30/06, eric kustarz <eric.kustarz at sun.com> wrote: > > >>Joe Little wrote: >> >> >> >>>My concern was the stated poor performance of gtar on zfs vs gtar on >>>ufs. Looks like there is perchance a bug there. >>> >>> >>> >>> >>> >>What build did you see a negative performance difference between zfs and >>ufs for *tar? This was addressed in build 35 with: >>http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6350001 >> >>eric >> >> >>
On Thu, Mar 30, 2006 at 02:20:38PM -0800, eric kustarz wrote:> ah, cool - yeah, the performance difference mentioned in this thread was > *not* between zfs and ufs, it was between using tar or zfs backup. tar > is going to be slower since it operates on all files (not blocks > changed) and has to do several syscalls per-file.Eric, I haven''t been following this thread very closely, but is there a way to perform incremental backups with zfs? perhaps using snapshots or clones? "zfs backup" is great but I can''t see a way to do any kind of incremental backups with it. perhaps I''ve just missed something obvious? grant.
grant beattie wrote:> On Thu, Mar 30, 2006 at 02:20:38PM -0800, eric kustarz wrote: > >> ah, cool - yeah, the performance difference mentioned in this thread was >> *not* between zfs and ufs, it was between using tar or zfs backup. tar >> is going to be slower since it operates on all files (not blocks >> changed) and has to do several syscalls per-file. > > Eric, > > I haven''t been following this thread very closely, but is there a way > to perform incremental backups with zfs? perhaps using snapshots or > clones? > > "zfs backup" is great but I can''t see a way to do any kind of > incremental backups with it. perhaps I''ve just missed something > obvious? >zfs backup -i <old snapshot> <new snapshot>
On Thu, 2006-03-30 at 19:11, grant beattie wrote:> "zfs backup" is great but I can''t see a way to do any kind of > incremental backups with it. perhaps I''ve just missed something > obvious?Give it two snapshots of the same filesystem and it will back up the delta between the two. from the man page: zfs backup [-i snapshot] snapshot2 Creates a backup of snapshot2. The backup is written to standard output, which can be redirected to a file or to a different machine (for example, using ssh(1)). By default, a full backup is performed. -i snapshot Perform an incremental backup from snapshot to snapshot2. - Bill
On Thu, Mar 30, 2006 at 07:44:21PM -0500, Bill Sommerfeld wrote:> On Thu, 2006-03-30 at 19:11, grant beattie wrote: > > "zfs backup" is great but I can''t see a way to do any kind of > > incremental backups with it. perhaps I''ve just missed something > > obvious? > > Give it two snapshots of the same filesystem and it will back up the > delta between the two.oops, I totally missed that. knew it would probably be something simple. :) grant.
Here you go: http://www.gnu.org/software/tar/manual/html_node/Standard.html#Standard On Thu, 2006-03-30 at 21:32 +0200, Joerg Schilling wrote:> Gregory Shaw <greg.shaw at sun.com> wrote: > > > > Is GNU tar''s file format an _open standard_ or open source product that > > > is a defacto standard? > > > > > > > An open standard. After all, it *is* gnu. > > Did you forget a smiley? > > I would at least expect some kind of documentation to make a standard. > > J?rg >
grant beattie <grant at grunta.com> wrote:> I haven''t been following this thread very closely, but is there a way > to perform incremental backups with zfs? perhaps using snapshots or > clones?You definitely should be able to do this with "star". If you like to backup ZFS ACLs in this case, you would need to wait a few weeks. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Gregory Shaw <greg.shaw at sun.com> wrote:> Here you go: > > http://www.gnu.org/software/tar/manual/html_node/Standard.html#StandardWell, nice to see that they did finally _start_ to add documentation. This must have happened recently after they found the star docs. Note that this still cannot be called a documentation as it e.g. does not imclude a description of the way how GNU tar violates the TAR structuring rules when sparse files are archived. In other words: the file does not include more information as could be found in gnutar/tar.h which is not sufficient to reimplement the format. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily