zfs discuss - Jan 2010 - zfs send/receive as backup - reliability?

Hi,

I am considering building a modest sized storage system with zfs. Some of the
data on this is quite valuable, some small subset to be backed up
"forever", and I am evaluating back-up options with that in mind.

My understanding is that zfs send approximately captures the copy-on-write file
system block-level dump, and zfs receive plays it back to rebuild the file
system, and this can be used among other things for back-ups. I call the dump
"stream" below.

How reliable is this? I don''t mind the fact I would have to replay
entire file system instead of individual files. My concern is that for whatever
reason I''d lose ability to play the stream back, and would not be able
to restore possibly years from now.

Is the format documented? Is it possible to interpret the data with independent
tools, like it''s possible with tar/pax/cpio archives? Even if no such
tool exists now, could I for example write a user space tool using currently
existing open source zfs user space library that was able to extract useful
information from the data stream? I realise this could be somewhat complex,
especially incremental dumps - but just how hard?

How exactly does the stream have to match the file system to restore? My
assumption zfs requires an exact match: you can only restore at exact point you
backed up from. (Fine by me, just need to know.)

Follow-up: does one corrupted incremental back-up set invalidate all incremental
back-up sets until the next full back-up point (or higher-level incremental
point)?

Assuming the zfs send data stream hasn''t been corrupted, have there
been instances where it''s not possible to restore file system by
playing it back via zfs receive?

Have there been cases where some bug has caused "zfs send" data become
corrupted so that restoration is no longer possible? (Either zfs on-disk file
system bug, or something in zfs send not doing the right thing.)

Is it possible to make dry-run restoration attempt at back-up time to verify the
restoration would succeed?

Would you recommend zfs send/receive as a back-up strategy for highly valuable
data at this point in time? Ignoring the individual file vs. whole file system
aspect, how reliable you rank it compared to tar/pax?

Regards,
Lassi

> I am considering building a modest sized storage system with zfs. Some
> of the data on this is quite valuable, some small subset to be backed
> up "forever", and I am evaluating back-up options with that in
mind.
You don''t need to store the "zfs send" data stream on your
backup media.
This would be annoying for the reasons mentioned - some risk of being able
to restore in future (although that''s a pretty small risk) and
inability to
restore with any granularity, i.e. you have to restore the whole FS if you
restore anything at all.

A better approach would be "zfs send" and pipe directly to "zfs
receive" on
the external media.  This way, in the future, anything which can read ZFS
can read the backup media, and you have granularity to restore either the
whole FS, or individual things inside there.

Plus, the only way to guarantee the integrity of a "zfs send" data
stream is
to perform a "zfs receive" on that data stream.  So by performing a
successful receive, you''ve guaranteed the datastream is not corrupt. 
Yet.

Edward Ned Harvey <solaris at nedharvey.com> wrote:
> > I am considering building a modest sized storage system with zfs. Some
> > of the data on this is quite valuable, some small subset to be backed
> > up "forever", and I am evaluating back-up options with that
in mind.
>
> You don''t need to store the "zfs send" data stream on
your backup media.
NO, zfs send is not a backup.
>From a backup, you could restore individual files.
J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

>
>
>
> NO, zfs send is not a backup.
>
> From a backup, you could restore individual files.
>
> J?rg
>
>
I disagree.

It is a backup.  It''s just not "an enterprise backup
solution"
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100116/bed02a1f/attachment.html>

On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote:
>> I am considering building a modest sized storage system with zfs.  
>> Some
>> of the data on this is quite valuable, some small subset to be backed
>> up "forever", and I am evaluating back-up options with that
in mind.
>
> You don''t need to store the "zfs send" data stream on
your backup
> media.
> This would be annoying for the reasons mentioned - some risk of  
> being able
> to restore in future (although that''s a pretty small risk) and  
> inability to
> restore with any granularity, i.e. you have to restore the whole FS  
> if you
> restore anything at all.
>
> A better approach would be "zfs send" and pipe directly to
"zfs
> receive" on
> the external media.  This way, in the future, anything which can  
> read ZFS
> can read the backup media, and you have granularity to restore  
> either the
> whole FS, or individual things inside there.
There have also been comments about the extreme fragility of the data  
stream compared to other archive formats. In general it is strongly  
discouraged for these purposes.

--Toby

>
> Plus, the only way to guarantee the integrity of a "zfs send"
data
> stream is
> to perform a "zfs receive" on that data stream.  So by performing
a
> successful receive, you''ve guaranteed the datastream is not  
> corrupt.  Yet.
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain <toby at telegraphics.com.au>
wrote:
> On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote:
>
>>> I am considering building a modest sized storage system with zfs.
Some
>>> of the data on this is quite valuable, some small subset to be
backed
>>> up "forever", and I am evaluating back-up options with
that in mind.
>>
>> You don''t need to store the "zfs send" data stream
on your backup media.
>> This would be annoying for the reasons mentioned - some risk of being
able
>> to restore in future (although that''s a pretty small risk) and
inability
>> to
>> restore with any granularity, i.e. you have to restore the whole FS if
you
>> restore anything at all.
>>
>> A better approach would be "zfs send" and pipe directly to
"zfs receive"
>> on
>> the external media. ?This way, in the future, anything which can read
ZFS
>> can read the backup media, and you have granularity to restore either
the
>> whole FS, or individual things inside there.
>
> There have also been comments about the extreme fragility of the data
stream
> compared to other archive formats. In general it is strongly discouraged
for
> these purposes.
>
Yet it is used in ZFS flash archives on Solaris 10 and are slated for
use in the successor to flash archives.  This initial proposal seems
to imply using the same mechanism for a system image backup (instead
of just system provisioning).

http://mail.opensolaris.org/pipermail/caiman-discuss/2010-January/015909.html

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On 16-Jan-10, at 6:51 PM, Mike Gerdts wrote:
> On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain  
> <toby at telegraphics.com.au> wrote:
>> On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote:
>>
>>>> I am considering building a modest sized storage system with  
>>>> zfs. Some
>>>> of the data on this is quite valuable, some small subset to be
>>>> backed
>>>> up "forever", and I am evaluating back-up options
with that in
>>>> mind.
>>>
>>> You don''t need to store the "zfs send" data
stream on your backup
>>> media.
>>> This would be annoying for the reasons mentioned - some risk of  
>>> being able
>>> to restore in future (although that''s a pretty small risk)
and
>>> inability
>>> to
>>> restore with any granularity, i.e. you have to restore the whole  
>>> FS if you
>>> restore anything at all.
>>>
>>> A better approach would be "zfs send" and pipe directly
to "zfs
>>> receive"
>>> on
>>> the external media.  This way, in the future, anything which can  
>>> read ZFS
>>> can read the backup media, and you have granularity to restore  
>>> either the
>>> whole FS, or individual things inside there.
>>
>> There have also been comments about the extreme fragility of the  
>> data stream
>> compared to other archive formats. In general it is strongly  
>> discouraged for
>> these purposes.
>>
>
> Yet it is used in ZFS flash archives on Solaris 10
I can see the temptation, but isn''t it a bit under-designed? I think  
Mr Nordin might have ranted about this in the past...

--Toby

> and are slated for
> use in the successor to flash archives.  This initial proposal seems
> to imply using the same mechanism for a system image backup (instead
> of just system provisioning).
>
> http://mail.opensolaris.org/pipermail/caiman-discuss/2010-January/ 
> 015909.html
>
> -- 
> Mike Gerdts
> http://mgerdts.blogspot.com/

> NO, zfs send is not a backup.
Understood, but perhaps you didn''t read my whole message.  Here, I will
spell out the whole discussion:

If you "zfs send > somefile" it is well understood there are two
big
problems with this method of backup.  #1 If a single bit error is introduced
into the file, then the whole data stream is corrupt.  #2 If you want to
restore just a subset of the filesystem, you cannot.  The only option
available is to restore the whole filesystem.

Instead, it is far preferable to "zfs send | zfs receive"  ...  That
is,
receive the data stream on external media as soon as you send it.  By
receiving the data stream onto external media, instead of just saving the
datastream as a file on external media ... You solve both of the above
problems.  Obviously this is only possible with external disks, and not
possible with tapes.

Toby Thain <toby at telegraphics.com.au> wrote:
> > Yet it is used in ZFS flash archives on Solaris 10
>
> I can see the temptation, but isn''t it a bit under-designed? I
think
> Mr Nordin might have ranted about this in the past...
Isn''t flash cpio based and thus not prepared for the future? Cpio coes
not
support sparse files and is unable to archive files > 8 GB.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Edward Ned Harvey <solaris at nedharvey.com> wrote:
> > NO, zfs send is not a backup.
>
> Understood, but perhaps you didn''t read my whole message.  Here, I
will
> spell out the whole discussion:
...
> Instead, it is far preferable to "zfs send | zfs receive"  ... 
That is,
> receive the data stream on external media as soon as you send it.  By
> receiving the data stream onto external media, instead of just saving the
> datastream as a file on external media ... You solve both of the above
> problems.  Obviously this is only possible with external disks, and not
> possible with tapes.
Here is the big difference. For a professional backup people still typically
use tapes although tapes have become expensive.

I still believe that a set of compressed incremental star archives give you 
more features.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

>
> Cpio coes not
> support sparse files and is unable to archive files > 8 GB.
>
> J?rg
>
>
 I found this out the hard way last time i used it.  I was backing up all my
data from one system to another using cpio and i had a bunch of movies over
8GB (720p and 1080p mkv files)

none of them worked.  I never use cpio anymore.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100117/08b5b584/attachment.html>

Thomas Burgess <wonslung at gmail.com> wrote:
>  I found this out the hard way last time i used it.  I was backing up all
my
> data from one system to another using cpio and i had a bunch of movies over
> 8GB (720p and 1080p mkv files)
>
> none of them worked.  I never use cpio anymore.
What dou you use instead?

BTW: I recommend "star" and to use either the H=exustar or -dump
option.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Sun, Jan 17, 2010 at 05:31:39AM -0500, Edward Ned Harvey
wrote:
> Instead, it is far preferable to "zfs send | zfs receive"  ... 
That is,
> receive the data stream on external media as soon as you send it. 
Agree 100% - but..

.. it''s hard to beat the convenience of a "backup file"
format, for
all sorts of reasons, including media handling, integration with other
services, and network convenience. 

Consider then, using a zpool-in-a-file as the file format, rather than
zfs send streams.

Make a backup pool out of N conveniently-sized files, sparse to start
with if it suits you.  Go ahead and make it a raidz[23] pool, in case
some of your backup media goes bad. zfs recv your backups into this
pool.  zfs export the pool, to quiesce the file contents. 

If the files are themselves in zfs, snapshot that filesystem now for
good measure; then you can immediately bring your backup pool online
again, as well as have local reference copies of old backups, as
storage space allows. 

Send these files to whatever backup system/service you want to use.  
Handle them like any other large archive file format.

If you choose file sizes well, you can burn them to dvd''s or write
them to tapes.  Multiple smaller files (rather than one file per
medium) can be easier to handle, but its best to make multiple raidz
vdevs and arrange a file from each vdev per medium (so you can still
recover from a lost dvd/tape). 

If you have a non-zfs remote storage server, rsync works well to
update the remote copies incrementally after each backup cycle (and to
bring back older versions later if needed).  Lots of cloud backup
providers exist now that can do similar incremental replication.

gpg sign and encrypt the files before sending, if you need to. Some
day soon, zfs crypto will allow backups encrypted within the pool,
without defeating incremental replication of the files (as gpg will).

Another option is to mount these files directly on whatever generic NAS
devices you want to hold the backups, and import the pool from there.
I''d be wary, but if you were to consider that, fortunately
there''s a
good testing tool (zfs scrub) to help you be sure the NAS service is
reliable.  

You do need all (or at least most) of your files available in order to
do a restore.  Given that you should be preferring to use local
snapshots for small recovery jobs anyway, that''s not really a burden
for a full recovery.  At this point, you get back all the snapshots
that were in the backup pool at the time it was saved.

The zfs send stream format used to not be committed, though it appears
this has recently changed.  It still has all the other drawbacks
previously noted. The zpool format is committed and does have a tested
and supported backwards compatibility/upgrade path. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100118/5c2d27d3/attachment.bin>

>
>
>
> What dou you use instead?
>
*
*
**tar cvf - /some/dir | (cd /some/other/dir; tar xf -)

>
> BTW: I recommend "star" and to use either the H=exustar or -dump
option.
>
> J?rg
>
>
i will have to check it out.  I recently migrated to opensolaris from
FreeBSD and i have a LOT to learn.  I am really enjoying Opensolaris so far.


so star is better than tar?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100117/ed478fba/attachment.html>

> I still believe that a set of compressed incremental star archives give
> you
> more features.
Big difference there is that in order to create an incremental star archive,
star has to walk the whole filesystem or folder that''s getting backed
up,
and do a "stat" on every file to see which files have changed since
the last
backup run.  If you have a large filesystem, that can take a very long time.

I recently switched to ZFS specifically for this reason.  Previously, I was
doing a nightly rsync on 1Tb of data.  It required 10 hrs every night to
run, and copy typically a few hundred megs that had changed that day.  Now,
I run incremental zfs send & receive, and it completes typically in a minute
or two.

> Consider then, using a zpool-in-a-file as the file format, rather than
> zfs send streams.
That''s a pretty cool idea.  Then you''ve still got the entire
zfs volume
inside of a file, but you''re able to mount and extract individual files
if
you want, and you''re able to pipe your zfs send directly to zfs
receive, and
basically you get all the benefits of zfs send & receive, but also get all
the benefits of being able to treat your backup like a file or a set of
files.

Pretty cool.

YMMV. At a recent LOSUG meeting we were told of a case where rsync was  
faster than an incremental zfs send/recv. But I think that was for a  
mail server with many tiny files (i.e. changed blocks are very easy to  
find in files with very few blocks).

However, I don''t see why further ZFS perfomance work couldn''t
close
that gap, since rsync will always need to compare directories and  
timestamps.

Phil

On 18 Jan 2010, at 08:07, Edward Ned Harvey <solaris at nedharvey.com>  
wrote:
>> I still believe that a set of compressed incremental star archives  
>> give
>> you
>> more features.
>
> Big difference there is that in order to create an incremental star  
> archive,
> star has to walk the whole filesystem or folder that''s getting  
> backed up,
> and do a "stat" on every file to see which files have changed
since
> the last
> backup run.  If you have a large filesystem, that can take a very  
> long time.
>
> I recently switched to ZFS specifically for this reason.   
> Previously, I was
> doing a nightly rsync on 1Tb of data.  It required 10 hrs every  
> night to
> run, and copy typically a few hundred megs that had changed that  
> day.  Now,
> I run incremental zfs send & receive, and it completes typically in  
> a minute
> or two.
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Mon, Jan 18, 2010 at 3:59 AM, Phil Harman <phil.harman at gmail.com>
wrote:
> YMMV. At a recent LOSUG meeting we were told of a case where rsync was
> faster than an incremental zfs send/recv. But I think that was for a mail
> server with many tiny files (i.e. changed blocks are very easy to find in
> files with very few blocks).
>
> However, I don''t see why further ZFS perfomance work
couldn''t close that
> gap, since rsync will always need to compare directories and timestamps.
>
> Phil
>
> The best info i''ve read on this was on this blog:
http://richardelling.blogspot.com/2009/01/parallel-zfs-sendreceive.html

>
> On 18 Jan 2010, at 08:07, Edward Ned Harvey <solaris at
nedharvey.com> wrote:
>
>  I still believe that a set of compressed incremental star archives give
>>> you
>>> more features.
>>>
>>
>> Big difference there is that in order to create an incremental star
>> archive,
>> star has to walk the whole filesystem or folder that''s getting
backed up,
>> and do a "stat" on every file to see which files have changed
since the
>> last
>> backup run.  If you have a large filesystem, that can take a very long
>> time.
>>
>> I recently switched to ZFS specifically for this reason.  Previously, I
>> was
>> doing a nightly rsync on 1Tb of data.  It required 10 hrs every night
to
>> run, and copy typically a few hundred megs that had changed that day.
>>  Now,
>> I run incremental zfs send & receive, and it completes typically in
a
>> minute
>> or two.
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100118/f7e26de5/attachment.html>

On 18/01/2010 08:59, Phil Harman wrote:
> YMMV. At a recent LOSUG meeting we were told of a case where rsync was 
> faster than an incremental zfs send/recv. But I think that was for a 
> mail server with many tiny files (i.e. changed blocks are very easy to 
> find in files with very few blocks).
>
>
After changes around build 114 (iirc) this shouldn''t be the case
anymore
znd an incremental zfs send should always be able to at least match the 
performance of incremental rsync. Now with lots of files where only a 
small subset of them changes incremental zfs send should be much faster 
and that''s my observation.

-- 
Robert Milkowski
http://milek.blogspot.com

or you might do something like: 
http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

however in your case if all your clients are running zfs only 
filesystems then relaying just on zfs send|recv might be a good idea.


-- 
Robert Milkowski
http://milek.blogspot.com

Hi,
>> I am considering building a modest sized storage system with zfs. Some
>> of the data on this is quite valuable, some small subset to be backed
>> up "forever", and I am evaluating back-up options with that
in mind.
> 
> You don''t need to store the "zfs send" data stream on
your backup media.
> This would be annoying for the reasons mentioned - some risk of being able
> to restore in future (although that''s a pretty small risk) and
inability to
> restore with any granularity, i.e. you have to restore the whole FS if you
> restore anything at all.
> 
> A better approach would be "zfs send" and pipe directly to
"zfs receive" on
> the external media.  This way, in the future, anything which can read ZFS
> can read the backup media, and you have granularity to restore either the
> whole FS, or individual things inside there.
> 
> Plus, the only way to guarantee the integrity of a "zfs send"
data stream is
> to perform a "zfs receive" on that data stream.  So by performing
a
> successful receive, you''ve guaranteed the datastream is not
corrupt.  Yet.
Thanks for your feedback!

My plan is to have near-line back-up on zfs on another physically independent
media, much as you describe. Then another copy off-site on separate system
(disk), and third copy on WORM-like media in "chunks" somewhere else.
I am really looking to have the latter two sliced to fit non-disk media (tape,
optical disks, ...), or external storage not really usable as a filesystem (S3,
web hosting, ...).

The off-site disk is not in zfs-capable system, unless I set up a virtual
machine; but it won''t have enough physical disks for raidz2 anyway. I
am primarily looking for something I can store encrypted in dvd- or tape-sized
slices on at least two physically separate media. This backup would be used when
at least two other media have already been lost, so while convenience is a plus
I really desire reliability and longevity. (Yes I know tapes and DVDs die too.)

I appreciate the comments by others on zfs send not allowing individual file
restore, but as I wrote before this is not very important to me, at least not as
important than my other questions. Apropos, is anyone able to respond to those?
(Is format documented, independent tools, bugs known to have affected
send/restore, would you recommend over tar/pax in real life, etc.)

I am very interested in what people have actually done and experienced in real
life, including disaster recovery from multiple media failure.

Regards,
Lassi

Hi,
> Here is the big difference. For a professional backup people still
typically
> use tapes although tapes have become expensive.
> 
> I still believe that a set of compressed incremental star archives give you
> more features.
Thanks for your comments!

I think I roughly understand the feature trade-offs, and have some experience
with back-up solutions ranging from simple to enterprise.

I guess what I am after is, for data which really matters to its owners and
which they actually had to recover, did people use tar/pax archives (~ file
level standard archive format), dump/restore (~ semi-standard format based on
files/inodes) or zfs send/receive (~ file system block level dump), or something
else, and why? (Regardless of how these are implemented, hobby scripts or
enterprise tools, how they dealt with possible media failure issues, etc.)

Other than the file system vs. file restore, is there any concern in doing the
block level thing? "Concerns" as in "mind being in the line of
fire if it fails?" :-)

Regards,
Lassi

Hi,
> .. it''s hard to beat the convenience of a "backup file"
format, for
> all sorts of reasons, including media handling, integration with other
> services, and network convenience. 
Yes.
> Consider then, using a zpool-in-a-file as the file format, rather than
> zfs send streams.
This is an interesting suggestion :-)

Did I understand you correctly that once a slice is written, zfs won''t
rewrite it? In other words, I can have an infinitely growing pile of these
slices, and once zfs fills one file up (or one "raidz2" set of files),
I flush it to tape/optical disk/whatever, and zfs won''t change it
"ever after"? When I need more space, I just add more slices, but old
slices are effectively read-only?

Or did I misunderstand how you meant it work? It sounded very interesting but my
understanding on zfs is currently limited :-)

And perhaps most importantly: has anyone actually done this for their back-ups
and has success stories on restore after media failure?

Regards,
Lassi

>>>>> "mg" == Mike Gerdts <mgerdts at gmail.com>
writes:
>>>>> "tt" == Toby Thain <toby at
telegraphics.com.au> writes:
>>>>> "tb" == Thomas Burgess <wonslung at
gmail.com> writes:
    mg> Yet it is used in ZFS flash archives on Solaris 10 and are
    mg> slated for use in the successor to flash archives.

in FLAR, ``if a single bit''s flipped, the whole archive and any
incrementals descended from it are toast'''' is a desireable
characteristic.  FLAR is a replication tool, just like ''zfs send | zfs
receive'' together are, and the sensitivity''s desireable
because you
want a deterministicaly exact replica.

If zpools themselves had the characteristic, ``single bit flipped,
entire pool lost,'''' few would be happy with that.  In fact
it''s part
of the ZFS kool-aid cocktail that bitflips are detected and reported,
and that critical metadata is written several times far apart on the
pool so the damage of a single bitflip, even on an unredundant pool,
is limited.  Other things that aren''t ``enterprise backup
solutions'''',
like tarballs, are also resilient to single bit flips.  so, why
tolerate this fragility from a backup format?  It''s not appropriate.

The other problems are lack of a verification tool that doesn''t
involve ponderous amounts of kernel code revision-bound to hardware
drivers and such that may one day become unobtainable, the possibility
of painting yourself into a corner (ex. if you are backing up a 17TB
filesystem, you must provide 17TB of restore space or else there is no
way to access a 2kB file), and the stream format which is
intentionally tied to the relatively often-changing zfs version so
that it can make exact copies but at the cost of constraining the
restore environment which may be separated from the backup environment
by seven years and/or in the midst of a disaster in a way that
tarballs and cpios and so on don''t constrain.

Another problem is that the snv_112 man page says this:

-----8<-----
         The format of the stream is evolving. No backwards  com-
         patibility is guaranteed. You may not be able to receive
         your streams on future versions of ZFS.
-----8<-----

I think the new man page says something more generous.  The use case
here is, we should be able to:

   old solaris                    new solaris           newer solaris

   zfs send  ----------------->   zfs recv
                                      \
                                       \_____ zpool upgrade -->
                                          (not ''zfs upgrade'')


                                                     -- zfs send
                                                    |
                                                     -> zfs recv


   zfs recv  <----------------------------------------- zfs send


That is, the stream format should be totally deterministic given the
zfs version, and not depend on the zpool version nor the
kernel/userland version.  This way a single backup pool can hold the
backups of several different versions of solaris.  The alternative to
this level of guarantee is for your archival backup to involve a ball
of executable code like a LiveCD or a .VDI, and that''s just not on.
Seven to ten years, people---there are laws TODAY that require
archiving WORM media that long, and being able to read it, which means
no rewriting for a decade.  That''s a ``hard
requirement'''' as
Christopher says, _today,_ never mind what''s desireable or useful.
I''m not sure the new man page makes a promise quite that big as
accomodating the above chart, but IIRC it''s much better than the old
one.


anyway, zfs send and receive are very good replication tools, and
replication can be used for backup (by replicating into another zpool
*NOT* by storing the stream modulo the caveat above) or for system
install (by recv''ing multiple times like FLAR).

If you choose to store zfs send streams as backups, the least you can
do is warn those you advise that zfs send streams are different from
other kidns of backup stream, because sysadmin experience in how to
write backups is old and hard-won, and these tools diverge from it.
they diverge usefully---I''m not putting them down---I''m saying
*don''t
use them that way* unless you are able to think through
$subtleproblems, after which you''ll probably not want to use them that
way.

    tt> I can see the temptation, but isn''t it a bit under-designed?
I
    tt> think Mr Nordin might have ranted about this in the past...

no, I think it''s great for FLAR.  single-bit-self-destruct is exactly
what''s wanted for FLAR.

for those case where you could somehow magically capture and replay an
rsync stream it''s great.  It''s not a dumb design because IMHO
a single
format probably can''t do well for both replication and backup, but
it''s not a backup format, enterprise or otherwise.  What escalates my
objection into a rant is the way ``enterprise backup
solution'''' gets
tossed around as some kind of soundbite hatetorial prole phrase
substituting and blocking off exploring the actual use cases which
we''ve done over and over again on this list.  I bet the phrase is
still there on the si wiki, implying while sneakily and lawyer-like
not outright saying, that the tools are non-enterprise backup tools
like ''tar'', but they''re not, and a hundred web2.0
sneaker-wearing
douches scribbling away in bouncylogo textpad iwordpress meblog will
not make me wrong.

     >>  Cpio coes not support sparse files and is unable to archive
     >> files > 8 GB.

    tb>  I found this out the hard way last time i used it.  

yeah, I guess they all suck.  Also some of them, like tar, don''t have
checksums.

the solaris userland is ancient, so if you use gtar you''ll be
surprised less often, like IMHO you are less likely to have silently
truncated files, but then it''s maintained upstream so it''s
missing
support for all these forked files, mandatory labels, and windows
ACL''s that they''re piling into ZFS now for NAS features.

When we brought it up last time, I think we found no one knows of a
userland tool similar to ''ufsdump'' that''s capable of
serializing a ZFS
along with holes, large files, ``attribute'''' forks, windows
ACL''s, and
checksums of its own, and then restoring the stream in a
filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100118/cefd615f/attachment.bin>

On Jan 18, 2010, at 11:04 AM, Miles Nordin wrote:
...
> Another problem is that the snv_112 man page says this:
> 
> -----8<-----
>         The format of the stream is evolving. No backwards  com-
>         patibility is guaranteed. You may not be able to receive
>         your streams on future versions of ZFS.
> -----8<-----
> 
> I think the new man page says something more generous. 

The Solaris 10 10/09 zfs(1m) man page says:

         The format of the stream is committed. You will be  able
         to receive your streams on future versions of ZFS.

I''m not sure when that hit snv, but obviously it was after b112.
...
> the solaris userland is ancient, so if you use gtar you''ll be
> surprised less often, like IMHO you are less likely to have silently
> truncated files, but then it''s maintained upstream so
it''s missing
> support for all these forked files, mandatory labels, and windows
> ACL''s that they''re piling into ZFS now for NAS features.
OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead
of /usr/bin, so gnu tar is "the default."  As of b130 (I''m
not running
an older build currently) the included gnu tar is version 1.22 which
is the latest as released March 2009 at http://www.gnu.org/software/tar/
> When we brought it up last time, I think we found no one knows of a
> userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
> along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
> checksums of its own, and then restoring the stream in a
> filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
That is my understanding as well.  It seems that the backup vendors
are moving forward in a more-or-less vendor-specific way. This is
not necessarily a bad thing, since there are open source solutions.
But I see the requirements for backups being much more sophisticated
than ufsdump was 25 years ago.  hmmm... has ufsdump changed over 
the past 25 years? ;-)
 -- richard

On Mon, Jan 18, 2010 at 07:34:51PM +0100, Lassi Tuura
wrote:
> > Consider then, using a zpool-in-a-file as the file format, rather than
> > zfs send streams.
> 
> This is an interesting suggestion :-)
> 
> Did I understand you correctly that once a slice is written, zfs
> won''t rewrite it? In other words, I can have an infinitely growing
> pile of these slices, and once zfs fills one file up (or one
> "raidz2" set of files), I flush it to tape/optical disk/whatever,
> and zfs won''t change it "ever after"? When I need more
space, I just
> add more slices, but old slices are effectively read-only?
No.  The intention of this scheme is to have an ongoing backup pool,
with a rolling set of historical snapshots collected together.  I
wouldn''t suggest growing the pool indefinately, because you''ll
need an
indefinite number of pool backing files online when it comes time for
a distant-future restore.

Even if you were never to delete old snapshots, zfs will still
update some of the metadata in each file, or raidz set of files,
whenever you make changes to your backup pool.  You will need to sync
each of these files to your backup media or offsite storage, to have a
consistent pool to restore from.  

However, you can add new top-level vdevs (e.g. raidz set of files) as
your backup pool starts to fill.  Until you start to recycle space (by
deleting old snapshots from the backup pool) there won''t be much new
data written to the original full files.  Therefore, if your offsite
replication of these files is efficient for incremental changes
(e.g. rsync) then they will update quickly.

The same will happen for changes within the existing files, of course,
if they''re not yet full or if you are recycling space.  It''s
the same
data, all that changes is how it''s spread among the files.

If you''re writing to tapes or dvd''s, you''ll either
need to rewrite the
files completely every time, or layer some *other* incremental
mechanism on top (perhaps your existing large-scale enterprise VTL,
which you''d like to continue using, or are forced to continue using :).

I don''t mind rewriting tape media every time.. keep 2 or three sets
and just roll between them each time.  There are advantages to
having a predictable cycle - knowing that you''re going to need exactly
N tapes this week and they will take T time to write out.  You also
get the benefit of all other D-to-D-to-T schemes, that this can be
done at leisure asynchronously to host backup windows. For DVD''s its a
little more annoying, but at least the media is cheap. 

For these purposes, you can also consider removable hard disks as
tapes.  As I replace one generation of drives with the next, higher
capacity, I intend for the previous ones can move to backup service. 
> And perhaps most importantly: has anyone actually done this for
> their back-ups and has success stories on restore after media
> failure? 
For me it''s somewhere between concept and regular practice. It works,
I''ve tinkered with it and done intermittent runs at archiving off the
pool files and test imports and restores, but not written automation
or made it as regular practice as I should.  I''ve used it to drop
backups of my opensolaris hosts onto enterprise backed-up servers, in 
non-solaris customer environments.  

There are some wrinkles, like you can''t mount zpool files off
read-only DVD media directly - you have to copy them to r/w scratch
space because import wants to update metadata (is there an RFE for
read-only import? I forget).  This is mildly irritating at worst -
copying one or two dvd''s worth of files is easy, yet any more than
this in your backup pool, you''ll need to copy anyway for lack of many
dvd readers at once. 

I also don''t recommend files >1Gb in size for DVD media, due to
iso9660 limitations.  I haven''t used UDF enough to say much about any
limitations there.

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/7ff4a9c8/attachment.bin>

On 18/01/2010 18:28, Lassi Tuura wrote:
> Hi,
>
>    
>> Here is the big difference. For a professional backup people still
typically
>> use tapes although tapes have become expensive.
>>
>> I still believe that a set of compressed incremental star archives give
you
>> more features.
>>      
> Thanks for your comments!
>
> I think I roughly understand the feature trade-offs, and have some
experience with back-up solutions ranging from simple to enterprise.
>
> I guess what I am after is, for data which really matters to its owners and
which they actually had to recover, did people use tar/pax archives (~ file
level standard archive format), dump/restore (~ semi-standard format based on
files/inodes) or zfs send/receive (~ file system block level dump), or something
else, and why? (Regardless of how these are implemented, hobby scripts or
enterprise tools, how they dealt with possible media failure issues, etc.)
>
> Other than the file system vs. file restore, is there any concern in doing
the block level thing? "Concerns" as in "mind being in the line
of fire if it fails?" :-)
>
>    
What we are doing basically is:

1. incremental rsync from a client to a dedicated filesystem for that client
2. snapshot after rsync finished
3. go to #1 for next backup

a pool is a dynamic stripe across raidz-2 groups + hot spares.
Then selected clients along with all their backups (snapshots) are 
replicated to another device which is exactly the same hardware/software 
configuration.

Add to it a management of snapshots (retention policies), reporting, 
etc. and you have a pretty good backup solution which allows you to 
restore a single file or entire filesystem with a very easy access to 
any backup. And it works for different OSes as clients.

See more details at
http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

-- 
Robert Milkowski
http://milek.blogspot.com

On Mon, Jan 18, 2010 at 01:38:16PM -0800, Richard Elling
wrote:
> The Solaris 10 10/09 zfs(1m) man page says:
> 
>          The format of the stream is committed. You will be  able
>          to receive your streams on future versions of ZFS.
> 
> I''m not sure when that hit snv, but obviously it was after b112.
> ...
I only noticed it recently.  My guess was that it arrived together
with the format changes for "zfs send -D".   As handy as that change
is, it doesn''t address some of the other deficiencies involved in
trying to repurpose a replication stream as an archive format.
> > When we brought it up last time, I think we found no one knows of a
> > userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
> > along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
> > checksums of its own, and then restoring the stream in a
> > filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
It was precisely the realisation that the zpool-in-a-file format had
all of the desired characteristics that led to the scheme I described
elsewhere in this thread.  

(It wasn''t one of my criteria, but this goes up to and including the
"userland tool" component - although I''ve not tried, I
understand
there are userland versions of the zfs tools in the test suite.)

It has the advantage of being entirely common with the original, so it
will be well tested and keep in sync with new features (dedup, crypto,
next+N).   A focus on formalising this usage pattern would bring
further benefits in terms of features  (e.g, as a worthwhile use case
for "read only import") and of practices (documentation and process).

--
Dan.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/c3ad2ec4/attachment.bin>

> When we brought it up last time, I think we found no one knows of a
> userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
> along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
> checksums of its own, and then restoring the stream in a
> filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
This has been something I''ve been working on for the last few days and
I''ve not yet found an adequate solution. I''ve looked at tar,
cpio and pax as potential tools for backing up a ZFS filesystem to tape, but
each has limitations. As of today, Sun do not provide the ability to reliably
backup a Solaris installation without resorting to third party tools. This is
crazy!

The beauty of ufsdump/ufsrestore is that because it''s bundled with the
operating system, I can perform bare metal recovery using a Solaris DVD and
locally attached tape drive. It''s simple and arguably essential for
system administrators.

From my perspective, Sun really need to create a zfsdump/zfsrestore set of
commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.

Does anyone know the process to formally request this (we have a support
contract), or would I be wasting my time?

Thanks

JR

_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/17a8d897/attachment.html>

Thomas Burgess <wonslung at gmail.com> wrote:
> so star is better than tar?
Star is the oldest OSS tar implementation (it started in 1982).
Star is (in contrary to Sun''s tar and to GNU tar) able to create
archives with attributes from recent POSIX standards and it implements
aprox. twice as many features than GNU tar without forcing you to learn
twice as many options ;-)

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Edward Ned Harvey <solaris at nedharvey.com> wrote:
> > I still believe that a set of compressed incremental star archives
give
> > you
> > more features.
>
> Big difference there is that in order to create an incremental star
archive,
> star has to walk the whole filesystem or folder that''s getting
backed up,
> and do a "stat" on every file to see which files have changed
since the last
> backup run.  If you have a large filesystem, that can take a very long
time.
Star implements this in a very effective way (by using libfind) that is even 
faster that the find(1) implementation from Sun.

The big advantage with star is that you get OS and filesystem independent
archives that are compatible with recent POSIX standards and thus can be read 
on any POSIX.1-2001 compliant OS even if you don''t have a star binary
handy.
> I run incremental zfs send & receive, and it completes typically in a
minute
> or two.
Did you make a test with star?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Lassi Tuura <lat at cern.ch> wrote:
> I guess what I am after is, for data which really matters to its owners and
which they actually had to recover, did people use tar/pax archives (~ file
level standard archive format), dump/restore (~ semi-standard format based on
files/inodes) or zfs send/receive (~ file system block level dump), or something
else, and why? (Regardless of how these are implemented, hobby scripts or
enterprise tools, how they dealt with possible media failure issues, etc.)
star combines the advantages from ufsdump/ufsrestore (true incremental dumps)
with the advantages of a POSIX standard achive format.

Note that star is even at least 30% faster that ufsdump (although ufsdump
reads the raw disk device while star uses the official OS filesystem interface).

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Miles Nordin <carton at Ivy.NET> wrote:
> When we brought it up last time, I think we found no one knows of a
> userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
> along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
> checksums of its own, and then restoring the stream in a
> filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
Star is such a tool. It privides all the features known from ufsdump
but it is OS and filesystem indepentent. 

Once Sun shows interest in star, there will of course be also support 
for NTFS-ACLs.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Richard Elling <richard.elling at gmail.com> wrote:
> OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead
> of /usr/bin, so gnu tar is "the default."  As of b130
(I''m not running
> an older build currently) the included gnu tar is version 1.22 which
> is the latest as released March 2009 at http://www.gnu.org/software/tar/
This causes Indiana to by default offer tar implementation that creates archives
that are in conflict with the POSIX standard.

Also note that GNU tar is unable to be used as a backup utility:

-	It does neither support ACLs not any other file attributes.

-	It failes even with simples modifications in the original filesystem
	and people will become aware of this problem at the time when
	they try to restore a backup that will not work in many cases.

> That is my understanding as well.  It seems that the backup vendors
> are moving forward in a more-or-less vendor-specific way. This is
> not necessarily a bad thing, since there are open source solutions.
> But I see the requirements for backups being much more sophisticated
> than ufsdump was 25 years ago.  hmmm... has ufsdump changed over 
> the past 25 years? ;-)
ufsdump did (slightly) change as it now allows you to specify a subdirectory
instead of the whole filesystem.

But what features are you missing?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Daniel Carosone <dan at geek.com.au> wrote:
> I also don''t recommend files >1Gb in size for DVD media, due to
> iso9660 limitations.  I haven''t used UDF enough to say much about
any
> limitations there.
ISO-9660 supports files up to 8 TB. Do you have a bigger pool?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

+1

for zfsdump/zfsrestore



Julian Regel wrote:
> 
>> When we brought it up last time, I think we found no one knows of a
>> userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
>> along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
>> checksums of its own, and then restoring the stream in a
>> filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
> 
> This has been something I''ve been working on for the last few days
and I''ve not yet found an adequate solution. I''ve looked at
tar, cpio and pax as potential tools for backing up a ZFS filesystem to tape,
but each has limitations. As of today, Sun do not provide the ability to
reliably backup a Solaris installation without resorting to third party tools.
This is crazy!
> 
> The beauty of ufsdump/ufsrestore is that because it''s bundled with
the operating system, I can perform bare metal recovery using a Solaris DVD and
locally attached tape drive. It''s simple and arguably essential for
system administrators.
> 
> From my perspective, Sun really need to create a zfsdump/zfsrestore set of
commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.
> 
> Does anyone know the process to formally request this (we have a support
contract), or would I be wasting my time?
> 
> Thanks
> 
> JR
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> 
> 
> 
>       
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Jan 19, 2010, at 1:53 AM, Julian Regel wrote:
> > When we brought it up last time, I think we found no one knows of a
> > userland tool similar to ''ufsdump'' that''s
capable of serializing a ZFS
> > along with holes, large files, ``attribute'''' forks,
windows ACL''s, and
> > checksums of its own, and then restoring the stream in a
> > filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
> 
> This has been something I''ve been working on for the last few days
and I''ve not yet found an adequate solution. I''ve looked at
tar, cpio and pax as potential tools for backing up a ZFS filesystem to tape,
but each has limitations. As of today, Sun do not provide the ability to
reliably backup a Solaris installation without resorting to third party tools.
This is crazy!
> 
> The beauty of ufsdump/ufsrestore is that because it''s bundled with
the operating system, I can perform bare metal recovery using a Solaris DVD and
locally attached tape drive. It''s simple and arguably essential for
system administrators.
Yep. And it was invented because there was no option other than tar
at the time. Today, there are many very comprehensive backup solutions
on the market including open source. Some are nicely integrated, such 
as NexentaStor and Zmanda.
> From my perspective, Sun really need to create a zfsdump/zfsrestore set of
commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.
In my crystal ball, I see a divergence away from traditional backup 
solution approaches. Today you can feel comfortable with backing up
ZFS file systems because they provide a POSIX file system interface.
Other datasets don''t resemble POSIX file systems at all, and I see
several
different types of datasets with the opportunity to become popular with
the basic ZVol getting a lot of attention lately. The win will go to the 
vendor who can provide the best value for the various datasets in the ZFS
environment.
> Does anyone know the process to formally request this (we have a support
contract), or would I be wasting my time?
You can pile onto CR 5004379, want comprehensive backup strategy
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379

I can''t answer the latter, but judging by the date of the CR, I
wouldn''t hold my
breath. Give the fine folks at Zmanda a look.

Also, the ADM project seems to be dead. Unfortunate?
http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM
 -- richard

On Tue, Jan 19, 2010 at 11:36 AM, Richard Elling
<richard.elling at gmail.com>wrote:
> On Jan 19, 2010, at 1:53 AM, Julian Regel wrote:
> > > When we brought it up last time, I think we found no one knows of
a
> > > userland tool similar to ''ufsdump''
that''s capable of serializing a ZFS
> > > along with holes, large files, ``attribute''''
forks, windows ACL''s, and
> > > checksums of its own, and then restoring the stream in a
> > > filesystem-agnostic read/write/lseek/... manner like
''ufsrestore''.
> >
> > This has been something I''ve been working on for the last few
days and
> I''ve not yet found an adequate solution. I''ve looked at
tar, cpio and pax as
> potential tools for backing up a ZFS filesystem to tape, but each has
> limitations. As of today, Sun do not provide the ability to reliably backup
> a Solaris installation without resorting to third party tools. This is
> crazy!
> >
> > The beauty of ufsdump/ufsrestore is that because it''s bundled
with the
> operating system, I can perform bare metal recovery using a Solaris DVD and
> locally attached tape drive. It''s simple and arguably essential
for system
> administrators.
>
> Yep. And it was invented because there was no option other than tar
> at the time. Today, there are many very comprehensive backup solutions
> on the market including open source. Some are nicely integrated, such
> as NexentaStor and Zmanda.
>
> > From my perspective, Sun really need to create a zfsdump/zfsrestore
set
> of commands that perform block level backups of a specified filesystem (or
> snapshot) and that preserves ACLs, atime, sparse files, handles multiple
> tapes (many of us still use and rely on tape!) etc.
>
> In my crystal ball, I see a divergence away from traditional backup
> solution approaches. Today you can feel comfortable with backing up
> ZFS file systems because they provide a POSIX file system interface.
> Other datasets don''t resemble POSIX file systems at all, and I see
several
> different types of datasets with the opportunity to become popular with
> the basic ZVol getting a lot of attention lately. The win will go to the
> vendor who can provide the best value for the various datasets in the ZFS
> environment.
>
> > Does anyone know the process to formally request this (we have a
support
> contract), or would I be wasting my time?
>
> You can pile onto CR 5004379, want comprehensive backup strategy
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379
>
> I can''t answer the latter, but judging by the date of the CR, I
wouldn''t
> hold my
> breath. Give the fine folks at Zmanda a look.
>
> Also, the ADM project seems to be dead. Unfortunate?
> http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM
>  -- richard
>
>
I''m likely missing something, so please, fill me in.  Aren''t
they simply
asking for the functionality already provided by NDMP?

-- 
--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/5e1e7ac2/attachment.html>

>> The beauty of ufsdump/ufsrestore is that because it''s bundled
with the operating system, I can perform bare metal recovery using a Solaris DVD
and locally attached tape drive. It''s simple and arguably essential for
system administrators.
> Yep. And it was invented because there was no option other than tar
> at the time. Today, there are many very comprehensive backup solutions
> on the market including open source. Some are nicely integrated, such 
> as NexentaStor and Zmanda.
When I look at the documentation for Zmanda
(http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
it states that the command used to backup a ZFS filesystem depends on whether
backing up ACLs are required (the default is not to(!)). If backing up ACLs are
required - which they are for us - then the native /usr/bin/tar command is used.
Given that /usr/bin/tar doesn''t handle sparse files correctly and
updates atime when creating archives, it''s not possible to create a
complete copy of a filesystem without making intrusive changes to the structure
of the data and/or metadata.

So it''s arguable that ufsdump was a significant improvement over tar,
and that the current archiving solutions for ZFS are actually a step backwards.
> From my perspective, Sun really need to create a zfsdump/zfsrestore set of
commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.
> In my crystal ball, I see a divergence away from traditional backup 
> solution approaches. Today you can feel comfortable with backing up
> ZFS file systems because they provide a POSIX file system interface.
Based on what I''ve seen in other comments, you might be right.
Unfortunately, I don''t feel comfortable backing up ZFS filesystems
because the tools aren''t there to do it (built into the operating
system or using Zmanda/Amanda).

I know tape backup isn''t sexy, but it''s a reality for many of
us and it''s not going away anytime soon.

JR


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/87a7e734/attachment.html>

Julian Regel wrote:
>
> Based on what I''ve seen in other comments, you might be right. 
> Unfortunately, I don''t feel comfortable backing up ZFS filesystems
> because the tools aren''t there to do it (built into the operating 
> system or using Zmanda/Amanda).
>
Commercial backup solutions are available for ZFS.
> I know tape backup isn''t sexy, but it''s a reality for
many of us and
> it''s not going away anytime soon.
>
True, but I wonder how viable its future is.  One of my clients requires 
17 LT04 types for a full backup, which cost more and takes up more space 
than the equivalent in removable hard drives.

In the past few years growth in hard drive capacities has outstripped 
tapes to the extent that removable hard drives and ZFS snapshots have 
become a more cost effective and convenient backup media.

What do people with many tens of TB use for backup these days?

-- 
Ian.

Julian Regel <jrmailgate-zfsdiscuss at yahoo.co.uk> wrote:
> When I look at the documentation for Zmanda
(http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
it states that the command used to backup a ZFS filesystem depends on whether
backing up ACLs are required (the default is not to(!)). If backing up ACLs are
required - which they are for us - then the native /usr/bin/tar command is used.
Given that /usr/bin/tar doesn''t handle sparse files correctly and
updates atime when creating archives, it''s not possible to create a
complete copy of a filesystem without making intrusive changes to the structure
of the data and/or metadata.
>
> So it''s arguable that ufsdump was a significant improvement over
tar, and that the current archiving solutions for ZFS are actually a step
backwards.
>
> > From my perspective, Sun really need to create a zfsdump/zfsrestore
set of commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.
Sun''s tar is not able to archive files > 8 GB in a way that can be
read on any
other OS unless you use star. This is because Sun''s tar does not
implement
support for the POSIX.1-2001 extended tar format.

Sun''s tar also writes ACLs in a way that is 100% non-portable. Star
cannot
understand them and probably never will be able to understand this format as it
is not well defined for a portable program like star.

Star supports to do backups without affecting atime (on Solaris) since 15 years 
already and star supports the withdrawn POSIX draft ACLs in a OS independent 
way. This allows to use star for platform independent ACL support in case you 
are using UFS on Solaris.

Star will in future support NTFS ACLs in a OS independent way. If someone likes 
to contribute, he is of course welcome. As I am currently working on 
cdrtools-3.0-final, star is currently on maintenance only.

What we need for the future is a OS/FS independent program that implements the
needed features.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Ian Collins <ian at ianshome.com> wrote:
> Julian Regel wrote:
> >
> > Based on what I''ve seen in other comments, you might be
right.
> > Unfortunately, I don''t feel comfortable backing up ZFS
filesystems
> > because the tools aren''t there to do it (built into the
operating
> > system or using Zmanda/Amanda).
> >
> Commercial backup solutions are available for ZFS.
On what archiving programs are these solutions based on?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Joerg Schilling wrote:
> Ian Collins <ian at ianshome.com> wrote:
>
>   
>> Julian Regel wrote:
>>     
>>> Based on what I''ve seen in other comments, you might be
right.
>>> Unfortunately, I don''t feel comfortable backing up ZFS
filesystems
>>> because the tools aren''t there to do it (built into the
operating
>>> system or using Zmanda/Amanda).
>>>
>>>       
>> Commercial backup solutions are available for ZFS.
>>     
>
> On what archiving programs are these solutions based on?
>
>   
I''m sure they use their own formats.  The only one I have crossed paths
with is NetVault.

-- 
Ian.

On Wed, 20 Jan 2010, Ian Collins wrote:
> Commercial backup solutions are available for ZFS.
>> I know tape backup isn''t sexy, but it''s a reality for
many of us and it''s
>> not going away anytime soon.
>> 
> True, but I wonder how viable its future is.  One of my clients requires 17
> LT04 types for a full backup, which cost more and takes up more space than 
> the equivalent in removable hard drives.
>
> In the past few years growth in hard drive capacities has outstripped tapes
> to the extent that removable hard drives and ZFS snapshots have become a
more
> cost effective and convenient backup media.
In all of these discussions, people seem to forget some very important 
criteria.  That important criteria is the time required for a full 
recovery from backup.

If the company is not able to survive more than a week with total 
disruption of business, but the full restore will take three weeks, 
then the backup has been rendered 100% ineffective.

When planning for recovery from disaster, the time to recover is 
exceedingly important.

When using the backup to filesystem approach, that backup could be 
done to a remote mirrored system (filesystems + hardware) which is 
sufficiently distant to protect against any local disaster.  If there 
is a disaster in the local data center, that system could be 
immediately put on line (assuming adequate connectivity), or that 
system could be loaded on a truck for overnight delivery as a 
replacement to the data center.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

Joerg Schilling wrote:
> Julian Regel <jrmailgate-zfsdiscuss at yahoo.co.uk> wrote:
>
>   
>> When I look at the documentation for Zmanda
(http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
it states that the command used to backup a ZFS filesystem depends on whether
backing up ACLs are required (the default is not to(!)). If backing up ACLs are
required - which they are for us - then the native /usr/bin/tar command is used.
Given that /usr/bin/tar doesn''t handle sparse files correctly and
updates atime when creating archives, it''s not possible to create a
complete copy of a filesystem without making intrusive changes to the structure
of the data and/or metadata.
>>
>> So it''s arguable that ufsdump was a significant improvement
over tar, and that the current archiving solutions for ZFS are actually a step
backwards.
>>
>>     
>>> From my perspective, Sun really need to create a zfsdump/zfsrestore
set of commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.
>>>       
>
> Sun''s tar is not able to archive files > 8 GB in a way that can
be read on any
> other OS unless you use star. This is because Sun''s tar does not
implement
> support for the POSIX.1-2001 extended tar format.
>
> Sun''s tar also writes ACLs in a way that is 100% non-portable.
Star cannot
> understand them and probably never will be able to understand this format
as it
> is not well defined for a portable program like star.
>
>   
Is that because they are NFSv4 ACLs?  tar uses the formatting routines 
from libacl to archive them.

-- 
Ian.

Ian Collins <ian at ianshome.com> wrote:
> > Sun''s tar also writes ACLs in a way that is 100%
non-portable. Star cannot
> > understand them and probably never will be able to understand this
format as it
> > is not well defined for a portable program like star.
> >
> >   
> Is that because they are NFSv4 ACLs?  tar uses the formatting routines 
> from libacl to archive them.
The correct way to archivbe ACLs would be to put them into extended POSIX tar
attrubutes as star does.

See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.
ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz

The ACL format used by Sun is undocumented......


J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

>>>>> "jk" == Jerry K <sun.mail.list47 at
oryx.cc> writes:
    jk> +1 
    jk> for zfsdump/zfsrestore

-1

I don''t think a replacement for the ufsdump/ufsrestore tool is really
needed.  From now on, backups just go into Another Zpool.

We need the zfs send stream format commitment (stream format depends
only on zfs version, not on zpool version or kernel version), but
that''s enough.


If people are really still backing up to tapes or DVD''s, just use file
vdev''s, export the pool, and then copy the unmounted vdev onto the
tape or DVD.  The requirement that your backup be staged on a disk
isn''t an obstacle in the backup direction: Amanda already has this
requirement.  Amanda, when I used it, did *not* have this requirement
in the restore direction so one would have to keep that in mind: if
using tapes, he needs a scratch disk as big as the biggest tape file
on the DR site or the development environment or Compliance Extractor
Station or wherever the restore is happening.

File vdev''s have all the wanted characteristics: bitflip resiliency,
checksums, ability to represent all the opaque mysteryfeatures of
inner ZFS''s, snapshot/clone efficiency, and importability by future
ZFS kernels ~forever.  It still has the all-or-nothing-restore
downside, but maybe we can live with that.  Those who can''t might
stick to spinning-disk backups.

It might be nice to have a ``read-only vdev'''' like a mac os
compressed
disk image that can occupy the same pool next to a read-write
vdev---this would be neat for livecd''s and incrementals---but
it''s a
neat feature, not something blocking a frequent/unavoidable sysadmin
task.

What''s needed IMHO is non-ZFS-specific tools like tar/pax/cpio/rsync
that understand all this new ACL and opaque-fork garbage that
filesystems are growing (and not just Sun''s either).  It''s
needed to
copy between NFSv4 and ZFS, and also to split/combine subdirectories
into separate/common ZFS filesystems without losing any of the
newfangled mysterybaggage.  It has as much to do with shaking corner
cases and awkwardnesses out of the newfangled API''s as it does with
actually accomplishing a sysadmin task: the best documentation for the
weird appendages in various modern Unix filesystems would probably be
the tool itself, if we had it.  

Without this kind of tool and the API shakeout that making it would
require, our systems are slowly turning into shitty Windows boxes that
need some black magic proprietary tool like Ghost or PQDI to capture
all the silly out-of-band garbage their installations depend on and/or
accumulate.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/d32fc30b/attachment.bin>

>>>>> "ic" == Ian Collins <ian at ianshome.com>
writes:
     >> I know tape backup isn''t sexy, but it''s a
reality for many of
     >> us and it''s not going away anytime soon.

    ic> True, but I wonder how viable its future is.  One of my
    ic> clients requires 17 LT04 types for a full backup, which cost
    ic> more and takes up more space than the

but Ian, aiui SOX requires backups onto a write-once non-eraseable
medium, because of all the discovery shennanigans and
claimed-incompetence around the time of the Worldcomm and Enron
bankruptcy near the start of Bush II.  The law in practice is sort of
like a tiger that chases the herd and eats the sick animal that''s
lagging behind, and currently I think the herd is using ``WORM
tape''''
which is a tape that''s made to appear to be append-only by the tape
drive''s firmware.  These tapes exist now and must be retained for
seven years, so even if you invented a newfangled gadget meeting the
WORM requirements TODAY so that you can stay in the middle of the herd
without using tape, you still will not get rid of tape for seven
years.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100119/431aa3f5/attachment.bin>

On Tue, Jan 19, 2010 at 12:16:01PM +0100, Joerg Schilling
wrote:
> Daniel Carosone <dan at geek.com.au> wrote:
> 
> > I also don''t recommend files >1Gb in size for DVD media,
due to
> > iso9660 limitations.  I haven''t used UDF enough to say much
about any
> > limitations there.
> 
> ISO-9660 supports files up to 8 TB. 
However, some implementations have (or had?) limitations which meant
that they couldn''t read such disks.  Some of those platforms even had
problems (or required remembering arcane options and flags for "large
file support") when copying these files back to hard disk.  

Linux was a prime culprit, though not the only one.  I didn''t use
linux, but I never knew what might be at hand when needing to read
disks for recovery, so I wanted them to be portable.  I envisaged
using multiple machines to read more disks in parallel, for example.

It may not be as relevant for this use case, nor perhaps for more
modern platforms, but it was a habit I developed earlier for other
kinds of archives. 

I apologise, though, for misattributing this limitation to the
filesystem spec.  I misremembered the reason why I chose the limit.
It was a practical limit, and the practicalities may well have changed
since.
> Do you have a bigger pool?
I certainly don''t have a bigger DVD media :-)

(Yes, I know iso9660 also has multivolume support, but I''m similarly
wary of being able to read it back on all platforms, since it''s not
encountered often, and there''s no need when the source files can be
split to better effect.)

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/691e58b1/attachment.bin>

There is a tendency to conflate "backup" and "archive", both
generally
and in this thread. They have different requirements.  

Backups should enable quick restore of a full operating image with all
the necessary system level attributes. They concerned with SLA and
uptime and outage and data loss window criteria. 

Archives are usually much more concerned with application data, which
may have been specially prepared for the future user of the
archive. Portability is often more important here, both in archive
format but also in data schema within, even at the expense of some
kinds of technical integrity or completeness (e.g. ACLs).

Adding regulatory requirements into the mix further complicates matters.

If one solution can address the union of all requirements, then you''re
in luck, but that''s not always the case.  Sometimes the best way to
resolve the tension is to save the data differently for different
purposes. 

--
Dan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 194 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/d8810313/attachment.bin>

> Message: 3
> Date: Tue, 19 Jan 2010 15:48:52 -0500
> From: Miles Nordin <carton at Ivy.NET>
> To: zfs-discuss at opensolaris.org
> Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability?
> Message-ID: <oqpr55lt1n.fsf at castrovalva.Ivy.NET>
> Content-Type: text/plain; charset="us-ascii"
> 
> I don''t think a replacement for the ufsdump/ufsrestore tool is
really
> needed.  From now on, backups just go into Another Zpool.
> 
> We need the zfs send stream format commitment (stream format depends
> only on zfs version, not on zpool version or kernel version), but
> that''s enough.
> 
> 
> If people are really still backing up to tapes or DVD''s, just use
file
> vdev''s, export the pool, and then copy the unmounted vdev onto the
> tape or DVD.  The requirement that your backup be staged on a disk
> isn''t an obstacle in the backup direction: Amanda already has this
> requirement.  Amanda, when I used it, did *not* have this requirement
> in the restore direction so one would have to keep that in mind: if
> using tapes, he needs a scratch disk as big as the biggest tape file
> on the DR site or the development environment or Compliance Extractor
> Station or wherever the restore is happening.

While this idea may be fine for a home user?those us of who have customers in
the enterprise still have a need for tape backups.

And some of those enterprises require backup mechanism that can be easily used
in a DR situation.

ufsdump/restore was perfect in that regard.  The lack of equivalent
functionality is a big problem for the situations where this functionality is a
business requirement.

For example, one customer, local government, requires a backup that can be taken
offsite and used in a DR situation.  They have 2 sun servers.  They have very
little Solaris knowledge. So whatever I provide them has to be easy and easily
documented.  Lack of "zfsdump/restore" means I either have to forgo
moving them to ZFS root, or have to add in a third party application?like
I''m really going to meet their requirements with Amanda or Backula?

This lack in Solaris is just plain ludicrous to at least some parts of the
business/enterprise world.

On Jan 19, 2010, at 4:26 PM, Allen Eastwood wrote:
>> Message: 3
>> Date: Tue, 19 Jan 2010 15:48:52 -0500
>> From: Miles Nordin <carton at Ivy.NET>
>> To: zfs-discuss at opensolaris.org
>> Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability?
>> Message-ID: <oqpr55lt1n.fsf at castrovalva.Ivy.NET>
>> Content-Type: text/plain; charset="us-ascii"
>> 
>> I don''t think a replacement for the ufsdump/ufsrestore tool is
really
>> needed.  From now on, backups just go into Another Zpool.
>> 
>> We need the zfs send stream format commitment (stream format depends
>> only on zfs version, not on zpool version or kernel version), but
>> that''s enough.
>> 
>> 
>> If people are really still backing up to tapes or DVD''s, just
use file
>> vdev''s, export the pool, and then copy the unmounted vdev onto
the
>> tape or DVD.  The requirement that your backup be staged on a disk
>> isn''t an obstacle in the backup direction: Amanda already has
this
>> requirement.  Amanda, when I used it, did *not* have this requirement
>> in the restore direction so one would have to keep that in mind: if
>> using tapes, he needs a scratch disk as big as the biggest tape file
>> on the DR site or the development environment or Compliance Extractor
>> Station or wherever the restore is happening.
> 
> 
> While this idea may be fine for a home user?those us of who have customers
in the enterprise still have a need for tape backups.
> 
> And some of those enterprises require backup mechanism that can be easily
used in a DR situation.
> 
> ufsdump/restore was perfect in that regard.  The lack of equivalent
functionality is a big problem for the situations where this functionality is a
business requirement.
How quickly we forget ufsdump''s limitations :-).  For example, it is
not supported
for use on an active file system (known data corruption possibility) and 
UFS snapshots are, well, a poor hack and often not usable for backups.
As the ufsdump(1m) manpage says,
     The ufsdump command can only be used on unmounted file  sys-
     tems,  or  those  mounted  read-only.  Attempting  to dump a
     mounted, read-write file system might  result  in  a  system
     disruption  or the inability to restore files from the dump.
     Consider using the fssnap(1M) command to create a file  sys-
     tem  snapshot  if  you  need a point-in-time image of a file
     system that is mounted.

So, if you are taking ufsdumps of an active file system for business
requirements,
then perhaps you should rethink the solution.
> For example, one customer, local government, requires a backup that can be
taken offsite and used in a DR situation.  They have 2 sun servers.  They have
very little Solaris knowledge. So whatever I provide them has to be easy and
easily documented.  Lack of "zfsdump/restore" means I either have to
forgo moving them to ZFS root, or have to add in a third party application?like
I''m really going to meet their requirements with Amanda or Backula?
Many people use send/recv or AVS for disaster recovery on the inexpensive
side. Obviously, enterprise backup systems also provide DR capabilities.
Since ZFS has snapshots that actually work, and you can use send/receive
or other backup solutions on snapshots, I assert the "problem" is low
priority.
> This lack in Solaris is just plain ludicrous to at least some parts of the
business/enterprise world.
Methinks you never read the ufsdump man page? :-)
 -- richard

On Jan 19, 2010, at 18:48 , Richard Elling wrote:
> 
> Many people use send/recv or AVS for disaster recovery on the inexpensive
> side. Obviously, enterprise backup systems also provide DR capabilities.
> Since ZFS has snapshots that actually work, and you can use send/receive
> or other backup solutions on snapshots, I assert the "problem" is
low priority.
> 

 What I have issue with is the idea that no one uses/should use tape any more. 
There are places for tape and it still has value as a backup device.  In many
cases in the past, ufsdump, despite it''s many issues, was able to
restore working OS''s, or individual files.  Perfect, not by a long
shot.  But it did get the job done.

As was pointed out earlier, all I needed was a Solaris CD (or network boot) and
I could restore.  Entire OS gone, boot and ufsrestore.  Critical files deleted,
same thing?and I can restore just the file(s) I need.  And while it''s
been a few years since I''ve read the man page on ufsdump, ufsrestore
and fssnap, those tools have proven useful when dealing with a downed system.

I use zfs send/recv in the enterprise and in smaller environments all time and
it''s is excellent.

Have a look at how awesome the functionally is in this example.

http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs

Regards,

Mike
-- 
This message posted from opensolaris.org

Joerg Schilling wrote:
> Ian Collins <ian at ianshome.com> wrote:
>
>   
>>> Sun''s tar also writes ACLs in a way that is 100%
non-portable. Star cannot
>>> understand them and probably never will be able to understand this
format as it
>>> is not well defined for a portable program like star.
>>>
>>>   
>>>       
>> Is that because they are NFSv4 ACLs?  tar uses the formatting routines 
>> from libacl to archive them.
>>     
>
> The correct way to archivbe ACLs would be to put them into extended POSIX
tar
> attrubutes as star does.
>
> See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
> documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.
> ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz
>
> The ACL format used by Sun is undocumented......
>
>   
man acltotext

-- 
Ian.

Allen Eastwood wrote:
> On Jan 19, 2010, at 18:48 , Richard Elling wrote:
>
>   
>> Many people use send/recv or AVS for disaster recovery on the
inexpensive
>> side. Obviously, enterprise backup systems also provide DR
capabilities.
>> Since ZFS has snapshots that actually work, and you can use
send/receive
>> or other backup solutions on snapshots, I assert the
"problem" is low priority.
>>
>>     
>
>
>  What I have issue with is the idea that no one uses/should use tape any
more.  There are places for tape and it still has value as a backup device.  In
many cases in the past, ufsdump, despite it''s many issues, was able to
restore working OS''s, or individual files.  Perfect, not by a long
shot.  But it did get the job done.
> As was pointed out earlier, all I needed was a Solaris CD (or network boot)
and I could restore.  Entire OS gone, boot and ufsrestore.  Critical files
deleted, same thing?and I can restore just the file(s) I need.  And while
it''s been a few years since I''ve read the man page on ufsdump,
ufsrestore and fssnap, those tools have proven useful when dealing with a downed
system.
>   
For a full recovery, you can archive a send stream and receive it back.  
With ZFS snapshots, the need for individual file recovery from tape is 
much reduced.  The backup server I manage for a large client has 60 days 
of snaps and I can''t remember when they had to go to tape to recover a
file.

-- 
Ian.

> Star implements this in a very effective way (by using libfind) that is
> even
> faster that the find(1) implementation from Sun.
Even if I just "find" my filesystem, it will run for 7 hours.  But zfs
can
create my whole incremental snapshot in a minute or two.  There is no way
star or any other user-space utility that walks the filesystem can come
remotely close to this performance.  Such performance can only be
implemented at the filesystem level, or lower.

On Jan 19, 2010, at 22:54 , Ian Collins wrote:
> Allen Eastwood wrote:
>> On Jan 19, 2010, at 18:48 , Richard Elling wrote:
>> 
>>  
>>> Many people use send/recv or AVS for disaster recovery on the
inexpensive
>>> side. Obviously, enterprise backup systems also provide DR
capabilities.
>>> Since ZFS has snapshots that actually work, and you can use
send/receive
>>> or other backup solutions on snapshots, I assert the
"problem" is low priority.
>>> 
>>>    
>> 
>> 
>> What I have issue with is the idea that no one uses/should use tape any
more.  There are places for tape and it still has value as a backup device.  In
many cases in the past, ufsdump, despite it''s many issues, was able to
restore working OS''s, or individual files.  Perfect, not by a long
shot.  But it did get the job done.
> 
>> As was pointed out earlier, all I needed was a Solaris CD (or network
boot) and I could restore.  Entire OS gone, boot and ufsrestore.  Critical files
deleted, same thing?and I can restore just the file(s) I need.  And while
it''s been a few years since I''ve read the man page on ufsdump,
ufsrestore and fssnap, those tools have proven useful when dealing with a downed
system.
>>  
> For a full recovery, you can archive a send stream and receive it back. 
With ZFS snapshots, the need for individual file recovery from tape is much
reduced.  The backup server I manage for a large client has 60 days of snaps and
I can''t remember when they had to go to tape to recover a file.
> 
> -- 
> Ian.
> 
Let''s see?

For full recovery, I have to zfs send to something, preferably that understands
tape (yes, I know I can send to tape directly, but how well does zfs send handle
the end of the tape? auto-changers?).  Then for individual file recovery, I have
snaphots?which I also have to get on to tape?if I want to have them available on
something other than the boot devices.

Now?to recover the entire OS, perhaps not so bad?but that''s one tool. 
And to recover the one file, say a messed up /etc/system, that''s
preventing my OS from booting?  Have to get that snapshot where I can use it
first?oh and restoring individual files and not the entire snapshot?

At best, it''s an unwieldy process.  But does it offer the simplicity
that ufsdump/ufsrestore (or dump/restore on how many Unix variants?) did?  No
way.

I suppose it might be fair to say, strictly speaking, that backup/restore
probably should be dealt with as a "ZFS" issue.  It''s kind of
ironic that Microsoft offers a backup utility and Sun is basically dropping
theirs.  It might be better to frame the debate somewhere else, but zfs
send/receive and snapshots are not the same as a built-in OS utility for backing
up and restoring the OS and OS files.

A simple, effective dump/restore that deals with all the supported file systems,
can deal with tape or disk, and allows for complete OS restore or individual
file restore, and can be run from a install CD/DVD.  As much as I love ZFS and
as many problems as it does solve, leaving this out was a mistake, IMO.

Allen Eastwood wrote:
> On Jan 19, 2010, at 22:54 , Ian Collins wrote:
>
>   
>> Allen Eastwood wrote:
>>     
>>> On Jan 19, 2010, at 18:48 , Richard Elling wrote:
>>>
>>>  
>>>       
>>>> Many people use send/recv or AVS for disaster recovery on the
inexpensive
>>>> side. Obviously, enterprise backup systems also provide DR
capabilities.
>>>> Since ZFS has snapshots that actually work, and you can use
send/receive
>>>> or other backup solutions on snapshots, I assert the
"problem" is low priority.
>>>>
>>>>    
>>>>         
>>> What I have issue with is the idea that no one uses/should use tape
any more.  There are places for tape and it still has value as a backup device. 
In many cases in the past, ufsdump, despite it''s many issues, was able
to restore working OS''s, or individual files.  Perfect, not by a long
shot.  But it did get the job done.
>>>       
>>> As was pointed out earlier, all I needed was a Solaris CD (or
network boot) and I could restore.  Entire OS gone, boot and ufsrestore. 
Critical files deleted, same thing?and I can restore just the file(s) I need. 
And while it''s been a few years since I''ve read the man page
on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing
with a downed system.
>>>  
>>>       
>> For a full recovery, you can archive a send stream and receive it back.
With ZFS snapshots, the need for individual file recovery from tape is much
reduced.  The backup server I manage for a large client has 60 days of snaps and
I can''t remember when they had to go to tape to recover a file.
>>
>> -- 
>> Ian.
>>
>>     
>
> Let''s see?
>
> For full recovery, I have to zfs send to something, preferably that
understands tape (yes, I know I can send to tape directly, but how well does zfs
send handle the end of the tape? auto-changers?).
I keep a stream (as a file) of my root pool on a USB stick.  It could be 
on tape, but root pools are small.
> Then for individual file recovery, I have snaphots?which I also have to get
on to tape?if I want to have them available on something other than the boot
devices.
>
>   
No, just keep the snapshots in place.  If a file is lost, just gab it 
form the snapshot directory.  If the root filesystem is munted, roll 
back to the last snapshot.
> Now?to recover the entire OS, perhaps not so bad?but that''s one
tool.  And to recover the one file, say a messed up /etc/system, that''s
preventing my OS from booting?  Have to get that snapshot where I can use it
first?oh and restoring individual files and not the entire snapshot?
>
>   
As I said, roll back.  Boot form install media, import the root pool, 
get the file from a snapshot, or roll back to the last good snapshot, 
export and reboot.
> At best, it''s an unwieldy process.  But does it offer the
simplicity that ufsdump/ufsrestore (or dump/restore on how many Unix variants?)
did?  No way.
>
>   
It certainly does for file recovery.  Do you run incremental dumps every 
hour, or every 15 minutes?  Periodic snapshots are quick and cheep.  As 
I said before, careful use of snapshots all be removes the need to 
recover files from tape.  We have 60 days of 4 hourly and 24 hourly 
snapshots in place, so the odds on finding a recent copy of a lost file 
are way better than they would be on daily incrementals.  I certainly 
don''t miss the pain of loading a sequence of incrementals to recover 
lost data.

So ZFS solves the problems in a different way.  For fat-finger recovery, 
it''s way better than ufsdump/ufsrestore.
> A simple, effective dump/restore that deals with all the supported file
systems, can deal with tape or disk, and allows for complete OS restore or
individual file restore, and can be run from a install CD/DVD.  As much as I
love ZFS and as many problems as it does solve, leaving this out was a mistake,
IMO.
>   
It possibly was, but it has encouraged us to find better solutions.

-- 
Ian.

On 19 jan 2010, at 20.11, Ian Collins wrote:
> Julian Regel wrote:
>> 
>> Based on what I''ve seen in other comments, you might be right.
Unfortunately, I don''t feel comfortable backing up ZFS filesystems
because the tools aren''t there to do it (built into the operating
system or using Zmanda/Amanda).
>> 
> Commercial backup solutions are available for ZFS.
>> I know tape backup isn''t sexy, but it''s a reality for
many of us and it''s not going away anytime soon.
>> 
> True, but I wonder how viable its future is.  One of my clients requires 17
LT04 types for a full backup, which cost more and takes up more space than the
equivalent in removable hard drives.
> 
> In the past few years growth in hard drive capacities has outstripped tapes
to the extent that removable hard drives and ZFS snapshots have become a more
cost effective and convenient backup media.
LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so.
LTO drivers cost a little, but so do disk shelves. So, now that there is no big
price issue, there is choice instead. Use it!

Hard drives are good for random access - both restore of individual files and
partial rewrite.

Hard drivers aren''t faster than tape for data transfer, but they might
be cheaper to run in parallel and therefore you could potentially gain speed.
Hard drives have shorter seek time, which may be important.

Hard drives are probably bad for storing for longer times - especially - you
will never know how long it could be stored before it will fail. A month?
Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes
are supposed to be able to keep it''s data for at least 30 years of
stored properly. Hard drives are probably best when used online or at least very
often.

So - it is wrong to say that one is better or cheaper than the other. They have
different properties, and could be used to solve different problems.

/ragge s

Richard Elling <richard.elling at gmail.com> wrote:
> > 
> > ufsdump/restore was perfect in that regard.  The lack of equivalent
functionality is a big problem for the situations where this functionality is a
business requirement.
>
> How quickly we forget ufsdump''s limitations :-).  For example, it
is not supported
> for use on an active file system (known data corruption possibility) and 
> UFS snapshots are, well, a poor hack and often not usable for backups.
> As the ufsdump(1m) manpage says,
It seems you forgot that zfs also needs snapshots. There is nothing bad with 
snapshots.

When I was talking with Jeff Bonwick in September 2004 (before ZFS became 
public), the only feature that was missing in Solaris for a 100% correct
backup based on star was an interface for holey files, so we designed it.

I believe the only mistake from ufsdumps is that is does not use standard
OS interfaces and that it does not use a standard archive format. You get
both with star and star is even faster than ufsdump.


J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Ian Collins <ian at ianshome.com> wrote:
> > The correct way to archivbe ACLs would be to put them into extended
POSIX tar
> > attrubutes as star does.
> >
> > See http://cdrecord.berlios.de/private/man/star/star.4.html for a
format
> > documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha,
e.g.
> > ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz
> >
> > The ACL format used by Sun is undocumented......
> >
> >   
> man acltotext
We are talking about TAR and I did give a pointer to the star archive format 
documentation, so it is obvious that I was talking about the ACL format from
Sun tar. This format is not documented.



J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Edward Ned Harvey <solaris at nedharvey.com> wrote:
> > Star implements this in a very effective way (by using libfind) that
is
> > even
> > faster that the find(1) implementation from Sun.
>
> Even if I just "find" my filesystem, it will run for 7 hours. 
But zfs can
> create my whole incremental snapshot in a minute or two.  There is no way
> star or any other user-space utility that walks the filesystem can come
> remotely close to this performance.  Such performance can only be
> implemented at the filesystem level, or lower.
You claim that it is fast for you but this is because it is block oriented and 
because you probably changed only few data. 

If you like to have a backup that allows to access files, you need a file based 
backup and I am sure that even a filesystem level scan for recently changed 
files will not be much faster than what you may achive with e.g. star.

Note that ufsdump directly accesees the raw disk device and thus _is_ at 
filesystem leven but still is slower than star on UFS.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

While I can appreciate that ZFS snapshots are very useful in being able to
recover files that users might have deleted, they do not do much to help when
the entire disk array experiences a crash/corruption or catches fire. Backing up
to a second array helps if a) the array is off-site and for many of us the cost
of remote links with sufficient bandwidth is still prohibitive,or b) on the
local network but sufficiently far away from the original array such that the
fire does not corrupt damage the backup as well.

This leaves some form of removable storage. I''m not sure I''m
aware of any enterprise-level removable disk solution, primarily because disk
isn''t really designed to be used for offsite backup whereas tape is.

The biggest problem with tape was finding a sufficiently large window in which
to perform the backup. ZFS snapshots completely solves this issue, but Sun have
failed to provide the mechanism to protect the data off-site.



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/f9b5ff85/attachment.html>

> If you like to have a backup that allows to access files, you need a file
based
> backup and I am sure that even a filesystem level scan for recently changed
> files will not be much faster than what you may achive with e.g. star.
>
> Note that ufsdump directly accesees the raw disk device and thus _is_ at 
> filesystem leven but still is slower than star on UFS.
While I am sure that star is technically a fine utility, the problem is that it
is effectively an unsupported product.

If our customers find a bug in their backup that is caused by a failure in a Sun
supplied utility, then they have a legal course of action. The
customer''s system administrators are covered because they were using
tools provided by the vendor. The wrath of the customer would be upon Sun, not
the supplier (us) or the supplier''s technical lead (me).

If the system administrator has chosen star (or if the supplier recommends
star), then the conversation becomes a lot more awkward. From the perspective of
the business, the system administrator will have acted irresponsibly by choosing
a tool that has no vendor support. Alternatively, the supplier will be held
responsible for recommending a product that has broken the customer''s
ability to restore, and with no legal recourse, I wouldn''t dare touch
it. Sorry.

This is why Sun need to provide the solution themselves (or adopt and provide
support for star or similar third party products).

JR



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/391a6419/attachment.html>

Julian Regel <jrmailgate-zfsdiscuss at yahoo.co.uk> wrote:
> > If you like to have a backup that allows to access files, you need a
file based
>
> > backup and I am sure that even a filesystem level scan for recently
changed
> > files will not be much faster than what you may achive with e.g. star.
> >
> > Note that ufsdump directly accesees the raw disk device and thus _is_
at
> > filesystem leven but still is slower than star on UFS.
>
> While I am sure that star is technically a fine utility, the problem is
that it is effectively an unsupported product.
>From this viewpoint, you may call most of Solaris "unsupported".
> If our customers find a bug in their backup that is caused by a failure in
a Sun supplied utility, then they have a legal course of action. The
customer''s system administrators are covered because they were using
tools provided by the vendor. The wrath of the customer would be upon Sun, not
the supplier (us) or the supplier''s technical lead (me).
Do you really believe that Sun will help such a customer?
There are many bugs in Solaris (I remember e.g. some showstopper
bugs in the multimedia area) that are not fixed although they are known
since a very long time (more than a year).

There is a bug in ACL handling in Sun''s tar (reported by me in 2004 or
even
before) that is not fixed. As a result in many cases ACLs are not restored.

Note that bugs in star are fixed much faster and looking back at the 28 years
of history with star, I know of not a single bug that took more than 3 months
to get a fix. Typically, bugs are fixed withing less than a week - many bugs
even within a few hours. This is a support quality that Sun does not offer.

So please explain us where you see a problem with star......

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

>> While I am sure that star is technically a fine utility, the problem is
that it is effectively an unsupported product.
>From this viewpoint, you may call most of Solaris "unsupported".
From the perspective of the business, the contract with Sun provides that
support.
>> If our customers find a bug in their backup that is caused by a failure
in a Sun supplied utility, then they have a legal course of action. The
>>customer''s system administrators are covered because they were
using tools provided by the vendor. The wrath of the customer would be upon
>>Sun, not the supplier (us) or the supplier''s technical lead
(me).
>Do you really believe that Sun will help such a customer?
>There are many bugs in Solaris (I remember e.g. some showstopper
>bugs in the multimedia area) that are not fixed although they are known
>since a very long time (more than a year).
>There is a bug in ACL handling in Sun''s tar (reported by me in 2004
or even
>before) that is not fixed. As a result in many cases ACLs are not restored.
If Sun don''t fix a critical bug that is affecting the availability of
server that is under support, then it becomes a problem for the legal
department. In the ACL example, it''s possible the effected users
didn''t have a support contract.
>Note that bugs in star are fixed much faster and looking back at the 28
years
>of history with star, I know of not a single bug that took more than 3
months
>to get a fix. Typically, bugs are fixed withing less than a week - many bugs
>even within a few hours. This is a support quality that Sun does not offer.
Possibly, but there is no guarantee that it will be fixed, no-one to call when
there is a problem, no-one to escalate the problem to if it is ignored, and no
company to sue if it all goes wrong.
>So please explain us where you see a problem with star......
Hopefully my above comments explain sufficiently. It''s not a technical
issue with star, it''s a business issue. The rules there are very
different and not based on merit (this is also why many companies prefer running
their mission critical apps on Red Hat Enterprise Linux instead of CentOS, even
though technically they are almost identical).

JR



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/8b634eb9/attachment.html>

Julian Regel <jrmailgate-zfsdiscuss at yahoo.co.uk> wrote:
> >> While I am sure that star is technically a fine utility, the
problem is that it is effectively an unsupported product.
>
> >From this viewpoint, you may call most of Solaris
"unsupported".
>
> From the perspective of the business, the contract with Sun provides that
support.
>From a perspective of reality, such a contract will not help.
> >Do you really believe that Sun will help such a customer?
> >There are many bugs in Solaris (I remember e.g. some showstopper
> >bugs in the multimedia area) that are not fixed although they are known
> >since a very long time (more than a year).
> >There is a bug in ACL handling in Sun''s tar (reported by me in
2004 or even
> >before) that is not fixed. As a result in many cases ACLs are not
restored.
>
> If Sun don''t fix a critical bug that is affecting the availability
of
> server that is under support, then it becomes a problem for the legal
> department. In the ACL example, it''s possible the effected users
didn''t have a support contract.
What you seem to point out is that in case of a problem for a customer with a 
contract, the legal department gets involved. Unfortunately, laywers do not fix 
bugs.....
> >Note that bugs in star are fixed much faster and looking back at the 28
years
> >of history with star, I know of not a single bug that took more than 3
months
> >to get a fix. Typically, bugs are fixed withing less than a week - many
bugs
> >even within a few hours. This is a support quality that Sun does not
offer.
>
> Possibly, but there is no guarantee that it will be fixed, no-one to call
when there is a problem, no-one to escalate the problem to if it is ignored, and
no company to sue if it all goes wrong.
Escalating a problem does not fix it. 
> >So please explain us where you see a problem with star......
>
> Hopefully my above comments explain sufficiently. It''s not a
technical issue with star, it''s a business issue. The rules there are
very different and not based on merit (this is also why many companies prefer
running their mission critical apps on Red Hat Enterprise Linux instead of
CentOS, even though technically they are almost identical).
Now we are back to reality.

A person that is interested in a solution will usually check what happened
in similar cases before. If you compare star with Sun supplied tools with
this background, Sun cannot outperform star.

"Red Hat Enterprise Linux" may offer something you cannot get with
CentOS.
But I don''t see that Sun can offer something you don''t get
with star.

Let me make another reality check:

Many people use GNU tar for backup purposes, but my first automated test case
with incremental backups using GNU tar did fail so miserably that I was unable
to use GNU tar as a test reference at all.

On the other side, I am doing incremental backup _and_ restore tests with 
gigabytes of real delta data on a dayly base since 2004 and I did hot see any 
problem since April 2005.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Joerg Schilling wrote:
> Julian Regel <jrmailgate-zfsdiscuss at yahoo.co.uk> wrote:
> 
>>> If you like to have a backup that allows to access files, you need
a file based
>>> backup and I am sure that even a filesystem level scan for recently
changed
>>> files will not be much faster than what you may achive with e.g.
star.
>>>
>>> Note that ufsdump directly accesees the raw disk device and thus
_is_ at
>>> filesystem leven but still is slower than star on UFS.
>> While I am sure that star is technically a fine utility, the problem is
that it is effectively an unsupported product.
> 
> From this viewpoint, you may call most of Solaris "unsupported".
what is that supposed to mean?

Michael
-- 
Michael Schuster        http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

On 19/01/2010 19:11, Ian Collins wrote:
> Julian Regel wrote:
>>
>> Based on what I''ve seen in other comments, you might be right.
>> Unfortunately, I don''t feel comfortable backing up ZFS
filesystems
>> because the tools aren''t there to do it (built into the
operating
>> system or using Zmanda/Amanda).
>>
> Commercial backup solutions are available for ZFS.
>> I know tape backup isn''t sexy, but it''s a reality for
many of us and
>> it''s not going away anytime soon.
>>
> True, but I wonder how viable its future is.  One of my clients 
> requires 17 LT04 types for a full backup, which cost more and takes up 
> more space than the equivalent in removable hard drives.
>
> In the past few years growth in hard drive capacities has outstripped 
> tapes to the extent that removable hard drives and ZFS snapshots have 
> become a more cost effective and convenient backup media.
>
> What do people with many tens of TB use for backup these days?
>
http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

-- 
Robert Milkowski
http://milek.blogspot.com

On 20/01/2010 10:48, Ragnar Sundblad wrote:
> On 19 jan 2010, at 20.11, Ian Collins wrote:
>
>    
>> Julian Regel wrote:
>>      
>>> Based on what I''ve seen in other comments, you might be
right. Unfortunately, I don''t feel comfortable backing up ZFS
filesystems because the tools aren''t there to do it (built into the
operating system or using Zmanda/Amanda).
>>>
>>>        
>> Commercial backup solutions are available for ZFS.
>>      
>>> I know tape backup isn''t sexy, but it''s a reality
for many of us and it''s not going away anytime soon.
>>>
>>>        
>> True, but I wonder how viable its future is.  One of my clients
requires 17 LT04 types for a full backup, which cost more and takes up more
space than the equivalent in removable hard drives.
>>
>> In the past few years growth in hard drive capacities has outstripped
tapes to the extent that removable hard drives and ZFS snapshots have become a
more cost effective and convenient backup media.
>>      
> LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or
so. LTO drivers cost a little, but so do disk shelves. So, now that there is no
big price issue, there is choice instead. Use it!
>
> Hard drives are good for random access - both restore of individual files
and partial rewrite.
>
> Hard drivers aren''t faster than tape for data transfer, but they
might be cheaper to run in parallel and therefore you could potentially gain
speed. Hard drives have shorter seek time, which may be important.
>
> Hard drives are probably bad for storing for longer times - especially -
you will never know how long it could be stored before it will fail. A month?
Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes
are supposed to be able to keep it''s data for at least 30 years of
stored properly. Hard drives are probably best when used online or at least very
often.
>
> So - it is wrong to say that one is better or cheaper than the other. They
have different properties, and could be used to solve different problems.
>
>
>    
It is actually not that easy.

Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.

Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare 
+ 2x OS disks.
The four raidz2 group form a single pool. This would provide well over 
30TB of logical storage per each box.

Now you rsync all the data from your clients to a dedicated filesystem 
per client, then create a snapshot.
All snapshots are replicated to a 2nd x4540 so even if you would loose 
entire box/data for some reason you would still have a spare copy.

Now compare it to a cost of a library, lto drives, tapes, software + 
licenses, support costs, ...

See more details at 
http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

-- 
Robert Milkowski
http://milek.blogspot.com

On 20/01/2010 11:26, Joerg Schilling wrote:
> Edward Ned Harvey<solaris at nedharvey.com>  wrote:
>
>    
>>> Star implements this in a very effective way (by using libfind)
that is
>>> even
>>> faster that the find(1) implementation from Sun.
>>>        
>> Even if I just "find" my filesystem, it will run for 7 hours.
But zfs can
>> create my whole incremental snapshot in a minute or two.  There is no
way
>> star or any other user-space utility that walks the filesystem can come
>> remotely close to this performance.  Such performance can only be
>> implemented at the filesystem level, or lower.
>>      
> You claim that it is fast for you but this is because it is block oriented
and
> because you probably changed only few data.
>
> If you like to have a backup that allows to access files, you need a file
based
> backup and I am sure that even a filesystem level scan for recently changed
> files will not be much faster than what you may achive with e.g. star.
>
>    
Not really - I mean if you do an incremental zfs send and on the other 
side receive it so a filesystem/snapshot is created then you have an 
access to each individual file while having the benefit of doing block 
level incremental zfs send which in many environments will be waaaaay 
faster then rsync/start/...

-- 
Robert Milkowski
http://milek.blogspot.com

On Wed, January 20, 2010 09:23, Robert Milkowski wrote:
> Now you rsync all the data from your clients to a dedicated filesystem
> per client, then create a snapshot.

Is there an rsync out there that can reliably replicate all file
characteristics between two ZFS/Solaris systems?  I haven''t found one. 
The ZFS ACLs seem to be beyond all of them, in particular.

(Losing just that, and preserving the data, is clearly far, far better
than losing everything!  And a system build *knowing* it was losing the
protections could preserve them some other way.)

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Wed, January 20, 2010 04:48, Ragnar Sundblad wrote:
> LTO media is still cheaper than equivalent sized disks, maybe a factor 5
> or so. LTO drivers cost a little, but so do disk shelves. So, now that
> there is no big price issue, there is choice instead. Use it!
Depends on the scale you''re operating at.

Backing up my 800GB home data pool onto a couple of external 1TB USB
drives is *immensely* cheaper than buying tape equipment.

At enough bigger scales, I accept that tape is still cheaper.  Makes
sense, since the tapes are relatively simple compared to drives, and you
only need a small number of drives to use a large number of tapes.

I think hard drives are still cheaper at small-enterprise levels, actually.

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Wed, 20 Jan 2010, Julian Regel wrote:
> 
> If our customers find a bug in their backup that is caused by a 
> failure in a Sun supplied utility, then they have a legal course of 
> action. The customer''s system administrators are covered because 
> they were using tools provided by the vendor. The wrath of the 
> customer would be upon Sun, not the supplier (us) or the
supplier''s
> technical lead (me).
I would love to try whatever you are smoking because it must be really 
good stuff.  It would be a bold new step for me, but the benefits are 
clear.

While your notions of the transitive protection offered by vendor 
"support" are interesting, I will be glad to meet you in the 
unemployment line then we can share some coffee and discuss the good 
old days.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

>It is actually not that easy.
>
>Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
>
>Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare 
>+ 2x OS disks.
>The four raidz2 group form a single pool. This would provide well over 
>30TB of logical storage per each box.
>
>Now you rsync all the data from your clients to a dedicated filesystem 
>per client, then create a snapshot.
>All snapshots are replicated to a 2nd x4540 so even if you would loose 
>entire box/data for some reason you would still have a spare copy.
>
>Now compare it to a cost of a library, lto drives, tapes, software + 
>licenses, support costs, ...
>
>See more details at 
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
I''ve just read your presentation Robert. Interesting stuff.

I''ve also just done a pen and paper exercise to see how much 30TB of
tape would cost as a comparison to your disk based solution.

Using list prices from Sun''s website (and who pays list..?), an SL48
with 2 x LTO3 drives would cost ?14000. I couldn''t see a price on an
LTO4 equipped SL48 despite the Sun website saying it''s a supported
option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48
tapes in the library (14.4TB native per library). To match the 30TB in your
solution, we''d need two libraries totalling ?28000.

You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently
bought a pack of 20 tapes for ?340, so five packs would be ?1700.


So you could provision a tape backup for just under ?30000 (~$49000). In
comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price
?30900. I''ve not factored in backup software since you could use an
open source solution such as Amanda or Bacula.

Which isn''t to say tape would be a "better" solution since
it''s going to be slower to restore etc. But it does show that tape can
work out cheaper, especially since the cost of a high speed WAN link
isn''t required.

JR



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/87d07c43/attachment.html>

>>>>> "ae" == Allen Eastwood <mixal at
paconet.us> writes:
>>>>> "ic" == Ian Collins <ian at ianshome.com>
writes:
     >> If people are really still backing up to tapes or DVD''s,
just
     >> use file vdev''s, export the pool, and then copy the
unmounted
     >> vdev onto the tape or DVD.

    ae> And some of those enterprises require backup mechanism that
    ae> can be easily used in a DR situation.

    ae> ufsdump/restore was perfect in that regard.  The lack of
    ae> equivalent functionality is a big problem for the situations
    ae> where this functionality is a business requirement.

    ae> For example, one customer, local government, requires a backup
    ae> that can be taken offsite and used in a DR situation.

Were you confused by some part of:

 Use file vdevs, epxort the pool, and then copy the unmounted vdev
 onto the tape.

or do you find that this doesn''t do what you want?  because it seems
fine to me.  And the fact that it doesn''t need any extra tools means
it''s unlikely to break (1) far into the future or (2) for a few
unlucky builds, and (3) that the restore environment is simple and
doesn''t involve prepopulating some Legato Database with the TOC of
every tape in the library or some such nonsense, which ought to all be
among your ``requirements'''' but if you''re
substituting for those,
``works in exactly the way we were used to it working before''''
then
you may as well use ''zfs send'' since you''re more
concerned with
identical-feeling invocatoin syntax than the problems I mentioned.

    ic> For a full recovery, you can archive a send stream and receive
    ic> it back.

You can send the stream to the tape, transport the tape to the DR
site, and receive it.  You can do this weekly as part of your offsite
backup plan provided that you receive each tape you transport
immediately.  Then the data should be permanently stored on disk at
the DR site, and the tapes used only for transport.

If you store the backup permanently on tape then it''s a step backwards
from tar/cpio/ufsrestore because the ''zfs send'' format is more
fragile
and has to be restored entire.  If you receive the tape immediately
this is an improvement because under the old convention tapes could be
damaged in transit, or over the years by heat/dust/sunlight, without
your knowledge, while on disks it''s simple to scrub periodically.

I am not trying to take away your tapes, Allen, so please quote Ian
instead if that''s the thing you object to.  I''ve instead
suggested a
different way to use them if you really do need them archivally: store
file vdev''s on them.  If you''re just using them to replicate
data to
the DR site then you needn''t even go as far as my workaround.

I do agree that there''s a missing tool: it''s not possible to
copy one
subdirectory to another while preserving holes, forkey extended
attributes, and ACL''s.  Also if Windows ACL''s are going to be
stored
right in the filesystem, then Windows ACL''s probably ought to be
preserved over an rsync pipe between Solaris and EnTee, or a
futuristic tarball written on one and extracted on the other.  I don''t
agree that the missing tool is designed primarily for the narrow
use-case of writing to ancient backup tapes: it''s a more general tool.
or, really, it''s just a matter of documenting and committing the
extra-OOB-gunk APIs and then fixing rsync and GNUtar.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/2d122dd6/attachment.bin>

On 20/01/2010 16:22, Julian Regel wrote:
> >It is actually not that easy.
> >
> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on
LTO.
> >
> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot
spare
> >+ 2x OS disks.
> >The four raidz2 group form a single pool. This would provide well over
> >30TB of logical storage per each box.
> >
> >Now you rsync all the data from your clients to a dedicated filesystem
> >per client, then create a snapshot.
> >All snapshots are replicated to a 2nd x4540 so even if you would loose
> >entire box/data for some reason you would still have a spare copy.
> >
> >Now compare it to a cost of a library, lto drives, tapes, software +
> >licenses, support costs, ...
> >
> >See more details at
> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>
> I''ve just read your presentation Robert. Interesting stuff.
>
> I''ve also just done a pen and paper exercise to see how much 30TB
of
> tape would cost as a comparison to your disk based solution.
>
> Using list prices from Sun''s website (and who pays list..?), an
SL48
> with 2 x LTO3 drives would cost ?14000. I couldn''t see a price on
an
> LTO4 equipped SL48 despite the Sun website saying it''s a supported
> option. Each LTO3 has a native capacity of 300GB and the SL48 can hold 
> up to 48 tapes in the library (14.4TB native per library). To match 
> the 30TB in your solution, we''d need two libraries totalling
?28000.
>
> You would also need 100 LTO3 tapes to provide 30TB of native storage. 
> I recently bought a pack of 20 tapes for ?340, so five packs would be 
> ?1700.
>
> So you could provision a tape backup for just under ?30000 (~$49000). 
> In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
> list price ?30900. I''ve not factored in backup software since you 
> could use an open source solution such as Amanda or Bacula.
>
> Which isn''t to say tape would be a "better" solution
since it''s going
> to be slower to restore etc. But it does show that tape can work out 
> cheaper, especially since the cost of a high speed WAN link isn''t 
> required.
>
> JR
>
You would also need to add at least one server to your library with fc 
cards.
Then with most software you would need more tapes due to data 
fragmentation and a need to do regular full backups (with zfs+rsync you 
only do a full backup once).

So in best case a library will cost about the same as disk based 
solution but generally will be less flexible, etc. If you would add any 
enterprise software on top of it (Legato, NetBackup, ...) then the price 
would change dramaticallly. Additionally with ZFS one could start using 
deduplication (in testing already).


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/389b7f58/attachment.html>

On 20/01/2010 17:21, Robert Milkowski wrote:
> On 20/01/2010 16:22, Julian Regel wrote:
>> >It is actually not that easy.
>> >
>> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on
LTO.
>> >
>> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot 
>> spare
>> >+ 2x OS disks.
>> >The four raidz2 group form a single pool. This would provide well
over
>> >30TB of logical storage per each box.
>> >
>> >Now you rsync all the data from your clients to a dedicated
filesystem
>> >per client, then create a snapshot.
>> >All snapshots are replicated to a 2nd x4540 so even if you would
loose
>> >entire box/data for some reason you would still have a spare copy.
>> >
>> >Now compare it to a cost of a library, lto drives, tapes, software
+
>> >licenses, support costs, ...
>> >
>> >See more details at
>> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>>
>> I''ve just read your presentation Robert. Interesting stuff.
>>
>> I''ve also just done a pen and paper exercise to see how much
30TB of
>> tape would cost as a comparison to your disk based solution.
>>
>> Using list prices from Sun''s website (and who pays list..?),
an SL48
>> with 2 x LTO3 drives would cost ?14000. I couldn''t see a price
on an
>> LTO4 equipped SL48 despite the Sun website saying it''s a
supported
>> option. Each LTO3 has a native capacity of 300GB and the SL48 can 
>> hold up to 48 tapes in the library (14.4TB native per library). To 
>> match the 30TB in your solution, we''d need two libraries
totalling
>> ?28000.
>>
>> You would also need 100 LTO3 tapes to provide 30TB of native storage. 
>> I recently bought a pack of 20 tapes for ?340, so five packs would be 
>> ?1700.
>>
>> So you could provision a tape backup for just under ?30000 (~$49000). 
>> In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
>> list price ?30900. I''ve not factored in backup software since
you
>> could use an open source solution such as Amanda or Bacula.
>>
>> Which isn''t to say tape would be a "better" solution
since it''s going
>> to be slower to restore etc. But it does show that tape can work out 
>> cheaper, especially since the cost of a high speed WAN link
isn''t
>> required.
>>
>> JR
>>
> You would also need to add at least one server to your library with fc 
> cards.
> Then with most software you would need more tapes due to data 
> fragmentation and a need to do regular full backups (with zfs+rsync 
> you only do a full backup once).
>
> So in best case a library will cost about the same as disk based 
> solution but generally will be less flexible, etc. If you would add 
> any enterprise software on top of it (Legato, NetBackup, ...) then the 
> price would change dramaticallly. Additionally with ZFS one could 
> start using deduplication (in testing already).
>
>
What I really mean is that a disk based solution used to be much more 
expensive than tapes but currently they are comparable in costs while 
often the disk based is more flexible.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/77a98b66/attachment.html>

On Jan 20, 2010, at 3:15 AM, Joerg Schilling wrote:
> Richard Elling <richard.elling at gmail.com> wrote:
> 
>>> 
>>> ufsdump/restore was perfect in that regard.  The lack of equivalent
functionality is a big problem for the situations where this functionality is a
business requirement.
>> 
>> How quickly we forget ufsdump''s limitations :-).  For example,
it is not supported
>> for use on an active file system (known data corruption possibility)
and
>> UFS snapshots are, well, a poor hack and often not usable for backups.
>> As the ufsdump(1m) manpage says,
> 
> It seems you forgot that zfs also needs snapshots. There is nothing bad
with
> snapshots.
Yes, snapshots are a good thing. But most people who try fssnap 
on the UFS root file system will discover that it doesn''t work; for 
reasons mentioned in the NOTES section of fssnap_ufs(1m). 
fssnap_ufs is simply a butt-ugly hack. So if you believe you can
reliably use ufsdump to store a DR copy of root for a 7x24x365 
production environment, then you probably believe the Backup
Fairy will leave a coin under your pillow when your restore fails :-)

Fortunately, ZFS snapshot do the right thing.
 -- richard

Julian Regel wrote:
> >It is actually not that easy.
> >
> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on
LTO.
> >
> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot
spare
> >+ 2x OS disks.
> >The four raidz2 group form a single pool. This would provide well over
> >30TB of logical storage per each box.
> >
> >Now you rsync all the data from your clients to a dedicated filesystem
> >per client, then create a snapshot.
> >All snapshots are replicated to a 2nd x4540 so even if you would loose
> >entire box/data for some reason you would still have a spare copy.
> >
> >Now compare it to a cost of a library, lto drives, tapes, software +
> >licenses, support costs, ...
> >
> >See more details at
> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>
> I''ve just read your presentation Robert. Interesting stuff.
>
> I''ve also just done a pen and paper exercise to see how much 30TB
of
> tape would cost as a comparison to your disk based solution.
>
> Using list prices from Sun''s website (and who pays list..?), an
SL48
> with 2 x LTO3 drives would cost ?14000. I couldn''t see a price on
an
> LTO4 equipped SL48 despite the Sun website saying it''s a supported
> option. Each LTO3 has a native capacity of 300GB and the SL48 can hold 
> up to 48 tapes in the library (14.4TB native per library). To match 
> the 30TB in your solution, we''d need two libraries totalling
?28000.
>
> You would also need 100 LTO3 tapes to provide 30TB of native storage. 
> I recently bought a pack of 20 tapes for ?340, so five packs would be 
> ?1700.
>
> So you could provision a tape backup for just under ?30000 (~$49000). 
> In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
> list price ?30900. I''ve not factored in backup software since you 
> could use an open source solution such as Amanda or Bacula.
>
A more apples to apples comparison would be to compare the storage 
only.  Both removable drive and tape options require a server with FC or 
SCSI ports, so that can be excluded from the comparison.

So for 30TB, assuming 2TB drives @ ~?100 with a pool built of 6 drive 
raidz vdevs 18 drives would be required plus 2 16 drive shelves.  So 
each backup set would cost about ?1800.  So there''s not a great deal of
difference.  With drives you also get the added benefit of keeping all 
your incrementals (as snapshots) on the archive set.

HDD price per GB will continue to drop faster than tape, so it will be 
interesting to do the same comparison in 12 months.

-- 
Ian.

Joerg Schilling wrote:
> Ian Collins <ian at ianshome.com> wrote:
>
>   
>>> The correct way to archivbe ACLs would be to put them into extended
POSIX tar
>>> attrubutes as star does.
>>>
>>> See http://cdrecord.berlios.de/private/man/star/star.4.html for a
format
>>> documentation or have a look at
ftp://ftp.berlios.de/pub/star/alpha, e.g.
>>> ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz
>>>
>>> The ACL format used by Sun is undocumented......
>>>
>>>   
>>>       
>> man acltotext
>>     
>
> We are talking about TAR and I did give a pointer to the star archive
format
> documentation, so it is obvious that I was talking about the ACL format
from
> Sun tar. This format is not documented.
>
>   
It is, Sun''s ZFS ACL aware tools use acltotext() to format ACLs.

-- 
Ian.

>>>>> "jr" == Julian Regel <jrmailgate-zfsdiscuss at
yahoo.co.uk> writes:
    jr> While I am sure that star is technically a fine utility, the
    jr> problem is that it is effectively an unsupported product.

I have no problems with this whatsoever.

    jr> If our customers find a bug in their backup that is caused by
    jr> a failure in a Sun supplied utility, then they have a legal
    jr> course of action. The customer''s system administrators are
    jr> covered because they were using tools provided by the
    jr> vendor. The wrath of the customer would be upon Sun, not the
    jr> supplier (us) or the supplier''s technical lead (me).

We were just talking about this somewhere else, actually: ``if
something goes wrong, its their ass. but if nothing ever gets done,
its nobody''s fault.''''  It''s sad for me how
much money is to be made
supporting broken corporate cultures like that.

I''m not saying you''re wrong, just that you might not want to
contribute to such a culture because you''ve chosen to endure it for a
scratch.  You need to have a better way to evaluate employees than
micromanagement-by-the-clueless and vindictive hindsight.  But the
point that there''s money to be made by bleeding it out of ossified
broken American companies is well-taken.

    jr> From the perspective of the business, the system administrator
    jr> will have acted irresponsibly by choosing a tool that has no
    jr> vendor support.

From the perspective of MY business, I would much rather have the dark
OOB acl/fork/whatever-magic that''s gone into ZFS and NFSv4 supported
in standard tools like rsync and GNUtar.  This is, for example, what
Apple achieved with CUPS and why I can share printers between Ubuntu
and Mac OS effortlessly, and this increases the amount of money I''m
willing to give Apple for their proprietary platform.  The purpose of
the tool I''m discussing definitely includes the same level of
cooperation, so working with the existing best-in-class and
most-popular tools, and reasonableness, might be better than brittle
CYA support in some fringey ''/opt/SUNWbkpkit/bin/VendorCP -Rf''
tool.

Even if you get your cyaCP tool you may find it doesn''t achieve the
ass-covering you wanted because these tools can be cheeky little
bastards.  Most of the other quirky little balkanized-platform
Solaris-only tools are littered with straightjacketing assertions to
avoid ``call generators'''' and push the blame back onto the
sysadmin,
then there is some ``all bets are off'''' flag to allowe you to
actually
accomplish job, like ''NOINUSECHECK=1 format -e''.  

Honestly...why bother playing this game?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100120/68babb2c/attachment.bin>

On Jan 20, 2010, at 12:21, Robert Milkowski wrote:
> On 20/01/2010 16:22, Julian Regel wrote:
>>
> [...]
>> So you could provision a tape backup for just under ?30000 (~ 
>> $49000). In comparison, the cost of one X4540 with ~ 36TB usable  
>> storage is UK list price ?30900. I''ve not factored in backup  
>> software since you could use an open source solution such as Amanda  
>> or Bacula.
> [...]
> You would also need to add at least one server to your library with  
> fc cards.
> Then with most software you would need more tapes due to data  
> fragmentation and a need to do regular full backups (with zfs+rsync  
> you only do a full backup once).
>
> So in best case a library will cost about the same as disk based  
> solution but generally will be less flexible, etc. If you would add  
> any enterprise software on top of it (Legato, NetBackup, ...) then  
> the price would change dramaticallly. Additionally with ZFS one  
> could start using deduplication (in testing already).
Regardless of the economics of tape, nowadays you generally need to go  
to disk first because trying to stream at 120 MB/s (LTO-4) really  
isn''t practical over the network, directly from the client.

So in the end you''ll be starting with disk (either DAS or VTL or  
whatever), and generally going to tape if you need to keep stuff  
that''s older than (say) 3-6 months. Tape also doesn''t rotate
while
it''s sitting there, so if it''s going to be sitting around for
a while
(e.g., seven years) better to use tape than something that sucks up  
power.

LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB  
and (uncompressed) writes at 180 MB/s. The only way to realistically  
feed that is from disk.

Ian Collins <ian at ianshome.com> wrote:
> > We are talking about TAR and I did give a pointer to the star archive
format
> > documentation, so it is obvious that I was talking about the ACL
format from
> > Sun tar. This format is not documented.
> >
> >   
> It is, Sun''s ZFS ACL aware tools use acltotext() to format ACLs.
Please don''t reply without checking facts.

The fact that you know that there is salt in the soup does not give
you the whole list of ingredients.... Please look into the Sun tar format
to understand that you are wrong.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Miles Nordin <carton at Ivy.NET> wrote:
> From the perspective of MY business, I would much rather have the dark
> OOB acl/fork/whatever-magic that''s gone into ZFS and NFSv4
supported
> in standard tools like rsync and GNUtar.  This is, for example, what
GNU tar does not support any platform speficic feature on any OS.
Don''t expect that GNU tar will ever add such properties......

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Wed, Jan 20, 2010 at 2:52 PM, David Magda <dmagda at ee.ryerson.ca>
wrote:
>
> On Jan 20, 2010, at 12:21, Robert Milkowski wrote:
>
>> On 20/01/2010 16:22, Julian Regel wrote:
>>>
>> [...]
>>>
>>> So you could provision a tape backup for just under ?30000
(~$49000). In
>>> comparison, the cost of one X4540 with ~ 36TB usable storage is UK
list
>>> price ?30900. I''ve not factored in backup software since
you could use an
>>> open source solution such as Amanda or Bacula.
>>
>> [...]
>> You would also need to add at least one server to your library with fc
>> cards.
>> Then with most software you would need more tapes due to data
>> fragmentation and a need to do regular full backups (with zfs+rsync you
only
>> do a full backup once).
>>
>> So in best case a library will cost about the same as disk based
solution
>> but generally will be less flexible, etc. If you would add any
enterprise
>> software on top of it (Legato, NetBackup, ...) then the price would
change
>> dramaticallly. Additionally with ZFS one could start using
deduplication (in
>> testing already).
>
> Regardless of the economics of tape, nowadays you generally need to go to
> disk first because trying to stream at 120 MB/s (LTO-4) really
isn''t
> practical over the network, directly from the client.
I remember for about 5 years ago (before LT0-4 days) that streaming
tape drives would go to great lengths to ensure that the drive kept
streaming - because it took so much time to stop, backup and stream
again.  And one way the drive firmware accomplished that was to write
blocks of zeros when there was no data available.  This also occurred
when the backup source was sending a bunch of small files, which took
longer to stream and did''nt produce enough data to keep the drive
writing useful data.  And if you had the tape hardware setup to do
compression, then, assuming a normal 2:1 compression ratio, you''d need
to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec.  The
net result was the consumption of a lot more tape than a
back-of-the-napkin calculation told you was required.

Obviously at higher compression ratios or with the higher stream data
write rates you quote below - this problem becomes more troublesome.
So I agree with your conclusion: "The only way to realistically feed
that is from disk."
> So in the end you''ll be starting with disk (either DAS or VTL or
whatever),
> and generally going to tape if you need to keep stuff that''s older
than
> (say) 3-6 months. Tape also doesn''t rotate while it''s
sitting there, so if
> it''s going to be sitting around for a while (e.g., seven years)
better to
> use tape than something that sucks up power.
>
> LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB and
> (uncompressed) writes at 180 MB/s. The only way to realistically feed that
> is from disk.
>
> _______________________________________________
Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX al at logical-approach.com
                   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/

On 20 jan 2010, at 17.22, Julian Regel wrote:
> >It is actually not that easy.
> >
> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on
LTO.
> >
> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot
spare
> >+ 2x OS disks.
> >The four raidz2 group form a single pool. This would provide well over 
> >30TB of logical storage per each box.
> >
> >Now you rsync all the data from your clients to a dedicated filesystem 
> >per client, then create a snapshot.
> >All snapshots are replicated to a 2nd x4540 so even if you would loose 
> >entire box/data for some reason you would still have a spare copy.
> >
> >Now compare it to a cost of a library, lto drives, tapes, software + 
> >licenses, support costs, ...
> >
> >See more details at 
> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
> 
> I''ve just read your presentation Robert. Interesting stuff.
> 
> I''ve also just done a pen and paper exercise to see how much 30TB
of tape would cost as a comparison to your disk based solution.
> 
> Using list prices from Sun''s website (and who pays list..?), an
SL48 with 2 x LTO3 drives would cost ?14000. I couldn''t see a price on
an LTO4 equipped SL48 despite the Sun website saying it''s a supported
option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48
tapes in the library (14.4TB native per library). To match the 30TB in your
solution, we''d need two libraries totalling ?28000.
LTO3 has native capacity 400 GB, LTO4 has 800. The price is about the same per
tape and per drive, a little higher for LTO4.
> You would also need 100 LTO3 tapes to provide 30TB of native storage. I
recently bought a pack of 20 tapes for ?340, so five packs would be ?1700.
Or rather 37 LTO4 tapes, and only one 48 tape library. But that doesn''t
matter, the interesting part is that one now can use whatever best solves the
problem at hand.
> So you could provision a tape backup for just under ?30000 (~$49000). In
comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price
?30900. I''ve not factored in backup software since you could use an
open source solution such as Amanda or Bacula.
> 
> Which isn''t to say tape would be a "better" solution
since it''s going to be slower to restore etc. But it does show that
tape can work out cheaper, especially since the cost of a high speed WAN link
isn''t required.
Reading from tape is normally faster than reading from (a single) disk. Seek
time of course isn''t.

/ragge

On 21 jan 2010, at 00.20, Al Hopper wrote:
> I remember for about 5 years ago (before LT0-4 days) that streaming
> tape drives would go to great lengths to ensure that the drive kept
> streaming - because it took so much time to stop, backup and stream
> again.  And one way the drive firmware accomplished that was to write
> blocks of zeros when there was no data available.  This also occurred
> when the backup source was sending a bunch of small files, which took
> longer to stream and did''nt produce enough data to keep the drive
> writing useful data.  And if you had the tape hardware setup to do
> compression, then, assuming a normal 2:1 compression ratio, you''d
need
> to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec.  The
> net result was the consumption of a lot more tape than a
> back-of-the-napkin calculation told you was required.
> 
> Obviously at higher compression ratios or with the higher stream data
> write rates you quote below - this problem becomes more troublesome.
> So I agree with your conclusion: "The only way to realistically feed
> that is from disk."
Yes! Modern LTO drives can typically vary their speed about a factor four
or so, so even if you can''t keep up with the tape drive maximum speed,
it will typically work pretty good anyway. If you can''t keep up even
then,
it will have to stop, back up a bit, and restart, which will be _very_
slow. Having a disk system deliver data at 240 MB/s at the same time
as you are writing to it can be a bit of a challenge. 

I haven''t seen drives that fill out with zeros. Sounds like an ugly
solution, but maybe it could be useful in some strange case.

/ragge s

Ragnar Sundblad <ragge at csc.kth.se> wrote:
> Yes! Modern LTO drives can typically vary their speed about a factor four
> or so, so even if you can''t keep up with the tape drive maximum
speed,
> it will typically work pretty good anyway. If you can''t keep up
even then,
> it will have to stop, back up a bit, and restart, which will be _very_
> slow. Having a disk system deliver data at 240 MB/s at the same time
> as you are writing to it can be a bit of a challenge. 
And star implements a FIFO that is written in a way that dramatically reduces 
the sawtooth behavior seen typically with other applications. You just need to 
tell star to use a say 2 GB FIFO and star will be able to keep the tame 
streaming for a longer time before it waits until there is enough data for the 
next longer streaming period.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On 20/01/2010 15:45, David Dyer-Bennet wrote:
> On Wed, January 20, 2010 09:23, Robert Milkowski wrote:
>
>    
>> Now you rsync all the data from your clients to a dedicated filesystem
>> per client, then create a snapshot.
>>      
>
> Is there an rsync out there that can reliably replicate all file
> characteristics between two ZFS/Solaris systems?  I haven''t found
one.
> The ZFS ACLs seem to be beyond all of them, in particular.
>
>    
No it doesn''t support ZFS ACLs - fortunately it is not an issue for us.

On 20/01/2010 19:20, Ian Collins wrote:
> Julian Regel wrote:
>> >It is actually not that easy.
>> >
>> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on
>> LTO.
>> >
>> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot 
>> spare
>> >+ 2x OS disks.
>> >The four raidz2 group form a single pool. This would provide well
over
>> >30TB of logical storage per each box.
>> >
>> >Now you rsync all the data from your clients to a dedicated
filesystem
>> >per client, then create a snapshot.
>> >All snapshots are replicated to a 2nd x4540 so even if you would
loose
>> >entire box/data for some reason you would still have a spare copy.
>> >
>> >Now compare it to a cost of a library, lto drives, tapes, software
+
>> >licenses, support costs, ...
>> >
>> >See more details at
>> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>>
>> I''ve just read your presentation Robert. Interesting stuff.
>>
>> I''ve also just done a pen and paper exercise to see how much
30TB of
>> tape would cost as a comparison to your disk based solution.
>>
>> Using list prices from Sun''s website (and who pays list..?),
an SL48
>> with 2 x LTO3 drives would cost ?14000. I couldn''t see a price
on an
>> LTO4 equipped SL48 despite the Sun website saying it''s a
supported
>> option. Each LTO3 has a native capacity of 300GB and the SL48 can 
>> hold up to 48 tapes in the library (14.4TB native per library). To 
>> match the 30TB in your solution, we''d need two libraries
totalling
>> ?28000.
>>
>> You would also need 100 LTO3 tapes to provide 30TB of native storage. 
>> I recently bought a pack of 20 tapes for ?340, so five packs would be 
>> ?1700.
>>
>> So you could provision a tape backup for just under ?30000 (~$49000). 
>> In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
>> list price ?30900. I''ve not factored in backup software since
you
>> could use an open source solution such as Amanda or Bacula.
>>
> A more apples to apples comparison would be to compare the storage 
> only.  Both removable drive and tape options require a server with FC 
> or SCSI ports, so that can be excluded from the comparison.
>
>
>
I think one should actually compare whole solutions - including servers, 
fc infrastructure, tape drives, robots, software costs, rack space, ...

Servers like x4540 are ideal for zfs+rsync backup solution - very 
compact, good $/GB ratio, enough CPU power for its capacity, allow to 
easily scale it horizontally, and it is not too small and not too big. 
Then thanks to its compactness they are very easy to administer.

Depending on an anvironment one could deploy them always in paris - one 
in one datacenter and 2nd one in other datacanter with ZFS send based 
replication of all backups (snapshots). Or one may replicate 
(cross-replicate) only selected clients if needed.


-- 
Robert Milkowski
http://milek.blogspot.com

Robert Milkowski wrote:
> On 20/01/2010 19:20, Ian Collins wrote:
>> Julian Regel wrote:
>>> >It is actually not that easy.
>>> >
>>> >Compare a cost of 2x x4540 with 1TB disks to equivalent
solution on
>>> LTO.
>>> >
>>> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x
hot
>>> spare
>>> >+ 2x OS disks.
>>> >The four raidz2 group form a single pool. This would provide
well over
>>> >30TB of logical storage per each box.
>>> >
>>> >Now you rsync all the data from your clients to a dedicated
filesystem
>>> >per client, then create a snapshot.
>>> >All snapshots are replicated to a 2nd x4540 so even if you
would loose
>>> >entire box/data for some reason you would still have a spare
copy.
>>> >
>>> >Now compare it to a cost of a library, lto drives, tapes,
software +
>>> >licenses, support costs, ...
>>> >
>>> >See more details at
>>> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>>>
>>> I''ve just read your presentation Robert. Interesting
stuff.
>>>
>>> I''ve also just done a pen and paper exercise to see how
much 30TB of
>>> tape would cost as a comparison to your disk based solution.
>>>
>>> Using list prices from Sun''s website (and who pays
list..?), an SL48
>>> with 2 x LTO3 drives would cost ?14000. I couldn''t see a
price on an
>>> LTO4 equipped SL48 despite the Sun website saying it''s a
supported
>>> option. Each LTO3 has a native capacity of 300GB and the SL48 can 
>>> hold up to 48 tapes in the library (14.4TB native per library). To 
>>> match the 30TB in your solution, we''d need two libraries
totalling
>>> ?28000.
>>>
>>> You would also need 100 LTO3 tapes to provide 30TB of native 
>>> storage. I recently bought a pack of 20 tapes for ?340, so five 
>>> packs would be ?1700.
>>>
>>> So you could provision a tape backup for just under ?30000 
>>> (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable 
>>> storage is UK list price ?30900. I''ve not factored in
backup
>>> software since you could use an open source solution such as Amanda
>>> or Bacula.
>>>
>> A more apples to apples comparison would be to compare the storage 
>> only.  Both removable drive and tape options require a server with FC 
>> or SCSI ports, so that can be excluded from the comparison.
>>
>
> I think one should actually compare whole solutions - including 
> servers, fc infrastructure, tape drives, robots, software costs, rack 
> space, ...
>
> Servers like x4540 are ideal for zfs+rsync backup solution - very 
> compact, good $/GB ratio, enough CPU power for its capacity, allow to 
> easily scale it horizontally, and it is not too small and not too big. 
> Then thanks to its compactness they are very easy to administer.
Until you try to pick one up and put it in a fire safe!
> Depending on an anvironment one could deploy them always in paris - 
> one in one datacenter and 2nd one in other datacanter with ZFS send 
> based replication of all backups (snapshots). Or one may replicate 
> (cross-replicate) only selected clients if needed.
>
Yes, I agree.  That''s how my client''s systems are configured
(pairs).
We also have another with an attached tape library.

-- 
Ian.

Robert Milkowski wrote:
> I think one should actually compare whole solutions - including servers, 
> fc infrastructure, tape drives, robots, software costs, rack space, ...
> 
> Servers like x4540 are ideal for zfs+rsync backup solution - very 
> compact, good $/GB ratio, enough CPU power for its capacity, allow to 
> easily scale it horizontally, and it is not too small and not too big. 
> Then thanks to its compactness they are very easy to administer.
> 
> Depending on an anvironment one could deploy them always in paris - one 
> in one datacenter and 2nd one in other datacanter with ZFS send based 
> replication of all backups (snapshots). Or one may replicate 
> (cross-replicate) only selected clients if needed.
Something else that often sells the 4500/4540 relates to internal 
company politics. Often, inside a company storage has to be provisioned 
from the company''s storage group, using very expensive SAN based 
storage, indeed so expensive by the time the company''s storage group 
have added their overhead onto the already expensive SAN, that whole 
projects become unviable. Instead, teams find they can order
4500/4540''s
which slip under the radar as servers (or even PCs), and they now have 
affordable storage for their projects, which makes them viable once more.

-- 
Andrew

On 21/01/2010 09:07, Ian Collins wrote:
> Robert Milkowski wrote:
>> On 20/01/2010 19:20, Ian Collins wrote:
>>> Julian Regel wrote:
>>>> >It is actually not that easy.
>>>> >
>>>> >Compare a cost of 2x x4540 with 1TB disks to equivalent
solution
>>>> on LTO.
>>>> >
>>>> >Each x4540 could be configured as: 4x 11 disks in raidz-2 +
2x hot
>>>> spare
>>>> >+ 2x OS disks.
>>>> >The four raidz2 group form a single pool. This would
provide well
>>>> over
>>>> >30TB of logical storage per each box.
>>>> >
>>>> >Now you rsync all the data from your clients to a dedicated
>>>> filesystem
>>>> >per client, then create a snapshot.
>>>> >All snapshots are replicated to a 2nd x4540 so even if you
would
>>>> loose
>>>> >entire box/data for some reason you would still have a
spare copy.
>>>> >
>>>> >Now compare it to a cost of a library, lto drives, tapes,
software +
>>>> >licenses, support costs, ...
>>>> >
>>>> >See more details at
>>>>
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
>>>>
>>>> I''ve just read your presentation Robert. Interesting
stuff.
>>>>
>>>> I''ve also just done a pen and paper exercise to see
how much 30TB
>>>> of tape would cost as a comparison to your disk based solution.
>>>>
>>>> Using list prices from Sun''s website (and who pays
list..?), an
>>>> SL48 with 2 x LTO3 drives would cost ?14000. I
couldn''t see a price
>>>> on an LTO4 equipped SL48 despite the Sun website saying
it''s a
>>>> supported option. Each LTO3 has a native capacity of 300GB and
the
>>>> SL48 can hold up to 48 tapes in the library (14.4TB native per 
>>>> library). To match the 30TB in your solution, we''d
need two
>>>> libraries totalling ?28000.
>>>>
>>>> You would also need 100 LTO3 tapes to provide 30TB of native 
>>>> storage. I recently bought a pack of 20 tapes for ?340, so five
>>>> packs would be ?1700.
>>>>
>>>> So you could provision a tape backup for just under ?30000 
>>>> (~$49000). In comparison, the cost of one X4540 with ~ 36TB
usable
>>>> storage is UK list price ?30900. I''ve not factored in
backup
>>>> software since you could use an open source solution such as
Amanda
>>>> or Bacula.
>>>>
>>> A more apples to apples comparison would be to compare the storage 
>>> only.  Both removable drive and tape options require a server with 
>>> FC or SCSI ports, so that can be excluded from the comparison.
>>>
>>
>> I think one should actually compare whole solutions - including 
>> servers, fc infrastructure, tape drives, robots, software costs, rack 
>> space, ...
>>
>> Servers like x4540 are ideal for zfs+rsync backup solution - very 
>> compact, good $/GB ratio, enough CPU power for its capacity, allow to 
>> easily scale it horizontally, and it is not too small and not too 
>> big. Then thanks to its compactness they are very easy to administer.
>
> Until you try to pick one up and put it in a fire safe!
Then you backup to tape from x4540 whatever data you need.
In case of enterprise products you save on licensing here as you need a 
one client license per x4540 but in fact can backup data from many 
clients which are there.

:)

>> Until you try to pick one up and put it in a fire safe!
>Then you backup to tape from x4540 whatever data you need.
>In case of enterprise products you save on licensing here as you need a one
client license per x4540 but in fact can >backup data from many clients which
are there.
Which brings up full circle...

What do you then use to backup to tape bearing in mind that the Sun-provided
tools all have significant limitations?

I guess you need to use a third party tool and watch carefully that they provide
complete backups.

JR
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100121/bc66d0dc/attachment.html>

On Jan 21, 2010, at 3:55 AM, Julian Regel wrote:
> >> Until you try to pick one up and put it in a fire safe!
> 
> >Then you backup to tape from x4540 whatever data you need.
> >In case of enterprise products you save on licensing here as you need a
one client license per x4540 but in fact can >backup data from many clients
which are there.
> 
> Which brings up full circle...
> 
> What do you then use to backup to tape bearing in mind that the
Sun-provided tools all have significant limitations?
Poor choice of words.  Sun resells NetBackup and (IIRC) that which was 
formerly called NetWorker.  Thus, Sun does provide enterprise backup
solutions.

If I may put on my MBA hat, the competition is not ufsdump.  ufsdump has
nearly zero market penetration and no prospects for improving its market
share. Making another ufsdump will also gain no market share.  The market
leaders are the likes of EMC, IBM, and Symantec with their heterogenous
backup support. If Sun wanted to provide a better solution that might gain
market share against the others, then it would also need to be heterogenous.
So I think it would be hard to make a business case for a whole new backup
solution. A less costly and less risky approach is to work with the market 
leaders to better integrate with dataset replication.  Caveat: this may already
be available, I haven''t looked recently.
> I guess you need to use a third party tool and watch carefully that they
provide complete backups.
This is a good idea anyway.
 -- richard

Julian Regel wrote:
>
> >> Until you try to pick one up and put it in a fire safe!
>
> >Then you backup to tape from x4540 whatever data you need.
> >In case of enterprise products you save on licensing here as you need 
> a one client license per x4540 but in fact can >backup data from many 
> clients which are there.
>
> Which brings up full circle...
>
> What do you then use to backup to tape bearing in mind that the 
> Sun-provided tools all have significant limitations?
>
In addition to Richard''s comments, I doubt many medium to large 
businesses would use ufsdump/restore as their backup solution.

-- 
Ian.

Ian Collins <ian at ianshome.com> wrote:
> In addition to Richard''s comments, I doubt many medium to large 
> businesses would use ufsdump/restore as their backup solution.
ufsdump/restore is a reliable backup sulution as long as you install
a mirrored root fs (in case you are using UFS as root fs).
In fact many sites now use a basic backup bethod that is much less
reliable than ufsdump/restore. The fact that they use a high level front 
end does not necessarily remove limitations from the software that is used
as backend. I would never trust a backup sulution that uses e.g. GNU tar
as backend.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

On Thu, Jan 21, 2010 at 11:28 AM, Richard Elling
<richard.elling at gmail.com> wrote:
> On Jan 21, 2010, at 3:55 AM, Julian Regel wrote:
>> >> Until you try to pick one up and put it in a fire safe!
>>
>> >Then you backup to tape from x4540 whatever data you need.
>> >In case of enterprise products you save on licensing here as you
need a one client license per x4540 but in fact can >backup data from many
clients which are there.
>>
>> Which brings up full circle...
>>
>> What do you then use to backup to tape bearing in mind that the
Sun-provided tools all have significant limitations?
>
> Poor choice of words. ?Sun resells NetBackup and (IIRC) that which was
> formerly called NetWorker. ?Thus, Sun does provide enterprise backup
> solutions.
(Symantec nee Veritas) NetBackup and (EMC nee Legato) Networker are
different products that compete in the enterprise backup space.

Under the covers NetBackup uses gnu tar to gather file data for the
backup stream.  At one point (maybe still the case), one of the
claimed features of netbackup is that if a tape is written without
multiplexing, you can use gnu tar to extract data.  This seems to be
most useful when you need to recover master and/or media servers and
to be able to extract your data after you no longer use netbackup.

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins
wrote:
> True, but I wonder how viable its future is.  One of my clients
> requires 17 LT04 types for a full backup, which cost more and takes
> up more space than the equivalent in removable hard drives.
What kind of removable hard drives are you getting that are cheaper than
tape?

LTO4 media is less than 2.5 cents/GB for us (before compression,
acquisition cost only).

-- 
Darren

On Thu, Jan 21, 2010 at 12:38:56AM +0100, Ragnar Sundblad
wrote:
> On 21 jan 2010, at 00.20, Al Hopper wrote:
> > I remember for about 5 years ago (before LT0-4 days) that streaming
> > tape drives would go to great lengths to ensure that the drive kept
> > streaming - because it took so much time to stop, backup and stream
> > again.  And one way the drive firmware accomplished that was to write
> > blocks of zeros when there was no data available.
> I haven''t seen drives that fill out with zeros. Sounds like an
ugly
> solution, but maybe it could be useful in some strange case.
It was closer to 15 years ago than 5, but this may be a reference to the
first release of the DLT 7000.  That version came out with only 4MB as a
RAM buffer, which is insufficient to buffer at speed during a stop/start
cycle.  It didn''t write zeros, but it would disable the on-drive
compression to try to keep the speed of bits being written to tape up.
So the effect was similar in that the capacity of the media was reduced.
The later versions had 8MB buffers and that behavior was removed.

-- 
Darren

A Darren Dunham wrote:
> On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins wrote:
>   
>> True, but I wonder how viable its future is.  One of my clients
>> requires 17 LT04 types for a full backup, which cost more and takes
>> up more space than the equivalent in removable hard drives.
>>     
>
> What kind of removable hard drives are you getting that are cheaper than
> tape?
>
>   
It''s not the raw cost per GB, it''s the way the tapes are used.
To aid
recovery times, a number of different backup sets (groups of 
filesystems) are written, so the tapes aren''t all used to capacity.

-- 
Ian.

On 21/01/2010 11:55, Julian Regel wrote:
>
> >> Until you try to pick one up and put it in a fire safe!
>
> >Then you backup to tape from x4540 whatever data you need.
> >In case of enterprise products you save on licensing here as you need 
> a one client license per x4540 but in fact can >backup data from many 
> clients which are there.
>
> Which brings up full circle...
>
> What do you then use to backup to tape bearing in mind that the 
> Sun-provided tools all have significant limitations?
>
> I guess you need to use a third party tool and watch carefully that 
> they provide complete backups.
>
in this specific environment I was talking about NetBackup is used.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100125/0c19fa79/attachment.html>

uep,
This solution seems like the best and most efficient way of handling large
filesystems. My biggest question however is, when backing this up to tape, can
it be split across several tapes? I will be using bacula to back this up. Will i
need to tar or star this filesystem before writing it to tape? The next question
I have is since I am using a primary and a secondary server, using zfs send/recv
how would I incorporate this solution to then backing the secondary SAN to tape?
Will I need double the hard disk space as I will need a file based copy of the
ZFS filesystem? I know it is a lot of questions but I thought the solution would
work perfect in my environment.

Thanks,
Greg
-- 
This message posted from opensolaris.org