zfs discuss - Apr 2007 - Preferred backup mechanism for ZFS?

It seems that neither Legato nor NetBackup seem to lend themselves well to the
notion of lots of file systems within storage pools from an administration
perspective.  Is there a preferred methodology for doing traditional backups to
tape from ZFS where there are hundreds or thousands of filesystems?  Is there a
disk-to-disk-to-tape method? Sorry for the murky question.
 
 
This message posted from opensolaris.org

On 4/18/07, Bill Sprouse <bill.sprouse at sun.com>
wrote:
>  It seems that neither Legato nor NetBackup seem to lend themselves well to
the notion of lots of file systems within storage pools from an administration
perspective.  Is there a preferred methodology for doing traditional backups to
tape from ZFS where there are hundreds or thousands of filesystems?  Is there a
disk-to-disk-to-tape method? Sorry for the murky question.

It is a good question.

I asked this over a year ago and the result was an RFE whose number I
lost. Initially I wanted a way to  do a dump to tape like ufsdump.  I
don''t know if this makes sense anymore because the tape market is
crashing slowly. People just don''t backup 300MB per night anymore. We
are looking at terabytes of data and I don''t know how to backup a
terabyte a night.  I don''t even know how the big banks are going to
address the data explosion that shows no signs of slowing.

I recall, back in 1997, I was at camp Microsoft out in Seattle where I
was given a full tour of the facility and got to meet with Jeff Raikes
and a stack of techies. Top notch world class techies and we were
looking at Microsoft Exchange and its database backend or lack
thereof. I simply refused to buy the idea that it was a "zero
administration" system at the time. That was a buzzword back then. The
other issue on the table was backup. I recall being quite adamant that
any system must be able to be backed up in a fashion that allows  the
state of a system or software set to be restored precisely at some
point later in time. I was stunned at the time to learn that there
were some 4 terabytes of data in all those compaq racks that MS had
and no way to do a backup. At all. To me, at the time, that was just
bad engineering.  Perhaps I was myopic and could not see what the ZFS
designers can plainly see; there are no backups anymore.

So now here we are ten years later with a new filesystem and I have no
way to back it up in such a fashion that I can restore it perfectly. I
can take snapshots. I can do a strange send and receive but the
process is not stable From zfs (1M) we see :

     The format of the stream is evolving. No backwards  compati-
     bility  is  guaranteed.  You may not be able to receive your
     streams on future versions of ZFS.

This leaves us with tar or cpio or Joerg Schillings star.  But if you
have many man file systems and they are larger than a LTO tape ( or
whatever media du jour ) then you will need to figure out a way to do
the backups.  Somehow.

If there were *ever* a discussion that needs to happen then it
certainly is the state of backups with ZFS.

How shall we begin ?

Maybe with a definition of what a "backup" is and then some way to
achieve it. As far as I know the only real backup is one that can be
tossed into a vault and locked away for seven years.  Or any arbitrary
amount of time within in reason. Like a decade or a century.   But
perhaps a backup today will have as much meaning as papertape over
time.

Can we discuss this with a few objectives ?  Like define "backup" and
then describe mechanisms that may achieve one?  Or a really big
question that I guess I have to ask, do we even care anymore?

Dennis

On Wed, Apr 18, 2007 at 03:47:55PM -0400, Dennis Clarke
wrote:
> Maybe with a definition of what a "backup" is and then some way
to
> achieve it. As far as I know the only real backup is one that can be
> tossed into a vault and locked away for seven years.  Or any arbitrary
> amount of time within in reason. Like a decade or a century.   But
> perhaps a backup today will have as much meaning as papertape over
> time.
> 
> Can we discuss this with a few objectives ?  Like define "backup"
and
> then describe mechanisms that may achieve one?  Or a really big
> question that I guess I have to ask, do we even care anymore?
As far as ZFS is concerned any discussion of how you''ll read
today''s
media a decade into the future is completely OT :)

"zfs send" as backup is probably not generally acceptable: you
can''t
expect to extract a single file out of it (at least not out of an
incremental zfs send), but that''s certainly done routinely with
ufsdump,
tar, cpio, ...

Also, why not just punt to NDMP?

Nico
--

> Can we discuss this with a few objectives ?  Like define "backup"
and
> then describe mechanisms that may achieve one?  Or a really big
> question that I guess I have to ask, do we even care anymore?
</lurk>
Personally I think you would benefit from some slightly different terms. 
I would differentiate between backups and archives.  Here I am trying to 
move away from tape based backups.  We do backups to a disk based system. 
There is no technical reason why this couldn''t be done with zfs send |
zfs
receive.

Then there are archives.  I am exceptionally fortunate that my group
don''t
really have to deal with these.  If I did then, despite the significant 
drop in price for hard drive based storage, I''d still go for a tape
based
system.  I don''t know of anyway to do this usefully under zfs.

I don''t believe that there are any good/useful solutions which are free
that will store both the data and all the potential meta-data in the 
filesystem in a recoverable way.  I''ve never been impressed enough with
any of the commercial solutions, and certainly not enough to give someone 
money for them.

<lurk>

Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

On 4/18/07, Nicolas Williams <Nicolas.Williams at sun.com>
wrote:
> On Wed, Apr 18, 2007 at 03:47:55PM -0400, Dennis Clarke wrote:
> > Maybe with a definition of what a "backup" is and then some
way to
> > achieve it. As far as I know the only real backup is one that can be
> > tossed into a vault and locked away for seven years.  Or any arbitrary
> > amount of time within in reason. Like a decade or a century.   But
> > perhaps a backup today will have as much meaning as papertape over
> > time.
> >
> > Can we discuss this with a few objectives ?  Like define
"backup" and
> > then describe mechanisms that may achieve one?  Or a really big
> > question that I guess I have to ask, do we even care anymore?
>
> As far as ZFS is concerned any discussion of how you''ll read
today''s
> media a decade into the future is completely OT :)
probably.  Media should have a shelf life of seven years minimum and
probably a whole lot longer. The technology ( QIC, 4mm DAT, DLT etc
etc ) should be available and around for a long long time.
>
> "zfs send" as backup is probably not generally acceptable: you
can''t
> expect to extract a single file out of it (at least not out of an
> incremental zfs send), but that''s certainly done routinely with
ufsdump,
> tar, cpio, ...
>
> Also, why not just punt to NDMP?
.. lets look at it.

http://www.ndmp.org/products/

thats a fair list of companies there.  The SDK looks to be alpha stage
or maybe beta :

Q: What good is the NDMP SDK?

The NDMP software developers kit is developed to prototype new NDMP
functionality added and provides a functional (although fairly basic)
implementation of an NDMP client and NDMP server.

The objective of the SDK is to facilitate rapid development of NDMP
compliant clients and servers on a variety of platforms.

Third parties are welcome to download and make use of the provided
source code within your products (subject to copyright notices
supplied) or as example/reference code.


OKay .. thats a good candidate to look at.

dc

On 4/18/07, J.P. King <jpk28 at cam.ac.uk> wrote:
>
> > Can we discuss this with a few objectives ?  Like define
"backup" and
> > then describe mechanisms that may achieve one?  Or a really big
> > question that I guess I have to ask, do we even care anymore?
>
> </lurk>
> Personally I think you would benefit from some slightly different terms.
> I would differentiate between backups and archives.  Here I am trying to
> move away from tape based backups.  We do backups to a disk based system.
> There is no technical reason why this couldn''t be done with zfs
send | zfs
> receive.
  Okay .. that is disk to disk or system to system. I can only assume
that you have large pipes of bandwidth ( 10 GE ) to move data around
with.
> Then there are archives.  I am exceptionally fortunate that my group
don''t
> really have to deal with these.  If I did then, despite the significant
> drop in price for hard drive based storage, I''d still go for a
tape based
> system.  I don''t know of anyway to do this usefully under zfs.
me neither.

I just finished installing Solaris 10 and ZFS at a manufacturing site
that needs fast cheap storage. Its real tough to argue with ZFS once
you see it in action. They were sold and I went ahead with a few
terabytes of storage attached to a Sun UltraSparc server.  Everyone is
happy until the question came up about "how to back it up?"

This is where a netbackup solution is being used. A Netbackup machine
mounts the various ZFS filesystems via NFS and then dumps to tape.
Nightly.
> I don''t believe that there are any good/useful solutions which are
free
> that will store both the data and all the potential meta-data in the
> filesystem in a recoverable way.
I think that star ( Joerg Schilling ) has a good grasp on all the
metadata. Or do you mean the underlying structure that allows you to
restore to bare metal or bare disks ?

Dennis

On Wed, Apr 18, 2007 at 04:32:18PM -0400, Dennis Clarke
wrote:
> On 4/18/07, J.P. King <jpk28 at cam.ac.uk> wrote:
> >
> >> Can we discuss this with a few objectives ?  Like define
"backup" and
> >> then describe mechanisms that may achieve one?  Or a really big
> >> question that I guess I have to ask, do we even care anymore?
> >
> ></lurk>
> >Personally I think you would benefit from some slightly different
terms.
> >I would differentiate between backups and archives.  Here I am trying
to
> >move away from tape based backups.  We do backups to a disk based
system.
> >There is no technical reason why this couldn''t be done with
zfs send | zfs
> >receive.
> 
>  Okay .. that is disk to disk or system to system. I can only assume
> that you have large pipes of bandwidth ( 10 GE ) to move data around
> with.
Why do you think that you''d need bigger pipes to backup to disk than to
tape?  Same source, same amount of data, and same backup time window,
same bandwidth requirement.  Unless backup to disk allows for much
higher bandwidths than backup to tape (but I don''t believe that), but
then you only need larger pipes if you''d wanted to take advantage of
that.

Nicolas Williams wrote:
> Also, why not just punt to NDMP?
While I like NDMP, the protocol is just a transport for blobs of data
in vendor-specific data formats.  We could put a ufsdump or star or
''zfs send'' bag-o-bits in there, and call it ours.  So
it''s a part of
a solution, but not a complete thing.

Rob T

On Wed, Apr 18, 2007 at 04:32:18PM -0400, Dennis Clarke
wrote:
> I just finished installing Solaris 10 and ZFS at a manufacturing site
> that needs fast cheap storage. Its real tough to argue with ZFS once
> you see it in action. They were sold and I went ahead with a few
> terabytes of storage attached to a Sun UltraSparc server.  Everyone is
> happy until the question came up about "how to back it up?"
> 
> This is where a netbackup solution is being used. A Netbackup machine
> mounts the various ZFS filesystems via NFS and then dumps to tape.
> Nightly.
But netbackup should work fine for ZFS (has support for ZFS ACLs been
added?).

ZFS can improve the situation by speeding up incremental dumps since it
knows just what changed (ah, but it doesn''t quite -- it knows what
blocks and what dnodes changed, but there''s no dnode->{dir dnode,
link
name} index, so currently working out what files to backup may not be
much faster by taking advantage of native ZFS features than by the
traditional fs walk, stat and compare approach.

Nico
--

On Wed, Apr 18, 2007 at 02:42:15PM -0600, Robert Thurlow
wrote:
> Nicolas Williams wrote:
> 
> >Also, why not just punt to NDMP?
> 
> While I like NDMP, the protocol is just a transport for blobs of data
> in vendor-specific data formats.  We could put a ufsdump or star or
> ''zfs send'' bag-o-bits in there, and call it ours.  So
it''s a part of
> a solution, but not a complete thing.
Thanks for setting me straight on that.

> Okay .. that is disk to disk or system to system. I can only assume
> that you have large pipes of bandwidth ( 10 GE ) to move data around
> with.
System to system.  No, we have 100Mbit to the backup system.  The systems 
being backed up are small though, they are primarily people''s desktops.
The backup servers (we have two, one onsite, the other offsite) have 
approx 3TB of space, and that backs up systems for 2 to 4 weeks.
> This is where a netbackup solution is being used. A Netbackup machine
> mounts the various ZFS filesystems via NFS and then dumps to tape.
> Nightly.
I don''t believe, although it has been a long time since I looked at 
netbackup, that it will store all the meta data.  Does it cope with ACLs?
If you are using NFSv4 then it might have a fighting chance... ok a quick 
search finds me:

http://www.sun.com/software/solaris/faqs/zfs.xml#q9

This says that networker should cope with ACLs RealSoonNow (mid 2006).  It 
also says something which I didn''t know, namely that sun''s tar
and cpio
will cope with ACLs... I guess they have a pax which works too.
> I think that star ( Joerg Schilling ) has a good grasp on all the
> metadata. Or do you mean the underlying structure that allows you to
> restore to bare metal or bare disks ?
I meant everything from the different time stamps to ACLs.  Something 
which was sufficient to do a restore which would be indistinguishable to 
the user or to any apps that I might wish to run.
> Dennis
Julian
--
Julian King
Computer Officer, University of Cambridge, Unix Support

> > Also, why not just punt to NDMP?
> 
> While I like NDMP, the protocol is just a transport for blobs of data
> in vendor-specific data formats.  We could put a ufsdump or star or
> ''zfs send'' bag-o-bits in there, and call it ours.  So
it''s a part of
> a solution, but not a complete thing.
NDMP does support more than just a blob of data.  It can also supply
position information for individual files within the blob, the ability
to restore a subset of files, and the way to give position information
for those files within the datastream.

Right now I don''t know that any of those features are available with
zfs
send/receive.  

-- 
Darren Dunham                                           ddunham at taos.com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

On 18-Apr-07, at 5:22 PM, J.P. King wrote:
>
>> Can we discuss this with a few objectives ?  Like define
"backup" and
>> then describe mechanisms that may achieve one?  Or a really big
>> question that I guess I have to ask, do we even care anymore?
>
> </lurk>
> Personally I think you would benefit from some slightly different  
> terms. I would differentiate between backups and archives.  Here I  
> am trying to move away from tape based backups.  We do backups to a  
> disk based system. There is no technical reason why this couldn''t
> be done with zfs send | zfs receive.
>
> Then there are archives.
Your distinction is one I''ve always found helpful.
> I am exceptionally fortunate that my group don''t really have to  
> deal with these.  If I did then, despite the significant drop in  
> price for hard drive based storage, I''d still go for a tape based
> system.  I don''t know of anyway to do this usefully under zfs.
Except by simply using ZFS itself: ZFS on redundant hard drives has  
much better archival properties than previous generations of  
filesystem, so it may be worth reconsidering.

--Toby
>
> I don''t believe that there are any good/useful solutions which are
> free that will store both the data and all the potential meta-data  
> in the filesystem in a recoverable way.  I''ve never been impressed
> enough with any of the commercial solutions, and certainly not  
> enough to give someone money for them.
>
> <lurk>
>
> Julian
> --
> Julian King
> Computer Officer, University of Cambridge, Unix Support
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Apr 18, 2007, at 12:47 PM, Dennis Clarke wrote:
>
> Maybe with a definition of what a "backup" is and then some way
to
> achieve it. As far as I know the only real backup is one that can be
> tossed into a vault and locked away for seven years.  Or any arbitrary
> amount of time within in reason. Like a decade or a century.   But
> perhaps a backup today will have as much meaning as papertape over
> time.
>
> Can we discuss this with a few objectives ?  Like define "backup"
and
> then describe mechanisms that may achieve one?  Or a really big
> question that I guess I have to ask, do we even care anymore?
>
This is actually a good point.  ZFS seems to be able to do a credible  
job of keeping itself intact and free from corruption in a day to day  
operational sense and provides for those conveniences like  
snapshots.  It may be that we need to overcome our innate fear of  
spinning rust failure and trust in the force, er, technology.   
However, there still is need for a good way to provide archival  
storage on a regular basis for regulatory and reference reasons as  
well as disaster recovery archives which live off site in  a vault  
someplace.  There is also the need to minimize consumption of  
expensive fast disk by use of nearline cheap SATA or off line tape (D- 
D-T, SAMFS).  It seems like the industry has developed ways to  
achieve this over the last 20 years (50 years?), but we just don''t  
have access to this with ZFS yet in a convenient and easy to manage  
way.  I''m hoping someone will tell me I''m way wrong and
provide a
nice explanation of this works.

On Apr 18, 2007, at 4:53 PM, J.P. King wrote:
> http://www.sun.com/software/solaris/faqs/zfs.xml#q9
>
> This says that networker should cope with ACLs RealSoonNow (mid  
> 2006).  It also says something which I didn''t know, namely that  
> sun''s tar and cpio will cope with ACLs... I guess they have a pax
> which works too.
Just dropping in to say that the Networker 7.3.2 client knows of ZFS  
ACLs, and has been available since last fall in both Legato Networker- 
proper and Sun EBS.

/dale

Hi

This is a bit off topic...but as  Bill mentioned SAM-FS...my job at Sun 
is working in a group focused on ISV''s in the archiving space (Symantec
Enterprise Vault, Open Text LEA, CA Message Manager, FileNet, Mobius, 
AXS-One etc etc)...we see a strong interest from some of them in using 
SAM-FS because is solves the problem of backing up _archived _data. I 
just completed some interesting projects with Symantec using SAMBA + 
SAM-FS (Enterprise Vault is a Windows App).

One thing we have considered is using SAM-FS & ZFS together on a 
server....the archived data is written into SAM-FS by the application 
which later copies it to  ZFS (which is then a mid-tier disk archive) 
and tape (last tier). SAM-FS is being Open Sourced real soon (for people 
who care about that) also runs on x64 now.

SAM-FS + ZFS + Thumper (Sun Fire x4500) + tape could be a nice back end 
for an archiving application.

Rgds

Tim

Bill Sprouse said the following :
>
> On Apr 18, 2007, at 12:47 PM, Dennis Clarke wrote:
>
>>
>> Maybe with a definition of what a "backup" is and then some
way to
>> achieve it. As far as I know the only real backup is one that can be
>> tossed into a vault and locked away for seven years.  Or any arbitrary
>> amount of time within in reason. Like a decade or a century.   But
>> perhaps a backup today will have as much meaning as papertape over
>> time.
>>
>> Can we discuss this with a few objectives ?  Like define
"backup" and
>> then describe mechanisms that may achieve one?  Or a really big
>> question that I guess I have to ask, do we even care anymore?
>>
>
> This is actually a good point.  ZFS seems to be able to do a credible 
> job of keeping itself intact and free from corruption in a day to day 
> operational sense and provides for those conveniences like snapshots.  
> It may be that we need to overcome our innate fear of spinning rust 
> failure and trust in the force, er, technology.  However, there still 
> is need for a good way to provide archival storage on a regular basis 
> for regulatory and reference reasons as well as disaster recovery 
> archives which live off site in  a vault someplace.  There is also the 
> need to minimize consumption of expensive fast disk by use of nearline 
> cheap SATA or off line tape (D-D-T, SAMFS).  It seems like the 
> industry has developed ways to achieve this over the last 20 years (50 
> years?), but we just don''t have access to this with ZFS yet in a 
> convenient and easy to manage way.  I''m hoping someone will tell
me
> I''m way wrong and provide a nice explanation of this works.
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

"Dennis Clarke" <blastwave at gmail.com> wrote:
>      The format of the stream is evolving. No backwards  compati-
>      bility  is  guaranteed.  You may not be able to receive your
>      streams on future versions of ZFS.
This is a good point even if your claim on ZFSs "send/receive" format
may be wrong. The POSIX.1-2001 extended tar archive format is a format
that you may safely rely on. It is infinitely expandable and this is important 
for future enhancements.
> This leaves us with tar or cpio or Joerg Schillings star.  But if you
The archive format supported by the current "Solaris tar" is not
sufficient for
doing backups. A lot of important information is missing that is needed in order
to create incremental backups.

cpio is a "dead end" archive format. There are already 4 completely
incompatible
cpio archive formats. There are currently two different "revetn" cpio
archive
formats: 

-	The POSIX variant. It supports a max. filesize of 8 GB -2 byte but
	limits UID/GID as well as dev/ino to a range from 0 ... 262143

-	The Svr4 variant. It allows UID/GID as well as dev/ino to be in the
	range 0 ... 4294967295 but it limits file size to 4 GB - 2 byte.

and it is hard to further enhance the limited format.

In addition, Sun "Solaris tar" as well as cpio do not support sparse
files.

Star supports all needed features and in addition implements incremental backups
using the same basic algorithm als ufsdump.
> have many man file systems and they are larger than a LTO tape ( or
> whatever media du jour ) then you will need to figure out a way to do
> the backups.  Somehow.
With star, you may do incremental backups and multi volume archives 
that span more than one tape.

> How shall we begin ?
>
> Maybe with a definition of what a "backup" is and then some way
to
> achieve it. As far as I know the only real backup is one that can be
> tossed into a vault and locked away for seven years.  Or any arbitrary
> amount of time within in reason. Like a decade or a century.   But
> perhaps a backup today will have as much meaning as papertape over
> time.
A real backup can be done using star on a snaphot.
> Can we discuss this with a few objectives ?  Like define "backup"
and
> then describe mechanisms that may achieve one?  Or a really big
> question that I guess I have to ask, do we even care anymore?
If you have questions on backup features and whether they are already 
implemented in star or possible/impossible in the future, please ask.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

Nicolas Williams <Nicolas.Williams at sun.com> wrote:
> "zfs send" as backup is probably not generally acceptable: you
can''t
> expect to extract a single file out of it (at least not out of an
> incremental zfs send), but that''s certainly done routinely with
ufsdump,
> tar, cpio, ...
Then an incremental star backup looks like the right solution.

The incremental is still a valid tar archive and you may extract any single
file using any recent tar implementation. Only if you like to use the 
incremental features (that allow you to deal with renamed or removed files),
you need star.


J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

"Dennis Clarke" <blastwave at gmail.com> wrote:

> > I don''t believe that there are any good/useful solutions
which are free
> > that will store both the data and all the potential meta-data in the
> > filesystem in a recoverable way.
>
> I think that star ( Joerg Schilling ) has a good grasp on all the
> metadata. Or do you mean the underlying structure that allows you to
> restore to bare metal or bare disks ?
Star currently does not support extended attribute files but all other
metadata. Of you are in doubt whether the metadata you are thinking of
is supported, please ask.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       schilling at fokus.fraunhofer.de     (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

Hello Tim,

Thursday, April 19, 2007, 10:32:53 AM, you wrote:

TT> Hi

TT> This is a bit off topic...but as  Bill mentioned SAM-FS...my job at Sun
TT> is working in a group focused on ISV''s in the archiving space
(Symantec
TT> Enterprise Vault, Open Text LEA, CA Message Manager, FileNet, Mobius, 
TT> AXS-One etc etc)...we see a strong interest from some of them in using
TT> SAM-FS because is solves the problem of backing up _archived _data. I 
TT> just completed some interesting projects with Symantec using SAMBA + 
TT> SAM-FS (Enterprise Vault is a Windows App).

TT> One thing we have considered is using SAM-FS & ZFS together on a 
TT> server....the archived data is written into SAM-FS by the application 
TT> which later copies it to  ZFS (which is then a mid-tier disk archive) 
TT> and tape (last tier). SAM-FS is being Open Sourced real soon (for people
TT> who care about that) also runs on x64 now.

TT> SAM-FS + ZFS + Thumper (Sun Fire x4500) + tape could be a nice back end
TT> for an archiving application.

Does SAM-FS work with ZFS right now? I thought that it works only with
QFS...?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Hi Robert

no, SAM-FS does not work with ZFS, it is still tied to QFS i.e. the SAM 
piece cannot archive data straight out of a ZFS file system.

I am a big fan of SAM-FS so forgive me if I expand a little on the topic...

The way a solution like this would work is that you would build a SAM-FS 
file system on some of the disks and a ZFS file system on some others. 
Maybe SAM-FS would be on "fast" disks and ZFS on "slow"
disks.

Your application would write files into the SAM-FS file system. In my 
work this application would be an archiving application that has 
extracted data (e.g emails) out of a business application and now wants 
somewhere to put the archived data.

SAM-FS can then make copies of the files (back them up) in to the ZFS 
file system and to tape (at the same or another time) up to a maximum of 
4 copies...so from SAM-FS''s perspective ZFS is being treated as a disk 
archive....SAM-FS does not care that it writing to a ZFS file system 
..it just sees a directory that it can treat as a target to make backup 
copies of data into. ZFS is nice here because it can be used with JBODs 
and can do software RAID.

SAM-FS (and it''s little brother QFS which is the file system without
the
archiving), both do software RAID, but only RAID-0...so the underlying 
"disks" need to be h/ware RAID protected or logical volumes built with
something like Solaris Volume Manager.

So (you may ask) why would anyone want to do this ? Well...think of the 
SAM-FS file system as a cache...once it has backed up a file to a disk 
and/or tape archive it can be configured to "release" the
file''s data
blocks and just leave a stub behind. The file is now viewed as off-line 
(Hey, this looks like HSM!) - There are a rich set of rules you can 
define around when this happens - the simplest is to do it when the file 
system passes a high %full watermark...in short..IT NEVER FILLS UP!

So that is pretty cool...so here is the next really nice bit, the files 
are backed up continuously (once again. there are a rich set of rules 
around this)...no huge backup windows while millions of small files are 
backed up like a conventional backup product...SAM-FS will back them up 
throughout the day...also, NO  INCREMENTAL BACKUPS required...we are 
working at a per file level here. If a  file is written and never 
changes SAM-FS will make copies of it and then we are done..we never 
copy it again unless it changes...there is just one image of the file 
system.

SAM-FS handles all the interaction with tape libraries etc...no extra 
s/ware required.

So, what if you open a released file ? Once you read past the end of the 
stub SAM-FS stages the rest of the data back from archives....it is 
transparent...and it really is...if the file is having to come back from 
tape the long delays can cause application issues sometimes but files 
coming back from disk archives come back so fast that there are no problems.

You can access SAM-FS via NFS and CIFS/SAMBA safely. NFS client need to 
support correct handling of the NFS EJUKEBOX error code...Solaris''s
does.

Now, what if you loose a whole file system with millions of files in it. 
Recovering that from backups is no fun. part of managing SAM-FS is 
taking regular backups of the metadata...this is effectively a snapshot. 
If a  file system is lost due to catastrophic h/ware failure then you 
can restore the meta into an empty SAM-FS file system (which can be 
smaller or larger than the original) in a  few minutes and bingo, all 
the data is there again..with all the files off-line.

SAM-FS management is all via the command line or via a really nice Web 
based UI. I won''t pretend that it is as easy as ZFS to create a SAM-FS 
file system ''cause it aint''..but heck, it''s fun once
you get with
it...and the GUI certainly makes life much easier.

So..what is the "bad" stuff, other than no RAID-Z equivalent, well:
you
cannot grow a SAM-FS file system on-line..it needs to be unmounted 
first; No snapshots (other than the meta data based ones); you have to 
back up the meta data regularly as that is effectively the backup 
catalog which can be a drag but the procedures are well defined.

There is other stuff that SAM-FS can do such as shared Global File 
system support in SAN''s etc...but I have gone on enough!!

Rgds

Tim



Robert Milkowski said the following :
> Hello Tim,
>
> Thursday, April 19, 2007, 10:32:53 AM, you wrote:
>
> TT> Hi
>
> TT> This is a bit off topic...but as  Bill mentioned SAM-FS...my job at
Sun
> TT> is working in a group focused on ISV''s in the archiving
space (Symantec
> TT> Enterprise Vault, Open Text LEA, CA Message Manager, FileNet,
Mobius,
> TT> AXS-One etc etc)...we see a strong interest from some of them in
using
> TT> SAM-FS because is solves the problem of backing up _archived _data.
I
> TT> just completed some interesting projects with Symantec using SAMBA +
> TT> SAM-FS (Enterprise Vault is a Windows App).
>
> TT> One thing we have considered is using SAM-FS & ZFS together on a
> TT> server....the archived data is written into SAM-FS by the
application
> TT> which later copies it to  ZFS (which is then a mid-tier disk
archive)
> TT> and tape (last tier). SAM-FS is being Open Sourced real soon (for
people
> TT> who care about that) also runs on x64 now.
>
> TT> SAM-FS + ZFS + Thumper (Sun Fire x4500) + tape could be a nice back
end
> TT> for an archiving application.
>
> Does SAM-FS work with ZFS right now? I thought that it works only with
> QFS...?
>
>   
--

Hello Nicolas,

Wednesday, April 18, 2007, 10:12:17 PM, you wrote:

NW> On Wed, Apr 18, 2007 at 03:47:55PM -0400, Dennis Clarke
wrote:
>> Maybe with a definition of what a "backup" is and then some
way to
>> achieve it. As far as I know the only real backup is one that can be
>> tossed into a vault and locked away for seven years.  Or any arbitrary
>> amount of time within in reason. Like a decade or a century.   But
>> perhaps a backup today will have as much meaning as papertape over
>> time.
>> 
>> Can we discuss this with a few objectives ?  Like define
"backup" and
>> then describe mechanisms that may achieve one?  Or a really big
>> question that I guess I have to ask, do we even care anymore?
NW> As far as ZFS is concerned any discussion of how you''ll read
today''s
NW> media a decade into the future is completely OT :)

NW> "zfs send" as backup is probably not generally acceptable: you
can''t
NW> expect to extract a single file out of it (at least not out of an
NW> incremental zfs send), but that''s certainly done routinely with
ufsdump,
NW> tar, cpio, ...

Depends where you ''zfs send'' those data and what you do on the
other
side. If you just zfs send <---remote host---> zfs recv
and then also make snapshots you have actually very good backup in
many aspects much better than using tapes.
The only real drawback is management (custom zfs send|recv scripts
compared to Legato, Tivolli, ...).

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Dennis Clarke wrote:
> So now here we are ten years later with a new filesystem and I have no
> way to back it up in such a fashion that I can restore it perfectly. I
> can take snapshots. I can do a strange send and receive but the
> process is not stable From zfs (1M) we see :
> 
>     The format of the stream is evolving. No backwards  compati-
>     bility  is  guaranteed.  You may not be able to receive your
>     streams on future versions of ZFS.
The format of the stream may not be stable. So you can''t dump the
stream
to a file somewhere and expect to receive from it sometime in the future.

But if you stash it away as a pool on the same machine or elsewhere, it 
is not an issue.

# zfs send [-i b] pool/fs at a | [ssh host] zfs receive poolB/received/fs at a

Right? I think this is quite cool! :)

-Manoj

Hi Tim,

I  run a setup of SAM-FS for our main file server and we loved the
backup/restore parts that you described.

The main concerns I have with SAM fronting the entire conversation is
data integrity. Unlike ZFS, SAMFS does not do end to end checksumming.

We have considered the setup you proposed (samfs copy1 -> ZFS) but you
will run into problem with fs-cache.  Being only a copy, ZFS probably
do not need much caching but will win the battle for memory due to the
way its cache is managed.  Unless there is a visible memory shortfall,
ZFS will starve (sorry guys) samfs from memory it could use as cache.
Also, ZFS''s data integrity feature is limited by the use of 2nd hand
data.


-- 
Just me,
Wire
<http://prstat.blogspot.com>

On 4/19/07, Tim Thomas <Tim.Thomas at sun.com>
wrote:
> Hi Robert
>
> no, SAM-FS does not work with ZFS, it is still tied to QFS i.e. the SAM
> piece cannot archive data straight out of a ZFS file system.
>
> I am a big fan of SAM-FS so forgive me if I expand a little on the topic...
>
> The way a solution like this would work is that you would build a SAM-FS
> file system on some of the disks and a ZFS file system on some others.
> Maybe SAM-FS would be on "fast" disks and ZFS on "slow"
disks.
>
> Your application would write files into the SAM-FS file system. In my
> work this application would be an archiving application that has
> extracted data (e.g emails) out of a business application and now wants
> somewhere to put the archived data.
>
> SAM-FS can then make copies of the files (back them up) in to the ZFS
> file system and to tape (at the same or another time) up to a maximum of
> 4 copies...so from SAM-FS''s perspective ZFS is being treated as a
disk
> archive....SAM-FS does not care that it writing to a ZFS file system
> ..it just sees a directory that it can treat as a target to make backup
> copies of data into. ZFS is nice here because it can be used with JBODs
> and can do software RAID.
>
> SAM-FS (and it''s little brother QFS which is the file system
without the
> archiving), both do software RAID, but only RAID-0...so the underlying
> "disks" need to be h/ware RAID protected or logical volumes built
with
> something like Solaris Volume Manager.
>
> So (you may ask) why would anyone want to do this ? Well...think of the
> SAM-FS file system as a cache...once it has backed up a file to a disk
> and/or tape archive it can be configured to "release" the
file''s data
> blocks and just leave a stub behind. The file is now viewed as off-line
> (Hey, this looks like HSM!) - There are a rich set of rules you can
> define around when this happens - the simplest is to do it when the file
> system passes a high %full watermark...in short..IT NEVER FILLS UP!
>
> So that is pretty cool...so here is the next really nice bit, the files
> are backed up continuously (once again. there are a rich set of rules
> around this)...no huge backup windows while millions of small files are
> backed up like a conventional backup product...SAM-FS will back them up
> throughout the day...also, NO  INCREMENTAL BACKUPS required...we are
> working at a per file level here. If a  file is written and never
> changes SAM-FS will make copies of it and then we are done..we never
> copy it again unless it changes...there is just one image of the file
> system.
>
> SAM-FS handles all the interaction with tape libraries etc...no extra
> s/ware required.
>
> So, what if you open a released file ? Once you read past the end of the
> stub SAM-FS stages the rest of the data back from archives....it is
> transparent...and it really is...if the file is having to come back from
> tape the long delays can cause application issues sometimes but files
> coming back from disk archives come back so fast that there are no
problems.
>
> You can access SAM-FS via NFS and CIFS/SAMBA safely. NFS client need to
> support correct handling of the NFS EJUKEBOX error
code...Solaris''s does.
>
> Now, what if you loose a whole file system with millions of files in it.
> Recovering that from backups is no fun. part of managing SAM-FS is
> taking regular backups of the metadata...this is effectively a snapshot.
> If a  file system is lost due to catastrophic h/ware failure then you
> can restore the meta into an empty SAM-FS file system (which can be
> smaller or larger than the original) in a  few minutes and bingo, all
> the data is there again..with all the files off-line.
>
> SAM-FS management is all via the command line or via a really nice Web
> based UI. I won''t pretend that it is as easy as ZFS to create a
SAM-FS
> file system ''cause it aint''..but heck, it''s fun
once you get with
> it...and the GUI certainly makes life much easier.
>
> So..what is the "bad" stuff, other than no RAID-Z equivalent,
well: you
> cannot grow a SAM-FS file system on-line..it needs to be unmounted
> first; No snapshots (other than the meta data based ones); you have to
> back up the meta data regularly as that is effectively the backup
> catalog which can be a drag but the procedures are well defined.
>
> There is other stuff that SAM-FS can do such as shared Global File
> system support in SAN''s etc...but I have gone on enough!!
>
> Rgds
>
> Tim
...

Hello Wee,

Friday, April 20, 2007, 4:50:08 AM, you wrote:

WYT> Hi Tim,

WYT> I  run a setup of SAM-FS for our main file server and we loved the
WYT> backup/restore parts that you described.

WYT> The main concerns I have with SAM fronting the entire conversation is
WYT> data integrity. Unlike ZFS, SAMFS does not do end to end checksumming.

WYT> We have considered the setup you proposed (samfs copy1 -> ZFS) but
you
WYT> will run into problem with fs-cache.  Being only a copy, ZFS probably
WYT> do not need much caching but will win the battle for memory due to the
WYT> way its cache is managed.  Unless there is a visible memory shortfall,
WYT> ZFS will starve (sorry guys) samfs from memory it could use as cache.
WYT> Also, ZFS''s data integrity feature is limited by the use of 2nd
hand
WYT> data.

You can limit how much memory zfs can use for its caching.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
> You can limit how much memory zfs can use for its caching.
>
Indeed, but that memory will still be locked.  How can you tell the
system to be "flexible" with the caching?

I deem that archiving will not present a cache challenge but we will
want zfs to prefetch (or do whatever magic) when staging in files.  We
do not want to limit ZFS''s cache but we want to tell the system to
prefer SAMFS''s cache to ZFS''s.


-- 
Just me,
Wire ...
<http://prstat.blogspot.com>

> Initially I wanted a way to  do a dump to tape like ufsdump.  I
> don''t know if this makes sense anymore because the tape market is
> crashing slowly.
It makes sense if you need to keep backups for more than a handful of years
(think regulatory requirements or scientific data), or if cost is important.
Storing tape is much cheaper than keeping disks running. (Storing disks
isn''t practical over long periods of time; not only does the signal on
the media degrade, but so do some components.)
> People just don''t backup 300MB per night anymore. We
> are looking at terabytes of data and I don''t know how
> to backup a terabyte a night.
If you''re actually generating a terabyte per day of data, I''m
impressed.  :-)

Tape seems a reasonable way to back that up, in any case. A T10000 stores 500 GB
on each tape and runs at 120 MB/sec, so a terabyte would take roughly 2.5 hours
to backup with a single tape drive. LTO-4 is in the same ballpark. Of course,
that assumes your disk system can keep up.

The SAM-QFS approach of continuous archiving makes a lot of sense here since it
effectively lets backups run continuously. I don''t know how much Sun
can say about the work going on to add SAM to ZFS.
> Or a really big question that I guess I have to ask, do we even care
anymore?
If we''re serious about disaster recovery, we do.

In particular, remote replication is NOT a substitute for backups.

Anton
 
 
This message posted from opensolaris.org

Hi Wee
>
> I  run a setup of SAM-FS for our main file server and we loved the
> backup/restore parts that you described.
That is great to hear.
>
> The main concerns I have with SAM fronting the entire conversation is
> data integrity. Unlike ZFS, SAMFS does not do end to end checksumming.
My initial reaction is that the world has got by without file systems 
that can do this for a long time...so I don''t see the absence of this
as
a big deal. On the other hand, it hard to argue against a feature that 
improves data integrity, so I will not. Anyway, SAM-FS has been enhanced 
in this respect...in SAM-FS 4.6 you can enable the following...

/If required, you can enable data verification for archive copies. This 
feature checks
for data corruption on any data that is copied to secondary and/or 
tertiary media.
The data verification process performs a read-after-write verification 
test, and
records a confirmation of data validity in the metadata properties for 
that file.
An ssum option is used to mark files and directories as needing to be 
verified. Child
directories inherit the data verification properties of their parent. 
The normal
checksum method is employed to verify copies written to tape or disk 
archive.
Use the ssum -e command to set data verification for a file or 
directory. This forces
the generation and use of checksums for archiving and staging, and 
prevents the
release of the file until all archive copies have been created and their 
checksums
verified. Only a superuser can set this attribute on a file or directory./

This is  taken from the Sun StorageTek SAM Archive Configuration and 
Administration Guide Version 4, Update 6 (SAM-FS 4.6 was released April 
6th). You can get all the SAM-FS 4.6 docs from here...

http://www.sun.com/products-n-solutions/hardware/docs/Software/Storage_Software/Sun_SAM-FS_and_Sun_SAM-QFS_Software/index.html

This checksum model is different than ZFS and is more like the way a 
backup product verifies its backups.
>
> We have considered the setup you proposed (samfs copy1 -> ZFS) but you
> will run into problem with fs-cache.  Being only a copy, ZFS probably
> do not need much caching but will win the battle for memory due to the
> way its cache is managed.  Unless there is a visible memory shortfall,
> ZFS will starve (sorry guys) samfs from memory it could use as cache.
> Also, ZFS''s data integrity feature is limited by the use of 2nd
hand
> data.
>
I don''t know enough about how ZFS manages memory other than what I have
seen on this alias (I just joined a couple of weeks ago) which seems to 
indicate it is a  memory hog...as is VxFS so we are in good company. I 
am not against keeping data in memory so long as it has also been 
written to somewhere non-volatile as well so that data is not lost if 
the lights go out... and applications don''t fight for memory to run. I 
recall stories from years ago where VxFS hogged so much memory on a Sun 
Cluster node that the Cluster services stalled and the cluster failed over!

I need to go read some white papers on this...but I assume that 
something like direct I/O (which UFS, VxFS and QFS all have) is in the 
plans for ZFS so we don''t end up double buffering data for apps like 
databases ? - that is just ugly.

Rgds

Tim

Hello Wee,

Friday, April 20, 2007, 5:20:00 AM, you wrote:

WYT> On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
>> You can limit how much memory zfs can use for its caching.
>>
WYT> Indeed, but that memory will still be locked.  How can you tell the
WYT> system to be "flexible" with the caching?

It shouldn''t be locked but in reality it can.

WYT> I deem that archiving will not present a cache challenge but we will
WYT> want zfs to prefetch (or do whatever magic) when staging in files.  We
WYT> do not want to limit ZFS''s cache but we want to tell the system
to
WYT> prefer SAMFS''s cache to ZFS''s.

I don''t know how SAM-FS works (I''ve never used it) but
I''m surprised
that you started paging via using swap - or perhaps you meant
something else.

If qfs uses standard page cache perhaps increasing segmap would also
help. It''s still static however.

By limit ZFS arc you are not disabling prefetching or any other
features.

First I would be most interested what exactly happened when your
server started to crawl ''coz you can''t "swap out"
page cache or arc
cache...





-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Hello Anton,

Friday, April 20, 2007, 9:02:12 AM, you wrote:
>> Initially I wanted a way to  do a dump to tape like ufsdump.  I
>> don''t know if this makes sense anymore because the tape market
is
>> crashing slowly.
ABR> It makes sense if you need to keep backups for more than a
ABR> handful of years (think regulatory requirements or scientific
ABR> data), or if cost is important. Storing tape is much cheaper than
ABR> keeping disks running. (Storing disks isn''t practical over long
ABR> periods of time; not only does the signal on the media degrade, but so
do some components.)
>> People just don''t backup 300MB per night anymore. We
>> are looking at terabytes of data and I don''t know how
>> to backup a terabyte a night.
ABR> If you''re actually generating a terabyte per day of data,
I''m impressed.  :-)

ABR> Tape seems a reasonable way to back that up, in any case. A
ABR> T10000 stores 500 GB on each tape and runs at 120 MB/sec, so a
ABR> terabyte would take roughly 2.5 hours to backup with a single
ABR> tape drive. LTO-4 is in the same ballpark. Of course, that
ABR> assumes your disk system can keep up.

ABR> The SAM-QFS approach of continuous archiving makes a lot of
ABR> sense here since it effectively lets backups run continuously. I
ABR> don''t know how much Sun can say about the work going on to add
SAM to ZFS.
>> Or a really big question that I guess I have to ask, do we even care
anymore?
ABR> If we''re serious about disaster recovery, we do.

ABR> In particular, remote replication is NOT a substitute for backups.

I can''t entirely agree - it really depends.
If you do remote replication and also provide snapshoting it will work
extremely well. And your "restore" would be MUCH more efficient than
from tape. Then if your primary array is down you just switch to
secondary - depending on environment it could be all you need.
With tapes not only you will have to wait for restore you also need a
working array so you have a place to restore.

Of course if you need to take your backup outside then that''s
different.

I''m really disappointed that our try at adding zfs async replication
hasn''t worked out. We''ll have to settle with ''while [
1 ]; do
snapshot; zfs send -i | zfs recv ; sleep 10s; done'' ...


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

To clarify, there are at least two issues with remote replication vs. backups in
my mind. (Feel free to joke about the state of my mind!  ;-)

The first, which as you point out can be alleviated with snapshots, is the
ability to "go back" in time. If an accident wipes out a file, the
missing file will shortly be deleted on the remote end. Snapshots help you here
... as long as you can keep sufficient space online. If your turnover is 1
TB/day and you require the ability to go back to the end of any week in the past
year, that''s 52 TB.

The second is protection against file system failures. If a bug in file system
code, or damage to the metadata structures on disk, results in the master being
unreadable, then it could easily be replicated to the remote system. (Consider a
bug which manifests itself only when 10^9 files have been created; both file
systems will shortly fail.) Keeping backups in a file system independent manner
(e.g. tar format, netbackup format, etc.) protects against this.

If you''re not concerned about the latter, and you can afford to keep
all of your backups on rotating rust (and have sufficient CPU & I/O
bandwidth at the remote site to scrub those backups), and have sufficient
bandwidth to actually move data between sites (for 1 TB/day, assuming continuous
modification, that''s 11 MB/second if data is never rewritten during the
day, or potentially much more in a real environment) then remote replication
could work.
 
 
This message posted from opensolaris.org

Hello Anton,

Friday, April 20, 2007, 3:54:52 PM, you wrote:

ABR> To clarify, there are at least two issues with remote
ABR> replication vs. backups in my mind. (Feel free to joke about the state
of my mind!  ;-)

ABR> The first, which as you point out can be alleviated with
ABR> snapshots, is the ability to "go back" in time. If an accident
ABR> wipes out a file, the missing file will shortly be deleted on the
ABR> remote end. Snapshots help you here ... as long as you can keep
ABR> sufficient space online. If your turnover is 1 TB/day and you
ABR> require the ability to go back to the end of any week in the past year,
that''s 52 TB.

Really depends. With ZFS snapshots in order to consume 1TB by snapshot
you would have deleted 1TB of files or make 1TB modification to files
(or both with 1TB in SUM). There certainly are such workload.
But if you just put new data (append to files, or write new files)
then snapshots practically won''t consume any storage. In that case it
works perfectly.


ABR> The second is protection against file system failures. If a bug
ABR> in file system code, or damage to the metadata structures on
ABR> disk, results in the master being unreadable, then it could
ABR> easily be replicated to the remote system. (Consider a bug which
ABR> manifests itself only when 10^9 files have been created; both
ABR> file systems will shortly fail.) Keeping backups in a file system
ABR> independent
ABR> ABR>  manner (e.g. tar format, netbackup format, etc.) protects
against this.

Lets say I agree. :)


ABR> If you''re not concerned about the latter, and you can afford to
ABR> keep all of your backups on rotating rust (and have sufficient
ABR> CPU & I/O bandwidth at the remote site to scrub those backups),
ABR> and have sufficient bandwidth to actually move data between sites
ABR> (for 1 TB/day, assuming continuous modification, that''s 11
ABR> MB/second if data is never rewritten during the day, or
ABR> potentially much more in a real environment) then remote replication
could work.

You need exactly the same bandwidth as with any other classical backup
solution - it doesn''t matter how at the end you need to copy all those
data (differential) out of the box regardless if it''s a tape or a
disk.

However instead of doing backup during the night, which you want to do
so there will be limited impact on production performance, with
replication you can do it continuously 24x7. The actual performance
impact will be minimal as you should get most data from memory without
touching much of disks on sending side. That also means you actually
need much less throughput available to remote side. Also with frequent
enough snapshoting you have your backup basically every 30 minutes or
every one hour.


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Tim Thomas wrote:
> I don''t know enough about how ZFS manages memory other than what I
have
> seen on this alias (I just joined a couple of weeks ago) which seems to 
> indicate it is a  memory hog...as is VxFS so we are in good company. I 
> am not against keeping data in memory so long as it has also been 
> written to somewhere non-volatile as well so that data is not lost if 
> the lights go out... and applications don''t fight for memory to
run. I
> recall stories from years ago where VxFS hogged so much memory on a Sun 
> Cluster node that the Cluster services stalled and the cluster failed over!
Even after many years, I can still get mileage from this one :-)
	http://www.sun.com/blueprints/0400/ram-vxfs.pdf

ZFS behaves differently, however, so the symptoms and prescriptions are
slightly different.
> I need to go read some white papers on this...but I assume that 
> something like direct I/O (which UFS, VxFS and QFS all have) is in the 
> plans for ZFS so we don''t end up double buffering data for apps
like
> databases ? - that is just ugly.
Before you get very far down this path, it gets regularly rehashed here,
so Roch Bourbonnaise and Bob Sneed wrote some good blogs on the topic.
Especially:
	http://blogs.sun.com/bobs/entry/one_i_o_two_i
	http://blogs.sun.com/roch/entry/zfs_and_directio

  -- richard

> You need exactly the same bandwidth as with any other
> classical backup solution - it doesn''t matter how at the end you
need
> to copy all those data (differential) out of the box regardless if
it''s
> a tape or a disk.
Sure.  However, it''s somewhat cheaper to buy 100 MB/sec of
local-attached tape than 100 MB/sec of long-distance networking.  (The pedant in
me points out that you also need to move the tape to the remote site, which
isn''t entirely free....)

Anton
 
 
This message posted from opensolaris.org

On April 20, 2007 9:54:07 AM +0100 Tim Thomas <Tim.Thomas at Sun.COM>
wrote:
> My initial reaction is that the world has got by without file systems
> that can do [end-to-end data integrity] for a long time...so I
don''t see
> the absence of this as a big deal.
How about

My initial reaction is that the world has got by without [email|cellphone|
other technology] for a long time ... so not a big deal.

-frank

> My initial reaction is that the world has got by without 
> [email|cellphone|
> other technology] for a long time ... so not a big deal. 
Well, I did say I viewed it as an indefensible position :-)

Now shall we debate if the world is a better place because of cell 
phones :-P

Anton B. Rang wrote:
>>You need exactly the same bandwidth as with any other
>>classical backup solution - it doesn''t matter how at the end
you need
>>to copy all those data (differential) out of the box regardless if
it''s
>>a tape or a disk.
>>    
>>
>
>Sure.  However, it''s somewhat cheaper to buy 100 MB/sec of
local-attached tape than 100 MB/sec of long-distance networking.  (The pedant in
me points out that you also need to move the tape to the remote site, which
isn''t entirely free....)
>
>  
>
But a tape in a van is a very high bandwidth connection :)

Ian

> But a tape in a van is a very high bandwidth connection :)
Australia used to get it''s usenet feed on FedExed 9-tracks.

--lyndon

   The two most common elements in the universe are Hydrogen and stupidity.
   			-- Harlan Ellison

On 20-Apr-07, at 5:54 AM, Tim Thomas wrote:
> Hi Wee
>>
>> I  run a setup of SAM-FS for our main file server and we loved the
>> backup/restore parts that you described.
> That is great to hear.
>>
>> The main concerns I have with SAM fronting the entire conversation is
>> data integrity. Unlike ZFS, SAMFS does not do end to end  
>> checksumming.
> My initial reaction is that the world has got by without file  
> systems that can do this for a long time.
Indeed. Progress is one-way like that...
> ..so I don''t see the absence of this as a big deal.
Except that it dilutes the ZFS promise to, "we can keep your data  
safe... unless we have to restore any of it from a backup." It''s a
big deal if you want the integrity promise to extend beyond the pool  
itself! Or so it seems to me.

--T
>
> Rgds
>
> Tim
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Sat, Apr 21, 2007 at 11:14:02AM +1200, Ian Collins
wrote:
> 
> >Sure.  However, it''s somewhat cheaper to buy 100 MB/sec of
local-attached tape than 100 MB/sec of long-distance networking.  (The pedant in
me points out that you also need to move the tape to the remote site, which
isn''t entirely free....)
> >
> But a tape in a van is a very high bandwidth connection :)
What''s the old quote?  "Never underestimate the bandwidth of a
stationwagon full of tapes."  ;)

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

Lyndon Nerenberg wrote:
>> But a tape in a van is a very high bandwidth connection :)
>
> Australia used to get it''s usenet feed on FedExed 9-tracks.
But you had to put them in the reader upside down and read them back to 
front.

Torrey McMahon wrote:
> Lyndon Nerenberg wrote:
>
>>> But a tape in a van is a very high bandwidth connection :)
>>
>>
>> Australia used to get it''s usenet feed on FedExed 9-tracks.
>
>
> But you had to put them in the reader upside down and read them back
> to front.
>
>
Not necessary, upside down was enough, the spools span in the opposite
direction.

Ian

On 4/20/07, Tim Thomas <Tim.Thomas at sun.com>
wrote:
> My initial reaction is that the world has got by without file systems
> that can do this for a long time...so I don''t see the absence of
this as
> a big deal. On the other hand, it hard to argue against a feature that
I admit that this is "typically" not a big deal but as we move more
data, it is not so much of whether a data corruption happens but when
it will be serious enough.  I do not have any statistics on this
though.
> improves data integrity, so I will not. Anyway, SAM-FS has been enhanced
> in this respect...in SAM-FS 4.6 you can enable the following...
> ...
>
http://www.sun.com/products-n-solutions/hardware/docs/Software/Storage_Software/Sun_SAM-FS_and_Sun_SAM-QFS_Software/index.html
>
> This checksum model is different than ZFS and is more like the way a
> backup product verifies its backups.
I briefly went through the release notes.  This feature provides
checksumming still of 2nd hand data but definitely warrants a good
look since it is going to be cheaper than using ZFS for the same job.
ZFS will require redundancy at each archive copy.

What interests me a lot is
http://www.sun.com/products-n-solutions/hardware/docs/html/819-7938-10/index.html#69093
"New Recycling Tool for Archive Copy Retention".  We wrote a mini-hack
to prevent recycling of tape volumes that potentially still holds
relevant backups.
> >
> > We have considered the setup you proposed (samfs copy1 -> ZFS) but
you
> > will run into problem with fs-cache.  Being only a copy, ZFS probably
> > do not need much caching but will win the battle for memory due to the
> > way its cache is managed.  Unless there is a visible memory shortfall,
> > ZFS will starve (sorry guys) samfs from memory it could use as cache.
> > Also, ZFS''s data integrity feature is limited by the use of
2nd hand
> > data.
> >
> I don''t know enough about how ZFS manages memory other than what I
have
> seen on this alias (I just joined a couple of weeks ago) which seems to
> indicate it is a  memory hog...as is VxFS so we are in good company. I
> am not against keeping data in memory so long as it has also been
> written to somewhere non-volatile as well so that data is not lost if
> the lights go out... and applications don''t fight for memory to
run. I
> recall stories from years ago where VxFS hogged so much memory on a Sun
> Cluster node that the Cluster services stalled and the cluster failed over!
ZFS implements ARC (Adaptive Replacement Cache) using kernel pages and
has mechanism to release memory back to the system when memory is low.
 What I''m saying is that in the fight for memory, SAMFS will lose.
I''m a huge fan of ZFS but in this case, it''s not the primary
file
system so we should have some semantics to tell it say so.
> I need to go read some white papers on this...but I assume that
> something like direct I/O (which UFS, VxFS and QFS all have) is in the
> plans for ZFS so we don''t end up double buffering data for apps
like
> databases ? - that is just ugly.
Not sure about the whitepapers but I''m sure the ZFS developers can
answer that :P.


-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
> Hello Wee,
>
> Friday, April 20, 2007, 5:20:00 AM, you wrote:
>
> WYT> On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
> >> You can limit how much memory zfs can use for its caching.
> >>
>
> WYT> Indeed, but that memory will still be locked.  How can you tell the
> WYT> system to be "flexible" with the caching?
>
> It shouldn''t be locked but in reality it can.
It will be "locked" because there is nothing in its view that is
claiming memory.  The machine is a file server and does very little
else that demands memory.  cachelist will lose to zio_buffer.
> I don''t know how SAM-FS works (I''ve never used it) but
I''m surprised
> that you started paging via using swap - or perhaps you meant
> something else.t
>
> If qfs uses standard page cache perhaps increasing segmap would also
> help. It''s still static however.
How do you increase segmap?
> By limit ZFS arc you are not disabling prefetching or any other
> features.
>
> First I would be most interested what exactly happened when your
> server started to crawl ''coz you can''t "swap
out" page cache or arc
> cache...
It wasn''t a case of swapping.  In fact, the problem is exactly the
reverse. There was not enough memory demand to kick ARC to scale back
on its memory usage.  In our test runs with ZFS serving as copy1 for
SAMFS, we saw ZFS using up 1.5-2GB of RAM even when the system is
idle.  The cache is leftover from the previous archiving activities
and is unlikely to be reused.  I''d rather the memory be used by SAMFS.


-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Hello Wee,

Sunday, April 22, 2007, 11:25:23 AM, you wrote:

WYT> On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
>> Hello Wee,
>>
>> Friday, April 20, 2007, 5:20:00 AM, you wrote:
>>
>> WYT> On 4/20/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
>> >> You can limit how much memory zfs can use for its caching.
>> >>
>>
>> WYT> Indeed, but that memory will still be locked.  How can you tell
the
>> WYT> system to be "flexible" with the caching?
>>
>> It shouldn''t be locked but in reality it can.
WYT> It will be "locked" because there is nothing in its view that
is
WYT> claiming memory.  The machine is a file server and does very little
WYT> else that demands memory.  cachelist will lose to zio_buffer.
>> I don''t know how SAM-FS works (I''ve never used it)
but I''m surprised
>> that you started paging via using swap - or perhaps you meant
>> something else.t
>>
>> If qfs uses standard page cache perhaps increasing segmap would also
>> help. It''s still static however.
WYT> How do you increase segmap?

bash-3.00# mdb -k
Loading modules: [ unix krtld genunix dtrace specfs ufs sd pcisch md ip sctp
usba fcp fctl qlc ssd crypto lofs zfs random ptm cpc nfs
]
> segmap_percent/D
segmap_percent:
segmap_percent: 12

(it''s static IIRC)
>> By limit ZFS arc you are not disabling prefetching or any other
>> features.
>>
>> First I would be most interested what exactly happened when your
>> server started to crawl ''coz you can''t "swap
out" page cache or arc
>> cache...
WYT> It wasn''t a case of swapping.  In fact, the problem is exactly
the
WYT> reverse. There was not enough memory demand to kick ARC to scale back
WYT> on its memory usage.  In our test runs with ZFS serving as copy1 for
WYT> SAMFS, we saw ZFS using up 1.5-2GB of RAM even when the system is
WYT> idle.  The cache is leftover from the previous archiving activities
WYT> and is unlikely to be reused.  I''d rather the memory be used by
SAMFS.

Then limit the arc size (see list archives on how to do it).



-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

On 4/23/07, Robert Milkowski <rmilkowski at task.gda.pl>
wrote:
> bash-3.00# mdb -k
> Loading modules: [ unix krtld genunix dtrace specfs ufs sd pcisch md ip
sctp usba fcp fctl qlc ssd crypto lofs zfs random ptm cpc nfs ]
> > segmap_percent/D
> segmap_percent:
> segmap_percent: 12
>
> (it''s static IIRC)
segmap_percent is only referenced in
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sun4/os/startup.c#2270.

You are right that it is static but that means you cannot tune that
unless you run your own kernel.
> Then limit the arc size (see list archives on how to do it).
I know how to do that.  Doing so effectively partitions available
cache memory but that''s not what I want.


-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Wee Yeh Tan wrote:
> On 4/23/07, Robert Milkowski <rmilkowski at task.gda.pl> wrote:
>> bash-3.00# mdb -k
>> Loading modules: [ unix krtld genunix dtrace specfs ufs sd pcisch md 
>> ip sctp usba fcp fctl qlc ssd crypto lofs zfs random ptm cpc nfs ]
>> > segmap_percent/D
>> segmap_percent:
>> segmap_percent: 12
>>
>> (it''s static IIRC)
> 
> segmap_percent is only referenced in
>
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sun4/os/startup.c#2270.
> 
> 
> You are right that it is static but that means you cannot tune that
> unless you run your own kernel.
I took a quick look at startup.c. Looks like you should be able to set 
this value in /etc/system. You should not have to compile your own kernel.

Regards,
Manoj

Hello Manoj,

Monday, April 23, 2007, 5:58:43 AM, you wrote:

MJ> Wee Yeh Tan wrote:
>> On 4/23/07, Robert Milkowski <rmilkowski at task.gda.pl> wrote:
>>> bash-3.00# mdb -k
>>> Loading modules: [ unix krtld genunix dtrace specfs ufs sd pcisch
md
>>> ip sctp usba fcp fctl qlc ssd crypto lofs zfs random ptm cpc nfs ]
>>> > segmap_percent/D
>>> segmap_percent:
>>> segmap_percent: 12
>>>
>>> (it''s static IIRC)
>> 
>> segmap_percent is only referenced in
>>
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sun4/os/startup.c#2270.
>> 
>> 
>> You are right that it is static but that means you cannot tune that
>> unless you run your own kernel.
MJ> I took a quick look at startup.c. Looks like you should be able to set
MJ> this value in /etc/system. You should not have to compile your own
kernel.

Definitely you can.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

On 4/23/07, Manoj Joseph <manoj at clusterfs.com>
wrote:
> Wee Yeh Tan wrote:
> > On 4/23/07, Robert Milkowski <rmilkowski at task.gda.pl> wrote:
> >> bash-3.00# mdb -k
> >> Loading modules: [ unix krtld genunix dtrace specfs ufs sd pcisch
md
> >> ip sctp usba fcp fctl qlc ssd crypto lofs zfs random ptm cpc nfs ]
> >> > segmap_percent/D
> >> segmap_percent:
> >> segmap_percent: 12
> >>
> >> (it''s static IIRC)
> >
> > segmap_percent is only referenced in
> >
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sun4/os/startup.c#2270.
> >
> >
> > You are right that it is static but that means you cannot tune that
> > unless you run your own kernel.
>
> I took a quick look at startup.c. Looks like you should be able to set
> this value in /etc/system. You should not have to compile your own kernel.
Ok, you got me here... segmep_percent is only mentioned on 2 occassions,
#121 sets segmap_percent to 12
#2270 uses segmap_percent to calculate the mapsize.

I didn''t spot anything that reads it from /etc/system.  Appreciate any
pointers.


-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Wee Yeh Tan wrote:
> I didn''t spot anything that reads it from /etc/system.  Appreciate
any
> pointers.
The beauty, and curse, of /etc/system is that modules do not need to create
an explicit reader.
  -- richard

On 4/24/07, Richard Elling <Richard.Elling at sun.com>
wrote:
> Wee Yeh Tan wrote:
> > I didn''t spot anything that reads it from /etc/system. 
Appreciate any
> > pointers.
>
> The beauty, and curse, of /etc/system is that modules do not need to create
> an explicit reader.
Grr.... I suspected after I replied that the mechanism should be
similiar to what mdb uses so assuming I''m looking at the right place
this time...
 http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/os/modsysfile.c#1172



-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Wee Yeh Tan wrote:
> On 4/24/07, Richard Elling <Richard.Elling at sun.com> wrote:
>> Wee Yeh Tan wrote:
>> > I didn''t spot anything that reads it from /etc/system. 
Appreciate any
>> > pointers.
>>
>> The beauty, and curse, of /etc/system is that modules do not need to 
>> create
>> an explicit reader.
> 
> Grr.... I suspected after I replied that the mechanism should be
> similiar to what mdb uses so assuming I''m looking at the right
place
> this time...
>
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/os/modsysfile.c#1172
Yes, this looks more appropriate.  IIRC, this section was rewritten for
Solaris 10, and many old, irritating problems were fixed along the way :-)
... but it is still a loaded weapon... be careful where you point it.
  -- richard