Note to readers: There are multiple topics discussed herein. Please identify which idea(s) you are responding to, should you respond. Also make sure to take in all of this before responding. Something you want to discuss may already be covered at a later point in this e-mail, including NDMP and ZFS ACLs. It''s loooong. It seems to me that something is being overlooked (either by myself or others) in all of these discussions about backing up ZFS pools... The one thing that I keep thinking, and which I have yet to see discredited, is that ZFS file systems use POSIX semantics. So, unless you are using specific features (notably ACLs, as Paul Henson is), you should be able to backup those file systems using well known tools. The ZFS Best Practices Guide speaks to this in section 4.4 (specifically 4.4.3[1]) and there have been various posters who have spoken of using other tools. (Star comes to mind, most prominently.) The Best Practices Guide is also very clear about send and receive NOT being designed explicitly for backup purposes. I find it odd that so many people seem to want to force this point. ZFS appears to have been designed to allow the use of well known tools that are available today to perform backups and restores. I''m not sure how many people are actually using NFS v4 style ACLs, but those people have the most to worry about when it comes to using tar or NetBackup or Networker or Amanda or Bacula or star to backup ZFS file systems. Everyone else, which appears to be the majority of people, have many tools to choose from, tools they''ve used for a long time in various environments on various platforms. The learning curve doesn''t appear to be as steep as most people seem to make it out to be. I honestly think many people may be making this issue more complex than it needs to be. Maybe the people having the most problems are those who are new to Solaris, but if you have any real *nix experience, Solaris shouldn''t be that difficult to figure out, especially for those with System V experience. The Linux folks? Well, I sorta feel sorry for you and I sorta don''t. So, am I missing something? It wouldn''t surprise me if I am. What am I missing? The other things I have been thinking about are NDMP support and what tools out there support NFS v4 ACLs. Has anyone successfully used NDMP support with ZFS? If so, what did you do? How did you configure your system, including any custom coding you did? From the looks of the NDMP project on os.org, NDMP was integrated in build 102[3] but it appears to only be NDMP v4 not the latest, v5. Maybe NDMP support would placate some of those screaming for the send stream to be a tape backup format? As for ACLs[2], the list of tools supporting NFS v4 ACLs seems to be pretty small. I plan to spend some quality time with RFC 3530 to get my head around NFS v4, and ACLs in particular. star seems to be fairly adept, with the exception of the NFS v4 ACL support. Hopefully that is forthcoming? Again, I think those people who are not using ZFS ACLs can probably perform actual tape backups (should they choose to) with existing tools. If I''m mistaken or missing something, I invite someone to please point it out. Finally, there''s backup of ZVOLs. I don''t know what the commercial tool support for backing up ZVOLs looks like but I know this is the *perfect* place for NDMP. Backing up ZVOLs should be priority #1 for NDMP support in (Open)Solaris, I think. Looking through the symbols in libzfs.so, I don''t see anything specifically related to backup of ZVOLs in the existing code. How are people handling ZVOL backups today? Not to be too flip, but star looks like it might be the perfect tape backup software if it supported NDMP, NFS v4 ACLs and ZVOLs. Just thinking out loud... [1] http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Using_ZFS_With_Enterprise_Backup_Solutions [2] http://docs.sun.com/app/docs/doc/819-5461/ftyxi?l=en&a=view [3] http://hub.opensolaris.org/bin/view/Project+ndmp/ Aside: I see so many posts to this list about backup strategy for ZFS file systems, and I continue to be amazed by how few people check the archives for previous discussions before they start a new one. So many of the conversations are repeated over and over, with good information being spread over multiple threads? I personally find it interesting that so few people read first before posting. Few even seem to bother to do so much (little?) as a Google search which would yield several previous discussions on the topic of ZFS pool backups to tape. Oh well. -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/85500c4d/attachment.html>
On 03/17/2010 08:28 PM, Khyron wrote:> The Best Practices Guide is also very clear about send and receive NOT > being > designed explicitly for backup purposes. I find it odd that so many > people seem to > want to force this point. ZFS appears to have been designed to allow > the use of > well known tools that are available today to perform backups and > restores. I''m not > sure how many people are actually using NFS v4 style ACLs, but those > people have > the most to worry about when it comes to using tar or NetBackup or > Networker or > Amanda or Bacula or star to backup ZFS file systems. Everyone else, > which appears > to be the majority of people, have many tools to choose from, tools > they''ve used > for a long time in various environments on various platforms. The > learning curve > doesn''t appear to be as steep as most people seem to make it out to > be. I honestly > think many people may be making this issue more complex than it needs > to be. > > Maybe the people having the most problems are those who are new to > Solaris, but > if you have any real *nix experience, Solaris shouldn''t be that > difficult to figure out, > especially for those with System V experience. The Linux folks? > Well, I sorta feel > sorry for you and I sorta don''t. > > > -- > "You can choose your friends, you can choose the deals." - Equity Private > > "If Linux is faster, it''s a Solaris bug." - Phil Harman > > Blog - http://whatderass.blogspot.com/ > Twitter - @khyron4evaMy only comment would be that your assumption is that most people are backing up their stuff via gzipped archives or using 3rd party solutions. I would argue (at least in my environment) that the majority of backups pre-zfs was doing using ufsdump. When ZFS came along it obsoleted ufsdump and their was no direct replacement. For my situation we have created some custom python code to wrap send/receive stuff between our machine''s pools and our backup server pool. It would be nice, however, if some sort of enterprise level backup solution in the style of ufsdump was introduced to ZFS. -- Steve
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17.03.2010 12:28, Khyron wrote:> Note to readers: There are multiple topics discussed herein. Please > identify which > idea(s) you are responding to, should you respond. Also make sure to > take in all of > this before responding. Something you want to discuss may already be > covered at > a later point in this e-mail, including NDMP and ZFS ACLs. It''s loooong. > > It seems to me that something is being overlooked (either by myself or > others) in all > of these discussions about backing up ZFS pools... > > The one thing that I keep thinking, and which I have yet to see > discredited, is that > ZFS file systems use POSIX semantics. So, unless you are using specific > features > (notably ACLs, as Paul Henson is), you should be able to backup those > file systems > using well known tools. The ZFS Best Practices Guide speaks to this in > section 4.4 > (specifically 4.4.3[1]) and there have been various posters who have > spoken of using > other tools. (Star comes to mind, most prominently.) > > The Best Practices Guide is also very clear about send and receive NOT > being > designed explicitly for backup purposes. I find it odd that so many > people seem to > want to force this point. ZFS appears to have been designed to allow > the use of > well known tools that are available today to perform backups and > restores. I''m not > sure how many people are actually using NFS v4 style ACLs, but those > people have > the most to worry about when it comes to using tar or NetBackup or > Networker or > Amanda or Bacula or star to backup ZFS file systems. Everyone else, > which appears > to be the majority of people, have many tools to choose from, tools > they''ve used > for a long time in various environments on various platforms. The > learning curve > doesn''t appear to be as steep as most people seem to make it out to be. > I honestly > think many people may be making this issue more complex than it needs to be. > > Maybe the people having the most problems are those who are new to > Solaris, but > if you have any real *nix experience, Solaris shouldn''t be that > difficult to figure out, > especially for those with System V experience. The Linux folks? Well, > I sorta feel > sorry for you and I sorta don''t. > > So, am I missing something? It wouldn''t surprise me if I am. What am I > missing? > > The other things I have been thinking about are NDMP support and what > tools out > there support NFS v4 ACLs. > > Has anyone successfully used NDMP support with ZFS? If so, what did you > do? How > did you configure your system, including any custom coding you did? > From the looks > of the NDMP project on os.org <http://os.org>, NDMP was integrated in > build 102[3] but it appears > to only be NDMP v4 not the latest, v5. Maybe NDMP support would placate > some of > those screaming for the send stream to be a tape backup format? > > As for ACLs[2], the list of tools supporting NFS v4 ACLs seems to be > pretty small. I > plan to spend some quality time with RFC 3530 to get my head around NFS > v4, and > ACLs in particular. star seems to be fairly adept, with the exception > of the NFS v4 > ACL support. Hopefully that is forthcoming? Again, I think those > people who are > not using ZFS ACLs can probably perform actual tape backups (should they > choose > to) with existing tools. If I''m mistaken or missing something, I invite > someone to > please point it out. > > Finally, there''s backup of ZVOLs. I don''t know what the commercial tool > support > for backing up ZVOLs looks like but I know this is the *perfect* place > for NDMP. > Backing up ZVOLs should be priority #1 for NDMP support in > (Open)Solaris, I think. > Looking through the symbols in libzfs.so, I don''t see anything > specifically related to > backup of ZVOLs in the existing code. How are people handling ZVOL backups > today? > > Not to be too flip, but star looks like it might be the perfect tape > backup software > if it supported NDMP, NFS v4 ACLs and ZVOLs. Just thinking out loud... > > [1] > http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#Using_ZFS_With_Enterprise_Backup_Solutions > > [2] http://docs.sun.com/app/docs/doc/819-5461/ftyxi?l=en&a=view > <http://docs.sun.com/app/docs/doc/819-5461/ftyxi?l=en&a=view> > > [3] http://hub.opensolaris.org/bin/view/Project+ndmp/ > > Aside: I see so many posts to this list about backup strategy for ZFS > file systems, > and I continue to be amazed by how few people check the archives for > previous > discussions before they start a new one. So many of the conversations are > repeated over and over, with good information being spread over multiple > threads? > I personally find it interesting that so few people read first before > posting. Few > even seem to bother to do so much (little?) as a Google search which > would yield > several previous discussions on the topic of ZFS pool backups to tape. > > Oh well.How does backing up the NFSv4 acls help you backup up a zvol (shared for iSCSI)? Please enlighten me. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkugy7MACgkQSBMQn1jNM7btJgCghy0ClEjSeRHzDHEucmgMJQHr DYoAoOuh9bUKVLV6cX0kP88sQOHmw8g+ =5MDK -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17.03.2010 13:31, Svein Skogen wrote:> On 17.03.2010 12:28, Khyron wrote: >> Note to readers: There are multiple topics discussed herein. Please >> identify which*SNIP*> > How does backing up the NFSv4 acls help you backup up a zvol (shared for > iSCSI)? Please enlighten me. > > //Svein >Again I replied before fully reading the mail I replied to (had I read it properly I''d have spotted you already mentioned this). My apologies to the list for the redundant posting. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkugzLwACgkQSBMQn1jNM7advQCeN90sNaKKyYMVERnXFXDBtYD0 KbcAn2rOSt+/b2WfNi3ZHzs5c03+6Y2r =08EZ -----END PGP SIGNATURE-----
Joerg Schilling
2010-Mar-17 13:01 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Stephen Bunn <scbunn at sbunn.org> wrote:> between our machine''s pools and our backup server pool. It would be > nice, however, if some sort of enterprise level backup solution in the > style of ufsdump was introduced to ZFS.Star can do the same as ufsdump does but independent of OS and filesystem. Star is currently missing support for ZFS ACLs and for extended attributes from Solaris. If you are interested, make a test. If you need support for ZFS ACLs or Solaris extendd attributes, send me a note. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Exactly! This is what I meant, at least when it comes to backing up ZFS datasets. There are tools available NOW, such as Star, which will backup ZFS datasets due to the POSIX nature of those datasets. As well, Amanda, Bacula, NetBackup, Networker and probably some others I missed. Re-inventing the wheel is not required in these cases. As I said in my original e-mail, Star is probably perfect once it gets ZFS (e.g. NFS v4) ACL and NDMP support (e.g. accepting NDMP input streams and ouputting onto tape). ZVOLs are the piece I''m still not sure about though. So I repeat my question: how are people backing up ZVOLs today? (If Star could do ZVOLs as well as NDMP and ZFS ACLs, then it literally *is* perfect.) On Wed, Mar 17, 2010 at 09:01, Joerg Schilling < Joerg.Schilling at fokus.fraunhofer.de> wrote:> Stephen Bunn <scbunn at sbunn.org> wrote: > > > between our machine''s pools and our backup server pool. It would be > > nice, however, if some sort of enterprise level backup solution in the > > style of ufsdump was introduced to ZFS. > > Star can do the same as ufsdump does but independent of OS and filesystem. > > Star is currently missing support for ZFS ACLs and for extended attributes > from Solaris. If you are interested, make a test. If you need support for > ZFS > ACLs or Solaris extendd attributes, send me a note. > > J?rg > > -- > EMail:joerg at schily.isdn.cs.tu-berlin.de<EMail%3Ajoerg at schily.isdn.cs.tu-berlin.de>(home) J?rg Schilling D-13353 Berlin > js at cs.tu-berlin.de (uni) > joerg.schilling at fokus.fraunhofer.de (work) Blog: > http://schily.blogspot.com/ > URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily >-- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/8f1bf223/attachment.html>
Edward Ned Harvey
2010-Mar-17 13:37 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> The one thing that I keep thinking, and which I have yet to see > discredited, is that > ZFS file systems use POSIX semantics.? So, unless you are using > specific features > (notably ACLs, as Paul Henson is), you should be able to backup those > file systems > using well known tools.?This is correct. Many people do backup using tar, star, rsync, etc.> The Best Practices Guide is also very clear about send and receive NOT > being > designed explicitly for backup purposes.? I find it odd that so many > people seem to > want to force this point.? ZFS appears to have been designed to allow > the use of > well known tools that are available today to perform backups and > restores.? I''m not > sure how many people are actually using NFS v4 style ACLs, but those > people have > the most to worry about when it comes to using tar or NetBackup or > Networker or > Amanda or Bacula or star to backup ZFS file systems.? Everyone else, > which appears > to be the majority of people, have many tools to choose from, tools > they''ve used > for a long time in various environments on various platforms.? The > learning curve > doesn''t appear to be as steep as most people seem to make it out to > be.? I honestly > think many people may be making this issue more complex than it needs > to be.I think what you''re saying is: Why bother trying to backup with "zfs send" when the recommended practice, fully supportable, is to use other tools for backup, such as tar, star, Amanda, bacula, etc. Right? The answer to this is very simple. #1 "zfs send" is much faster. Particularly for incrementals on large numbers of files. #2 "zfs send" will support every feature of the filesystem, including things like filesystem properties, hard links, symlinks, and objects which are not files, such as character special objects, fifo pipes, and so on. Not to mention ACL''s. If you''re considering some other tool (rsync, star, etc), you have to read the man pages very carefully to formulate the exact backup command, and there''s no guarantee you''ll find a perfect backup command. There is a certain amount of comfort knowing that the people who wrote "zfs send" are the same people who wrote the filesystem. It''s simple, and with no arguments, and no messing around with man page research, it''s guaranteed to make a perfect copy of the whole filesystem. Did I mention fast? ;-) Prior to zfs, I backed up my file server via rsync. It''s 1TB of mostly tiny files, and it ran for 10 hours every night, plus 30 hours every weekend. Now, I use zfs send, and it runs for an average 7 minutes every night, depending on how much data changed that day, and I don''t know - 20 hours I guess - every month.
To be sure, Ed, I''m not asking: Why bother trying to backup with "zfs send" when there are fully supportable and working options available right NOW? Rather, I am asking: Why do we want to adapt "zfs send" to do something it was never intended to do, and probably won''t be adapted to do (well, if at all) anytime soon instead of optimizing existing technologies for this use case? But I got it. "zfs send" is fast. Let me ask you this, Ed...where do you "zfs send" your data to? Another pool? Does it go to tape eventually? If so, what is the setup such that it goes to tape? I apologize for asking here, as I''m sure you described it in one of the other threads I mentioned, but I''m not able to go digging in those threads at the moment. I ask this because I see an opportunity to kill 2 birds with one stone. With proper NDMP support and "zfs send" performance, why can''t you get the advantages of "zfs send" without trying to shoehorn "zfs send" into a use it''s not designed for? Maybe NDMP support needs to be a higher focus of the ZFS team? I noticed not many people even seem to be asking for it, never mind screaming for it. However, I did say this in my original e-mail - that I see NDMP support as being a way to handle the calls for "zfs send" to tape. Maybe we can broaden the conversation at this point. For all of those who use NDMP today to backup Filers, be they NetApp, EMC, or other vendors'' devices...how is your experience with NDMP? *IS* anyone using NDMP? If you have the option of using NDMP and you don''t, why don''t you? Backing up file servers directly to tape seems to be an obvious WIN, so if people aren''t doing it, I''m curious why they aren''t. That''s any kind of file server, because (Open)Solaris will increasingly be applied in this role. That was pretty much the goal of the Fishworks team, IIRC. So this looks like an opportunity by Sun (Oracle) to take a neglected backup technology and make it a must-have backup technology, by making it integrate smoothly with ZFS and high performance. On Wed, Mar 17, 2010 at 09:37, Edward Ned Harvey <solaris2 at nedharvey.com>wrote:> > The one thing that I keep thinking, and which I have yet to see > > discredited, is that > > ZFS file systems use POSIX semantics. So, unless you are using > > specific features > > (notably ACLs, as Paul Henson is), you should be able to backup those > > file systems > > using well known tools. > > This is correct. Many people do backup using tar, star, rsync, etc. > > > > The Best Practices Guide is also very clear about send and receive NOT > > being > > designed explicitly for backup purposes. I find it odd that so many > > people seem to > > want to force this point. ZFS appears to have been designed to allow > > the use of > > well known tools that are available today to perform backups and > > restores. I''m not > > sure how many people are actually using NFS v4 style ACLs, but those > > people have > > the most to worry about when it comes to using tar or NetBackup or > > Networker or > > Amanda or Bacula or star to backup ZFS file systems. Everyone else, > > which appears > > to be the majority of people, have many tools to choose from, tools > > they''ve used > > for a long time in various environments on various platforms. The > > learning curve > > doesn''t appear to be as steep as most people seem to make it out to > > be. I honestly > > think many people may be making this issue more complex than it needs > > to be. > > I think what you''re saying is: Why bother trying to backup with "zfs send" > when the recommended practice, fully supportable, is to use other tools for > backup, such as tar, star, Amanda, bacula, etc. Right? > > The answer to this is very simple. > #1 "zfs send" is much faster. Particularly for incrementals on large > numbers of files. > #2 "zfs send" will support every feature of the filesystem, including > things like filesystem properties, hard links, symlinks, and objects which > are not files, such as character special objects, fifo pipes, and so on. > Not to mention ACL''s. If you''re considering some other tool (rsync, star, > etc), you have to read the man pages very carefully to formulate the exact > backup command, and there''s no guarantee you''ll find a perfect backup > command. There is a certain amount of comfort knowing that the people who > wrote "zfs send" are the same people who wrote the filesystem. It''s > simple, > and with no arguments, and no messing around with man page research, it''s > guaranteed to make a perfect copy of the whole filesystem. > > Did I mention fast? ;-) Prior to zfs, I backed up my file server via > rsync. It''s 1TB of mostly tiny files, and it ran for 10 hours every night, > plus 30 hours every weekend. Now, I use zfs send, and it runs for an > average 7 minutes every night, depending on how much data changed that day, > and I don''t know - 20 hours I guess - every month. > >-- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/1c3003c9/attachment.html>
Edward Ned Harvey
2010-Mar-17 14:15 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> I think what you''re saying is: Why bother trying to backup with "zfs > send" > when the recommended practice, fully supportable, is to use other tools > for > backup, such as tar, star, Amanda, bacula, etc. Right? > > The answer to this is very simple. > #1 ... > #2 ...Oh, one more thing. "zfs send" is only discouraged if you plan to store the data stream and do "zfs receive" at a later date. If instead, you are doing "zfs send | zfs receive" onto removable media, or another server, where the data is immediately fed through "zfs receive" then it''s an entirely viable backup technique.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17.03.2010 15:15, Edward Ned Harvey wrote:>> I think what you''re saying is: Why bother trying to backup with "zfs >> send" >> when the recommended practice, fully supportable, is to use other tools >> for >> backup, such as tar, star, Amanda, bacula, etc. Right? >> >> The answer to this is very simple. >> #1 ... >> #2 ... > > Oh, one more thing. "zfs send" is only discouraged if you plan to store the > data stream and do "zfs receive" at a later date. > > If instead, you are doing "zfs send | zfs receive" onto removable media, or > another server, where the data is immediately fed through "zfs receive" then > it''s an entirely viable backup technique.... Fine. Tell me how to make a zpool on LTO-3 tapes. ;) Hence my earlier questions about the possibility of simply adding a parameter to the send/receive commands to use the ALREADY BUILT IN code for providing some sort of FEC to the stream. It _REALLY_ would solve the "store" problem. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkug5OEACgkQSBMQn1jNM7aO9ACfeRk+IWz+itts93TIoh8vLl1S aIcAn3A+/YWmqhe1tEUx6UKutz4w0BIy =MXk7 -----END PGP SIGNATURE-----
David Dyer-Bennet
2010-Mar-17 14:53 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Wed, March 17, 2010 06:28, Khyron wrote:> The Best Practices Guide is also very clear about send and receive > NOT being designed explicitly for backup purposes. I find it odd > that so many people seem to want to force this point. ZFS appears > to have been designed to allow the use of well known tools that are > available today to perform backups and restores. I''m not sure how > many people are actually using NFS v4 style ACLs, but those people > have the most to worry about when it comes to using tar or NetBackup > or Networker or Amanda or Bacula or star to backup ZFS file systems. > Everyone else, which appears to be the majority of people, have many > tools to choose from, tools they''ve used for a long time in various > environments on various platforms. The learning curve doesn''t > appear to be as steep as most people seem to make it out to be. I > honestly think many people may be making this issue more complex > than it needs to be.Anybody using the in-kernel CIFS is also concerned with the ACLs, and I think that''s the big issue. Also, snapshots. For my purposes, I find snapshots at some level a very important part of the backup process. My old scheme was to rsync from primary ZFS pool to backup ZFS pool, and snapshot both pools (with somewhat different retention schedules). My new scheme, forced by the ACL issues, is to use ZFS send/receive (but I haven''t been able to make it work yet), including snapshots. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Edward Ned Harvey
2010-Mar-17 15:19 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> Why do we want to adapt "zfs send" to do something it was never > intended > to do, and probably won''t be adapted to do (well, if at all) anytime > soon instead of > optimizing existing technologies for this use case?The only time I see or hear of anyone using "zfs send" in a way it wasn''t intended is when people store the datastream on tape or a filesystem, instead of feeding it directly into "zfs receive." Although it''s officially discouraged for this purpose, there is value in doing so, and I can understand why some people sometimes (including myself) would have interest in doing this. So let''s explore the reasons it''s discouraged to store a "zfs send" datastream: #1 If a single bit goes bad, the whole dataset is bad. #2 You can only receive the whole filesystem. You cannot granularly restore a single file or directory. Now, if you acknowledge these two points, let''s explore why somebody might want to do it anyway: To counter #1: Let''s acknowledge that storage media is pretty reliable. We''ve all seen tapes and disks go bad, but usually they don''t. If you''ve got a new tape archive every week or every month... The probability of *all* of those tapes having one or more bad bits is astronomically low. Nonzero risk, but a calculated risk. To counter #2: There are two basic goals for backups. (a) to restore some stuff upon request, or (b) for the purposes of DR, to guarantee your manager that you''re able to get the company back into production quickly after a disaster. Such as the building burning down. ZFS send to tape does not help you in situation (a). So we can conclude that "zfs send" to tape is not sufficient as an *only* backup technique. You need something else, and at most, you might consider "zfs send" to tape as an augmentation to your other backup technique. Still ... If you''re in situation (b) then you want as many options available to you as possible. I''ve helped many people and/or companies before, who ... Had backup media, but didn''t have the application that wrote the backup media and therefore couldn''t figure out how to restore. ... Had a backup system that was live synchronizing the master file server to a slave file server, and then when something blew up the master, it propagated and deleted the slave too. In this case, the only thing that saved them was an engineer who had copied the whole directory a week ago onto his iPod, if you can believe that. ... Had backup tapes but no tape drive ... Had archives on DVD, and the DVD''s were nearly all bad ... Looked through the backups only to discover something critical had been accidentally excluded ... Point is, having as many options available as possible is worthwhile in the disaster situation. Please see below for some more info, as it ties into some more of what you''ve said ...> But I got it.? "zfs send" is fast.? Let me ask you this, Ed...where do > you "zfs send" > your data to? Another pool?? Does it go to tape eventually?? If so, > what is the setup > such that it goes to tape?? I apologize for asking here, as I''m sure > you described it > in one of the other threads I mentioned, but I''m not able to go digging > in those > threads at the moment.Here is my backup strategy: I use "zfs send | ssh somehost ''zfs receive''" to send nightly incrementals to a secondary backup server. This way, if something goes wrong with the primary fileserver, I can simply change the IP address of the secondary, and let it assume the role of the primary. With the unfortunate loss of all of today''s data ... going back to last night. I have had to do this once before, in the face of primary fileserver disaster and service contract SLA failure by Netapp... All the users were very pleased that I was able to get them back into production using last night''s data in less than a few minutes.>From the secondary server, I "zfs send | zfs receive" onto removable harddisks. This is ideal to restore either individual files, or the whole filesystem. No special tools would be necessary to restore on any random ZFS server in the future, and nothing could be faster. In fact, you wouldn''t even need to restore if you wanted to in a pinch, you could work directly on the external disks. However, removable disks are not very reliable compared to tapes, and the disks are higher cost per GB, and require more volume in the safe deposit box, so the external disk usage is limited... Only going back for 2-4 weeks of archive... So there is also a need for tapes. Once every so often, from the secondary server, I "zfs send" the whole filesystem onto tape for archival purposes. This would only be needed after a disaster, and also the failure or overwriting of the removable disks. We have so many levels of backups, this is really unnecessary, but it makes me feel good. And finally just because the data is worth millions of dollars, I also use NetBackup to write tapes from the secondary server. This way, nobody could ever blame me if the data were ever lost somehow. I won''t get sued or have criminal charges pressed against me; my reputation will remain intact. I''m protecting against the possibility of me being an idiot.> I ask this because I see an opportunity to kill 2 birds with one > stone.? With proper > NDMP support and "zfs send" performance, why can''t you get the > advantages of > "zfs send" without trying to shoehorn "zfs send" into a use it''s not > designed for?I could be speaking out of line, but I think fundamentally, NDMP support cannot have "zfs send" performance. Because "zfs send" is taking advantage of the copy-on-write to efficiently generate a datastream from disk blocks, it doesn''t even need to think about the filesystem or the files; it''s just reading a prescribed set of disk blocks, as fast and as sequential as possible. No need to search and examine files to see what''s changed; you already have a simple list of disk blocks to read. Whereas NDMP, tar, star, rsync, and everything else that I know of are all fundamentally a file-level backup system, thus necessitating a far less efficient random access, with filesystem reading, seeking, and processing required. But sure, if it''s possible, NDMP or any other protocol or format performing like "zfs send" would be awesome. ;-)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17.03.2010 16:19, Edward Ned Harvey wrote: *snip> Still ... If you''re in situation (b) then you want as many options available > to you as possible. I''ve helped many people and/or companies before, who > ... Had backup media, but didn''t have the application that wrote the backup > media and therefore couldn''t figure out how to restore. ... Had a backup > system that was live synchronizing the master file server to a slave file > server, and then when something blew up the master, it propagated and > deleted the slave too. In this case, the only thing that saved them was an > engineer who had copied the whole directory a week ago onto his iPod, if you > can believe that. ... Had backup tapes but no tape drive ... Had > archives on DVD, and the DVD''s were nearly all bad ... Looked through the > backups only to discover something critical had been accidentally excludedI can add a few items on your list, that you''ve probably seen, but forgot to list: - -Using a tape format that really isn''t a standard (anybody remember the first generation of DDS with compression, that ... probably could only be restored on the exact identical drive that wrote them?) - -Enabling the fancy "security feature" and not keeping a safe copy of the encryption keys they''re using on the tapes? - -Forgetting that a disaster-recovery setup SHOULD start out with pen+paper planning "what to do _WHEN_ disaster strikes." Including making a step by step list. This way you can plan what prerequisites your disaster recovery has. How do you bootstrap your recovery? What do you need up and running to read your tapes? What routines do you have for bringing offline media off site for safekeeping? (and this "plan for disaster" is the reasons for my many questions on these lists lately. I simply *will not* switch to a new solution that I cannot properly plan for a disaster recovery of!). What I''m getting at, is that most of the disaster recovery procedure should actually be done _BEFORE_ disaster strikes. And use that work to do your best to make disasters unlikely. If your backup solution depends on a running system, bring a livecd or similar in the same suitcase as the tapes. If you don''t you''ll have a major problem when you NEED the backup. Proper backups for disaster-recovery should be offline media. Disaster isn''t only "place burning to the ground, switch to secondary location". It can also be a rogue employee (or outsider) data-bombing your infrastructure. If those backups are online-and-connected, there is little reason to trust that they weren''t destroyed as well. If you back up to a mounted file system, using a cronjob, mount the fs for the backup and dismount it after verify. Don''t keep it eternally mounted, since a situation corrupting the mounted data then could corrupt the backup as well. This falls back to "plan for disaster". Include "human failures" and "sysadmin accidents" in the disaster plan. And document every step. This because there''s no guarantee the sysadmin manages to get out of his dungeon when the building burns to the ground. A proper disaster recovery plan would mean that anyone competent can read the hardcopy of the plan, and restore the system. And so on. Maybe someone should add some "plan for disaster" to the "best current practices" zfs-page? ;) //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkug9u8ACgkQSBMQn1jNM7bBmwCg/i+Wcd9rXmWmhm6HReaP7XRe IFwAoNMg8UzbX+gs2ZvAzvplEtwehfbR =0Xlw -----END PGP SIGNATURE-----
> >> I think what you''re saying is: Why bother trying to backup with "zfs >> send" >> when the recommended practice, fully supportable, is to use other tools >> for >> backup, such as tar, star, Amanda, bacula, etc. Right? >> >> The answer to this is very simple. >> #1 ... >> #2 ... >> > Oh, one more thing. "zfs send" is only discouraged if you plan to store the > data stream and do "zfs receive" at a later date. >This is no longer the case. The send stream format is now versioned in such a way that future versions of Solaris will be able to read send streams generated by earlier versions of Solaris. The comment in the zfs(1M) manpage discouraging the use of send streams for later restoration has been removed. This versioning leverages the ZFS object versioning that already exists (the versioning that allows earlier version pools to be read by later versions of zfs), plus versioning and feature flags in the stream header. Lori
>>>>> "k" == Khyron <khyron4eva at gmail.com> writes:k> Star is probably perfect once it gets ZFS (e.g. NFS v4) ACL nope, because snapshots are lost and clones are expanded wrt their parents, and the original tree of snapshots/clones can never be restored. we are repeating, though. This is all in the archives. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/b94f0e9d/attachment.bin>
>>>>> "la" == Lori Alt <Lori.Alt at Sun.COM> writes:la> This is no longer the case. The send stream format is now la> versioned in such a way that future versions of Solaris will la> be able to read send streams generated by earlier versions of la> Solaris. Your memory of the thread is selective. This is only one of the several problems with it. If you are not concerned with bitflip gremlins on tape, then all the baloney about checksums and copies=2 metadata and insisting on zpool-level redundancy is just a bunch of opportunistic FUD. la> The comment in the zfs(1M) manpage discouraging the la> use of send streams for later restoration has been removed. The man page never warned of all the problems, nor did the si wiki. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/b6687045/attachment.bin>
David Dyer-Bennet
2010-Mar-17 17:18 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Wed, March 17, 2010 10:19, Edward Ned Harvey wrote:> However, removable disks are not very > reliable compared to tapes, and the disks are higher cost per GB, and > require more volume in the safe deposit box, so the external disk usage is > limited... Only going back for 2-4 weeks of archive...Last 5 times I checked, tapes were vastly more expensive than disks, even if you ignored the drive cost. Let''s see...currently buy.com lists an lt30 cartridge at about $25, for 400GB; a 2TB drive is about $170. So yeah, they''re a little cheaper than disk now (IF you ignore the drive cost; the cheap ones seem to be around $1800). I had a very satisfactory QIC-60 tape streamer in the 1980s, and a couple of rather unsatisfactory DAT drives later. I''ve been backing up to disk since then, seems to be a lot easier, quicker, cheaper, and more stable, on my scale. When you go to much bigger setups, the trade-offs change; the value of the space in the safe starts to show up as significant, the drive cost becomes less of an issue, and so forth. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 17.03.2010 18:18, David Dyer-Bennet wrote:> > On Wed, March 17, 2010 10:19, Edward Ned Harvey wrote: > >> However, removable disks are not very >> reliable compared to tapes, and the disks are higher cost per GB, and >> require more volume in the safe deposit box, so the external disk usage is >> limited... Only going back for 2-4 weeks of archive... > > Last 5 times I checked, tapes were vastly more expensive than disks, even > if you ignored the drive cost. Let''s see...currently buy.com lists an > lt30 cartridge at about $25, for 400GB; a 2TB drive is about $170. So > yeah, they''re a little cheaper than disk now (IF you ignore the drive > cost; the cheap ones seem to be around $1800). > > I had a very satisfactory QIC-60 tape streamer in the 1980s, and a couple > of rather unsatisfactory DAT drives later. I''ve been backing up to disk > since then, seems to be a lot easier, quicker, cheaper, and more stable, > on my scale. When you go to much bigger setups, the trade-offs change; > the value of the space in the safe starts to show up as significant, the > drive cost becomes less of an issue, and so forth.Please, don''t compare proper backup drives to that rotating head non-standard catastrophy... DDS was (in)famous for being a delayed-fuse tape-shredder. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuhECYACgkQSBMQn1jNM7YcEACgzRyMvBRYwoaWkwIBujnPL06S 3LsAoLpr9YOBop4UBkz9JUrUQ9q8kVPz =w+1I -----END PGP SIGNATURE-----
On 03/18/10 03:53 AM, David Dyer-Bennet wrote:> Anybody using the in-kernel CIFS is also concerned with the ACLs, and I > think that''s the big issue. > >Especially in a paranoid organisation with 100s of ACEs!> Also, snapshots. For my purposes, I find snapshots at some level a very > important part of the backup process. My old scheme was to rsync from > primary ZFS pool to backup ZFS pool, and snapshot both pools (with > somewhat different retention schedules). My new scheme, forced by the ACL > issues, is to use ZFS send/receive (but I haven''t been able to make it > work yet), including snapshots. > >I''m sure the folks here can help with that. I have been using a two stage backup process with my main client, send/receive to a backup pool and spool to tape for off site archival. I use a pair (on connected, one off site) of removable drives as single volume pools for my own backups via send/receive. -- Ian.
Ian, When you say you spool to tape for off-site archival, what software do you use? On Wed, Mar 17, 2010 at 18:53, Ian Collins <ian at ianshome.com> wrote: <SNIP>> > I have been using a two stage backup process with my main client, > send/receive to a backup pool and spool to tape for off site archival. > > I use a pair (on connected, one off site) of removable drives as single > volume pools for my own backups via send/receive. > > -- > Ian. > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100317/44c3b311/attachment.html>
David Dyer-Bennet
2010-Mar-18 01:43 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 3/17/2010 17:53, Ian Collins wrote:> On 03/18/10 03:53 AM, David Dyer-Bennet wrote: > >> Also, snapshots. For my purposes, I find snapshots at some level a very >> important part of the backup process. My old scheme was to rsync from >> primary ZFS pool to backup ZFS pool, and snapshot both pools (with >> somewhat different retention schedules). My new scheme, forced by >> the ACL >> issues, is to use ZFS send/receive (but I haven''t been able to make it >> work yet), including snapshots. >> > I''m sure the folks here can help with that.At this point I''m waiting for the 2010.Spring; in the previous stable release I''m having large filesystems on the backup drives fail to destroy, and replication streams failing to complete, and so forth.> > I have been using a two stage backup process with my main client, > send/receive to a backup pool and spool to tape for off site archival. > > I use a pair (on connected, one off site) of removable drives as > single volume pools for my own backups via send/receive. >My own stuff is intended to be backed up by a short-cut combination -- zfs send/receive to an external drive, which I then rotate off-site (I have three of a suitable size). However, the only way that actually works so far is to destroy the pool (not just the filesystem) and recreate it from scratch, and then do a full replication stream. That works most of the time, hangs about 1/5. Anything else I''ve tried is much worse, with hangs approaching 100%. I''ve posted here a few times about it; as I say, I''m in waiting for next stable release mode, we''ll see how that does, and I''ll push much harder for some kind of resolution if I still have trouble then. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Daniel Carosone
2010-Mar-18 02:23 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Wed, Mar 17, 2010 at 08:43:13PM -0500, David Dyer-Bennet wrote:> My own stuff is intended to be backed up by a short-cut combination -- > zfs send/receive to an external drive, which I then rotate off-site (I > have three of a suitable size). However, the only way that actually > works so far is to destroy the pool (not just the filesystem) and > recreate it from scratch, and then do a full replication stream. That > works most of the time, hangs about 1/5. Anything else I''ve tried is > much worse, with hangs approaching 100%. > > I''ve posted here a few times about it; as I say, I''m in waiting for next > stable release mode, we''ll see how that does, and I''ll push much harder > for some kind of resolution if I still have trouble then.My guess, asserted without substantiation, is that this could well be related to the usb channel and drivers as much as to anything with zfs. In any case, if the new release as a package solves the problem, it doesn''t necessarily matter to you which specific change(s) within made the difference. If it still shows up in the current dev builds, or the release shortly, then some process of elimination to narrow down the problem area would be worthwhile. In particular, some umass bridge chips are frankly shite, and you may be unlucky enough to have those between your host and your backup disks. -- Dan. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 194 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100318/e8fd1db8/attachment.bin>
I vote for zfs needing a backup and restore command against a snapshot. backup command should output on stderr at least Full_Filename SizeBytes Modification_Date_1970secSigned so backup software can build indexes and stdout contains the data. The advantage of zfs providing the command is that as ZFS upgrades or new features are added backup vendors do not need to re-test their code. Could also mean that when encryption comes a long a property on pool could indicate if it is OK to decrypt the filenames only as part of a backup. restore would work the same way except you would pass a filename or a directory to restore etc. And backup software would send back the stream to zfs restore command. The other alternative is for zfs to provide a standard API for backups like Oracle does for RMAN. It would be very useful with snapshots across pools http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6916404 -- This message posted from opensolaris.org
Joerg Schilling
2010-Mar-18 09:17 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Damon Atkins <Damon_Atkins at yahoo.com.au> wrote:> I vote for zfs needing a backup and restore command against a snapshot. > > backup command should output on stderr at least > Full_Filename SizeBytes Modification_Date_1970secSigned > so backup software can build indexes and stdout contains the data.This is something that does not belong to stderr but to a separate stream that only allows to build a database. Stderr is used for error messages and warnings.> The advantage of zfs providing the command is that as ZFS upgrades or new features are added backup vendors do not need to re-test their code. Could also mean that when encryption comes a long a property on pool could indicate if it is OK to decrypt the filenames only as part of a backup. > > restore would work the same way except you would pass a filename or a directory to restore etc. And backup software would send back the stream to zfs restore command. > > The other alternative is for zfs to provide a standard API for backups like Oracle does for RMAN.You need to decide what you like to get......>From what I''ve read so far, zfs send is a block level api and thus cannot beused for real backups. As a result of being block level oriented, the interpretation of the data is done zfs and thus every new feature could be copied without changing the format. If you like to have a backup that is able to retrieve arbitrary single files, you need a backup api at file level. If you have such an api, you need to enhance the backup tool in many cases where the file metadata is enhanced in the filesystem. We need to discuss to find the best archive formats for ZFS(NTFS) ACLs and for extended file attributes. I invite erybody to join star development at: https://lists.berlios.de/mailman/listinfo/star-developers and http://mail.opensolaris.org/mailman/listinfo/star-discuss J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Joerg Schilling
2010-Mar-18 09:31 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Svein Skogen <svein at stillbilde.net> wrote:> Please, don''t compare proper backup drives to that rotating head > non-standard catastrophy... DDS was (in)famous for being a delayed-fuse > tape-shredder.DDS was a WOM (write only memory) type device. It did not report write errors and it had many read errors. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 10:31, Joerg Schilling wrote:> Svein Skogen <svein at stillbilde.net> wrote: > >> Please, don''t compare proper backup drives to that rotating head >> non-standard catastrophy... DDS was (in)famous for being a delayed-fuse >> tape-shredder. > > DDS was a WOM (write only memory) type device. It did not report write errors > and it had many read errors.Kind of like /dev/null. And about as useful in a restore situation. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuh83MACgkQSBMQn1jNM7bvrwCfUoIwu+YO8tfvb/mfSW063Wst jK0AoIhFdKig2bZd3RSOyEgTPTN3YNng =Gf9R -----END PGP SIGNATURE-----
Edward Ned Harvey
2010-Mar-18 12:32 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> My own stuff is intended to be backed up by a short-cut combination -- > zfs send/receive to an external drive, which I then rotate off-site (I > have three of a suitable size). However, the only way that actually > works so far is to destroy the pool (not just the filesystem) and > recreate it from scratch, and then do a full replication stream. That > works most of the time, hangs about 1/5. Anything else I''ve tried is > much worse, with hangs approaching 100%.Interesting, that''s precisely what we do at work, and it works 100% of the time. Solaris 10u8
Edward Ned Harvey
2010-Mar-18 12:37 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> From what I''ve read so far, zfs send is a block level api and thus > cannot be > used for real backups. As a result of being block level oriented, theWeirdo. The above "cannot be used for real backups" is obviously subjective, is incorrect and widely discussed here, so I just say "weirdo." I''m tired of correcting this constantly.> I invite erybody to join star development at:We know, you have an axe to grind. Don''t insult some other product just because it''s not the one you personally work on. Yours is better in some ways, and "zfs send" is better in some ways.
Joerg Schilling
2010-Mar-18 12:54 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Edward Ned Harvey <solaris2 at nedharvey.com> wrote:> > I invite erybody to join star development at: > > We know, you have an axe to grind. Don''t insult some other product just > because it''s not the one you personally work on. Yours is better in some > ways, and "zfs send" is better in some ways.If you have no technical issues to discuss, please stop insulting people/products. We are on OpenSolaris and we don''t like this kind of discussions on the mailing lists. Please act collaborative. It has been widely discussed here already that the output of zfs send cannot be used as a backup. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Darren J Moffat
2010-Mar-18 13:01 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 18/03/2010 12:54, Joerg.Schilling at fokus.fraunhofer.de wrote:> It has been widely discussed here already that the output of zfs send cannot be > used as a backup.First define exactly what you mean by "backup". Please don''t confuse "backup" and "archival" they aren''t the same thing. It would also help if the storage medium for the "backup" is defined and what the required access to it is - eg: full restore only, incremental restores, per file restore. The format is now committed and versioned. It is the only format that saves all of the information about a ZFS dataset (including its dataset properties) not just the data files, ACLs, extended attributes and system attributes. The steam itself even supports deduplication of the data blocks with in it. So exactly what makes it unsuitable for backup ? Is it the file format or the way the utility works ? If it is the format what is wrong with it ? If it is the utility what is needed to fix that ? -- Darren J Moffat
Carsten Aulbert
2010-Mar-18 13:04 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Hi all On Thursday 18 March 2010 13:54:52 Joerg Schilling wrote:> If you have no technical issues to discuss, please stop insulting > people/products. > > We are on OpenSolaris and we don''t like this kind of discussions on the > mailing lists. Please act collaborative. >May I suggest this to both of you.> It has been widely discussed here already that the output of zfs send > cannot be used as a backup.Depends on the exact definition of backup, e.g. if I may take this from wikipedia: "In information technology, a backup or the process of backing up refers to making copies of data so that these additional copies may be used to restore the original after a data loss event." In this regard zfs send *could* be a tool for a backup provided you have the means of decrypting/deciphering the blob coming out of it. OTOH if I used zfs send to replicate data to another machine/location together with zfs receive and put a label "backup" onto the receiver this would also count as a backup from where you can restore everything and/or partially. In case of ''star'' the blob coming out of it might also be useless if you don''t have star (or other tools) around for deciphering it - very unlikely, but still possible ;) Of course your (plural!) definition of backup may vary, thus I would propose first to settle on this before exchanging blows... Cheers Carsten
Joerg Schilling
2010-Mar-18 13:10 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Carsten Aulbert <carsten.aulbert at aei.mpg.de> wrote:> In case of ''star'' the blob coming out of it might also be useless if you don''t > have star (or other tools) around for deciphering it - very unlikely, but > still possible ;)I invite you to inform yourself about star and to test it yourself. Star''s backups are completely based on POSIX standard archive formats. If you don''t have star (which is not very probable as star is OpenSource), you may extract the incremental dumps from star using any standard POSIX compliant archiver. You just lose the information and ability to do incremental restores. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Joerg Schilling
2010-Mar-18 13:12 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Darren J Moffat <Darren.Moffat at oracle.com> wrote:> So exactly what makes it unsuitable for backup ? > > Is it the file format or the way the utility works ? > > If it is the format what is wrong with it ? > > If it is the utility what is needed to fix that ?This has been discussed many times in the past already. If you archive the incremental "star send" data streams, you cannot extract single files andit seems that this cannot be fixed without introducing a different archive format. Star implements incremental backups and restores based on POSIX compliant archives. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 14:12, Joerg Schilling wrote:> Darren J Moffat <Darren.Moffat at oracle.com> wrote: > >> So exactly what makes it unsuitable for backup ? >> >> Is it the file format or the way the utility works ? >> >> If it is the format what is wrong with it ? >> >> If it is the utility what is needed to fix that ? > > This has been discussed many times in the past already. > > If you archive the incremental "star send" data streams, you cannot > extract single files andit seems that this cannot be fixed without > introducing a different archive format. > > Star implements incremental backups and restores based on POSIX compliant > archives.And how does your favourite tool handle zvols? //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuiJ40ACgkQSBMQn1jNM7ZShgCfaSXEz2/SjsKwZYIJ6TAFRBzF QkAAoJeH7tLHjgL5ECzHhAtlig+qtnat =Pt1K -----END PGP SIGNATURE-----
Joerg Schilling
2010-Mar-18 13:18 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) wrote:> > This has been discussed many times in the past already. > > If you archive the incremental "star send" data streams, you cannot > extract single files andit seems that this cannot be fixed without > introducing a different archive format.Sorry for the typo: this should be "zfs send" J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Darren J Moffat
2010-Mar-18 13:28 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 18/03/2010 13:12, Joerg.Schilling at fokus.fraunhofer.de wrote:> Darren J Moffat<Darren.Moffat at oracle.com> wrote: > >> So exactly what makes it unsuitable for backup ? >> >> Is it the file format or the way the utility works ? >> >> If it is the format what is wrong with it ? >> >> If it is the utility what is needed to fix that ? > > This has been discussed many times in the past already.> If you archive the incremental "star send" data streams, you cannot > extract single files andit seems that this cannot be fixed without > introducing a different archive format.That assumes you are writing the ''zfs send'' stream to a file or file like media. In many cases people using ''zfs send'' for they backup strategy are they are writing it back out using ''zfs recv'' into another pool. In those cases the files can even be restored over NFS/CIFS by using the .zfs/snapshot directory For example: http://hub.opensolaris.org/bin/download/User+Group+losug/w%2D2009/Open%2DBackup%2Dwith%2DNotes.pdf> Star implements incremental backups and restores based on POSIX compliant > archives.ZFS filesystem have functionality beyond POSIX and some of that is really very important for some people (especially those using CIFS) Does Star (or any other POSIX archiver) backup: ZFS ACLs ? ZFS system attributes (as used by the CIFS server and locally) ? ZFS dataset properties (compression, checksum etc) ? If it doesn''t then it is providing an "archive" of the data in the filesystem, not a full/incremental copy of the ZFS dataset. Which depending on the requirements of the backup may not be enough. In otherwords you have data/metadata missing from your backup. The only tool I''m aware of today that provides a copy of the data, and all of the ZPL metadata and all the ZFS dataset properties is ''zfs send''. Just like (s)tar alone is not an enterprise backup tool, neither is ''zfs send''. Both of them need some scripting and infrastructure mangement around them to make a backup solution suitable for a given deployment. In some deployments maybe the correct answer is both. Each have their place (s)tar is a file/directory archiver, ''zfs send'' on the other hand is a ZFS dataset (not just ZPL fileystems since it works on ZVOLs and all future dataset types too) replication tool that happens to write out a "stream". -- Darren J Moffat
On Wed, Mar 17, 2010 at 9:15 AM, Edward Ned Harvey <solaris2 at nedharvey.com> wrote:>> I think what you''re saying is: ?Why bother trying to backup with "zfs >> send" >> when the recommended practice, fully supportable, is to use other tools >> for >> backup, such as tar, star, Amanda, bacula, etc. ? Right? >> >> The answer to this is very simple. >> #1 ?... >> #2 ?... > > Oh, one more thing. ?"zfs send" is only discouraged if you plan to store the > data stream and do "zfs receive" at a later date. > > If instead, you are doing "zfs send | zfs receive" onto removable media, or > another server, where the data is immediately fed through "zfs receive" then > it''s an entirely viable backup technique.Richard Elling made an interesting observation that suggests that storing a zfs send data stream on tape is a quite reasonable thing to do. Richard''s background makes me trust his analysis of this much more than I trust the typical person that says that zfs send output is poison. http://opensolaris.org/jive/thread.jspa?messageID=465973&tstart=0#465861 I think that a similar argument could be made for storing the zfs send data streams on a zfs file system. However, it is not clear why you would do this instead of just zfs send | zfs receive. -- Mike Gerdts http://mgerdts.blogspot.com/
A system with 100TB of data its 80% full and the a user ask their local system admin to restore a directory with large files, as it was 30days ago with all Windows/CIFS ACLS and NFSv4/ACLS etc. If we used zfs send, we need to go back to a zfs send some 30days ago, and find 80TB of disk space to be able to restore it. zfs send/recv is great for copy zfs from one zfs file system to another file system even across servers. But their needs to be a tool: * To restore an individual file or a zvol (with all ACLs/properties) * That allows backup vendors (which place backups on tape or disk or CD or ..) build indexes of what is contain in the backup (e.g. filename, owner, size modification dates, type (dir/file/etc) ) *Stream output suitable for devices like tape drives. *Should be able to tell if the file is corrupted when being restored. *May support recovery of corrupt data blocks within the stream. *Preferable gnutar command-line compatible *That admins can use to backup and transfer a subset of files e.g user home directory (which is not a file system) to another server or on to CD to be sent to their new office location, or ???? For backup vendors is the idea for them to use NDMP protocol to backup ZFS and all its properties/ACLs????? Or is a new tool required to achieve the above?? Cheers -- This message posted from opensolaris.org
On 18 mars 2010, at 15:51, Damon Atkins wrote:> A system with 100TB of data its 80% full and the a user ask their local system admin to restore a directory with large files, as it was 30days ago with all Windows/CIFS ACLS and NFSv4/ACLS etc. > > If we used zfs send, we need to go back to a zfs send some 30days ago, and find 80TB of disk space to be able to restore it. > > zfs send/recv is great for copy zfs from one zfs file system to another file system even across servers.Bingo ! The zfs send/recv scenario is for backup to another site or server. Backup in this context being a second copy stored independently from the original/master. In one scenario here, we have individual sites that have zvol backed iSCSI volumes based on small, high performance 15K disks in mirror vdevs for the best performance. I only keep about a week of daily snapshots locally. I use ZFS send/recv to a backup system where I have lots of cheap, slow SATA drives in RAIDZ6 where I can afford to accumulate a lot more historical snapshots. The interest is that you can use the same tools in an asymmetric manner, with high performance primary systems and one or a few big slow systems to store your backups. Now for instances where I need to go back and get a file back off an NFS published filesystem, I can just go browse the .zfs/snapshot directory as required - or search for it or whatever I want. It''s a live filesystem, not an inert object, dependent on external indices and hardware. I think that this is the fundamental disconnect in these discussions where people''s ideas (or requirements) of what constitutes "a backup" are conflicting. There are two major reasons and types of backups : one is to be able to minimize your downtime and get systems running again as quickly as possible. (the server''s dead - make it come back!). The other is the ability to go back in time and rescue data that has become lost, corrupted or otherwise unavailable often with very granular requirements. (I need this particular 12K file from August 12, 2009) For my purposes, most of my backup strategies are oriented towards Business Uptime and minimal RTO. Given the data volume I work with using lots of virtual machines, tape is strictly an archival tool. I just can''t restore fast enough, and it introduces way to many mechanical dependencies into the process (well I could if I had an unlimited budget). I can restart entire sites from a backup system by cloning a filesystem off a backup snapshot and presenting the volumes to the servers that need it. Granted, I won''t have the performance of a primary site, but it will work and people can get work done. This responds to the first requirement of minimal downtime. Going back in time is accomplished via lots of snapshots on the backup storage system. Which I can afford since I''m not using expensive disks here. Then you move up the stack into the contents of the volumes and here''s where you use your traditional backup tools to get data off the top of the stack - out of the OS that''s handling the contents of the volume that understands it''s particularities regarding ACLS and private volume formats like VMFS. zfs send/recv is for cloning data off the bottom of the stack without requiring the least bit of knowledge about what''s happening on top. It''s just like using any of the asynchronous replication tools that are used in SANs. And they make no bones about the fact that they are strictly a block-level thing and don''t even ask them about the contents. At best, they will try to coordinate filesystem snapshots and quiescing operations with the block level snapshots. Other backup tools take your data off the top of the stack in the context where it is used with a fuller understanding of the issues of stuff like ACLs. When dealing with zvols, ZFS should have no responsibility in trying to understand what you do in there other than supplying the blocks. VMFS, NTFS, btrfs, ext4, HFS+, XFS, JFS, ReiserFS and that''s just the tip of the iceberg... ZFS has muddied the waters by straddling the SAN and NAS worlds.> But their needs to be a tool: > * To restore an individual file or a zvol (with all ACLs/properties) > * That allows backup vendors (which place backups on tape or disk or CD or ..) build indexes of what is contain in the backup (e.g. filename, owner, size modification dates, type (dir/file/etc) ) > *Stream output suitable for devices like tape drives. > *Should be able to tell if the file is corrupted when being restored. > *May support recovery of corrupt data blocks within the stream. > *Preferable gnutar command-line compatible > *That admins can use to backup and transfer a subset of files e.g user home directory (which is not a file system) to another server or on to CD to be sent to their new office location, or ????Highly incomplete and in no particular order : Backup Exec NetBackup Bacula Amanda/Zmanda Retrospect Avamar Arkeia Teradactyl CommVault Acronis Atempo Conceptually, think of a ZFS system as a SAN Box with built-in asynchronous replication (free!) with block-level granularity. Then look at your other backup requirements and attach whatever is required to the top of the stack. Remembering that everyone''s requirements can be wildly or subtly different so doing it differently is just adapting to the environment. e.g. - I use ZFS systems at home and work and the tools and scale are wildly different and therefor so are the backup strategies ? but that''s mostly a budget issue at home... :-) Cheers Erik
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 14:28, Darren J Moffat wrote:> > > On 18/03/2010 13:12, Joerg.Schilling at fokus.fraunhofer.de wrote: >> Darren J Moffat<Darren.Moffat at oracle.com> wrote: >> >>> So exactly what makes it unsuitable for backup ? >>> >>> Is it the file format or the way the utility works ? >>> >>> If it is the format what is wrong with it ? >>> >>> If it is the utility what is needed to fix that ? >> >> This has been discussed many times in the past already. > >> If you archive the incremental "star send" data streams, you cannot >> extract single files andit seems that this cannot be fixed without >> introducing a different archive format. > > That assumes you are writing the ''zfs send'' stream to a file or file > like media. In many cases people using ''zfs send'' for they backup > strategy are they are writing it back out using ''zfs recv'' into another > pool. In those cases the files can even be restored over NFS/CIFS by > using the .zfs/snapshot directoryFor the archival of files, most utilities can be ... converted (probably by including additional metadata) to store those. The problem arises with zvols (which is where I''m considering zfs send for backup anyways). Since these volumes already are an all-or-nothing scenario restorewise, that argument against using send/receive is flawed from the getgo. ( to restore individual files from a zvol exported as an iSCSI disk, the backup software would have to go on the machine mounting the iSCSI disk, not as a backup of the zvol itself), which basically means that apart from the rollback of snapshots, the send/receive backupstream is only likely to be used in a disaster-rebuild situation, where "restores all of itself batch" is a feature, not a bug. In that scenario "restoring everything" _IS_ a feature, not a bug. As to your two questions above, I''ll try to answer them from my limited understanding of the issue. The format: Isn''t fault tolerant. In the least. One single bit wrong and the entire stream is invalid. A FEC wrapper would fix this. The utility: Can''t handle streams being split (in case of streams being larger that a single backup media). Both of these usually gets fended off with the "was never meant as a backup solution", "you''re trying to use it as ufsdump which it isn''t, on purpose, ufsdump is oldfashioned" and similar arguments. Often accompanied with creative suggestions such as using usb disks (Have you ever tried getting several terabytes back and forth over USB?), and then a helpful pointer to multiple-thousand-dollars worth of backup software, as an excuse for why noone should be considering adding proper backup features to zfs itself. The last paragraph may sound like I''m taking a jab at specific people, I''m not, really. But I''ve had my share of helpful people who have been anything but helpful. Most of them taking care not to put their answers on the lists, and quite a lot of them wanting to sell me services or software (or rainwear such as macintoshes) //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuiWrwACgkQSBMQn1jNM7YvSACg9+Nh3REdxML6cnc0cWDP5cbP co4AoKjmeYx3o4/iQhkW7/tgvfF1qPvN =bNBT -----END PGP SIGNATURE-----
Darren J Moffat
2010-Mar-18 17:21 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> As to your two questions above, I''ll try to answer them from my limited > understanding of the issue. > > The format: Isn''t fault tolerant. In the least. One single bit wrong and > the entire stream is invalid. A FEC wrapper would fix this.I''ve logged CR# "6936195 ZFS send stream while checksumed isn''t fault tollerant" to keep track of that.> The utility: Can''t handle streams being split (in case of streams being > larger that a single backup media).I think it should be possible to store the ''zfs send'' stream via NDMP and let NDMP deal with the tape splitting. Though that may need additional software that isn''t free (or cheap) to drive the parts of NDMP that are in Solaris. I don''t know enough about NDMP to be sure but I think that should be possible. -- Darren J Moffat
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 18:21, Darren J Moffat wrote:>> As to your two questions above, I''ll try to answer them from my limited >> understanding of the issue. >> >> The format: Isn''t fault tolerant. In the least. One single bit wrong and >> the entire stream is invalid. A FEC wrapper would fix this. > > I''ve logged CR# "6936195 ZFS send stream while checksumed isn''t fault > tollerant" to keep track of that. > >> The utility: Can''t handle streams being split (in case of streams being >> larger that a single backup media). > > I think it should be possible to store the ''zfs send'' stream via NDMP > and let NDMP deal with the tape splitting. Though that may need > additional software that isn''t free (or cheap) to drive the parts of > NDMP that are in Solaris. I don''t know enough about NDMP to be sure but > I think that should be possible. >And here I was thinking that the NDMP stack basically was tapedev and/or autoloader device via network? (i.e. not a backup utility at all but a method for the software managing the backup to attach the devices) //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuiYjEACgkQSBMQn1jNM7aQ0QCg6hAO3oCb0YcxBbRceTO1ubMv OhEAoOgFoY903MrazWcRq2HtHH72LXjF =8aWY -----END PGP SIGNATURE-----
Darren J Moffat
2010-Mar-18 17:28 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 18/03/2010 17:26, Svein Skogen wrote:>>> The utility: Can''t handle streams being split (in case of streams being >>> larger that a single backup media). >> >> I think it should be possible to store the ''zfs send'' stream via NDMP >> and let NDMP deal with the tape splitting. Though that may need >> additional software that isn''t free (or cheap) to drive the parts of >> NDMP that are in Solaris. I don''t know enough about NDMP to be sure but >> I think that should be possible. >> > > And here I was thinking that the NDMP stack basically was tapedev and/or > autoloader device via network? (i.e. not a backup utility at all but a > method for the software managing the backup to attach the devices)NDMP doesn''t define the format of what goes on the tape so it can help put the ''zfs send'' stream on the tape and thus deal with the lack of ''zfs send'' being able to handle tape media smaller than its stream size. -- Darren J Moffat
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 18:28, Darren J Moffat wrote:> On 18/03/2010 17:26, Svein Skogen wrote: >>>> The utility: Can''t handle streams being split (in case of streams being >>>> larger that a single backup media). >>> >>> I think it should be possible to store the ''zfs send'' stream via NDMP >>> and let NDMP deal with the tape splitting. Though that may need >>> additional software that isn''t free (or cheap) to drive the parts of >>> NDMP that are in Solaris. I don''t know enough about NDMP to be sure but >>> I think that should be possible. >>> >> >> And here I was thinking that the NDMP stack basically was tapedev and/or >> autoloader device via network? (i.e. not a backup utility at all but a >> method for the software managing the backup to attach the devices) > > NDMP doesn''t define the format of what goes on the tape so it can help > put the ''zfs send'' stream on the tape and thus deal with the lack of > ''zfs send'' being able to handle tape media smaller than its stream size.How would NDMP help with this any more than running a local pipe splitting the stream (and handling the robotics for feeding in the next tape)? I can see the point of NDMP when the tape library isn''t physically connected to the same box as the zpools, but feeding local data via a network servce seems to me to be just complicating things... But that''s just my opinion. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuiZBgACgkQSBMQn1jNM7Y3XACglyXvPiSd+iInxLaJVeY+lnUn GiAAn0xjL7KWfbwfwHz7gaKA8FtGNZOb =6yI8 -----END PGP SIGNATURE-----
Darren J Moffat
2010-Mar-18 17:37 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 18/03/2010 17:34, Svein Skogen wrote:> How would NDMP help with this any more than running a local pipe > splitting the stream (and handling the robotics for feeding in the next > tape)?Probably doesn''t in that case.> I can see the point of NDMP when the tape library isn''t physically > connected to the same box as the zpools, but feeding local data via a > network servce seems to me to be just complicating things...Indeed if the drive is local then you it may be adding a layer you don''t need. -- Darren J Moffat
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 18:37, Darren J Moffat wrote:> On 18/03/2010 17:34, Svein Skogen wrote: >> How would NDMP help with this any more than running a local pipe >> splitting the stream (and handling the robotics for feeding in the next >> tape)? > > Probably doesn''t in that case. > >> I can see the point of NDMP when the tape library isn''t physically >> connected to the same box as the zpools, but feeding local data via a >> network servce seems to me to be just complicating things... > > Indeed if the drive is local then you it may be adding a layer you don''t > need.I''d think "getting it to work locally" make the utility in a fashion that it can take a local library, OR a remote NDMP one, would be a priority. Maybe more so for database customers of midsized size, that doesn''t have the luxury of 64 datacenters on multiple continents. ;) Having the option of having "catastrophy restore" backups picked up every morning (as a stack of 8 tapes?) and brought offsite for safekeeping should probably be in the operating instructions for ... all oracle-customers not having the luxury of a multisite setup. ;) I''d suspect there are a whole lot more small customers than large ones. :p //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuiZngACgkQSBMQn1jNM7Y4kgCg7EeQpvZVBzBCcGwI5mdG1vrr 7MYAoNj/EiUTQzycz4bM+wSs9HWZD589 =Cthq -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 18.03.2010 17:49, erik.ableson wrote:> Conceptually, think of a ZFS system as a SAN Box with built-in asynchronous replication (free!) with block-level granularity. Then look at your other backup requirements and attach whatever is required to the top of the stack. Remembering that everyone''s requirements can be wildly or subtly different so doing it differently is just adapting to the environment. e.g. - I use ZFS systems at home and work and the tools and scale are wildly different and therefor so are the backup strategies ? but that''s mostly a budget issue at home... :-)I''ll answer this with some perspective from my own usage, so bear with me if my setup isn''t exactly enterprise-grade. Which is exactly what I''m considering it. Basically a semi-intelligent box that has a lot of disks, and an attached tape autoloader for dumping "then entire box died!" data to. In my case the zvols contains vmfs (one vmfs per vm, actually), so to the just-died you can add "and brought the rest of the network down with it". A typical one-man setup with as cheap a budget as possible. And as I''ve posted some weeks ago, I''ve ... had to test the restore bit. Having a "hit f8 during boot, and boot directly off the tape" I can live without (using a bootcd instead). But missing the "I''ve now started restore, it''ll manage itself and switch tapes when necessary" isn''t something I really want to miss out on. For the record, restore in my case takes appx 12 hours fully automated and managing a good 60MB/Sec all the way. Now, I''m willing to miss out on some of the fun, and add an extra server for simply handling the iSCSI bit, and then set up that to dump itself onto the Windows-server (who has the backup). That''s a NOK12K investment (I''ve checked) with an option for becoming NOK16K if I need more than 2 1000BaseT''s bundled (if two isn''t enough, adding another four sounds about right). The main storage box has an LSI 8308elp with 8 1T5 7200rpm barracudas in RAID50 with coldspares, and it''s ... near enough my bed to hear the alarm should a disk die on me. So strictly speaking I don''t need ZFS featurewise (the 8308 has patrol reads/scrubbing, and it pushes sufficient IO speeds to outperform the four NICS in the box). So probably for my needs QFS/SAM would''ve been a better solution. I really don''t know, since the available installer for QFS/SAM only plays nice with <Solaris10u8 and the four Marvell 88E8056 NICs on the mainboard only has dladm aggregate capable support in the YGE driver (not the yukonx legacy driver from Marvells site). Maybe someone knowing what details of the QFS/SAM was opened up could answer me if that would work at all for my needs. ;) For day-to-day things snapshots perform admirably. Most of the bulk storage isn''t the iSCSI zvols, but SMB/CIFS shares containing mostly .nef files. And all of those files are irreplacable. Not having a tape backup of them is simply not an option. It''s a hobbyist setup for someone who ... used to be in the IT/Telco business, ended up on disability pension for his troubles, and is trying to climb out of that pension by starting over as a photographer. So backup software costing me more than my total living budget for 6 months isn''t likely to be a viable solution for the problem either. However, still looking from my own stance, there are several people still in the business who consult with me on solutions on a much larger scale than mine, and with a budget that''d make my eyes water. I''d really like to point them towards (Open)Solaris, but without having personally seen (Open)Solaris actually do the job I''m advertising it for, that recommendation starts sounding increasingly hollow even to me. I suspect a lot of such oneman-shops are (often less than more officially) consulted for larger shops (old hand experience from the business counts as something), and thus having a setup "that works" could (by Oracle) be considered a good advertisement. If one-man-shops happen to be in the situation that they really don''t see the need for considering alternatives because (Open)Solaris "just plain works", then the signal:noise ratio between people recommending (Open)Solaris and those speaking about other things gets tweaked in (Open)Solaris''es favor. //Svein - -- - --------+-------------------+------------------------------- /"\ |Svein Skogen | svein at d80.iso100.no \ / |Solberg ?stli 9 | PGP Key: 0xE5E76831 X |2020 Skedsmokorset | svein at jernhuset.no / \ |Norway | PGP Key: 0xCE96CE13 | | svein at stillbilde.net ascii | | PGP Key: 0x58CD33B6 ribbon |System Admin | svein-listmail at stillbilde.net Campaign|stillbilde.net | PGP Key: 0x22D494A4 +-------------------+------------------------------- |msn messenger: | Mobile Phone: +47 907 03 575 |svein at jernhuset.no | RIPE handle: SS16503-RIPE - --------+-------------------+------------------------------- If you really are in a hurry, mail me at svein-mobile at stillbilde.net This mailbox goes directly to my cellphone and is checked even when I''m not in front of my computer. - ------------------------------------------------------------ Picture Gallery: https://gallery.stillbilde.net/v/svein/ - ------------------------------------------------------------ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.12 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkuicZ8ACgkQSBMQn1jNM7aToQCg6hxHJaY+xLHZg94bEkWYzkgk XtoAn2nsi8+wbbHq6kAfvFGwEMESqR6c =OpIM -----END PGP SIGNATURE-----
>>>>> "djm" == Darren J Moffat <Darren.Moffat at Oracle.COM> writes:djm> I''ve logged CR# "6936195 ZFS send stream while checksumed djm> isn''t fault tollerant" to keep track of that. Other tar/cpio-like tools are also able to: * verify the checksums without extracting (like scrub) * verify or even extract the stream using a small userland tool that writes files using POSIX functions, so that you can build the tool on not-Solaris or extract the data onto not-ZFS. The ''zfs send'' stream can''t be extracted without the solaris kernel, although yes the promise that newer kernels can extract older streams is a very helpful one. For example, ufsdump | ufsrestore could move UFS data into ZFS. but zfs send | zfs recv leaves us trapped on ZFS, even though migrating/restoring ZFS data onto a pNFS or Lustre backend is a realistic desire in the near term. * partial extract Personally, I could give up the third bullet point. Admittedly the second bullet is hard to manage while still backing up zvol''s, pNFS / Lustre data-node datasets, windows ACL''s, properties, snapshots/clones, u.s.w., so it''s kind of...if you want both vanilla and chocolate cake at once, you''re both going to be unhappy. But there should at least be *a* tool that can copy from zfs to NFSv4 while preserving windows ACL''s, and the tool should build on other OS''s that support NFSv4 and be capable of faithfully copying one NFSv4 tree to another preserving all the magical metadata. I know it sounds like ACL-aware rsync is unrelated to your (Darren) goal of tweaking ''zfs send'' to be appropriate for backups, but for example before ZFS I could make a backup on the machine with disks attached to it or on an NFS client, and get exactly the same stream out. Likewise, I could restore into an NFS client. Sticking to a clean API instead of dumping the guts of the filesystem, made the old stream formats more archival. The ``I need to extract a ZFS dataset so large that my only available container is a distributed Lustre filesystem'''' use-case is pretty squarely within the archival realm, is going to be urgent in a year or so if it isn''t already, and is accomodated by GNUtar, cpio, Amanda (even old ufsrestore Amanda), and all the big commercial backup tools. I admit it would be pretty damn cool if someone could write a purely userland version of ''zfs send'' and ''zfs recv'' that interact with the outside world using only POSIX file i/o and unix pipes but produce the standard deduped-ZFS-stream format, even if the hypothetical userland tool accomplishes this by including a FUSE-like amount of ZFS code and thus being quite hard to build. However, so far I don''t think the goals of a replication tool: ``make a faithful and complete copy, efficiently, or else give an error,'''' are compatible with the goals of an archival tool: ``extract robustly far into the future even in non-ideal and hard to predict circumstances such as different host kernel, different destination filesystem, corrupted stream, limited restore space.'''' -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100318/2470110f/attachment.bin>
>>>>> "c" == Miles Nordin <carton at Ivy.NET> writes: >>>>> "mg" == Mike Gerdts <mgerdts at gmail.com> writes:c> are compatible with the goals of an archival tool: sorry, obviously I meant ``not compatible''''. mg> Richard Elling made an interesting observation that suggests mg> that storing a zfs send data stream on tape is a quite mg> reasonable thing to do. Richard''s background makes me trust mg> his analysis of this much more than I trust the typical person mg> that says that zfs send output is poison. ssh and tape are perfect, yet whenever ZFS pools become corrupt Richard talks about scars on his knees from weak TCP checksums and lying disk drives and about creating a ``single protection domain'''' of zfs checksums and redundancy instead of a bucket-brigade of fail of tcp into ssh into $blackbox_backup_Solution(likely involving unchecksummed disk storage) into SCSI/FC into ECC tapes. At worst, lying then or lying now? At best, the whole thing still strikes me as a pattern of banging a bunch of arcania into whatever shape''s needed to fit the conclusion that ZFS is glorious and no further work is requried to make it perfect. and there is still no way to validate a tape without extracting it, which is last I worked with them, an optional but suggested part of $blackbox_backup_Solution (and one which, incidentally, helps with the bucket brigade problem Richard likes to point out). and the other archival problems of constraining the restore environment, and the fundamental incompatibility of goals between faithful replication and robust, future-proof archiving from my last post. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100318/d09be1ef/attachment.bin>
On 03/18/10 12:07 PM, Khyron wrote:> Ian, > > When you say you spool to tape for off-site archival, what software do > you > use? >NetVault. -- Ian.
On Mar 18, 2010, at 15:00, Miles Nordin wrote:> Admittedly the second bullet is hard to manage while still backing up > zvol''s, pNFS / Lustre data-node datasets, windows ACL''s, properties,Some commercial backup products are able to parse VMware''s VMDK files to get file system information of them. The product sits on the VMware host, slurps in the files (which can be snapshotted for quiesced backups), and if you want to restore, you can either put back the entire VMDK or simply restore just the particular file(s) that are of interest. Currently NetBackup only supports "parsing" NTFS for individual file restoration. Theoretically, zvols could be added to the list of parsable container formats. Though there would probably have to be some kind of API akin to VMware''s VCB or vStorage.
Edward Ned Harvey
2010-Mar-19 02:55 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> > From what I''ve read so far, zfs send is a block level api and thus > > cannot be > > used for real backups. As a result of being block level oriented, the > > Weirdo. The above "cannot be used for real backups" is obviously > subjective, is incorrect and widely discussed here, so I just say > "weirdo." > I''m tired of correcting this constantly.I apologize if I was insulting, and it''s clear that I was. Seriously, I apologize. I should have thought about that more before I sent it, and I should have been more considerate. To clarify, more accurately, from a technical standpoint, what I meant: There are circumstances, such as backup to removable disks, or time-critical incremental data streams, where the performance of incremental "zfs send" versus the performance of star, rsync, or any other file-based backup mechanism, "zfs send" is the clear winner ... There are circumstances where zfs send is enormously a winner. There are other circumstances, such as writing to tape, where star, or tar, or in other circumstances, where rsync or other tools may be the winner... And I don''t claim to know all the circumstances where something else beats "zfs send." There probably are many circumstances where some other tool beats "zfs send" in some way. The only point which I wish to emphasize is that it''s not fair to say unilaterally that one technique is always better than another technique. Each one has their own pros/cons.
Ahhh, this has been...interesting...some real "personalities" involved in this discussion. :p The following is long-ish but I thought a re-cap was in order. I''m sure we''ll never finish this discussion, but I want to at least have a new plateau or base from which to consider these questions. I''ve just read through EVERY post to this thread, so I want to recap the best points in the vein of the original thread, and set a new base for continuing the conversation. Personally, I''m less interested in the archival case; rather, I''m looking for the best way to either recover from a complete system failure or recover an individual file or file set from some backup media, most likely tape. Now let''s put all of this together, along with some definitions. First, the difference between archival storage (to tape or other) and backup. I think the best definition provided in this thread came from Darren Moffat as well. As Carsten Aulbert mentioned, this discussion is fairly useless until we start using the same terminology to describe a set of actions. For this discussion, I am defining archival as taking the data and placing it on some media - likely tape, but not necessarily - in the simplest format possible that could hopefully be read by another device in the future. This could exclude capturing NTFS/NFSv4/ZFS ACLs, Solaris extended attributes, or zpool properties (aka metadata for purposes of this discussion). With an archive, we may not go back and touch the data for a long time, if ever again. Backup, OTOH, is the act of making a perfect copy of the data to some media (in my interest tape, but again, not necessarily) which includes all of the metadata associated with that data. Such a copy would allow perfect re-creation of the data in a new environment, recovery from a complete system failure, or single file (or file set) recovery. With a backup, we have the expectation that we may need to return to it shortly after it is created, so we have to be able to trust it...now. Data restored from this backup needs to be an exact replica of the original source - ZFS pool and dataset properties, extended attributes, and ZFS ACLs included. Now that I hopefully have common definitions for this conversation (and I hope I captured Darren''s meaning accurately), I''ll divide this into 2 sections, starting with NDMP. NDMP: For those who are unaware (and to clarify my own understanding), I''ll take a moment to describe NDMP. NDMP was invented by NetApp to allow direct backup of their Filers to tape backup servers, and eventually onto tape. It is designed to remove the need for indirect backup by backing up the NFS or CIFS shared file systems on the clients. Instead, we backup the shared file systems directly from the Filer (or other file server - say Fishworks box or OpenSolaris server) to the backup server via the network. We avoid multiple copies of the shared file systems. NDMP is a network-based delivery mechanism to get data from a storage server to a backup server, which is why the backup software must also speak NDMP. Hopefully, my description is mostly accurate, and it is clear why this might be useful for people using (Open)Solaris + ZFS for tape backup or archival purposes. Darren Moffat made the point that NDMP could be used to do the tape splitting, but I''m not sure this is accurate. If "zfs send" from a file server running (Open)Solaris to a tape drive over NDMP is viable -- which it appears to be to me -- then the tape splitting would be handled by the tape backup application. In my world, that''s typically NetBackup or some similar enterprise offering. I see no reason why it couldn''t be Amanda or Bacula or Arkeia or something else. THIS is why I am looking for faster progress on NDMP. Now, NDMP doesn''t do you much good for a locally attached tape drive, as Darren and Svein pointed out. However, provided the software which is installed on this fictional server can talk to the tape in an appropriate way, then all you have to do is pipe "zfs send" into it. Right? What did I miss? ZVOLs and NTFS/NFSv4/ZFS ACLs: The answer is "zfs send" to both of my questions about ZVOLs and ACLs. At the center of all of this attention is "zfs send". As Darren Moffat pointed out, it has all the pieces to do a proper, complete and correct backup. The big remaining issue that I see is how do you place a "zfs send" stream on a tape in a reliable fashion. CR 6936195 would seem to handle one complaint from Svein, Miles Nordin and others about reliability of the send stream on the tape. Again, I think NDMP may help answer this question for file servers without attached tape devices. For those with attached tape devices, what''s the equivalent answer? Who is doing this, and how? I believe we''ve seen Ed Harvey say "NetBackup" and Ian Collins say "NetVault". Do these products capture all the metadata required to call this copy a "backup"? That''s my next question. Finally, Damon Atkins said: "But their needs to be a tool: * To restore an individual file or a zvol (with all ACLs/properties) * That allows backup vendors (which place backups on tape or disk or CD or ..) build indexes of what is contain in the backup (e.g. filename, owner, size modification dates, type (dir/file/etc) ) *Stream output suitable for devices like tape drives. *Should be able to tell if the file is corrupted when being restored. *May support recovery of corrupt data blocks within the stream. *Preferable gnutar command-line compatible *That admins can use to backup and transfer a subset of files e.g user home directory (which is not a file system) to another server or on to CD to be sent to their new office location, or ????" So far, I don''t think I''ve seen any reference to a tool that fits this description. Erik Ableson provided a list of software that he says can do all of this. Is that list (incomplete though it may be) accurate? Can they all create what I define as a backup (to tape media) of a ZFS pool? Note: Being a long time Unix user, I could do without GNU command line semantics, but that''s just a personal thing. Items 1 - 3 on the list are needs, while 4 - 7 would be great if not quite needs, IMO. Finally, if no such tool currently exists as described by Damon Atkins, what is required to create one? This is a curiosity question, admittedly. -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/726b4407/attachment.html>
Darren J Moffat
2010-Mar-19 09:42 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> Now, NDMP doesn''t do you much good for a locally attached tape drive, > as Darren and Svein pointed out. However, provided the software which is > installed on this fictional server can talk to the tape in an > appropriate way, > then all you have to do is pipe "zfs send" into it. Right? What did I > miss?Actually there is a case where NDMP is useful when the tape drive is locally attached. If the data server is an appliance that you can not (either technically or by policy or both) install any backup agents onto. The SS7000 falls into this category. The SS7000 allows for a locally attached tape drive. The backup control software runs on another machine and talks with the local NDMP to move the data from local disk to local tape. -- Darren J Moffat
Joerg Schilling
2010-Mar-19 14:57 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Darren J Moffat <Darren.Moffat at oracle.com> wrote:> That assumes you are writing the ''zfs send'' stream to a file or file > like media. In many cases people using ''zfs send'' for they backup > strategy are they are writing it back out using ''zfs recv'' into another > pool. In those cases the files can even be restored over NFS/CIFS by > using the .zfs/snapshot directoryIf you unpack the datastream from zfs send on a machine on a different location that is safe against e.g. a fire that destroys the main machine, you may call it a backup.> > Star implements incremental backups and restores based on POSIX compliant > > archives. > > ZFS filesystem have functionality beyond POSIX and some of that is > really very important for some people (especially those using CIFS)As I mentioned many times in the past, star in contrary to other archives I know has the right infrastructure that allows to add support for additional metadata easily. The main problem seems to be that some people from inside Sun signal that they are not interested in star and that this discourages customers that do not maintain their own sw infrastructure. Adding missing features on the other side only makes sens if there is interes in using these features.> Does Star (or any other POSIX archiver) backup: > ZFS ACLs ?Now that libsec finally supports the needed features, it only needs to be defined and implemented. I am waiting since a few years on a discussion to define the textual format to be used in the tar headers...> ZFS system attributes (as used by the CIFS server and locally) ?star does support such things for Linux and FreeBSD, the problem on Solaris is that the documentation of the interfaces for this Solaris local feature is poor. The was Sun tar archives the attibutes is non-portable. Could you point to documentation?> ZFS dataset properties (compression, checksum etc) ?Where is the documentation of the interfaces?> If it doesn''t then it is providing an "archive" of the data in the > filesystem, not a full/incremental copy of the ZFS dataset. Which > depending on the requirements of the backup may not be enough. In > otherwords you have data/metadata missing from your backup. > > The only tool I''m aware of today that provides a copy of the data, and > all of the ZPL metadata and all the ZFS dataset properties is ''zfs send''.I encourage you to collaborate... Provide information for documentation in the interfaces and help to discuss the archive format extensions for the missing features. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Joerg Schilling
2010-Mar-19 15:07 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Mike Gerdts <mgerdts at gmail.com> wrote:> > another server, where the data is immediately fed through "zfs receive" then > > it''s an entirely viable backup technique. > > Richard Elling made an interesting observation that suggests that > storing a zfs send data stream on tape is a quite reasonable thing to > do. Richard''s background makes me trust his analysis of this much > more than I trust the typical person that says that zfs send output is > poison.If it is on tape you can restore the whole filesystem if you have a new empty one to restore to but you cannot do all the typical usages of backups. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Darren J Moffat
2010-Mar-19 15:16 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 19/03/2010 14:57, Joerg.Schilling at fokus.fraunhofer.de wrote:> Darren J Moffat<Darren.Moffat at oracle.com> wrote: > >> That assumes you are writing the ''zfs send'' stream to a file or file >> like media. In many cases people using ''zfs send'' for they backup >> strategy are they are writing it back out using ''zfs recv'' into another >> pool. In those cases the files can even be restored over NFS/CIFS by >> using the .zfs/snapshot directory > > If you unpack the datastream from zfs send on a machine on a different location > that is safe against e.g. a fire that destroys the main machine, you may call > it a backup.I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet the implication is that a tar archive stored on a tape is considered a backup ?>> ZFS system attributes (as used by the CIFS server and locally) ? > > star does support such things for Linux and FreeBSD, the problem on Solaris is > that the documentation of the interfaces for this Solaris local feature is poor. > The was Sun tar archives the attibutes is non-portable. > > Could you point to documentation?getattrat(3C) / setattrat(3C) Even has example code in it. This is what ls(1) uses.>> ZFS dataset properties (compression, checksum etc) ? > > Where is the documentation of the interfaces?There isn''t any for those because the libzfs interfaces are currently still private. The best you can currently do is to parse the output of ''zfs list'' eg. zfs list -H -o compression rpool/export/home Not ideal but it is the only publicly documented interface for now. -- Darren J Moffat
Damon (and others)
For those wanting the ability to perform file backups/restores along with all
metadata, without resorting to third party applications, if you have a Sun
support contract, log a call asking that your organisation be added to the list
of users who wants to see RFE #5004379 "want comprehensive backup
strategy" implemented.
I logged this last month and was told there are now 5 organisations asking for
this. Considering this topic seems to crop up regularly on zfs-discuss,
I''m guessing the actual number of people is higher but people
don''t know how to register their interest.
JR
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100319/025c067d/attachment.html>
Joerg Schilling
2010-Mar-19 16:11 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Darren J Moffat <darrenm at opensolaris.org> wrote:> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet > the implication is that a tar archive stored on a tape is considered a > backup ?You cannot get a single file out of the zfs send datastream.> >> ZFS system attributes (as used by the CIFS server and locally) ? > > > > star does support such things for Linux and FreeBSD, the problem on Solaris is > > that the documentation of the interfaces for this Solaris local feature is poor. > > The was Sun tar archives the attibutes is non-portable. > > > > Could you point to documentation? > > getattrat(3C) / setattrat(3C) > > Even has example code in it. > > This is what ls(1) uses.It could be easily possible to add portable support integrated into the framework that already supports FreeBSD and Linux attributes.> >> ZFS dataset properties (compression, checksum etc) ? > > > > Where is the documentation of the interfaces? > > There isn''t any for those because the libzfs interfaces are currently > still private. The best you can currently do is to parse the output of > ''zfs list'' eg. > zfs list -H -o compression rpool/export/home > > Not ideal but it is the only publicly documented interface for now.As long as there is no interface that supports what I did discuss with Jeff Bonwick in September 2004: - A public interface to get the property state - A public interface to read the file raw in compressed form - A public interface to write the file raw in compressed form I am not sure whether this is of relevance for a backup. If there is a need to change the states, on a directory base, there is a need for an easy to use public interface. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Darren J Moffat
2010-Mar-19 16:33 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 19/03/2010 16:11, Joerg.Schilling at fokus.fraunhofer.de wrote:> Darren J Moffat<darrenm at opensolaris.org> wrote: > >> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet >> the implication is that a tar archive stored on a tape is considered a >> backup ? > > You cannot get a single file out of the zfs send datastream.I don''t see that as part of the definition of a backup - you obviously do - so we will just have to disagree on that.>>>> ZFS system attributes (as used by the CIFS server and locally) ? >>> >>> star does support such things for Linux and FreeBSD, the problem on Solaris is >>> that the documentation of the interfaces for this Solaris local feature is poor. >>> The was Sun tar archives the attibutes is non-portable. >>> >>> Could you point to documentation? >> >> getattrat(3C) / setattrat(3C) >> >> Even has example code in it. >> >> This is what ls(1) uses. > > It could be easily possible to add portable support integrated into the > framework that already supports FreeBSD and Linux attributes.Great, do you have a time frame for when you will have this added to star then ?>>>> ZFS dataset properties (compression, checksum etc) ? >>> >>> Where is the documentation of the interfaces? >> >> There isn''t any for those because the libzfs interfaces are currently >> still private. The best you can currently do is to parse the output of >> ''zfs list'' eg. >> zfs list -H -o compression rpool/export/home >> >> Not ideal but it is the only publicly documented interface for now. > > As long as there is no interface that supports what I did discuss with > Jeff Bonwick in September 2004: > > - A public interface to get the property stateThat would come from libzfs. There are private interfaces just now that are very likely what you need zfs_prop_get()/zfs_prop_set(). They aren''t documented or public though and are subject to change at any time.> - A public interface to read the file raw in compressed formI think you are missing something about how ZFS works here. Files aren''t in a compressed form. Some blocks of a file may be compressed if compression is enabled on the dataset. Note that for compression and checksum properties they only indicate what algorithm will be used to compress (or checksum) blocks for new writes. It doesn''t say what algorithm the blocks of a given file are compressed with. In fact for any given file some blocks may be compressed and some not. The reasons for a block not being compressed include: 1) it didn''t compress 2) it was written when compression=off 3) it didn''t compress enough. It is even possible that if the user changed the value of compression blocks within a file are compressed with a different algorithm. So you won''t ever get this because ZFS just doesn''t work like that. In fact even ''zfs send'' doesn''t even store compressed data. The ''zfs send'' stream has the blocks in the form that they exist in the in memory ARC ie uncompressed. In kernel it is possible to ask for a block in its RAW (ie compressed) form but that is only for consumers of arc_read() and zio_read() - way way way below the ZPL layer and applications like star.> - A public interface to write the file raw in compressed formNot even a private API exists for this. There is no capability to send a RAW (ie compressed) block to arc_write() or zio_write().> I am not sure whether this is of relevance for a backup. If there is a need > to change the states, on a directory base, there is a need for an easy to use > public interface.I don''t understand what you mean by that, can you give me an example. -- Darren J Moffat
On 19 mars 2010, at 17:11, Joerg Schilling wrote:>> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet >> the implication is that a tar archive stored on a tape is considered a >> backup ? > > You cannot get a single file out of the zfs send datastream.zfs send is a block-level transaction with no filesystem dependencies - it could be transmitting a couple of blocks that represent a portion of a file, not necessarily an entire file. And since it can also be used to host a zvol with any filesystem format imaginable it doesn''t want to know. Going back to star as an example - from the man page : "Star archives and extracts multiple files to and from a single file called a tarfile. A tarfile is usually a magnetic tape, but it can be any file. In all cases, appearance of a directory name refers to the files and (recursively) subdirectories of that directory." This process pulls files (repeat: files! not blocks) off of the top of a filesystem so it needs to be presented a filesystem with interpretable file objects (like almost all backup tools). ZFS confuses the issue by integrating volume management with filesystem management. zfs send is dealing with the volume and the blocks that represent the volume without any file-level dependence. It addresses an entirely different type of backup need, that is to be able to restore or mirror (especially mirror to another live storage system) an entire volume at a point in time. It does not replace the requirement for file-level backups which deal with a different level of granularity. Simply because the restore use-case is different. For example, on my Mac servers, I run two different backup strategies concurrently - one is bootable clone from which I can restart the computer immediately in the case of a drive failure. At the same time, I use the Time Machine backups for file level granularity that allows me to easily find a particular file at a particular moment. Before Time Machine, this role was fulfilled with Retrospect to a tape drive. However, a block-level dump to tape had little interest in the first use case since the objective is to minimize the RTO. For disaster recovery purposes any of these backup objects can be externalized. Offsite rotation of the disks used allow the management of the RPO. Remember that files exist in a filesystem context and need to be backed up in this context. Volumes exist in another context and can be replicated/backed up in this context. zfs send/recv = EMC MirrorView, NetApp Snap Mirror, EqualLogic Auto-replication, HP StorageWorks Continuous Access, DataCore AIM, etc. zfs send/recv ? star, Backup Exec, CommVault, ufsdump, bacula, zmanda, Retrospect, etc. Erik
David Dyer-Bennet
2010-Mar-19 17:19 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Fri, March 19, 2010 11:33, Darren J Moffat wrote:> On 19/03/2010 16:11, Joerg.Schilling at fokus.fraunhofer.de wrote: >> Darren J Moffat<darrenm at opensolaris.org> wrote: >> >>> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet >>> the implication is that a tar archive stored on a tape is considered a >>> backup ? >> >> You cannot get a single file out of the zfs send datastream. > > I don''t see that as part of the definition of a backup - you obviously > do - so we will just have to disagree on that.I used to. Now I think more in terms of getting it from a snapshot maintained online on the original storage server. The overall storage strategy has to include retrieving files lost due to user error over some time period, whether that''s months or years. And having to restore an entire 100TB backup to "spare disk" somewhere to get one file is clearly not on. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Darren J Moffat
2010-Mar-19 17:25 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On 19/03/2010 17:19, David Dyer-Bennet wrote:> > On Fri, March 19, 2010 11:33, Darren J Moffat wrote: >> On 19/03/2010 16:11, Joerg.Schilling at fokus.fraunhofer.de wrote: >>> Darren J Moffat<darrenm at opensolaris.org> wrote: >>> >>>> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet >>>> the implication is that a tar archive stored on a tape is considered a >>>> backup ? >>> >>> You cannot get a single file out of the zfs send datastream. >> >> I don''t see that as part of the definition of a backup - you obviously >> do - so we will just have to disagree on that. > > I used to. Now I think more in terms of getting it from a snapshot > maintained online on the original storage server.Exactly! The single file retrieval due to user error case is best achieved by an automated snapshot system. ZFS+CIFS even provides Windows Volume Shadow Services so that Windows users can do this on their own.> The overall storage strategy has to include retrieving files lost due to > user error over some time period, whether that''s months or years. And > having to restore an entire 100TB backup to "spare disk" somewhere to get > one file is clearly not on.Completely agree, no where was I suggesting that ''zfs send'' out to tape should be the whole backup strategy. I even pointed to a presentation given at LOSUG that shows how someone is doing this. I''ll say it again: neither ''zfs send'' or (s)tar is an enterprise (or even home) backup system on their own one or both can be components of the full solution. -- Darren J Moffat
David Dyer-Bennet
2010-Mar-19 17:54 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Fri, March 19, 2010 12:25, Darren J Moffat wrote:> On 19/03/2010 17:19, David Dyer-Bennet wrote: >> >> On Fri, March 19, 2010 11:33, Darren J Moffat wrote: >>> On 19/03/2010 16:11, Joerg.Schilling at fokus.fraunhofer.de wrote: >>>> Darren J Moffat<darrenm at opensolaris.org> wrote: >>>> >>>>> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape >>>>> yet >>>>> the implication is that a tar archive stored on a tape is considered >>>>> a >>>>> backup ? >>>> >>>> You cannot get a single file out of the zfs send datastream. >>> >>> I don''t see that as part of the definition of a backup - you obviously >>> do - so we will just have to disagree on that. >> >> I used to. Now I think more in terms of getting it from a snapshot >> maintained online on the original storage server. > > Exactly! The single file retrieval due to user error case is best > achieved by an automated snapshot system. ZFS+CIFS even provides > Windows Volume Shadow Services so that Windows users can do this on > their own.I''ll need to look into that, when I get a moment. Not familiar with Windows Volume Shadow Services, but having people at home able to do this directly seems useful.>> The overall storage strategy has to include retrieving files lost due to >> user error over some time period, whether that''s months or years. And >> having to restore an entire 100TB backup to "spare disk" somewhere to >> get >> one file is clearly not on. > > Completely agree, no where was I suggesting that ''zfs send'' out to tape > should be the whole backup strategy. I even pointed to a presentation > given at LOSUG that shows how someone is doing this.Sorry, didn''t mean to sound like I was arguing with you (or suggest we disagreed in that area); I intended to pontificate on the problem in general.> I''ll say it again: neither ''zfs send'' or (s)tar is an enterprise (or > even home) backup system on their own one or both can be components of > the full solution.I''m seeing what a lot of professional and serious amateur photographers are building themselves for storage on a mailing list I''m on. Nearly always it consists of two layers of storage servers, often with one off-site (most of them keep current photos on LOCAL disk, instead of my choice of working directly off storage server disk). I''m in the fortunate position of having my backups less than the size of a large single drive; so I''m rotating three backup drives, and intend to be taking one of them off-site regularly (still in the process of converting to this new scheme; the previous scheme used off-site optical disks). I use ZFS for the removable drives, so I can if necessary reach into them and drag out single files fairly easily if necessary (but "necessary" would require something happening to the online snapshot first). People with much bigger configurations look like they save money using tape for the archival / disaster restore storage, but it''s not economically viable at my level. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Erik, I don''t think there was any confusion about the block nature of "zfs send" vs. the file nature of star. I think what this discussion is coming down to is the best ways to utilize "zfs send" as a backup, since (as Darren Moffat has noted) it supports all the ZFS objects and metadata. I see 2 things coming out of this: 1. NDMP for putting "zfs send" streams on tape over the network. So the question I have now is for anyone who has used or is using NDMP on OSol. How well does it work? Pros? Cons? If people aren''t using it, why not? I think this is one area where there are some gains to be made on the OSol backup front. I still need to go back and look at the best ways to use local tape drives on OSol file servers running ZFS to capture ZFS objects and metadata (ZFS ACLs, ZVOLs, etc.). 2. A new tool is required to provide some of the functionality desired, at least as a supported backup method from Sun. While someone in the community may be interested in developing such a tool, Darren also noted that the requisite APIs are private currently and still in flux. They haven''t yet stabilized and been published. To Ed Harvey: Some questions about your use of NetBackup on your secondary server: 1. Do you successfully backup ZVOLs? We know NetBackup should be able to capture datasets (ZFS file systems) using straight POSIX semantics. 2. What version of NetBackup are you using? 3. You simply run the NetBackup agent locally on the (Open)Solaris server? I thank everyone who has participated in this conversation for sharing their thoughts, experiences and realities. It has been most informational. On Fri, Mar 19, 2010 at 13:11, erik.ableson <eableson at me.com> wrote:> On 19 mars 2010, at 17:11, Joerg Schilling wrote: > > >> I''m curious, why isn''t a ''zfs send'' stream that is stored on a tape yet > >> the implication is that a tar archive stored on a tape is considered a > >> backup ? > > > > You cannot get a single file out of the zfs send datastream. > > zfs send is a block-level transaction with no filesystem dependencies - it > could be transmitting a couple of blocks that represent a portion of a file, > not necessarily an entire file. And since it can also be used to host a > zvol with any filesystem format imaginable it doesn''t want to know. ><snip> -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100320/cc33ca65/attachment.html>
Edward Ned Harvey
2010-Mar-20 04:22 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> > ZFS+CIFS even provides > > Windows Volume Shadow Services so that Windows users can do this on > > their own. > > I''ll need to look into that, when I get a moment. Not familiar with > Windows Volume Shadow Services, but having people at home able to do > this > directly seems useful.I''d like to spin off this discussion into a new thread. Any replies to this one will surely just get buried in the (many messages) in this very long thread... New thread: ZFS+CIFS: Volume Shadow Services, or Simple Symlink?
Edward Ned Harvey
2010-Mar-20 04:37 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> > I''ll say it again: neither ''zfs send'' or (s)tar is an enterprise (or > > even home) backup system on their own one or both can be components > of > > the full solution.I would be pretty comfortable with a solution thusly designed: #1 A small number of external disks, "zfs send" onto the disks and rotate offsite. Then, you''re satisfying the ability to restore individual files, but you''re not satisfying the archivability, longevity of tapes. #2 Also, "zfs send" onto tapes. So if ever you needed something older than your removable disks, it''s someplace reliable, just not readily accessible if you only want a subset of files.> I''m in the fortunate position of having my backups less than the size > of a > large single drive; so I''m rotating three backup drives, and intend toIt''s of course convenient if your backup fits entirely inside a single removable disk, but that''s not a requirement. You could always use removable stripesets, or raidz, or whatever you wanted. For example, you could build a raidz removable volume out of 5 removable disks if you wanted. Just be sure you attach all 5 disks before you "zpool import"
Edward Ned Harvey
2010-Mar-20 04:57 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> 1. NDMP for putting "zfs send" streams on tape over the network.? SoTell me if I missed something here. I don''t think I did. I think this sounds like crazy talk. I used NDMP up till November, when we replaced our NetApp with a Solaris Sun box. In NDMP, to choose the source files, we had the ability to browse the fileserver, select files, and specify file matching patterns. My point is: NDMP is file based. It doesn''t allow you to spawn a process and backup a data stream. Unless I missed something. Which I doubt. ;-)> To Ed Harvey: > > Some questions about your use of NetBackup on your secondary server: > > 1. Do you successfully backup ZVOLs?? We know NetBackup should be able > to capture datasets (ZFS file systems) using straight POSIX semantics.I wonder if I''m confused by that question. "backup zvols" to me, would imply something at a lower level than the filesystem. No, we''re not doing that. We just specify "backup the following directory and all of its subdirectories." Just like any other typical backup tool. The reason we bought NetBackup is because it intelligently supports all the permissions, ACL''s, weird (non-file) file types, and so on. And it officially supports ZFS, and you can pay for an enterprise support contract. Basically, I consider the purchase cost of NetBackup to be insurance. Although I never plan to actually use it for anything, because all our bases are covered by "zfs send" to hard disks and tapes. I actually trust the "zfs send" solution more, but I can''t claim that I, or anything I''ve ever done, is 100% infallible. So I need a commercial solution too, just so I can point my finger somewhere if needed.> 2. What version of NetBackup are you using?I could look it up, but I''d have to VPN in and open up a console, etc etc. We bought it in November, so it''s whatever was current 4-5 months ago.> 3. You simply run the NetBackup agent locally on the (Open)Solaris > server?Yup. We''re doing no rocket science with it. Ours is the absolute most basic NetBackup setup you could possibly have. We''re not using 90% of the features of NetBackup. It''s installed on a Solaris 10 server, with locally attached tape library, and it does backups directly from local disk to local tape.
Responses inline below... On Sat, Mar 20, 2010 at 00:57, Edward Ned Harvey <solaris2 at nedharvey.com>wrote:> > 1. NDMP for putting "zfs send" streams on tape over the network. So > > Tell me if I missed something here. I don''t think I did. I think this > sounds like crazy talk. > > I used NDMP up till November, when we replaced our NetApp with a Solaris > Sun > box. In NDMP, to choose the source files, we had the ability to browse the > fileserver, select files, and specify file matching patterns. My point is: > NDMP is file based. It doesn''t allow you to spawn a process and backup a > data stream. > > Unless I missed something. Which I doubt. ;-) > >You clearly know more about NDMP than I do. I''m still learning. I forgot that you previously mentioned the file-based nature of NDMP. I''m still wondering about that in the longer term, but yeah, this is my mistake. I''ll end up doing some deeper diving on this topic, I can see. But this was just me seeking clarity. Maybe Fishworks appliances would benefit from the presence of NDMP but if you''re using a standard server running (Open)Solaris, it looks like a non-starter.> > > To Ed Harvey: > > > > Some questions about your use of NetBackup on your secondary server: > > > > 1. Do you successfully backup ZVOLs? We know NetBackup should be able > > to capture datasets (ZFS file systems) using straight POSIX semantics. > > I wonder if I''m confused by that question. "backup zvols" to me, would > imply something at a lower level than the filesystem. No, we''re not doing > that. We just specify "backup the following directory and all of its > subdirectories." Just like any other typical backup tool. > > The reason we bought NetBackup is because it intelligently supports all the > permissions, ACL''s, weird (non-file) file types, and so on. And it > officially supports ZFS, and you can pay for an enterprise support > contract. > > Basically, I consider the purchase cost of NetBackup to be insurance. > Although I never plan to actually use it for anything, because all our > bases > are covered by "zfs send" to hard disks and tapes. I actually trust the > "zfs send" solution more, but I can''t claim that I, or anything I''ve ever > done, is 100% infallible. So I need a commercial solution too, just so I > can point my finger somewhere if needed. >Yeah, I get all the reasons you state for using NetBackup. Makes total sense. And I asked this question to be clear about support for backing up ZVOLs outside if ZFS-specific tools e.g. zfs(1M). I didn''t actually think NetBackup could capture ZVOLs, for the reasons you listed, but I wanted to be absolutely clear. Asking the wrong questions is the leading cause of wrong answers, as a former boss of mine used to say.> > > > 2. What version of NetBackup are you using? > > I could look it up, but I''d have to VPN in and open up a console, etc etc. > We bought it in November, so it''s whatever was current 4-5 months ago. > >Ok. Thanks.> > > 3. You simply run the NetBackup agent locally on the (Open)Solaris > > server? > > Yup. We''re doing no rocket science with it. Ours is the absolute most > basic NetBackup setup you could possibly have. We''re not using 90% of the > features of NetBackup. It''s installed on a Solaris 10 server, with locally > attached tape library, and it does backups directly from local disk to > local > tape. > >This is an advantage of Solaris being a 1st class citizen in the NetBackup world. For a Unified Storage appliance, however, NDMP for file level backup may be a reasonable choice (as Darren postulated earlier). But if you just buy a server and install Solaris, then the NetBackup Solaris agent is the easiest route, as you''ve shown. Thanks again, Ed, for your time and generosity. And thank you to all contributors to this thread for indulging my curiosity. -- "You can choose your friends, you can choose the deals." - Equity Private "If Linux is faster, it''s a Solaris bug." - Phil Harman Blog - http://whatderass.blogspot.com/ Twitter - @khyron4eva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100320/dcf37f14/attachment.html>
> > I''ll say it again: neither ''zfs send'' or (s)tar is an > enterprise (or > even home) backup system on their own one or both can > be components of > the full solution. >Up to a point. zfs send | zfs receive does make a very good back up scheme for the home user with a moderate amount of storage. Especially when the entire back up will fit on a single drive which I think would cover the majority of home users. Using external drives and incremental zfs streams allows for extremely quick back ups of large amounts of data. It certainly does for me. http://chrisgerhard.wordpress.com/2007/06/01/rolling-incremental-backups/ -- This message posted from opensolaris.org
On Fri, Mar 19, 2010 at 11:57 PM, Edward Ned Harvey <solaris2 at nedharvey.com> wrote:>> 1. NDMP for putting "zfs send" streams on tape over the network.? So > > Tell me if I missed something here. ?I don''t think I did. ?I think this > sounds like crazy talk. > > I used NDMP up till November, when we replaced our NetApp with a Solaris Sun > box. ?In NDMP, to choose the source files, we had the ability to browse the > fileserver, select files, and specify file matching patterns. ?My point is: > NDMP is file based. ?It doesn''t allow you to spawn a process and backup a > data stream. > > Unless I missed something. ?Which I doubt. ?;-)5+ years ago the variety of NDMP that was available with the combination of NetApp''s OnTap and Veritas NetBackup did backups at the volume level. When I needed to go to tape to recover a file that was no longer in snapshots, we had to find space on a NetApp to restore the volume. It could not restore the volume to a Sun box, presumably because the contents of the backup used a data stream format that was proprietary to NetApp. An expired Internet Draft for NDMPv4 says: butype_name Specifies the name of the backup method to be used for the transfer (dump, tar, cpio, etc). Backup types are NDMP Server implementation dependent and MUST match one of the Data Server implementation specific butype_name strings accessible via the NDMP_CONFIG_GET_BUTYPE_INFO request. http://www.ndmp.org/download/sdk_v4/draft-skardal-ndmp4-04.txt It seems pretty clear from this that an NDMP data stream can contain most anything and is dependent on the device being backed up. -- Mike Gerdts http://mgerdts.blogspot.com/
Edward Ned Harvey
2010-Mar-20 13:29 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> > I''ll say it again: neither ''zfs send'' or (s)tar is an > > enterprise (or > > even home) backup system on their own one or both can > > be components of > > the full solution. > > > > Up to a point. zfs send | zfs receive does make a very good back up > scheme for the home user with a moderate amount of storage. Especially > when the entire back up will fit on a single drive which I think would > cover the majority of home users.I''ll repeat: There is nothing preventing you from creating an external zpool using more than one disk. Sure it''s convenient when your whole backup fits onto a single external disk, but not necessary.
Edward Ned Harvey
2010-Mar-20 13:33 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> 5+ years ago the variety of NDMP that was available with the > combination of NetApp''s OnTap and Veritas NetBackup did backups at the > volume level. When I needed to go to tape to recover a file that was > no longer in snapshots, we had to find space on a NetApp to restore > the volume. It could not restore the volume to a Sun box, presumably > because the contents of the backup used a data stream format that was > proprietary to NetApp. > > An expired Internet Draft for NDMPv4 says: > > butype_name > Specifies the name of the backup method to be used > for the > transfer (dump, tar, cpio, etc). Backup types are > NDMP Server > implementation dependent and MUST match one of the > Data > Server implementation specific butype_name > strings accessible > via the NDMP_CONFIG_GET_BUTYPE_INFO request. > > http://www.ndmp.org/download/sdk_v4/draft-skardal-ndmp4-04.txt > > It seems pretty clear from this that an NDMP data stream can contain > most anything and is dependent on the device being backed up.So it''s clear that at least the folks at ndmp.org were/are thinking about doing backups using techniques not necessarily based on filesystem. But ... Where''s the implementation? It doesn''t do any good if there''s just an RFC written somewhere that all the backup tools ignore. I was using BackupExec 11d with NDMP Option to backup my Netapp. This setup certainly couldn''t do anything other than file selection. I can''t generalize and say "nothing does," but ... Does anything? Does anything out there support non-file-based backup via NDMP?
On Mar 20, 2010, at 00:57, Edward Ned Harvey wrote:> I used NDMP up till November, when we replaced our NetApp with a > Solaris Sun > box. In NDMP, to choose the source files, we had the ability to > browse the > fileserver, select files, and specify file matching patterns. My > point is: > NDMP is file based. It doesn''t allow you to spawn a process and > backup a > data stream.Not quite. It can reference files, but only by specifying where they are in an opaque "data stream" (see ?2.3.5.2 of the NDMPv4 spec [1]):> The file locator data in the file history record is in a data > service (OS) specific format. To the DMA this information is an > opaque string. This means that the DMA will not attempt to interpret > it. In order to determine the location of a file in the backup data > stream, the DMA will send the complete file history record for the > corresponding file history record to the data service, the data > service will calculate the starting location and the length of the > byte string to be read from the original backup data stream. The DMA > will use this data to manipulate the tape service to retrieve the > selected data.So the backup software (DMA) simply knows the tape on which the file is on, and the starting byte of that tape, but if you want to restore a file from (say) a NetApp share or export, you have to send the bytes to another NetApp which can interpret the stream. It''s not like the byte stream is in a known format (tar, cpio, or zip) that can be interpreted by anyone. (Unless you reverse engineer the format of course.) After a filer ("NDMP Data Service") is told to start backing up, it can tell the backup software ("NDMP Data Management Application"--DMA) about files via the NDMP_FH_ADD_FILE command (see ?4.3.1 [1]). [1] http://www.ndmp.org/download/sdk_v4/draft-skardal-ndmp4-04.txt So technically Oracle can implement an NDMP service on (Open)Solaris, and backup vendors could interface with that and send the raw ZFS data stream to tape. As the Solaris kernel traverses the file system, and comes across directories and files, it would tell the backup software about the file (path, owner, group, etc.) and where it is in the stream sent to "tape" (LTO, VTL, etc.). On file restoration, the backup software would then have to send the (opaque-to-it) data stream from tape to another Solaris box that could interpret it. This is of course in the case of a CIFS share or NFS export, where the filer (NetApp, Sun 7000 series, Celerra) has some knowledge of the file names, and wouldn''t work on a raw LUN--unless the filer starts parsing the LUN for disk formats like is done with VMware''s VMDK format and NetBackup, where they can figure out the files.
On Mar 18, 2010, at 6:28 AM, Darren J Moffat wrote:> The only tool I''m aware of today that provides a copy of the data, and all of the ZPL metadata and all the ZFS dataset properties is ''zfs send''.AFAIK, this is correct. Further, the only type of tool that can backup a pool is a tool like dd. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com
Edward Ned Harvey
2010-Mar-21 13:40 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
> > The only tool I''m aware of today that provides a copy of the data, > and all of the ZPL metadata and all the ZFS dataset properties is ''zfs > send''. > > AFAIK, this is correct. > Further, the only type of tool that can backup a pool is a tool like > dd.How is it different to backup a pool, versus to backup all the zfs filesystems in a pool? Obviously, dd is not viable for most situations as a backup tool... Is there any reason anyone would ever do it for ZFS, aside from forensics? Oh, I know one difference. "dd" to backup the pool would also preserve all the zfs snaps. But if you want, you could have done that with "zfs send" too. So my question still stands: What can you backup in a zpool, using dd, that you can''t backup via "zfs send?" (As pointless as this may be, it''s academic.) ;-)
Richard Elling
2010-Mar-21 19:33 UTC
[zfs-discuss] VTL with ZFS [was: Thoughts on ZFS Pool Backup Strategies]
On Mar 17, 2010, at 6:57 AM, Khyron wrote:> Rather, I am asking: > > Why do we want to adapt "zfs send" to do something it was never intended > to do, and probably won''t be adapted to do (well, if at all) anytime soon instead of > optimizing existing technologies for this use case?So when you turn your ZFS box into a VTL, you can use it to store NDMP backups from another ZFS box and recurse until the cows come home :-) The Nexenta community has a plugin for turning your NexentaStor box into a VTL. http://www.nexenta.com/corp/blog/2010/02/19/vtl-for-zfs-based-zvols-cool-and-free-nexentastororg/ -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com
On Mar 21, 2010, at 6:40 AM, Edward Ned Harvey wrote:>>> The only tool I''m aware of today that provides a copy of the data, >> and all of the ZPL metadata and all the ZFS dataset properties is ''zfs >> send''. >> >> AFAIK, this is correct. >> Further, the only type of tool that can backup a pool is a tool like >> dd. > > How is it different to backup a pool, versus to backup all the zfs > filesystems in a pool? Obviously, dd is not viable for most situations as a > backup tool... Is there any reason anyone would ever do it for ZFS, aside > from forensics? > > Oh, I know one difference. "dd" to backup the pool would also preserve all > the zfs snaps. But if you want, you could have done that with "zfs send" > too. > > So my question still stands: What can you backup in a zpool, using dd, that > you can''t backup via "zfs send?" > > (As pointless as this may be, it''s academic.) ;-)The pool configuration, metadata (eg. DDT), and all of the datasets. Today, there are three dataset types readily available: file system, zvol, and pnfs. On the radar is a lustre dataset type. Each of the dataset types has a method of backup, even if only dd-like. The way I see it, ZFS defies traditional notions of "file system" backup because ZFS is much more than just a "file system." Not surprisingly, it looks a lot more like an Oracle database than a UFS file system, and the methods of backup for databases are similar to that provided by ZFS. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com
Constantin Gonzalez
2010-Mar-22 08:42 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Hi, I agree 100% with Chris. Notice the "on their own" part of the original post. Yes, nobody wants to run zfs send or (s)tar by hand. That''s why Chris''s script is so useful: You set it up and forget and get the job done for 80% of home users. On another note, I was positively surprised by the availability of Crash Plan for OpenSolaris: http://crashplan.com/ Their free service allows to back up your stuff to a friend''s system over the net in an encrypted way, the paid-for servide uses Crashplan''s data centers at a less than Amazon-S3 pricing. While this may not be everyone''s solution, I find it significant that they explicitly support OpenSolaris. This either means they''re OpenSolaris fans or that they see potential in OpenSolaris home server users. Cheers, Constantin On 03/20/10 01:31 PM, Chris Gerhard wrote:>> >> I''ll say it again: neither ''zfs send'' or (s)tar is an >> enterprise (or >> even home) backup system on their own one or both can >> be components of >> the full solution. >> > > Up to a point. zfs send | zfs receive does make a very good back up scheme for the home user with a moderate amount of storage. Especially when the entire back up will fit on a single drive which I think would cover the majority of home users. > > Using external drives and incremental zfs streams allows for extremely quick back ups of large amounts of data. > > It certainly does for me. http://chrisgerhard.wordpress.com/2007/06/01/rolling-incremental-backups/-- Sent from OpenSolaris, http://www.opensolaris.org/ Constantin Gonzalez Sun Microsystems GmbH, Germany Principal Field Technologist Blog: constantin.glez.de Tel.: +49 89/4 60 08-25 91 Twitter: @zalez Sitz d. Ges.: Sun Microsystems GmbH, Sonnenallee 1, 85551 Kirchheim-Heimstetten Amtsgericht Muenchen: HRB 161028 Geschaeftsfuehrer: Thomas Schroeder
David Dyer-Bennet
2010-Mar-22 16:01 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
On Sat, March 20, 2010 07:31, Chris Gerhard wrote:> Up to a point. zfs send | zfs receive does make a very good back up scheme > for the home user with a moderate amount of storage. Especially when the > entire back up will fit on a single drive which I think would cover the > majority of home users.My own fit on a single external drive; but I''ve noticed that I have a rather small configuration (1.2TB nominal, less than 800GB used). Most people I hear describing building home NAS setups put between 4 and 10 of the biggest drives they can buy in them -- much more capacity than mine (but then I built mine in 2006, too). I''m not clear how much of it they ever fill up :-).> Using external drives and incremental zfs streams allows for extremely > quick back ups of large amounts of data. > > It certainly does for me. > http://chrisgerhard.wordpress.com/2007/06/01/rolling-incremental-backups/So far, for me it allows for endless failures and a LOT of reboots to free stuck IO subsystems. Your script seems to be using a simple zfs send -i; what I''m trying to do is use an incremental replication stream, a -R -I thing (from memory; hope that''s right!). This should propagate (for example) my every-2-hours snapshots over onto the backup, even though I only back up to a given drive every two or three days (three backup drives, rotating one off-site). Unfortunately, though, it doesn''t work; hangs during the receive eventually. I''m waiting for the 2010.$Spring stable release to see how it behave there before I get really energetic about debugging; for now I''m just forcing pool recreation and full backups each time (destroying the filesystem also hangs). Given that full backups run through to completion, whereas incrementals fail even though they''re pushing a lot less data and take a lot less time, I''m not inclined to blame my USB hardware. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Joerg Schilling
2010-Mar-24 14:36 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
Darren J Moffat <darrenm at opensolaris.org> wrote:> > You cannot get a single file out of the zfs send datastream. > > I don''t see that as part of the definition of a backup - you obviously > do - so we will just have to disagree on that.If you need to set up a file server of the same size as the original one in order to be able to access a specific file from backup data, this could be sees as major handicap.> >> getattrat(3C) / setattrat(3C) > >> > >> Even has example code in it. > >> > >> This is what ls(1) uses. > > > > It could be easily possible to add portable support integrated into the > > framework that already supports FreeBSD and Linux attributes. > > Great, do you have a time frame for when you will have this added to > star then ?I need to write some missing 50 lines of code (formatting virgin BD-RE and BD-RE/DL media) in cdrecordl and publish cdrtools-3.0-final before I start working on other projects, but this will hopefully be soon.> > - A public interface to get the property state > > That would come from libzfs. There are private interfaces just now that > are very likely what you need zfs_prop_get()/zfs_prop_set(). They aren''t > documented or public though and are subject to change at any time.mmm, as the state of the compression flag may seriously affect media consumption, this seems to be an important part of the meta data in case of a backup. Is there no way to define an interface that will just evolve without becoming ncompatible? J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
On Wed, March 24, 2010 10:36, Joerg Schilling wrote:>> > - A public interface to get the property state >> >> That would come from libzfs. There are private interfaces just now that >> are very likely what you need zfs_prop_get()/zfs_prop_set(). They aren''t >> documented or public though and are subject to change at any time. > > mmm, as the state of the compression flag may seriously affect media > consumption, this seems to be an important part of the meta data in case > of a > backup. Is there no way to define an interface that will just evolve > without > becoming ncompatible?I think the larger question is: when will ZFS be stable enough that Oracle will say that libzfs is an officially supported interface? Once that happens it will probably be possible for third parties to start accessing ZFS in ways other than the POSIX interface. I''m guessing that support for crypto, device removal, and parity changing (RAID-Z1 <-> Z2 <-> Z3) need to be put in first (the latter two necessitating bp rewrite). I would hazard to guess it will be at least a year before it''s even considered and longer before it happens (Solaris 12? or maybe a latter update of Solaris 11?). Until that happens we''ll be stuck with working at the ZPL for most things.
Darren J Moffat
2010-Mar-25 09:34 UTC
[zfs-discuss] Thoughts on ZFS Pool Backup Strategies
>>> - A public interface to get the property state >> >> That would come from libzfs. There are private interfaces just now that >> are very likely what you need zfs_prop_get()/zfs_prop_set(). They aren''t >> documented or public though and are subject to change at any time. > > mmm, as the state of the compression flag may seriously affect media > consumption, this seems to be an important part of the meta data in case of a > backup. Is there no way to define an interface that will just evolve without > becoming ncompatible?Compression doesn''t really impact the tools consuming the POSIX layer interfaces if you look at the fields of struct stat. The POSIX layer (infact even ''zfs send'') *always* sees the uncompressed data. For example: The ZFS filesystem /one has compression=on stat /one/hamlet.txt /one/off/hamlet.txt File: `/one/hamlet.txt'' Size: 211179 Blocks: 310 IO Block: 131072 regular file Device: 2d900a9h/47775913d Inode: 5 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2010-03-25 09:25:25.456468675 +0000 Modify: 2010-03-25 09:25:25.489588537 +0000 Change: 2010-03-25 09:25:25.489588537 +0000 The ZFS filesystem /one/off has compression=off File: `/one/off/hamlet.txt'' Size: 211179 Blocks: 517 IO Block: 131072 regular file Device: 2d900aah/47775914d Inode: 5 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2010-03-25 09:25:45.946130734 +0000 Modify: 2010-03-25 09:25:45.946404528 +0000 Change: 2010-03-25 09:25:45.946404528 +0000 Note that while the number of blocks is much lower in the compression=on case the Size of the file is still the same. Since you have no way to read or write the compressed data it really shouldn''t mater that the number of blocks on disk are different if you are using the POSIX layer interfaces (which is all you can do from userland anyway - and even all the kernel parts of ''zfs send'' can do). Now if you are doing something like multiplying out the number of blocks by the blocksize I can see where things can be a problem. However that would be a big problem with ZFS even if compression wasn''t enabled because the block size isn''t fixed (512 - 128k in powers of 2). Now compare this with the same hamlet.txt file stored on UFS: stat /mnt/hamlet.txt File: `/mnt/hamlet.txt'' Size: 211179 Blocks: 432 IO Block: 8192 regular file Device: 2d80003h/47710211d Inode: 4 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2010-03-25 09:27:34.109975000 +0000 Modify: 2010-03-25 09:27:34.110729000 +0000 Change: 2010-03-25 09:27:34.110729000 +0000 So maybe I''m missing what the issue for you is, if so can you try and explain it to me by using an example. Thanks. -- Darren J Moffat