I''m having a good look at the zfs backup and restore command, trying to compare with traditional backups, especially ufsdump and restore, and I have a number of points that need resolving. I had a quick scan through the discussion threads, and a good look through the ZFS docs, but to no avail:- 1. Can I take a recursive backup of an entire storage pool with one command? (Can''t see a way to snapshot this, so I think I may end up with a lot of backups to take...) 2. How do I list a backup file/tape to see what''s in it? (i.e If I get rid of the snapshot on disk) 2A. How do I verify a backup? I need to be sure I can restore from it. (as per -v option in ufsdump) 3. What format is a backup written in, or is it a new format for ZFS? Couldn''t decipher it with dd/od. ufsdump, tar and cpio did could not read it. 4. If it''s new, presumably I can''t restore on to a system that doesn''t support ZFS? So if I have only one Solaris/ZFS system, and that goes down, I can''t get at my data? 5. Can I restore an individual file or directory from a backup? (Thought: Maybe I could use tar - but have not yet found out if it supports ZFS ACLs yet.) Any help with these questions would be appreciated Regards Mick p.s. ZFS looks good, but I''ve only played with it for a couple of days preparing course notes for admin classes - hence the interest in backups. This message posted from opensolaris.org
Mick Hosegood <mick at firstalt.co.uk> wrote:> I''m having a good look at the zfs backup and restore command, trying to compare with traditional backups, especially ufsdump and restore, and I have a number of points > that need resolving. I had a quick scan through the discussion threads, and a good look through the ZFS docs, but to no avail:- > 1. Can I take a recursive backup of an entire storage pool with one command? > (Can''t see a way to snapshot this, so I think I may end up with a lot of backups to take...) > 2. How do I list a backup file/tape to see what''s in it? (i.e If I get rid of the snapshot on disk) > 2A. How do I verify a backup? I need to be sure I can restore from it. (as per -v option in ufsdump) > 3. What format is a backup written in, or is it a new format for ZFS? Couldn''t decipher it with dd/od. ufsdump, tar and cpio did could not read it. > 4. If it''s new, presumably I can''t restore on to a system that doesn''t support ZFS? So if I have only one Solaris/ZFS system, and that goes down, I can''t get at my data? > 5. Can I restore an individual file or directory from a backup? > (Thought: Maybe I could use tar - but have not yet found out if it supports ZFS ACLs yet.)You cannot make useful backups with Sun''s tar as the way it stores ACLs in the archive in a way that does not allow to restore the ACLs in a typical server environment. This bug has been reported agains Sun''s tar many years ago.... If you make a backup with star, you may use the -diff option to verify the backup. But note that star did not yet add support to the new ACL interface from ZFS. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Hi Mick, Here''s a potentially useful reply (perhaps others will chime in to help me out on the bits I''m missing) On Mon, 2005-12-12 at 18:58 +0100, Joerg Schilling wrote:> > 1. Can I take a recursive backup of an entire storage pool with one command?I can''t see a way to do that either, though if you''re allowed to use a construction such as : # for i in `zfs list -H -o name -t filesystem`> do > DATE=`date +%d%m%y` > zfs snapshot $i@${DATE} > zfs backup $i@${DATE} > `echo $i | sed -e ''s#/#_#g''`.${DATE}.backup > donethen it''s close enough (and not that hard to type)> > 2. How do I list a backup file/tape to see what''s in it? (i.e If I > get rid of the snapshot on disk)I''m not sure, other than to restore the archive to somewhere and have a look at the files. That''s not the best, I''ll admit.> > 2A. How do I verify a backup? I need to be sure I can restore from > it. (as per -v option in ufsdump)zfs restore -vn -d [fs] < archive ? that said, I took an archive, scribbled nonsense into the end of it, and the restore command still didn''t complain. I sense a bug/RFE coming on ...> > 3. What format is a backup written in, or is it a new format for > ZFS? Couldn''t decipher it with dd/od. ufsdump, tar and cpio did could > not read it.Yeah, it''s a new format, as per the zfs man page it''s not yet stable. More details at : http://opensolaris.org/os/community/zfs/docs/man_zfs/> > 4. If it''s new, presumably I can''t restore on to a system that > > doesn''t support ZFS? > > So if I have only one Solaris/ZFS system, and that goes down, I > > can''t get at my data?Right. Of course, you could back up using tar or cpio though... I''d imagine that most people using Solaris in production would also have a backup or test server floating around - though I realise I''m dodging the question there ;-) [ my backup systems at home are my laptop and a rather old Dell machine, both run Solaris quite nicely ]> > 5. Can I restore an individual file or directory from a backup? > > (Thought: Maybe I could use tar - but have not yet found out if > > it supports ZFS ACLs yet.)Yes, you can use tar. As per the ACL man page at http://opensolaris.org/os/community/zfs/docs/man_acl/ several Solaris utilities have been updated to support the new ACLs.> You cannot make useful backups with Sun''s tar as the way it stores ACLs > in the archive in a way that does not allow to restore the ACLs in a typical > server environment.Show me a typical server environment :-) Yes, you can use tar. Oh, glad you like ZFS, I like it too: am starting to find it hard to work out how I lived without snapshots in the past! cheers, tim -- Tim Foster, Sun Microsystems Inc, Operating Platforms Group Engineering Operations http://blogs.sun.com/timf
On Mon, 2005-12-12 at 17:58, Joerg Schilling wrote:> You cannot make useful backups with Sun''s tar as the way it stores ACLs > in the archive in a way that does not allow to restore the ACLs in a typical > server environment.In a most large enterprise environments people don''t use "tar" for backups and there is no such thing as a typical server environment. The programs tar(1)/cpio(1)/pax(1) are not backup programs to most people who run big enterprise deployments they are archivers. No disrespect but if you suggested using star to anyone backing up thousands of machines to massive tape libraries (and doing it all over the network with over the wire encryption), they would laugh at you. To these people real backup software is something like Legato Networker or Veritas NetBackup, both of which are being updated to deal with the new features of ZFS.> This bug has been reported agains Sun''s tar many years ago....I''ve asked before and I''ll ask again what bug number is this?, I''ve looked and I can''t find any possible hits. I really would like to see if it is possible to fix this but unless you either tell us exactly what the problem is or give a bug number then we can''t help and this is just here say. -- Darren J Moffat
Tim Foster <Tim.Foster at Sun.COM> wrote:> > > 3. What format is a backup written in, or is it a new format for > > ZFS? Couldn''t decipher it with dd/od. ufsdump, tar and cpio did could > > not read it. > > Yeah, it''s a new format, as per the zfs man page it''s not yet stable. > More details at : > http://opensolaris.org/os/community/zfs/docs/man_zfs/If this is an archive format, why does it introduce a new format? Is there any need to do this? There is the POSIX tar format with POSIX.1-2001 extensions (called pax) and there is star that established well documented archive extensions on top of POSIX.1-2001 extensions (in a way that is acceptable for a future POSIX standard). Why not using the star format?> Right. Of course, you could back up using tar or cpio though... I''d > imagine that most people using Solaris in production would also have a > backup or test server floating around - though I realise I''m dodging the > question there ;-)I do not recommend cpio - it is an outdated archive format that has limitations that cannot be removed easily. I do not recommend to use Sun''s tar as it uses a coding for ACLs that is not useful and that will prevent you from restoring ACLs in many cases.> > > 5. Can I restore an individual file or directory from a backup? > > > (Thought: Maybe I could use tar - but have not yet found out if > > > it supports ZFS ACLs yet.) > > Yes, you can use tar. As per the ACL man page at > http://opensolaris.org/os/community/zfs/docs/man_acl/ > several Solaris utilities have been updated to support the new ACLs. > > > You cannot make useful backups with Sun''s tar as the way it stores ACLs > > in the archive in a way that does not allow to restore the ACLs in a typical > > server environment. > > Show me a typical server environment :-) Yes, you can use tar.If you don''t care whether you will be able to restore ACLs, you may use Sun''s tar..... BTW: I did just file a bug report against this problem. A description on the star ACL format is here: http://web.archive.org/web/20031017110652/http://acl.bestbits.at/star.html I hope that in future, problems like this one are avoided by talking with each other before a change is made...... J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
Darren J Moffat <Darren.Moffat at Sun.COM> wrote:> On Mon, 2005-12-12 at 17:58, Joerg Schilling wrote: > > > You cannot make useful backups with Sun''s tar as the way it stores ACLs > > in the archive in a way that does not allow to restore the ACLs in a typical > > server environment. > > In a most large enterprise environments people don''t use "tar" > for backups and there is no such thing as a typical server environment. > > The programs tar(1)/cpio(1)/pax(1) are not backup programs to most > people who run big enterprise deployments they are archivers. No > disrespect but if you suggested using star to anyone backing up > thousands of machines to massive tape libraries (and doing it all over > the network with over the wire encryption), they would laugh at you.If you believe so, why does Solaris still include ufsdump/ufsrstore? Star is at the same level as ufsdump/ufsrestore is .... ....but well, star - Supports more than just rcmd() based remote backup but this could already be inside ufsdump if it would use librmt form star as already decided. - Uses a FS/OS independent archive format. - Does the backup using a standard compliant archive format that can be read by all POSIX compliant platforms without being forced to buy and install a special Vendor unique backup tool first.> > This bug has been reported agains Sun''s tar many years ago.... > > I''ve asked before and I''ll ask again what bug number is this?, I''ve > looked and I can''t find any possible hits. I really would like > to see if it is possible to fix this but unless you either tell us > exactly what the problem is or give a bug number then we can''t help and > this is just here say.This was done together with case #36430232 (~ November 2001). It may be that the Sun employee (Helmut vom Sondern) did not forward the report......but this is not my fault. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
On Tue, 2005-12-13 at 14:45, Joerg Schilling wrote:> If you believe so, why does Solaris still include ufsdump/ufsrstore?Because we still ship UFS and removing it would cause a regression. I didn''t say that they are useless only that they are archivers not backup solutions for enterprise deployments. I wouldn''t put ufsdump in as a backup solution for enterprise deployment either.> Star is at the same level as ufsdump/ufsrestore is .... > > ....but well, star > > - Supports more than just rcmd() based remote backup but this > could already be inside ufsdump if it would use librmt form star > as already decided.You are missing the point. star alone is not a backup solution it is an archiver. You can use archivers to build a backup solution, and in fact IIRC there are modes of operation for Veritas NetBackup and Legato Networker where you can use an archiver like tar instead of their custom backup formats.> - Uses a FS/OS independent archive format.Yes and so do Legato Networker and Veritas NetBackup, they run on a huge range of platforms for client and server parts.> - Does the backup using a standard compliant archive format that > can be read by all POSIX compliant platforms without being > forced to buy and install a special Vendor unique backup tool > first.No they don''t but that isn''t what they are all about. You are missing the very important distinction between an archiver program like tar and an enterprise backup solution. Yes this is all about vendor lock-in but these products are far advanced of anything else out there. I know you and I have been over this discussion before and last time I pointed you to the sites for networker and netbackup.> > > This bug has been reported agains Sun''s tar many years ago.... > > > > I''ve asked before and I''ll ask again what bug number is this?, I''ve > > looked and I can''t find any possible hits. I really would like > > to see if it is possible to fix this but unless you either tell us > > exactly what the problem is or give a bug number then we can''t help and > > this is just here say. > > This was done together with case #36430232 (~ November 2001).That case referenced an already existing bug# 4426407 that was logged by one of our internal test engineers. The bug is marked as closed with a comment that it was fixed by the integration of ZFS into ON. If you look at the diffs it certainly appears that was the intent: http://cvs.opensolaris.org/source/diff/on/usr/src/lib/libsec/common/acltext.c?r2=1.14&r1=1.13 I don''t have time to verify this but at a quick glance it the Sun Services case (based on what very little of the German I could understand) it looks like this is what you are looking for. -- Darren J Moffat
Darren J Moffat <Darren.Moffat at Sun.COM> wrote:> No they don''t but that isn''t what they are all about. You are missing > the very important distinction between an archiver program like > tar and an enterprise backup solution. Yes this is all about vendor > lock-in but these products are far advanced of anything else out there.You seem to miss the fact that I never stated that star is a enterprise backup solution. Star is a backup program that tries to be as universal as possible. It is free and people may like to write some high level code that depends on the features of star and offers the features people like to see with a enterprise backup solution. I will not do it myself. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
On Tue, 2005-12-13 at 16:06, Joerg Schilling wrote:> Darren J Moffat <Darren.Moffat at Sun.COM> wrote: > > > > No they don''t but that isn''t what they are all about. You are missing > > the very important distinction between an archiver program like > > tar and an enterprise backup solution. Yes this is all about vendor > > lock-in but these products are far advanced of anything else out there. > > You seem to miss the fact that I never stated that star is a > enterprise backup solution. Star is a backup program that tries to > be as universal as possible.Okay I translated "typical server environment" into "enterprise" and went from there. There is no typical server environment but the closest thing for many people is an enterprise data centre. I think we are way off topic now and I''m going to shut up. -- Darren J Moffat
With regards the question:>> 5. Can I restore an individual file or directory from a backup?Are there any plans for a similar command line interface to the restore function as there is today with ufsrestore??? This message posted from opensolaris.org
Phil Coleman <phil.c.coleman at britishairways.com> wrote:> With regards the question: > > >> 5. Can I restore an individual file or directory from a backup? > > Are there any plans for a similar command line interface to the restore function as there is today with ufsrestore???If you do not yet need support for the current beta ACL interfaces, you could use incremental backups with star. ftp://ftp.berlios.de/pub/star/alpha/ J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily
> I''m having a good look at the zfs backup and restore > command, trying to compare with traditional backups, > especially ufsdump and restore, and I have a number > of points > that need resolving. I had a quick scan through the > discussion threads, and a good look through the ZFS > docs, but to no avail:-Sorry for taking so long to reply. It seems that this thread has strayed a bit from your original questions, but I''ll try to answer them here. First let me reiterate that ''zfs backup'' is not meant to be a full-fledged backup "solution". At this point, the primary intended use is for remote replication and data movement. Going forward, ''zfs backup'' may be the underlying infrastructure that a real enterprise backup solution is built on (or more likely, it could be a new data provider for an existing backup program). I think that a lot of the misperception is due to the unfortunate choice of name for this command. Perhaps ''zfs serialize'' or ''zfs send'' would have been better.> 1. Can I take a recursive backup of an entire storage > pool with one command? > (Can''t see a way to snapshot this, so I think I may > end up with a lot of backups to take...)There isn''t a good way to do this currently, but it''s something that we''d very much like to implement.> 2. How do I list a backup file/tape to see what''s in > it? (i.e If I get rid of the snapshot on disk)You can use ''zfs restore -vn <snap>'' which will give you verbose output (-v) but do nothing (-n). This will effectively tell you the original name of the snapshot, and the name of the new snapshot that would be restored if you didn''t use -n. For example: # zfs backup foo/bar at a | zfs restore -vn foo/asdf would restore full backup of foo/bar at a into foo/asdf at a> 2A. How do I verify a backup? I need to be sure I can > restore from it. (as per -v option in ufsdump)There isn''t currently a good way to do this, but eventually I think that ''zfs restore -n'' should verify the backup stream.> 3. What format is a backup written in, or is it a new > format for ZFS? Couldn''t decipher it with dd/od. > ufsdump, tar and cpio did could not read it.It is a new format specific to ZFS.> 4. If it''s new, presumably I can''t restore on to a > system that doesn''t support ZFS? So if I have only > one Solaris/ZFS system, and that goes down, I can''t > get at my data?That''s right.> 5. Can I restore an individual file or directory from > a backup?Not currently.> p.s. ZFS looks good, but I''ve only played with it for > a couple of days preparing course notes for admin > classes - hence the interest in backups.Thanks! --matt This message posted from opensolaris.org