hello team i have an issue with my ZFS system, i have 5 file systems and i need to take a daily backup of these onto tape. how best do you think i should do these? the smallest filesystem is about 50GB here is what i have been doing i take snapshots of the 5 file systems, i zfs send these into a directory gzip the the files and then tar them onto tape. this takes a considerable amount of time. my question is there a faster and better way of doing this? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110913/f15ed41a/attachment.html>
On Tue, Sep 13, 2011 at 9:48 AM, cephas maposah <maposah at gmail.com> wrote:> hello team > i have an issue with my ZFS system, i have 5 file systems and i need to take > a daily backup of these onto tape. how best do you think i should do these? > the smallest filesystem is about 50GB > here is what i have been doing i take snapshots of the 5 file systems, i zfs > send these into a directory gzip the the files and then tar them onto tape. > this takes a considerable amount of time. > my question is there a faster and better way of doing this?So you zfs send to a file and save the file to tape? Personally, I use zfs send and zfs receive to replicate the data to a second system, and then simply tar the replicated file system to tape. That way I''m not dependent on zfs to get the data back (and can get individual files/directories back if necessary, which it often is). And you have the ability to slot that copy of the data instantly into service if the primary copy fails. For tar, you can substitute a free or commercial backup solution. It works the same way. -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
On 09/13/11 09:00 PM, Peter Tribble wrote:> On Tue, Sep 13, 2011 at 9:48 AM, cephas maposah<maposah at gmail.com> wrote: >> hello team >> i have an issue with my ZFS system, i have 5 file systems and i need to take >> a daily backup of these onto tape. how best do you think i should do these? >> the smallest filesystem is about 50GB >> here is what i have been doing i take snapshots of the 5 file systems, i zfs >> send these into a directory gzip the the files and then tar them onto tape. >> this takes a considerable amount of time. >> my question is there a faster and better way of doing this? > So you zfs send to a file and save the file to tape? > > Personally, I use zfs send and zfs receive to replicate the data to > a second system, and then simply tar the replicated file system > to tape. That way I''m not dependent on zfs to get the data back > (and can get individual files/directories back if necessary, which > it often is). And you have the ability to slot that copy of the data > instantly into service if the primary copy fails. >I''d also add that unless you are sending tapes off-site, having a snapshots on a seconds system can save you a lot of incremental backups. -- Ian.
i do send the tapes offline. The idea of sending onto another system seems an excellent idea but i would still need to send them to tape. if i tar or dump the replicated filesystems this will chew up a lot of space since my filesystems are almost 900GB in total On Tue, Sep 13, 2011 at 11:07 AM, Ian Collins <ian at ianshome.com> wrote:> On 09/13/11 09:00 PM, Peter Tribble wrote: > >> On Tue, Sep 13, 2011 at 9:48 AM, cephas maposah<maposah at gmail.com> >> wrote: >> >>> hello team >>> i have an issue with my ZFS system, i have 5 file systems and i need to >>> take >>> a daily backup of these onto tape. how best do you think i should do >>> these? >>> the smallest filesystem is about 50GB >>> here is what i have been doing i take snapshots of the 5 file systems, i >>> zfs >>> send these into a directory gzip the the files and then tar them onto >>> tape. >>> this takes a considerable amount of time. >>> my question is there a faster and better way of doing this? >>> >> So you zfs send to a file and save the file to tape? >> >> Personally, I use zfs send and zfs receive to replicate the data to >> a second system, and then simply tar the replicated file system >> to tape. That way I''m not dependent on zfs to get the data back >> (and can get individual files/directories back if necessary, which >> it often is). And you have the ability to slot that copy of the data >> instantly into service if the primary copy fails. >> >> I''d also add that unless you are sending tapes off-site, having a > snapshots on a seconds system can save you a lot of incremental backups. > > -- > Ian. > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110913/5303b1c6/attachment-0001.html>
On Tue, Sep 13, 2011 at 3:48 PM, cephas maposah <maposah at gmail.com> wrote:> hello team > i have an issue with my ZFS system, i have 5 file systems and i need to take > a daily backup of these onto tape. how best do you think i should do these? > the smallest filesystem is about 50GBIt depends. You can backup the files (so it''d be the same whatever filesystem the files was on), or you can backup the send/recv stream.> here is what i have been doing i take snapshots of the 5 file systems, i zfs > send these into a directory gzip the the files and then tar them onto tape. > this takes a considerable amount of time. > my question is there a faster and better way of doing this?yes, that sucks as you need to write it to a temporary file first. Is your tape a real tape or VTL? If it''s VTL, it might be easier to just use it as disk so you can directly write the compressed zfs stream there without the need of temporary file first. Also, using lzop is faster than gzip (and less compression), so you might want to try that as well. -- Fajar
On Tue, Sep 13, 2011 at 4:37 PM, Fajar A. Nugraha <list at fajar.net> wrote:>> here is what i have been doing i take snapshots of the 5 file systems, i zfs >> send these into a directory gzip the the files and then tar them onto tape. >> this takes a considerable amount of time. >> my question is there a faster and better way of doing this? > > yes, that sucks as you need to write it to a temporary file first.EDIT: you CAN write the zfs stream directly to tape (using plain output redirection, without using tar) so you don''t need a temporary file (at least this is true in Linux), but it''s kinda awkward since you need to remember (or store somewhere else) where the zfs stream was from and what file number (on the tape) stores the stream. So again I suggest you use disk as backup media when possible. -- Fajar
On 09/13/11 09:20 PM, cephas maposah wrote:> On Tue, Sep 13, 2011 at 11:07 AM, Ian Collins <ian at ianshome.com > <mailto:ian at ianshome.com>> wrote: > > On 09/13/11 09:00 PM, Peter Tribble wrote: > > On Tue, Sep 13, 2011 at 9:48 AM, cephas > maposah<maposah at gmail.com <mailto:maposah at gmail.com>> wrote: > > hello team > i have an issue with my ZFS system, i have 5 file systems > and i need to take > a daily backup of these onto tape. how best do you think i > should do these? > the smallest filesystem is about 50GB > here is what i have been doing i take snapshots of the 5 > file systems, i zfs > send these into a directory gzip the the files and then > tar them onto tape. > this takes a considerable amount of time. > my question is there a faster and better way of doing this? > > So you zfs send to a file and save the file to tape? > > Personally, I use zfs send and zfs receive to replicate the > data to > a second system, and then simply tar the replicated file system > to tape. That way I''m not dependent on zfs to get the data back > (and can get individual files/directories back if necessary, which > it often is). And you have the ability to slot that copy of > the data > instantly into service if the primary copy fails. > > I''d also add that unless you are sending tapes off-site, having a > snapshots on a seconds system can save you a lot of incremental > backups. > > i do send the tapes offline. The idea of sending onto another system > seems an excellent idea but i would still need to send them to tape. > if i tar or dump the replicated filesystems this will chew up a lot of > space since my filesystems are almost 900GB in totalPipe the output of tar or what ever you use through gzip (as you were originally?). -- Ian.
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of cephas maposah > > here is what i have been doing i take snapshots of the 5 file systems, izfs> send these into a directory gzip the the files and then tar them ontotape.> this takes a considerable amount of time. > my question is there a faster and better way of doing this?zfs send | lzop | mbuffer will write data stream directly to tape, no need for intermediate storage. It has the disadvantage that you can''t restore individual files, nor exclude anything from backup, and if there''s any corruption the whole stream is useless. zfs send | zfs receive (on disk) Has the advantage that you can restore directly from snapshots if necessary (much faster & easier than restoring from tape) and it verifies the integrity of the data as it''s written. You can then use anything you want (tar |lzop | mbuffer) or some commercial product, to write to tape.
On Tue, Sep 13, 2011 at 5:00 AM, Peter Tribble <peter.tribble at gmail.com> wrote:> Personally, I use zfs send and zfs receive to replicate the data to > a second system, and then simply tar the replicated file system > to tape. That way I''m not dependent on zfs to get the data back > (and can get individual files/directories back if necessary, which > it often is). And you have the ability to slot that copy of the data > instantly into service if the primary copy fails.tar or rsync will work for data, but you will lose any ACLs The only tools I have found that work with zfs ACLs are the native zfs tools (zfs send / recv), the native Solaris tools (cp, mv, etc.), and Symantec NetBackup. I have not tried other commercial backup systems as we already have NBU in house. zfs send / recv is far, far faster than NBU. -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Sound Designer: Frankenstein, A New Musical (http://www.facebook.com/event.php?eid=123170297765140) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, RPI Players
Concerning native Solaris tools: Solaris tar takes a ''p'' flag that stores ACL information, and re-creates same when extracted. I''ve not made much use of it yet, but it apparently works on UFS and ZFS (although there are obvious difficulties restoring ZFS ACLs to UFS, etc.) Gtar on Linux extracts the resulting tar file, but doesn''t know what to do with the ACL entries. Current versions of Amanda support a ''suntar'' backup type. Haven''t explored that yet, but Amanda might be useful. Bryan On Tue, Sep 13, 2011 at 08:41:37AM -0400, Paul Kraus wrote:> On Tue, Sep 13, 2011 at 5:00 AM, Peter Tribble <peter.tribble at gmail.com> wrote: > > > Personally, I use zfs send and zfs receive to replicate the data to > > a second system, and then simply tar the replicated file system > > to tape. That way I''m not dependent on zfs to get the data back > > (and can get individual files/directories back if necessary, which > > it often is). And you have the ability to slot that copy of the data > > instantly into service if the primary copy fails. > > tar or rsync will work for data, but you will lose any ACLs > > The only tools I have found that work with zfs ACLs are the native zfs > tools (zfs send / recv), the native Solaris tools (cp, mv, etc.), and > Symantec NetBackup. I have not tried other commercial backup systems > as we already have NBU in house. > > zfs send / recv is far, far faster than NBU. > > -- > {--------1---------2---------3---------4---------5---------6---------7---------} > Paul Kraus > -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) > -> Sound Designer: Frankenstein, A New Musical > (http://www.facebook.com/event.php?eid=123170297765140) > -> Sound Coordinator, Schenectady Light Opera Company ( > http://www.sloctheater.org/ ) > -> Technical Advisor, RPI Players > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Bryan Hodgson Lehigh University bryan.hodgson at lehigh.edu Packard Lab 019 room 115 610-758-5001 19 Memorial Drive West Bethlehem, PA 18015-3016
2011/9/13 Paul Kraus <paul at kraus-haus.org>:> The only tools I have found that work with zfs ACLs are the native zfs > tools (zfs send / recv), the native Solaris tools (cp, mv, etc.), and > Symantec NetBackup. I have not tried other commercial backup systems > as we already have NBU in house.cpio, possibly? -- Frank Van Damme No part of this copyright message may be reproduced, read or seen, dead or alive or by any means, including but not limited to telepathy without the benevolence of the author.