I have a backup system using zfs send/receive (I know there are pros and cons to that, but it''s suitable for what I need). What I have now is a script which runs daily, do zfs send, compress and write it to a file, then transfer it with ftp to a remote host. It does full backup every 1st, and do incremental (with 1st as reference) after that. It works, but not quite resource-effective (for example, the full backup every month, and the big size of incremental backup on 30th). I''m thinking of changing it to a script which can automate replication of a zfs pool or filesystem via zfs send/receive to a remote host (via ssh or whatever). It should be smart enough to choose between full and incremental, and choose which snapshot to base the incremental stream from (in case a scheduled incremental is missed), preferably able to use snapshots created by zfs/auto-snapshot smf service. To prevent re-inventing the wheel, does such script exists already? I prever not to use AVS as I can''t use on existing zfs pool. Regards, Fajar
On Sat, Mar 28, 2009 at 11:20 AM, Fajar A. Nugraha <fajar at fajar.net> wrote:> I have a backup system using zfs send/receive (I know there are pros > and cons to that, but it''s suitable for what I need). > > What I have now is a script which runs daily, do zfs send, compress > and write it to a file, then transfer it with ftp to a remote host. It > does full backup every 1st, and do incremental (with 1st as reference) > after that. It works, but not quite resource-effective (for example, > the full backup every month, and the big size of incremental backup on > 30th). > > I''m thinking of changing it to a script which can automate replication > of a zfs pool or filesystem via zfs send/receive to a remote host (via > ssh or whatever). It should be smart enough to choose between full and > incremental, and choose which snapshot to base the incremental stream > from (in case a scheduled incremental is missed), preferably able to > use snapshots created by zfs/auto-snapshot smf service. > > To prevent re-inventing the wheel, does such script exists already? > I prever not to use AVS as I can''t use on existing zfs pool. > > Regards, > > Fajar > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >The ZFS automatic snapshot tools have the ability to execute any command after a snapshot has taken place, such as ZFS send. However, I ran into some issues, as it wasn''t strictly designed for that and didn''t handle errors very well. I have since modified some scripts out there, and rolled them into my own, you can see it here at pastebin.com: http://pastebin.com/m3871e478 Inspiration: http://blogs.sun.com/constantin/entry/zfs_replicator_script_new_edition http://blogs.sun.com/timf/en_IE/entry/zfs_automatic_snapshots_in_nv Those are some good resources, from that, you can make something work that is tailored to your environment. -- Brent Jones brent at servuhome.net
On Sun, Mar 29, 2009 at 3:40 AM, Brent Jones <brent at servuhome.net> wrote:> I have since modified some scripts out there, and rolled them into my > own, you can see it here at pastebin.com: > > http://pastebin.com/m3871e478Thanks Brent. Your script seems to handle failed replication and locking pretty well. It doesn''t seem to log WHY the replication failed though, so I think there should be something that captures stderr on line 91. One more question, is there anything on that script that requires ksh? A quick glance seems to indicate that it will work with bash as well. Regards, Fajar
On Sat, Mar 28, 2009 at 5:40 PM, Fajar A. Nugraha <fajar at fajar.net> wrote:> On Sun, Mar 29, 2009 at 3:40 AM, Brent Jones <brent at servuhome.net> wrote: >> I have since modified some scripts out there, and rolled them into my >> own, you can see it here at pastebin.com: >> >> http://pastebin.com/m3871e478 > > Thanks Brent. > > Your script seems to handle failed replication and locking pretty well. > It doesn''t seem to log WHY the replication failed though, so I think > there should be something that captures stderr on line 91. > > One more question, is there anything on that script that requires ksh? > A quick glance seems to indicate that it will work with bash as well. > > Regards, > > Fajar >I''ll see about capturing from stderr, something I should''ve added anyways. It would probably work under bash too, but some of the case checking came from the original Sun scripts, which were in KSH. I looked it up, and bash has -z string checking, so, everything ''should'' work under bash. I''ll test on Monday I''d love to see others improve on the original Sun ones, or parts of mine... I only say that because the "dependency" checking, and the "locking" were something that I didn''t see in any one elses scripts. I did it a pretty lame way I''m sure, hopefully someone can find a better way :) CC:ing opensolaris-discuss, as others are probably asking similar questions about snapshot replication. -- Brent Jones brent at servuhome.net