Hi, I have a very limited amount of bandwidth between main office and a colocated rack of servers in a managed datacenter. My hope is to be able to zfs send/recv small incremental changes on a nightly basis as a secondary offsite backup strategy. My question is about the initial "seed" of the data. Is it possible to use a portable drive to copy the initial zfs filesystem(s) to the remote location and then make the subsequent incrementals over the network? If so, what would I need to do to make sure it is an exact copy? Thank you, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110216/7d93584e/attachment-0001.html>
On Feb 16, 2011, at 7:38 AM, whitetr6 at gmail.com wrote:> Hi, I have a very limited amount of bandwidth between main office and a colocated rack of servers in a managed datacenter. My hope is to be able to zfs send/recv small incremental changes on a nightly basis as a secondary offsite backup strategy. My question is about the initial "seed" of the data. Is it possible to use a portable drive to copy the initial zfs filesystem(s) to the remote location and then make the subsequent incrementals over the network? If so, what would I need to do to make sure it is an exact copy? Thank you,Yes, and this is a good idea. Once you have replicated a snapshot, it will be an exact replica -- it is an all-or-nothing operation. You can then make more replicas or incrementally add snapshots. -- richard
On Feb 16, 2011, at 7:38 AM, whitetr6 at gmail.com wrote:> My question is about the initial "seed" of the data. Is it possible > to use a portable drive to copy the initial zfs filesystem(s) to the > remote location and then make the subsequent incrementals over the > network? If so, what would I need to do to make sure it is an exact > copy? Thank you,Yes, you can send the initial seed snapshot to a file on a portable disk. for example: # zfs send tank/volume at seed > /myexternaldrive/zfssnap.data If the volume of data is too much to fit on a single disk then you can create a new pool spread across the number of disks you require, make a duplicate of the snapshot onto your new pool. Then from the new pool you can run a new zfs send when connected to your offsite server. thanks Andy.
>From what I have read, this is not the best way to do it.Your best bet is to create a ZFS pool using the external device (or even better, devices) then zfs send | zfs receive. You can then do the same at your remote location. If you just send to a file, you may find it was a wasted trip (or postage, if you send it that way) as a single error in the file will result in a failure when you try to pull the data back. If you have 2 external devices (USB or eSata HDDs?) each capable of holding all your data, my personal choice would be to use them both in a mirror, transfer to that pool, then go to your remote site a do the same. If you need more devices, try a raidz across 3-4 (or more) devices. Only my opinion.> -----Original Message----- > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of a.smith at ukgrid.net > Sent: 16 February 2011 16:46 > To: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] ZFS send/recv initial data load > > On Feb 16, 2011, at 7:38 AM, whitetr6 at gmail.com wrote: > > > My question is about the initial "seed" of the data. Is it possible > > to use a portable drive to copy the initial zfs filesystem(s) to the > > remote location and then make the subsequent incrementals over the > > network? If so, what would I need to do to make sure it is an exact > > copy? Thank you, > > Yes, you can send the initial seed snapshot to a file on a portable > disk. for example: > > # zfs send tank/volume at seed > /myexternaldrive/zfssnap.data > > If the volume of data is too much to fit on a single disk then you can > create a new pool spread across the number of disks you require, make > a duplicate of the snapshot onto your new pool. Then from the new pool > you can run a new zfs send when connected to your offsite server. > > thanks Andy. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Am 16.02.11 16:38, schrieb whitetr6 at gmail.com:> Hi, I have a very limited amount of bandwidth between main office and > a colocated rack of servers in a managed datacenter. My hope is to be > able to zfs send/recv small incremental changes on a nightly basis as > a secondary offsite backup strategy. My question is about the initial > "seed" of the data. Is it possible to use a portable drive to copy the > initial zfs filesystem(s) to the remote location and then make the > subsequent incrementals over the network? If so, what would I need to > do to make sure it is an exact copy? Thank you, > MarkJust be aware that sending zfs snapshots not only involves the actual new data, that has been written to the source(s) but also the metadata. If you don''t need it, maybe disableing atime will take away some of that load, but I have learned that, depending on how the dataset gets accessed - in my case it''s all about file sharing via netatalk and samba, the stream resulting from an incremental zfs send as always significantly greater than the data, that has been altered or has been written newly onto the dataset.
On 02/16/11 07:38, whitetr6 at gmail.com wrote:> Is it possible to use a portable drive to copy the > initial zfs filesystem(s) to the remote location and then make the > subsequent incrementals over the network?Yes. > If so, what would I need to do> to make sure it is an exact copy? Thank you,Rough outline: plug removable storage into source or a system near the source. zpool create backup pool on removable storage use an appropriate combination of zfs send & zfs receive to copy bits. zpool export backup pool. unplug removable storage move it plug it in to remote server zpool import backup pool use zfs send -i to verify that incrementals work (I did something like the above when setting up my home backup because I initially dinked around with the backup pool hooked up to a laptop and then moved it to a desktop system). optional: use zpool attach to mirror the removable storage to something faster/better/..., then after the mirror completes zpool detach to free up the removable storage. - Bill
All of these responses have been very helpful and are much appreciated. Thank you all. Mark On Feb 16, 2011 2:54pm, Erik ABLESON <eableson at mac.com> wrote:> Check out :> http://www.infrageeks.com/groups/infrageeks/wiki/8fb35/zfs_autoreplicate_script.html> It also works to an external hard disk with localhost as the destination > server. Although I don''t know if that''s the latest version which skips > ssh if it detects localhost as a destination.> Cheers,> Erik> On 16 f?vr. 2011, at 16:38, whitetr6 at gmail.com wrote:> > Hi, I have a very limited amount of bandwidth between main office and a > colocated rack of servers in a managed datacenter. My hope is to be able > to zfs send/recv small incremental changes on a nightly basis as a > secondary offsite backup strategy. My question is about the > initial "seed" of the data. Is it possible to use a portable drive to > copy the initial zfs filesystem(s) to the remote location and then make > the subsequent incrementals over the network? If so, what would I need to > do to make sure it is an exact copy? Thank you,> > Mark_______________________________________________> > zfs-discuss mailing list> > zfs-discuss at opensolaris.org> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110216/83a628c0/attachment-0001.html>
On 2/16/2011 8:08 AM, Richard Elling wrote:> On Feb 16, 2011, at 7:38 AM, whitetr6 at gmail.com wrote: > >> Hi, I have a very limited amount of bandwidth between main office and a colocated rack of servers in a managed datacenter. My hope is to be able to zfs send/recv small incremental changes on a nightly basis as a secondary offsite backup strategy. My question is about the initial "seed" of the data. Is it possible to use a portable drive to copy the initial zfs filesystem(s) to the remote location and then make the subsequent incrementals over the network? If so, what would I need to do to make sure it is an exact copy? Thank you, > Yes, and this is a good idea. Once you have replicated a snapshot, it will be an > exact replica -- it is an all-or-nothing operation. You can then make more replicas > or incrementally add snapshots. > -- richard > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discussTo follow up on Richard''s post, what you want to do is a perfectly good way to deal with moving large amounts of data via Sneakernet. :-) I''d suggest that you create a full zfs filesystem on the external drive, and use ''zfs send/receive'' to copy a snapshot from the production box to there, rather than try to store just a file from the output of ''zfs send''. You can then ''zfs send/receive'' that backup snapshot from the external drive onto your remote backup machine when you carry the drive over there later. As Richard mentioned, that snapshot is unique, and it doesn''t matter that you "recovered" it onto an external drive first, then copied that snapshot over to the backup machine. It''s a frozen snapshot, so you''re all good for future incrementals. -- Erik Trimble Java System Support Mailstop: usca22-123 Phone: x17195 Santa Clara, CA