zfs discuss - Feb 2011 - Best way/issues with large ZFS send?

I''m preparing to replicate about 200TB of data between two data centers
using zfs send. We have ten 10TB zpools that are further broken down into zvols
of various sizes in each data center. One DC is primary and the other will be
the replication target and there is plenty of bandwidth between them (10 gig
dark fiber).

Are there any gotchas that I should be aware of? Also, at what level should I be
taking the snapshot to do the zfs send? At the primary pool level or at the zvol
level? Since the targets are to be exact replicas, I presume at the primary pool
level (e.g. "tank") rather than for every zvol (e.g. tank/prod/vol1)?

This is all using Solaris 11 Express, snv_151a.

Thanks,

Eff
-- 
This message posted from opensolaris.org

On Feb 16, 2011, at 6:05 AM, Eff Norwood wrote:
> I''m preparing to replicate about 200TB of data between two data
centers using zfs send. We have ten 10TB zpools that are further broken down
into zvols of various sizes in each data center. One DC is primary and the other
will be the replication target and there is plenty of bandwidth between them (10
gig dark fiber).
> 
> Are there any gotchas that I should be aware of? Also, at what level should
I be taking the snapshot to do the zfs send? At the primary pool level or at the
zvol level? Since the targets are to be exact replicas, I presume at the primary
pool level (e.g. "tank") rather than for every zvol (e.g.
tank/prod/vol1)?
There is no such thing as a pool snapshot. There are only dataset snapshots.

The trick to a successful snapshot+send strategy at this size is to start
snapping
early and often. You don''t want to send 200TB, you want to send 2TB,
100 times :-)

The performance is tends to be bursty, so the fixed record size of the zvols can
work to your advantage for capacity planning. Also, a buffer of some sort can
help
smooth out the utilization, see the threads on ZFS and mbuffer.
 -- richard

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Eff Norwood
> 
> Are there any gotchas that I should be aware of? Also, at what level
should I
> be taking the snapshot to do the zfs send? At the primary pool level or at
the
> zvol level? Since the targets are to be exact replicas, I presume at the
primary
> pool level (e.g. "tank") rather than for every zvol (e.g.
tank/prod/vol1)?
I don''t think you have any choice.  There is such a thing as "zfs
send" and
there is no such thing as "zpool send."  You can use -R for
replication
recursive, which is kind of like doing the zpool directly...  But not
exactly.

The only gotcha is to ensure your applications (whatever they are) are
resilient to power failure.  Because the zfs snapshot will literally produce
a block-level snapshot of the way the disks are right now, at this instant.
If you ever need to restore, then it will be as-if you sustained a power
loss and came back online.

The most notable such situation would probably be databases, if you have
any.  Ensure to use a database backup tool to export the database to a
backup file, and then send the filesystem.  Or else momentarily stop the
database services while you take the snapshot.

On 16 February, 2011 - Richard Elling sent me these 1,3K bytes:
> On Feb 16, 2011, at 6:05 AM, Eff Norwood wrote:
> 
> > I''m preparing to replicate about 200TB of data between two
data centers using zfs send. We have ten 10TB zpools that are further broken
down into zvols of various sizes in each data center. One DC is primary and the
other will be the replication target and there is plenty of bandwidth between
them (10 gig dark fiber).
> > 
> > Are there any gotchas that I should be aware of? Also, at what level
should I be taking the snapshot to do the zfs send? At the primary pool level or
at the zvol level? Since the targets are to be exact replicas, I presume at the
primary pool level (e.g. "tank") rather than for every zvol (e.g.
tank/prod/vol1)?
> 
> There is no such thing as a pool snapshot. There are only dataset
snapshots.
.. but you can make a single recursive snapshot call that affects all
datasets.
> The trick to a successful snapshot+send strategy at this size is to start
snapping
> early and often. You don''t want to send 200TB, you want to send
2TB, 100 times :-)
> 
> The performance is tends to be bursty, so the fixed record size of the
zvols can
> work to your advantage for capacity planning. Also, a buffer of some sort
can help
> smooth out the utilization, see the threads on ZFS and mbuffer.
>  -- richard
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

/Tomas
-- 
Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Ume?
`- Sysadmin at {cs,acc}.umu.se