zfs discuss - Nov 2005 - remote mirroring of changes over ZFS

Hello,

I am starting to explore ZFS and am noting one feature in particular - the
remote replication feature.

I will start by paraphrasing what _I_ think this is:

ZFS is allowing me to attach one (or more) remote (attached over tcp/ip network)
filesystems to the first (master?) filesystem, and all changes to the first one
are automatically replicated to the remote ones.  Sort of like a big long
continuous rsync.

First question - is the above interpretation correct ?

Second question, what happens if you have two filesystems linked in this manner,
and then the network link gets severed for some amount of time ?  Does the
master filesystem save up the changes and then sync them up once the link is
re-established ?  Is there a timeout after which that recovery cannot occur ?

If it can recover after a lost link, presumably a much higher than average
amount of data will flow from master->slave as the changes are caught up - is
there a cap you can put on this catch-up data rate so that it will not attempt
to monopolize the entire link until it is caught up ?  (that is, purposely
increase the amount of time that catch-up takes in order to avoid saturating the
link)

tganks.
This message posted from opensolaris.org

Currently, we support a "poor man''s" remote replication
through the use
of snapshots and backup/restore.  Basically, you can take a snapshot at
regular intervals, do a very quick backup over ssh, and restore it on
another machine.

This works, but is obviously not the most integrated/ideal solution.
We''ll be looking at ways of making this easier and more robust in the
future.

- Eric

On Mon, Nov 21, 2005 at 12:46:26PM -0800, User Name
wrote:
> Hello,
> 
> I am starting to explore ZFS and am noting one feature in particular -
> the remote replication feature.
> 
> I will start by paraphrasing what _I_ think this is:
> 
> ZFS is allowing me to attach one (or more) remote (attached over
> tcp/ip network) filesystems to the first (master?) filesystem, and all
> changes to the first one are automatically replicated to the remote
> ones.  Sort of like a big long continuous rsync.
> 
> First question - is the above interpretation correct ?
> 
> Second question, what happens if you have two filesystems linked in
> this manner, and then the network link gets severed for some amount of
> time ?  Does the master filesystem save up the changes and then sync
> them up once the link is re-established ?  Is there a timeout after
> which that recovery cannot occur ?
> 
> If it can recover after a lost link, presumably a much higher than
> average amount of data will flow from master->slave as the changes are
> caught up - is there a cap you can put on this catch-up data rate so
> that it will not attempt to monopolize the entire link until it is
> caught up ?  (that is, purposely increase the amount of time that
> catch-up takes in order to avoid saturating the link)
--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

On Mon, 21 Nov 2005, Eric Schrock wrote:
> Currently, we support a "poor man''s" remote replication
through the use
> of snapshots and backup/restore.  Basically, you can take a snapshot at
> regular intervals, do a very quick backup over ssh, and restore it on
> another machine.
> 
> This works, but is obviously not the most integrated/ideal solution.
> We''ll be looking at ways of making this easier and more robust in
the
> future.

I see - so these backup/restore of a (mounted ?) filesystem snapshot are
discreet events that are initiated by humans (either at that time, or
automated by cron, for instance) and seperate from the moment to moment
operation of the ZFS filesystem ?

Is there any form of continuous replication by actually linking a
filesystem, as I described, in the roadmap, or am I getting too ambitious?

Thanks.

On Mon, Nov 21, 2005 at 06:09:09PM -0500, user wrote:
> 
> I see - so these backup/restore of a (mounted ?) filesystem snapshot are
> discreet events that are initiated by humans (either at that time, or
> automated by cron, for instance) and seperate from the moment to moment
> operation of the ZFS filesystem ?
Yep.  Matt has a script that does this constantly; he will probably
polish it off and post it once he gets back from vacation.
> Is there any form of continuous replication by actually linking a
> filesystem, as I described, in the roadmap, or am I getting too ambitious?
Yes, this is on the roadmap.  Exactly when is still uncertain - there''s
a lot of other stuff we need to cover first.

- Eric

--
Eric Schrock, Solaris Kernel Development       http://blogs.sun.com/eschrock

>> Currently, we support a "poor man''s" remote
replication through the use
>> of snapshots and backup/restore.  Basically, you can take a snapshot at
>> regular intervals, do a very quick backup over ssh, and restore it on
>> another machine.
>> 
>> This works, but is obviously not the most integrated/ideal solution.
>> We''ll be looking at ways of making this easier and more robust
in the
>> future.
>
>
>I see - so these backup/restore of a (mounted ?) filesystem snapshot are
>discreet events that are initiated by humans (either at that time, or
>automated by cron, for instance) and seperate from the moment to moment
>operation of the ZFS filesystem ?
>
>Is there any form of continuous replication by actually linking a
>filesystem, as I described, in the roadmap, or am I getting too ambitious?
>
>Thanks.

Disclaimer: I have not tested the below and am not in a position
to do so for a few days at least.  No implied or express warranty,
your batteries may be going dead, and this concept may cause the
premature end of the world.  That disclaimed...


I know the man page discourages it, but zpool can use a regular
file as a vdev... what if we use a regular file as a vdev in a
2+ way zpool mirror, and use a regular file which is on a
remote NFS server we have mounted locally?

That would seem to be legal based on the docs and would get you
a realtime remote replicated mirror.

The questions are,
	a) does it really even work, and
	b) how well or badly does it really perform...

...and I can''t answer those at   the moment.



-george william herbert
Technical Architect, Taos Inc.
gherbert-zfs at retro.com / gherbert at taos.com

more speculation below...

George William Herbert wrote:
> 
>>>Currently, we support a "poor man''s" remote
replication through the use
>>>of snapshots and backup/restore.  Basically, you can take a snapshot
at
>>>regular intervals, do a very quick backup over ssh, and restore it
on
>>>another machine.
>>>
>>>This works, but is obviously not the most integrated/ideal solution.
>>>We''ll be looking at ways of making this easier and more
robust in the
>>>future.
>>
>>
>>I see - so these backup/restore of a (mounted ?) filesystem snapshot are
>>discreet events that are initiated by humans (either at that time, or
>>automated by cron, for instance) and seperate from the moment to moment
>>operation of the ZFS filesystem ?
>>
>>Is there any form of continuous replication by actually linking a
>>filesystem, as I described, in the roadmap, or am I getting too
ambitious?
>>
>>Thanks.
> 
> 
> 
> Disclaimer: I have not tested the below and am not in a position
> to do so for a few days at least.  No implied or express warranty,
> your batteries may be going dead, and this concept may cause the
> premature end of the world.  That disclaimed...
> 
> 
> I know the man page discourages it, but zpool can use a regular
> file as a vdev... what if we use a regular file as a vdev in a
> 2+ way zpool mirror, and use a regular file which is on a
> remote NFS server we have mounted locally?
Or, iSCSI, samba, flash drives, iPods, etc...
I suppose the answer will always be some mix of YMMV and "we''ve
never tested that"
  -- richard
> That would seem to be legal based on the docs and would get you
> a realtime remote replicated mirror.
> 
> The questions are,
> 	a) does it really even work, and
> 	b) how well or badly does it really perform...
> 
> ...and I can''t answer those at   the moment.
> 
> 
> 
> -george william herbert
> Technical Architect, Taos Inc.
> gherbert-zfs at retro.com / gherbert at taos.com
> 
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/zfs-discuss