zfs discuss - Apr 2006 - flag day: new names for ''zfs backup'' and ''zfs restore''

If you do not use ZFS, you can ignore this message.

The putback for:

	PSARC 2006/185 zfs send/receive
	6399918 zfs backup/restore should be send/receive

Changes the names of the ''zfs backup'' and ''zfs
restore'' subcommands to
''zfs send'' and ''zfs receive'' (''zfs
recv'' is permitted as an alias for
''zfs receive'').  If you have any scripts which use these
subcommands,
you will have to modify them to use the new names.  The zfs(1m) manpage
will be updated to reflect the new names (CR 6400540, targeted for build 38).

Read on for details of why we made this change:

These subcommands produce and consume byte streams which can be used to
store a copy of a snapshot, or send data between machines.  However,
they do not provide a comprehensive backup solution.  At a minimum, a
real solution would provide the ability to backup a filesystem and all
its descendent filesystems and snapshots, preserving any properties that
have been set.  A real solution would also probably be able to deal with
tape drives and breaking the stream into segments to put onto multiple
tapes / dvds / whatevers.

Given that these subcommands do not actually provide full backup and
restore capabilities, we have renamed them so as to avoid confusion.
The new names imply the intended usage -- to transmit filesystems
between machines, not only for backups but also for real-time remote
replication (in the latter case, we send snapshot deltas).

--matt

Hello Matthew,

Wednesday, April 5, 2006, 6:17:04 AM, you wrote:
MA> The new names imply the intended usage -- to transmit filesystems
MA> between machines, not only for backups but also for real-time remote
MA> replication (in the latter case, we send snapshot deltas).

I hear this not for the first time - real-time remote replication.
If you think about putting in a cron a script which would make a
snapshot and then send a strem with incramental data it''s hardly
real-time replication. Or am I missing something?


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Matthew Ahrens <ahrens at eng.sun.com> writes:
> These subcommands produce and consume byte streams which can be used to
> store a copy of a snapshot, or send data between machines.  However,
> they do not provide a comprehensive backup solution.  At a minimum, a
> real solution would provide the ability to backup a filesystem and all
> its descendent filesystems and snapshots, preserving any properties that
> have been set.  A real solution would also probably be able to deal with
Are there plans for doing something like this?  The reason I ask is that I
recently had a need for just this functionality while working on O/N builds
on my laptop: consider the following hierarchy of zfs
filesystems/snapshots/clones, providing me with a poor man''s SCM :-)

internal/export/os
internal/export/os/obj
internal/export/os/obj/on
internal/export/os/obj/on/20060327		clone of src/on/20060327 at dist
internal/export/os/obj/on/20060327-local	clone of src/on/20060327-local/all at
20060402
internal/export/os/src
internal/export/os/src/on
internal/export/os/src/on/20060327
internal/export/os/src/on/20060327 at dist
internal/export/os/src/on/20060327-local
internal/export/os/src/on/20060327-local/all	clone of src/on/20060327 at dist
						with modifications from
						filesystems below merged in
internal/export/os/src/on/20060327-local/all at 20060402
internal/export/os/src/on/20060327-local/onbld
internal/export/os/src/on/20060327-local/onbld/SUNWonbld clone with
							 specific changeset
internal/export/os/src/on/20060327-local/onbld/optbase	 likewise

If I use zfs send/receive to transfer this to a sparc system (or back it up
using traditional tools), I don''t preserve the snapshot and clone
relationships, multiplying the disk space consumption on the way.

While (with the appropriate snapshots thrown in) it should be possible to
script such a transfer, it would seem natural to me to support this out of
the box.

	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Wed, Apr 05, 2006 at 09:44:15AM +0200, Robert Milkowski
wrote:
> Hello Matthew,
> 
> Wednesday, April 5, 2006, 6:17:04 AM, you wrote:
> MA> The new names imply the intended usage -- to transmit filesystems
> MA> between machines, not only for backups but also for real-time remote
> MA> replication (in the latter case, we send snapshot deltas).
> 
> I hear this not for the first time - real-time remote replication.
> If you think about putting in a cron a script which would make a
> snapshot and then send a strem with incramental data it''s hardly
> real-time replication. Or am I missing something?
Yeah, sorry that should have read "best-effort remote replication".
To be called real-time it should at least have the option to ensure that
the other side has the same as what''s on disk locally before
proceeding.
(There''s another bug for that, 5036182.)

--matt

On Wed, Apr 05, 2006 at 12:43:14PM +0200, Rainer Orth
wrote:
> Matthew Ahrens <ahrens at eng.sun.com> writes:
> 
> > These subcommands produce and consume byte streams which can be used
to
> > store a copy of a snapshot, or send data between machines.  However,
> > they do not provide a comprehensive backup solution.  At a minimum, a
> > real solution would provide the ability to backup a filesystem and all
> > its descendent filesystems and snapshots, preserving any properties
that
> > have been set.  A real solution would also probably be able to deal
with
> 
> Are there plans for doing something like this?
...
> While (with the appropriate snapshots thrown in) it should be possible to
> script such a transfer, it would seem natural to me to support this out of
> the box.
This is something that we have some ideas for, and plan to do for a
subsequent (post-update 2) release.  However, this would be a great
project for someone from the community to work on! :-)  Even if it was
just a fancy shell script, this functionality would be really useful.

--matt

On Wed, Matthew Ahrens wrote:
> On Wed, Apr 05, 2006 at 12:43:14PM +0200, Rainer Orth wrote:
> > Matthew Ahrens <ahrens at eng.sun.com> writes:
> > 
> > > These subcommands produce and consume byte streams which can be
used to
> > > store a copy of a snapshot, or send data between machines. 
However,
> > > they do not provide a comprehensive backup solution.  At a
minimum, a
> > > real solution would provide the ability to backup a filesystem
and all
> > > its descendent filesystems and snapshots, preserving any
properties that
> > > have been set.  A real solution would also probably be able to
deal with
> > 
> > Are there plans for doing something like this?
> ...
> > While (with the appropriate snapshots thrown in) it should be possible
to
> > script such a transfer, it would seem natural to me to support this
out of
> > the box.
> 
> This is something that we have some ideas for, and plan to do for a
> subsequent (post-update 2) release.  However, this would be a great
> project for someone from the community to work on! :-)  Even if it was
> just a fancy shell script, this functionality would be really useful.
And if the replica or migration of the filesystem were tied into
the NFSv4 server appropriately, it could notify clients of the 
repilcas or the new location.

Spencer

Hello Matthew,

Wednesday, April 5, 2006, 7:13:36 PM, you wrote:

MA> On Wed, Apr 05, 2006 at 09:44:15AM +0200, Robert Milkowski
wrote:
>> Hello Matthew,
>> 
>> Wednesday, April 5, 2006, 6:17:04 AM, you wrote:
>> MA> The new names imply the intended usage -- to transmit
filesystems
>> MA> between machines, not only for backups but also for real-time
remote
>> MA> replication (in the latter case, we send snapshot deltas).
>> 
>> I hear this not for the first time - real-time remote replication.
>> If you think about putting in a cron a script which would make a
>> snapshot and then send a strem with incramental data it''s
hardly
>> real-time replication. Or am I missing something?
MA> Yeah, sorry that should have read "best-effort remote
replication".
MA> To be called real-time it should at least have the option to ensure that
MA> the other side has the same as what''s on disk locally before
proceeding.
MA> (There''s another bug for that, 5036182.)

Thats for the begining. Then it would be useful to have an synchronous
replication so IO is considered commited if on both ends it ended-up
on a physical storage. Or asynchronous with some kind of a delay.
Or based on snapshot or current state of filesystem - you just want to
run a command and replicate filesystem (fully starting from the
beginning or just differences - I gues it''s what we''ve got
now).

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com