zfs discuss - Jul 2009 - first use send/receive... somewhat confused.

Reading various bits of google output about send/receive I''m starting
to wonder if that process is maybe the wrong way to go at what I want
to do.

I have a filesystem z1/projects  I want to remove it from the z1 pool
and put it in a z2 pool ending up with z2/projects.  With all the same
data, same mountpoint (/) and same name (/projects).

The examples I see of send/receive end with a new data set but it has
the name format of a snapshot.

man zfs shows:

     zfs receive [-vnF] filesystem|volume|snapshot
     zfs receive [-vnF] -d filesystem
But it claims the second usage is for creating a backup.

The first usage appears it would result in a new dataset with an
unfortunate name as one finds in snapshosts with `@'' character.

The example from Opensolaris Bible shows.

  zfs send tank/fish at trip | zfs receive newpool/crab at trap

I don''t want a name with a @ in it... so just a little confused as to
which of the two or neither is appropriate for my usage.

I want to destroy the z1/projects fs when done moving to z2 pool.  So
do I simply zfs send z1/projects at most_recent|
                 zfs receive z2/projects at starter 

Then zfs rename  z2/projects at starter z2/projects
     zfs destroy z1/projects

I''m worried I''d end up with a read-only /projects or some
other
unfortunate thing.

Or would it be best to go the lonq way.

zfs create z2/projects
zfs set mountpoint /new/projects

rsync/ or cp -a the contents of /projects to /new/projects

zfs destroy z1/projects

zfs set mountpoint=/projects z2/projects

Harry Putnam wrote:
> Reading various bits of google output about send/receive I''m
starting
> to wonder if that process is maybe the wrong way to go at what I want
> to do.
>
> I have a filesystem z1/projects  I want to remove it from the z1 pool
> and put it in a z2 pool ending up with z2/projects.  With all the same
> data, same mountpoint (/) and same name (/projects).
>
> The examples I see of send/receive end with a new data set but it has
> the name format of a snapshot.
>
> man zfs shows:
>
>      zfs receive [-vnF] filesystem|volume|snapshot
>      zfs receive [-vnF] -d filesystem
> But it claims the second usage is for creating a backup.
>
> The first usage appears it would result in a new dataset with an
> unfortunate name as one finds in snapshosts with `@'' character.
>   
You can only send/receive snapshots.  However, on the receiving end,
there will also be a dataset of the name you choose.  Since you didn''t
share what commands you used, it is pretty impossible for us to
speculate what you might have tried.
> The example from Opensolaris Bible shows.
>
>   zfs send tank/fish at trip | zfs receive newpool/crab at trap
>
> I don''t want a name with a @ in it... so just a little confused as
to
> which of the two or neither is appropriate for my usage.
>
> I want to destroy the z1/projects fs when done moving to z2 pool.  So
> do I simply zfs send z1/projects at most_recent|
>                  zfs receive z2/projects at starter 
>
> Then zfs rename  z2/projects at starter z2/projects
>      zfs destroy z1/projects
>
> I''m worried I''d end up with a read-only /projects or some
other
> unfortunate thing.
>
> Or would it be best to go the lonq way.
>
> zfs create z2/projects
> zfs set mountpoint /new/projects
>
> rsync/ or cp -a the contents of /projects to /new/projects
>
> zfs destroy z1/projects
>
> zfs set mountpoint=/projects z2/projects
>   
The ZFS Administration Guide has additional examples.
 -- richard

On Jul 12, 2009, at 11:45 PM, Harry Putnam <reader at newsguy.com> wrote:
> Reading various bits of google output about send/receive I''m
starting
> to wonder if that process is maybe the wrong way to go at what I want
> to do.
>
> I have a filesystem z1/projects  I want to remove it from the z1 pool
> and put it in a z2 pool ending up with z2/projects.  With all the same
> data, same mountpoint (/) and same name (/projects).
>
> The examples I see of send/receive end with a new data set but it has
> the name format of a snapshot.
>
> man zfs shows:
>
>     zfs receive [-vnF] filesystem|volume|snapshot
>     zfs receive [-vnF] -d filesystem
> But it claims the second usage is for creating a backup.
>
> The first usage appears it would result in a new dataset with an
> unfortunate name as one finds in snapshosts with `@'' character.
>
> The example from Opensolaris Bible shows.
>
>  zfs send tank/fish at trip | zfs receive newpool/crab at trap
>
> I don''t want a name with a @ in it... so just a little confused as
to
> which of the two or neither is appropriate for my usage.
>
> I want to destroy the z1/projects fs when done moving to z2 pool.  So
> do I simply zfs send z1/projects at most_recent|
>                 zfs receive z2/projects at starter
>
> Then zfs rename  z2/projects at starter z2/projects
>     zfs destroy z1/projects
>
> I''m worried I''d end up with a read-only /projects or some
other
> unfortunate thing.
>
> Or would it be best to go the lonq way.
>
> zfs create z2/projects
> zfs set mountpoint /new/projects
>
> rsync/ or cp -a the contents of /projects to /new/projects
>
> zfs destroy z1/projects
>
> zfs set mountpoint=/projects z2/projects
It''s not like that at all.

Zfs send/recv only works on static datasets, so you need to take a  
snapshot and send that.

Once the data is copied you can delete the snapshots that will then  
exist on both pools.

If you have mount options set use the -u option on the recv to have it  
defer attempting mounting the conflicting datasets.

If you just want to rename a pool, then export it and the import it  
under a new name.

-Ross

You''re getting confused between snapshots and filesystems.  Read @ as
"at", it''s a snapshot of that filesystem at a particular
point in time.  eg, you could create snapshots with names like:

z1/projects at now
z1/projects at 13-July-2009
z1/projects at my-snapshot

They''re all snapshots of the z1/projects filesystem, and you can use
almost any name you like.

To send your filesystem to another machine, here''s a quick overview:

System1:  contains z1/projects
System2:  has an empty pool z2

1. Take a snapshot:
zfs snapshot z1/projects at now

2. Send the snapshot to z2
zfs send z1/projects at now | ssh user at remote.system zfs receive z2/projects
at now

3. End result:
Two identical filesystems on each system called:
z1/projects
z2/projects

Each of which has a snapshot of the data from the point you started:
z1/projects at now
z2/projects at now
-- 
This message posted from opensolaris.org

Richard Elling <richard.elling at gmail.com> writes:
> You can only send/receive snapshots.  However, on the receiving end,
> there will also be a dataset of the name you choose.  Since you
didn''t
> share what commands you used, it is pretty impossible for us to
> speculate what you might have tried.
I thought I made it clear I had not used any commands but gave two
detailed examples of different ways to attempt the move.

I see now the main thing that confused me is that sending a z1/projects at
something
to a new z2/projects at something would also result in z2/projects being
created.

That part was not at all clear to me from the man page.

Ross Walker <rswwalker at gmail.com> writes:
> Once the data is copied you can delete the snapshots that will then
> exist on both pools.
That''s the part right there that wasn''t apparent.

That
  zfs send z1/something at snap  |zfs receive z2/something at snap

Would also create z2/something
> If you have mount options set use the -u option on the recv to have it
> defer attempting mounting the conflicting datasets.
That little item right there... has probably saved me some serious
headaches since in my scheme... the resulting newpool/fs would have
the same mount point. ... thanks

> Richard Elling <richard.elling at gmail.com> writes:
>
>> You can only send/receive snapshots.  However, on the receiving end,
>> there will also be a dataset of the name you choose.  Since you
didn''t
>> share what commands you used, it is pretty impossible for us to
>> speculate what you might have tried.
>
> I thought I made it clear I had not used any commands but gave two
> detailed examples of different ways to attempt the move.
>
> I see now the main thing that confused me is that sending a
> z1/projects at something
> to a new z2/projects at something would also result in z2/projects being
> created.
>
> That part was not at all clear to me from the man page.
This will probably get me bombed with napalm but I often just
use star from J?rg Schilling because its dead easy :

  star -copy -p -acl -sparse -dump -C old_dir . new_dir

and you''re done.[1]

So long as you have both the new and the old zfs/ufs/whatever[2]
filesystems mounted. It doesn''t matter if they are static or not. If
anything changes on the filesystem then star will tell you about it.

-- 
Dennis

[1] -p means preserve meta-properties of the files/dirs etc.
    -acl means what it says. Grabs ACL data also.
    -sparse means what it says. Handles files with holes in them.
    -dump means be super careful about everything ( read the manpage )

[2] star doesn''t care if its zfs or ufs or a CDROM or a floppy.

Dennis Clarke <dclarke at blastwave.org> writes:
> This will probably get me bombed with napalm but I often just
> use star from J?rg Schilling because its dead easy :
>
>   star -copy -p -acl -sparse -dump -C old_dir . new_dir
>
> and you''re done.[1]
>
> So long as you have both the new and the old zfs/ufs/whatever[2]
> filesystems mounted. It doesn''t matter if they are static or not.
If
> anything changes on the filesystem then star will tell you about it.
I''m not sure I see how that is easier.

The command itself may be but it requires other moves not shown in
your command.

1) zfs create z2/projects

2)  star -copy -p -acl -sparse -dump -C old_dir . new_dir

As a bare minimum would be required.

whereas 
zfs send z1/projects at snap |zfs receive z2/projects at snap

Is all that is necessary using zfs send receive, and the new
filesystem z2/projects is created and populated with data from
z1/projects, not to mention a snapshot at z2/projects/.zfs/snapshot

> Dennis Clarke <dclarke at blastwave.org> writes:
>
>> This will probably get me bombed with napalm but I often just
>> use star from J??rg Schilling because its dead easy :
>>
>>   star -copy -p -acl -sparse -dump -C old_dir . new_dir
>>
>> and you''re done.[1]
>>
>> So long as you have both the new and the old zfs/ufs/whatever[2]
>> filesystems mounted. It doesn''t matter if they are static or
not. If
>> anything changes on the filesystem then star will tell you about it.
>
> I''m not sure I see how that is easier.
>
> The command itself may be but it requires other moves not shown in
> your command.
>
> 1) zfs create z2/projects
>
> 2)  star -copy -p -acl -sparse -dump -C old_dir . new_dir
>
> As a bare minimum would be required.
>
> whereas
> zfs send z1/projects at snap |zfs receive z2/projects at snap
>
> Is all that is necessary using zfs send receive, and the new
> filesystem z2/projects is created and populated with data from
> z1/projects, not to mention a snapshot at z2/projects/.zfs/snapshot
sort of depends on what you want to get done and both work.

dc

Harry Putnam <reader at newsguy.com> wrote:
> Dennis Clarke <dclarke at blastwave.org> writes:
>
> > This will probably get me bombed with napalm but I often just
> > use star from J?rg Schilling because its dead easy :
> >
> >   star -copy -p -acl -sparse -dump -C old_dir . new_dir
> >
> > and you''re done.[1]
> >
> > So long as you have both the new and the old zfs/ufs/whatever[2]
> > filesystems mounted. It doesn''t matter if they are static or
not. If
> > anything changes on the filesystem then star will tell you about it.
>
> I''m not sure I see how that is easier.
>
> The command itself may be but it requires other moves not shown in
> your command.
Could you please explain your claims?

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Joerg Schilling wrote:
> Harry Putnam <reader at newsguy.com> wrote:
> 
>> Dennis Clarke <dclarke at blastwave.org> writes:
>>
>>> This will probably get me bombed with napalm but I often just
>>> use star from J?rg Schilling because its dead easy :
>>>
>>>   star -copy -p -acl -sparse -dump -C old_dir . new_dir
>>>
>>> and you''re done.[1]
>>>
>>> So long as you have both the new and the old zfs/ufs/whatever[2]
>>> filesystems mounted. It doesn''t matter if they are static
or not. If
>>> anything changes on the filesystem then star will tell you about
it.
>> I''m not sure I see how that is easier.
>>
>> The command itself may be but it requires other moves not shown in
>> your command.
> 
> Could you please explain your claims?
star doesn''t (and shouldn''t) create the destination ZFS
filesystem like
the zfs recv would.  It also doesn''t preserve the dataset level would
do.

One the other hand using star (or rsync which is what I tend to do) 
gives more flexibility in that the source and destination filesystem 
types can be different or even not a filesystem!

zfs send|recv and [g,s]tar exist for different purposes, but there are 
some overlapping use cases either either could do the job.

-- 
Darren J Moffat

Darren J Moffat <darrenm at opensolaris.org> wrote:
> >>> use star from J?rg Schilling because its dead easy :
> >>>
> >>>   star -copy -p -acl -sparse -dump -C old_dir . new_dir
...
> star doesn''t (and shouldn''t) create the destination ZFS
filesystem like
> the zfs recv would.  It also doesn''t preserve the dataset level
would do.
As star is software that cleany lives above the filesystem layer, this is
what people would expect ;-)
> One the other hand using star (or rsync which is what I tend to do) 
> gives more flexibility in that the source and destination filesystem 
> types can be different or even not a filesystem!
star is highly optimized and it''s build in find(1) (using libfind)
gives you many interesting features.

zfs send seems to be tied to the zfs version and this is another reason
why zfs send | receive may not even work on a 100% zfs based playground.
> zfs send|recv and [g,s]tar exist for different purposes, but there are 
> some overlapping use cases either either could do the job.
It would be nice if there was a discussion that does mention features instead
of always proposing zfs send....

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de                (uni)  
       joerg.schilling at fokus.fraunhofer.de (work) Blog:
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily

Joerg Schilling wrote:
> Darren J Moffat <darrenm at opensolaris.org> wrote:
> 
>>>>> use star from J?rg Schilling because its dead easy :
>>>>>
>>>>>   star -copy -p -acl -sparse -dump -C old_dir . new_dir
> ...
> 
>> star doesn''t (and shouldn''t) create the destination
ZFS filesystem like
>> the zfs recv would.  It also doesn''t preserve the dataset
level would do.
> 
> As star is software that cleany lives above the filesystem layer, this is
> what people would expect ;-)
Indeed but that is why the "extra steps" are needed to create the 
destination ZFS filesystem.  Not a bad thing or a criticism of star just 
  a fact (and in answer to the question you asked).
>> One the other hand using star (or rsync which is what I tend to do) 
>> gives more flexibility in that the source and destination filesystem 
>> types can be different or even not a filesystem!
> 
> star is highly optimized and it''s build in find(1) (using libfind)
> gives you many interesting features.
I''m sure the authors of rsync could make a similar statement :-)
> zfs send seems to be tied to the zfs version and this is another reason
> why zfs send | receive may not even work on a 100% zfs based playground.
Indeed but it isn''t, like tar (and variants there of), an archiver but
a
means of providing replication of ZFS datasets based on ZFS snapshots 
and works at the ZFS DMU layer.
>> zfs send|recv and [g,s]tar exist for different purposes, but there are 
>> some overlapping use cases either either could do the job.
> 
> It would be nice if there was a discussion that does mention features
instead
> of always proposing zfs send....
In general I completely agree, however this particular thread (given its 
title) is about zfs send|recv :-)

-- 
Darren J Moffat

Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) writes:
> Harry Putnam <reader at newsguy.com> wrote:
>
>> Dennis Clarke <dclarke at blastwave.org> writes:
>>
>> > This will probably get me bombed with napalm but I often just
>> > use star from J?rg Schilling because its dead easy :
>> >
>> >   star -copy -p -acl -sparse -dump -C old_dir . new_dir
>> >
>> > and you''re done.[1]
>> >
>> > So long as you have both the new and the old zfs/ufs/whatever[2]
>> > filesystems mounted. It doesn''t matter if they are static
or not. If
>> > anything changes on the filesystem then star will tell you about
it.
>>
>> I''m not sure I see how that is easier.
>>
>> The command itself may be but it requires other moves not shown in
>> your command.
>
> Could you please explain your claims?
Well it may be a case of newbie shooting off mouth on basis of small
knowledge but the setup you showed with star does not create a zfs
filesystem.  I guess that would have to be done externally.  

Whereas send/receive does that part for you.

My first thought was rsync... as a long time linux user... thats where
I would usually turn... until other posters pointed out how
send/receive works.  i.e.  It creates a new zfs filesystem for you,
which was exactly what I was after.

So by using send/receive with -u I was able in one move to
1) create a zfs filesystem
2) mount it automatically
3) transfer the data to the new fs

`star'' only does the last one.... right?

I then had a few external chores like setting options or changing
mountpoint.   Something that would have had to be done using `star''
too.  (That is, before using star)

That alone was the basis of what you call my `claims''.

Would `star'' move or create the .zfs directory?