zfs discuss - Mar 2012 - Move files between ZFS folder/Datasets ?

Hello.

I can''t seem to find any good info about this case.

I''m running OpenIndiana 151 and have some files on a ZFS folder
(located
under /datapool/stuff).

I''m in the need to create a new ZFS folder (/datapool/temp) and move 
some files from stuff.

What''s the best way to move files between ZFS folders / Datasets ?


Thanks Allot.

Best regards,

Svavar O
Reykjvik - Iceland

On Wed, Mar 14, 2012 at 5:57 AM, Svavar ?rn Eysteinsson <svavar at
fiton.is> wrote:
> I''m running OpenIndiana 151 and have some files on a ZFS folder
(located
> under /datapool/stuff).
>
> I''m in the need to create a new ZFS folder (/datapool/temp) and
move some
> files from stuff.
>
> What''s the best way to move files between ZFS folders / Datasets ?
Is it just a couple files or an entire dataset?

Do the files / folders have any ACLs set?

If it is an entire dataset, then I would use zfs send | zfs recv.

If it is less than an entire dataset and there are NO ACLs set, then I
typically use rsync.

If it is less than an entire dataset and there ARE ACLs set, then your
choices are more limited. I tend to use the cp that ships with the OS.
I have also used NetBackup as it does properly handle the ACLs. Beware
of the GNU cp as it does not set the ACLs as of the last time I
checked.

-- 
{--------1---------2---------3---------4---------5---------6---------7---------}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, Troy Civic Theatre Company
-> Technical Advisor, RPI Players

2012-03-14 13:57, Svavar ?rn Eysteinsson wrote:
> Hello.
>
> I can''t seem to find any good info about this case.
>
> I''m running OpenIndiana 151 and have some files on a ZFS folder
(located
> under /datapool/stuff).
>
> I''m in the need to create a new ZFS folder (/datapool/temp) and
move
> some files from stuff.
>
> What''s the best way to move files between ZFS folders / Datasets ?
Unfortunately, at this time - only commands like "move"
or "rsync -avPHK --remove-source-files SRC/ DST/"
You should use Sun commands like cp, mv, cpio, arguably
Sun tar, to transfer files along with their non-POSIX
ACL lists if you really use those.

There were some hopes that dedup would speed up such
transfers between datasets (not really, unless you have
a good hardware config with LOTS of RAM and an L2ARC),
or a special zfs-move command would emerge to reassign
ownership of on-disk blocks from one dataset to another
quickly and space-sparingly, but such wishes were not
yet fulfilled...

//Jim

2012-03-14 16:38, Paul Kraus wrote:
> On Wed, Mar 14, 2012 at 5:57 AM, Svavar ?rn Eysteinsson<svavar at
fiton.is>  wrote:
>
>> I''m running OpenIndiana 151 and have some files on a ZFS
folder (located
>> under /datapool/stuff).
>>
>> I''m in the need to create a new ZFS folder (/datapool/temp)
and move some
>> files from stuff.
> If it is an entire dataset, then I would use zfs send | zfs recv.
By OP''s post I guess that he needs to move only part of dataset.
However, to move a whole dataset I''d use "zfs rename" ;)
I am not sure it would work for a root dataset of the pool,
but for child datasets I''ve done it a number of times like
in this example:

Say, I have "pool/zonename", and want it to become a sub-DS
like "pool/zones/zonename". I run:
   # zfs create pool/zones
   # zfs rename pool/zonename pool/zones/zonename
This can occasionally stumble due to actively used datasets
or their children (i.e. send/recv going on, clones or zoned
delegated datasets interfere), but usually it works.
For "zoned" datasets in particular, you can shut down the
zone, unset the "zoned" attribute, rename the datasets and
reset the "zoned" attribute, and update /etc/zones/*.xml
manifest (or use zonecfg to remove and redelegate a ZFS DS).

You can be creative in such scenarios ;)
HTH,
//Jim

Thanks guys.

I''m only planning to move some directories. Not the complete dataset.
Just a couble of Gb''s.

Would it be save to use "mv"

There are no ACL''s on the files.
(at least ls -v dosn''t show any acl information)


Using rsync -avPHK --remove-source-files SRC/ DST/
isn''t that just as copying files ? Extra load on the server instead of 
moving the files from
one place to another ?


Thanks allot people.

Best regards,

Svavar O




Jim Klimov wrote:
> 2012-03-14 13:57, Svavar ?rn Eysteinsson wrote:
>> Hello.
>>
>> I can''t seem to find any good info about this case.
>>
>> I''m running OpenIndiana 151 and have some files on a ZFS
folder (located
>> under /datapool/stuff).
>>
>> I''m in the need to create a new ZFS folder (/datapool/temp)
and move
>> some files from stuff.
>>
>> What''s the best way to move files between ZFS folders /
Datasets ?
>
> Unfortunately, at this time - only commands like "move"
> or "rsync -avPHK --remove-source-files SRC/ DST/"
> You should use Sun commands like cp, mv, cpio, arguably
> Sun tar, to transfer files along with their non-POSIX
> ACL lists if you really use those.
>
> There were some hopes that dedup would speed up such
> transfers between datasets (not really, unless you have
> a good hardware config with LOTS of RAM and an L2ARC),
> or a special zfs-move command would emerge to reassign
> ownership of on-disk blocks from one dataset to another
> quickly and space-sparingly, but such wishes were not
> yet fulfilled...
>
> //Jim

On Wed, Mar 14, 2012 at 9:12 AM, Svavar ?rn Eysteinsson <svavar at
fiton.is> wrote:
> Would it be save to use "mv"
mv is safe, I just prefer to use cp and rm when done if I have the space.
> There are no ACL''s on the files.
> (at least ls -v dosn''t show any acl information)
>
>
> Using rsync -avPHK --remove-source-files SRC/ DST/
> isn''t that just as copying files ? Extra load on the server
instead of
> moving the files from
> one place to another ?
    If the data is not being accessed you can just cp or mv, I usually
don''t have that luxury and need to start the move while people are
still using the files. I run a first rsync to copy all of them, then I
declare a very short outage window and do a final rsync to catch
anything that got changed. I do NOT use the --remove-source-files
option.

-- 
{--------1---------2---------3---------4---------5---------6---------7---------}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, Troy Civic Theatre Company
-> Technical Advisor, RPI Players

> Thanks guys.
> 
> I''m only planning to move some directories. Not the complete
dataset.
> Just a couble of Gb''s.
> 
> Would it be save to use "mv"
Of course, provided your system doesn''t crash during the move.
> Using rsync -avPHK --remove-source-files SRC/ DST/
> isn''t that just as copying files ? Extra load on the server
instead of
> moving the files from one place to another ?
You may not realize it but a mv consists of a copy and delete. rsync is nice
because you can check that it really got all your files by doing another
rsync with -c after the first one completes. It compares checksums of files
to make sure they actually are identical. With ZFS this is less of an issue
but it is a nice doublecheck if you really can''t afford to lose your
files.

Make sure you understand rsync syntax before using it live. Make some
directories in /tmp and copy stuff around. One thing about rsync is that a
trailing / means something and not having it means something else.

rsync with the -n option does a trial run

rsync -axvn /tmp/path/to/data /tmp/target # moves a directory named data to
the target dir

rsync -axvn /tmp/path/to/data/ /tmp/target # moves everything in the data
directory (but not the data directory itself!) to the target

rsync -axv # do it live and keep date/time/owner
rsync -axvc # do it again and compare checksums instead of date/time/size