zfs discuss - Sep 2009 - Desire simple but complete copy - How?

It appears that I have waded into a quagmire.  Every option I can find (cpio,
tar (Many versions!), cp, star, pax) has issues.  File size and filename or path
length, and ACLs are common shortfalls.  "Surely there is an easy
answer" he says naively!

I simply want to copy one zfs filesystem tree to another, replicating it
exactly.  Times, Permissions, hard links, symbolic links, sparse file holes,
ACLs, extended attributes, and anything I don''t know about.

Can you give me a commandline with parameters?  I will then study what they
mean.

If there is no correct answer, please give me your best recommendation and what
you know about it.

Thanks.  (I hope!)

--Ray
-- 
This message posted from opensolaris.org

> It appears that I have waded into a quagmire.  Every option I can find
> (cpio, tar (Many versions!), cp, star, pax) has issues.  File size and
> filename or path length, and ACLs are common shortfalls.  "Surely
there is
> an easy answer" he says naively!
>
> I simply want to copy one zfs filesystem tree to another, replicating it
> exactly.  Times, Permissions, hard links, symbolic links, sparse file
> holes, ACLs, extended attributes, and anything I don''t know about.
>
> Can you give me a commandline with parameters?  I will then study what
> they mean.
>
Have you ruled out using ''zfs send'' / ''zfs
receive'' for some reason? And
have you looked at rsync? I generally find rsync to be the easiest and
most reliable tool for replicating directory structures. You may want to
look at the GNU version of rsync (available at www.sunfreeware.com and
elsewhere).

Paul

On Wed, September 30, 2009 08:21, paul at paularcher.org
wrote:
>> It appears that I have waded into a quagmire.  Every option I can find
>> (cpio, tar (Many versions!), cp, star, pax) has issues.  File size and
>> filename or path length, and ACLs are common shortfalls.  "Surely
there
>> is
>> an easy answer" he says naively!
>>
>> I simply want to copy one zfs filesystem tree to another, replicating
it
>> exactly.  Times, Permissions, hard links, symbolic links, sparse file
>> holes, ACLs, extended attributes, and anything I don''t know
about.
>>
>> Can you give me a commandline with parameters?  I will then study what
>> they mean.
>>
>
> Have you ruled out using ''zfs send'' / ''zfs
receive'' for some reason? And
> have you looked at rsync? I generally find rsync to be the easiest and
> most reliable tool for replicating directory structures. You may want to
> look at the GNU version of rsync (available at www.sunfreeware.com and
> elsewhere).
I had to discard an rsync-based backup scheme when I went to CIFS; rsync
doesn''t handle extended attributes and ACLs, which CIFS uses.

And I haven''t been able to make incremental replication send/receive
work.
 Supposed to be working on that, but now I''m having trouble getting a
VirtualBox install that works (my real NAS is physical, but I''m using
virtual systems to test things).
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

David Dyer-Bennet wrote:
> And I haven''t been able to make incremental replication
send/receive work.
>  Supposed to be working on that, but now I''m having trouble
getting a
> VirtualBox install that works (my real NAS is physical, but I''m
using
> virtual systems to test things).
I''ve had good success practicing and debugging zfs stuff
by creating small pools based on files and tinkering with
those, e.g.

# mkfile 100m /root/zpool_test1
# zpool create test1 /root/zpool_test1
# mkfile 100m /root/zpool_test2
# zpool create test2 /root/zpool_test2

This can get you a source of non-production data on a
real server.

Rob T

On Wed, September 30, 2009 10:07, Robert Thurlow wrote:
> David Dyer-Bennet wrote:
>
>> And I haven''t been able to make incremental replication
send/receive
>> work.
>>  Supposed to be working on that, but now I''m having trouble
getting a
>> VirtualBox install that works (my real NAS is physical, but
I''m using
>> virtual systems to test things).
>
> I''ve had good success practicing and debugging zfs stuff
> by creating small pools based on files and tinkering with
> those, e.g.
>
> # mkfile 100m /root/zpool_test1
> # zpool create test1 /root/zpool_test1
> # mkfile 100m /root/zpool_test2
> # zpool create test2 /root/zpool_test2
>
> This can get you a source of non-production data on a
> real server.
That''s where I started, and it''s useful.  I left out a bit
above -- I had
proven to my satisfaction that incremental replication streams don''t
work
in the software version on my server.  I''m trying to test current
versions, before I commit to upgrading the pools and/or filesystems with
the live data in them.
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Wed, Sep 30, 2009 at 10:54 PM, David Dyer-Bennet <dd-b at dd-b.net>
wrote:
> On Wed, September 30, 2009 10:07, Robert Thurlow wrote:
>> David Dyer-Bennet wrote:
>>
>>> And I haven''t been able to make incremental replication
send/receive
>>> work.
>>> ?Supposed to be working on that, but now I''m having
trouble getting a
>>> VirtualBox install that works
>> I''ve had good success practicing and debugging zfs stuff
>> by creating small pools based on files and tinkering with
>> those, e.g.
> That''s where I started, and it''s useful. ?I left out a
bit above -- I had
> proven to my satisfaction that incremental replication streams
don''t work
> in the software version on my server. ?I''m trying to test current
> versions, before I commit to upgrading the pools and/or filesystems with
> the live data in them.
Are you using x86 or sparc? solaris or opensolaris?
If opensolaris on x86, you can use xvm (xen) to achieve the same
functionality as virtualbox.
If sparc T series, you can use LDOM.

-- 
Fajar

Fajar A. Nugraha wrote:
>
> Are you using x86 or sparc? solaris or opensolaris?
> If opensolaris on x86, you can use xvm (xen) to achieve the same
> functionality as virtualbox.
> If sparc T series, you can use LDOM.
>   
x86, OpenSolaris.  But I''m not terribly attracted to the idea of 
switching to another, less familiar, virtualization product in hopes 
that it will work.  I really rather expected Sun''s virtualization 
product to run Sun''s OS.

-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

On Thu, Oct 1, 2009 at 8:46 AM, David Dyer-Bennet <dd-b at dd-b.net>
wrote:
> Fajar A. Nugraha wrote:
> x86, OpenSolaris. ?But I''m not terribly attracted to the idea of
switching
> to another, less familiar, virtualization product in hopes that it will
> work. ?I really rather expected Sun''s virtualization product to
run Sun''s
> OS.
xvm is part of opensolaris, so it''s Sun''s virtualization
product :D

-- 
Fajar

Responding to paul at paularcher.org''s sep 30 2009 9:21 post:

For the entire file system, I have chosen zfs send/receive, per thread
"Best way to convert checksums".  I has concerns, they have been
answered.

Do my immediate need is answered.  The question remains as to how to copy
portions of trees, which zfs send/receive do not do.  that is, just copy stuff
around normally.  cp, tar, cpio, rsync are all great.  If one digs in, there are
various issues with various implementations, none of which are documented. 
Witness the warnings I have gotten that sun tar does not support sparse files. 
Is this true?  Was it true, has it been fixed?  There is another thread that
says something to the effect that if you tell sun tar to not copy acls that it
creates an empty acl with every file, where there was no acl at all in the
source!  Every implementation of every utility has what read like credible,
carefully documented rocks thrown at them.

My conclusion is to not use any system features that cannot be verified with a
simply file-contents comparison.  This will check the file contents, the
filename, and the pathname.  I can see the file owner, group, and dates. 
Sparseness, acls, and extended attributes I will simply not use.  I will have to
use non-solaris utilities for the compare or build my own in scripts because I
have "large files" and diff does not support them and cmp does not
recurse.

Thank you all for you insights and suggestions.  I consider this to be done, but
given the unsatisfying result, I would still welcome any new remarks.
-- 
This message posted from opensolaris.org

Responding to paul at paularcher.org''s sep 30 2009 9:21 post:

For the entire file system, I have chosen zfs send/receive, per thread
"Best way to convert checksums".  I has concerns, they have been
answered.

Do my immediate need is answered.  The question remains as to how to copy
portions of trees, which zfs send/receive do not do.  that is, just copy stuff
around normally.  cp, tar, cpio, rsync are all great.  If one digs in, there are
various issues with various implementations, none of which are documented. 
Witness the warnings I have gotten that sun tar does not support sparse files. 
Is this true?  Was it true, has it been fixed?  There is another thread that
says something to the effect that if you tell sun tar to not copy acls that it
creates an empty acl with every file, where there was no acl at all in the
source!  Every implementation of every utility has what read like credible,
carefully documented rocks thrown at them.

My conclusion is to not use any system features that cannot be verified with a
simply file-contents comparison.  This will check the file contents, the
filename, and the pathname.  I can see the file owner, group, and dates. 
Sparseness, acls, and extended attributes I will simply not use.  I will have to
use non-solaris utilities for the compare or build my own in scripts because I
have "large files" and diff does not support them and cmp does not
recurse.

Thank you all for you insights and suggestions.  I consider this to be done, but
given the unsatisfying result, I would still welcome any new remarks.
-- 
This message posted from opensolaris.org