Ray Clark
2009-Sep-29 19:57 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
The April 2009 "ZFS Administration Guide" states "...tar and cpio commands, to save ZFS files. All of these utilities save and restore ZFS file attributes and ACLs. I am running 8/07 (U4). Was this true for the U4 verison of ZFS and the tar and cpio shipped with U4? Also, I cannot seem to figure out how to find the ZFS admin manual applicable to U4. Could someone please shove me in the right direction? -- This message posted from opensolaris.org
Trevor Pretty
2009-Sep-29 20:27 UTC
Re: True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Ray Use this link it''s worth it''s weight in gold. The goolge search engine is so much better than what''s available at doc.sun.com http://www.google.com/custom?hl=en&client=google-coop&cof=S%3Ahttp%3A%2F%2Fwww.sun.com%3BCX%3ASun%2520Documentation%3BL%3Ahttp%3A%2F%2Flogos.sun.com%2Ftry%2Fimg%2Fsun_logo.gif%3BLH%3A31%3BLP%3A1%3B&q=&btnG=Search&cx=014942951012728127402%3Aulblnwea12w Simply search for: solaris 8/07 ZFS FYI: A while back Sun decided rather than having complete copies of each manual for each Solaris release which where 99.99% the same the manuals would just indicate was was new. The "What''s new" is always a good place to start. http://docs.sun.com/app/docs/doc/817-0547 Or simply run a file with ACLs through a tar and cpio pipe and see if they survive much quicker than reading!! examples in the respective man pages. example% cd fromdir; tar cf - .| (cd todir; tar xfBp -) example% find . -depth -print | cpio -pdlmv newdir Don''t forget the ACLs on ZFS are different to UFS. Trevor Ray Clark wrote: The April 2009 "ZFS Administration Guide" states "...tar and cpio commands, to save ZFS files. All of these utilities save and restore ZFS file attributes and ACLs. I am running 8/07 (U4). Was this true for the U4 verison of ZFS and the tar and cpio shipped with U4? Also, I cannot seem to figure out how to find the ZFS admin manual applicable to U4. Could someone please shove me in the right direction? www.eagle.co.nz This email is confidential and may be legally privileged. If received in error please destroy and immediately notify us. _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Joerg Schilling
2009-Sep-30 06:34 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Ray Clark <webclark at rochester.rr.com> wrote:> The April 2009 "ZFS Administration Guide" states "...tar and cpio commands, to save ZFS files. All of these utilities save and restore ZFS file attributes and ACLs.Be careful, Sun tar and Sun cpio do not support sparse files. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Ray Clark
2009-Oct-01 02:03 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Joerg, Thanks. As you (of all people) know, this area is quite a quagmire. I am confident that I don''t have any sparse files, or if I do that they are small and loosing this property would not be a big impact. I have determined that none of the files have extended attributes or ACLs. Some are greater than 4GB and have long paths, but Sun TAR supports both if I include the E option. I am trusting that because it is recommended in the ZFS Admin Guide that it is my safest option with respect to any ZFS idiosyncrasies, given its limitations. If only those were documented! My next problem is that I want to do an exhaustive file compare afterwards, and diff is not large-file aware. I always wonder if or how these applications that run across every OS known to man such as star can possibly be able to have the right code to work around the idiosyncrasies and exploit the capabilities of all of those OS''s. Should I consider star for the compare? For the copy? (Recognizing that it cannot do the ACLs, but I don''t have those). -- This message posted from opensolaris.org
Joerg Schilling
2009-Oct-01 06:33 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Ray Clark <webclark at rochester.rr.com> wrote:> Joerg, Thanks. As you (of all people) know, this area is quite a quagmire. I am confident that I don''t have any sparse files, or if I do that they are small and loosing this property would not be a big impact. I have determined that none of the files have extended attributes or ACLs. Some are greater than 4GB and have long paths, but Sun TAR supports both if I include the E option. I am trusting that because it is recommended in the ZFS Admin Guide that it is my safest option with respect to any ZFS idiosyncrasies, given its limitations. If only those were documented!Be careful! Sun tar creates non standard and thus non portable archives wich -E Only star can read them.....> My next problem is that I want to do an exhaustive file compare afterwards, and diff is not large-file aware.This is what star implements> I always wonder if or how these applications that run across every OS known to man such as star can possibly be able to have the right code to work around the idiosyncrasies and exploit the capabilities of all of those OS''s. Should I consider star for the compare? For the copy? (Recognizing that it cannot do the ACLs, but I don''t have those).Star of course supports ACLs. Star does not yet support ZFS ACLs and this is just a result of the fact that Sun did implement the same sort of design bugs in the first attempt to suport ACLs in ZFS. Future star versions will support ZFS ACLs as well. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Dennis Clarke
2009-Oct-01 08:52 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
> Ray Clark <webclark at rochester.rr.com> wrote: > >> Joerg, Thanks. As you (of all people) know, this area is quite a >> quagmire.<snip>> Be careful! Sun tar creates non standard and thus non portable archives > wich -E > Only star can read them..... > >> My next problem is that I want to do an exhaustive file compare >> afterwards, and diff is not large-file aware. > > This is what star implements > >> I always wonder if or how these applications that run across every OS >> known to man such as star can possibly be able to have the right code to >> work around the idiosyncrasies and exploit the capabilities of all of >> those OS''s. Should I consider star for the compare? For the copy? >> (Recognizing that it cannot do the ACLs, but I don''t have those). > > Star of course supports ACLs. Star does not yet support ZFS ACLs and this > is just a result of the fact that Sun did implement the same sort of > design bugs > in the first attempt to suport ACLs in ZFS. Future star versions will > support ZFS ACLs as well. > > J?rgI use star a great deal, daily in fact. I have two versions that I am using because one of them seems to mysteriously create ACL''s when I perform a copy from one directory to another. The two versions that I have are : # /opt/csw/bin/star --version star: star 1.5 (i386-pc-solaris2.8) Options: acl find remote Copyright (C) 1985, 88-90, 92-96, 98, 99, 2000-2009 J�g Schilling This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # /opt/schily/bin/star --version star: star 1.5a89 (i386-pc-solaris2.8) Options: acl find remote Copyright (C) 1985, 88-90, 92-96, 98, 99, 2000-2008 J�g Schilling This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. One of them is built by you ( 1.5 ) and the other by me ( 1.5a89 ) with smake and Studio 11. Recently I had to copy a large collection of files from one dir to another where the source was on ZFS and the destination was an NFSv4 share. The end result had ACLs in it that should not be there : # ls -lap total 94 drwxr-xr-x 6 16411 csw 6 Sep 29 00:51 ./ drwxr-xr-x 22 16411 csw 38 Sep 25 10:51 ../ drwxr-xr-x 5 16411 csw 12 Sep 29 02:32 gcc-4.3.4-SunOS_5.10-corert/ drwxr-xr-x 5 16411 csw 12 Sep 29 01:20 gcc-4.3.4-SunOS_5.10-g++rt/ drwxr-xr-x+ 5 16411 csw 19 Sep 29 03:50 gcc-4.3.4_SunOS_5.10-pkg/ drwxr-xr-x+ 22 root root 31 Sep 24 20:58 gcc-4.3.4_SunOS_5.10-release/ This seems impossible but this is what I see : # getfacl gcc-4.3.4_SunOS_5.10-pkg File system doesn''t support aclent_t style ACL''s. See acl(5) for more information on Solaris ACL support. The source dir looked like this : $ ls -ladE gcc-4.3.4_SunOS_5.10-pkg drwxr-xr-x 3 root root 5 2009-09-25 10:49:46.081206911 +0000 gcc-4.3.4_SunOS_5.10-pkg The output at the other end of a copy with star looks like this : # ls -lVdE gcc-4.3.4_SunOS_5.10-pkg drwxr-xr-x+ 5 16411 csw 19 2009-09-29 03:50:03.038056700 +0000 gcc-4.3.4_SunOS_5.10-pkg owner@:-----DaA--cC-s:------:allow owner@:--------------:------:deny group@:------a---c--s:------:allow group@:-----D-A---C--:------:deny everyone@:------a---c--s:------:allow everyone@:-----D-A---C--:------:deny owner@:--------------:------:deny owner@:rwxp---A-W-Co-:------:allow group@:-w-p----------:------:deny group@:r-x-----------:------:allow everyone@:-w-p---A-W-Co-:------:deny everyone@:r-x---a-R-c--s:------:allow Both the source and destination were on ZFS but the destination was a NFS mount on the host that performed the copy. The command was very typical : /opt/csw/bin/star -copy -p -acl -sparse -dump -C src_dir1 . dest_dir I am still baffled by the apparent ACL on a dir entry within a ZFS filesystem when the source never had ACLs at all. What I see is this : $ $HOME/bin/star_1.5a89 --version star: star 1.5a89 (i386-pc-solaris2.8) Options: acl find remote Copyright (C) 1985, 88-90, 92-96, 98, 99, 2000-2008 J�g Schilling This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ $ mkdir /home/dclarke/test/destination $ pwd /build/dclarke $ ls -ladin gcc-4.3.4_SunOS_5.10-release 509336 drwxr-xr-x 22 16411 101 31 Sep 25 11:40 gcc-4.3.4_SunOS_5.10-release $ $HOME/bin/star_1.5a89 -copy -p -acl -sparse -dump -time -fs=96m -fifostats -C /build/dclarke gcc-4.3.4_SunOS_5.10-release /home/dclarke/test/destination gcc-4.3.4_SunOS_5.10-release/configure.lineno is sparse gcc-4.3.4_SunOS_5.10-release/libiberty/hashtab.o is sparse gcc-4.3.4_SunOS_5.10-release/libiberty/strverscmp.o is sparse gcc-4.3.4_SunOS_5.10-release/libiberty/objalloc.o is sparse gcc-4.3.4_SunOS_5.10-release/libiberty/cplus-dem.o is sparse gcc-4.3.4_SunOS_5.10-release/libiberty/cp-demint.o is sparse . . . gcc-4.3.4_SunOS_5.10-release/prev-libiberty/vasprintf.o is sparse /home/dclarke/bin/star_1.5a89: fifo had 57001 puts 55368 gets. /home/dclarke/bin/star_1.5a89: fifo was 2 times empty and 33 times full. /home/dclarke/bin/star_1.5a89: fifo held 100669440 bytes max, size was 100669440 bytes /home/dclarke/bin/star_1.5a89: 0 blocks + 1593968128 bytes (total of 1593968128 bytes = 1556609.50k). /home/dclarke/bin/star_1.5a89: Total time 1735.341sec (897 kBytes/sec) $ I should mention really poor performance also. Look in the output dir and see the ACL indicator ? $ ls -la /home/dclarke/test/destination total 11 drwxr-xr-x 3 dclarke csw 3 Oct 1 08:17 . drwxr-xr-x 3 dclarke csw 5 Oct 1 08:16 .. drwxr-xr-x+ 22 dclarke csw 31 Sep 25 11:40 gcc-4.3.4_SunOS_5.10-release That should not be there. $ cd /home/dclarke/test/destination $ ls -lVdE gcc-4.3.4_SunOS_5.10-release drwxr-xr-x+ 22 dclarke csw 31 2009-09-25 11:40:09.491951000 +0000 gcc-4.3.4_SunOS_5.10-release owner@:-----DaA--cC-s:------:allow owner@:--------------:------:deny group@:------a---c--s:------:allow group@:-----D-A---C--:------:deny everyone@:------a---c--s:------:allow everyone@:-----D-A---C--:------:deny owner@:--------------:------:deny owner@:rwxp---A-W-Co-:------:allow group@:-w-p----------:------:deny group@:r-x-----------:------:allow everyone@:-w-p---A-W-Co-:------:deny everyone@:r-x---a-R-c--s:------:allow If I look down into that dir I see ACL''s on all the ddir entries : $ cd gcc-4.3.4_SunOS_5.10-release $ ls -l total 1119 -rw-r--r-- 1 dclarke csw 577976 Sep 24 02:59 Makefile drwxr-xr-x+ 4 dclarke csw 5 Sep 24 03:11 build-i386-pc-solaris2.10 -rw-r--r-- 1 dclarke csw 10 Sep 24 20:58 compare -rw-r--r-- 1 dclarke csw 30323 Sep 24 02:59 config.log -rwxr-xr-x 1 dclarke csw 31724 Sep 24 02:59 config.status -rwxr-xr-x 1 dclarke csw 400174 Sep 24 02:59 configure.lineno drwxr-xr-x+ 2 dclarke csw 20 Sep 24 20:58 fixincludes drwxr-xr-x+ 15 dclarke csw 535 Sep 25 03:55 gcc drwxr-xr-x+ 10 dclarke csw 10 Sep 24 21:23 i386-pc-solaris2.10 drwxr-xr-x+ 2 dclarke csw 32 Sep 25 10:39 intl drwxr-xr-x+ 3 dclarke csw 29 Sep 24 20:44 libcpp drwxr-xr-x+ 2 dclarke csw 15 Sep 24 20:44 libdecnumber drwxr-xr-x+ 4 dclarke csw 72 Sep 24 20:43 libiberty drwxr-xr-x+ 14 dclarke csw 532 Sep 24 20:41 prev-gcc drwxr-xr-x+ 4 dclarke csw 4 Sep 24 20:40 prev-i386-pc-solaris2.10 drwxr-xr-x+ 2 dclarke csw 32 Sep 24 19:56 prev-intl drwxr-xr-x+ 3 dclarke csw 29 Sep 24 19:57 prev-libcpp drwxr-xr-x+ 2 dclarke csw 15 Sep 24 19:58 prev-libdecnumber drwxr-xr-x+ 4 dclarke csw 72 Sep 24 19:56 prev-libiberty -rw-r--r-- 1 dclarke csw 13 Sep 24 02:59 serdep.tmp drwxr-xr-x+ 14 dclarke csw 520 Sep 24 19:53 stage1-gcc drwxr-xr-x+ 4 dclarke csw 4 Sep 24 19:51 stage1-i386-pc-solaris2.10 drwxr-xr-x+ 2 dclarke csw 32 Sep 24 03:09 stage1-intl drwxr-xr-x+ 3 dclarke csw 29 Sep 24 03:15 stage1-libcpp drwxr-xr-x+ 2 dclarke csw 15 Sep 24 03:16 stage1-libdecnumber drwxr-xr-x+ 4 dclarke csw 72 Sep 24 03:08 stage1-libiberty -rw-r--r-- 1 dclarke csw 7 Sep 24 20:58 stage_current -rw-r--r-- 1 dclarke csw 7 Sep 24 03:01 stage_final -rw-r--r-- 1 dclarke csw 7 Sep 24 20:58 stage_last I''ll delete this now .. I can''t use it with those strange ACL''s there. $ cd /home/dclarke/test $ rm -rf destination I''ll do some more testing with star 1.5a89 and let you know what I see. -- Dennis Clarke dclarke at opensolaris.ca <- Email related to the open source Solaris dclarke at blastwave.org <- Email related to open source for Solaris
Joerg Schilling
2009-Oct-01 09:23 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Dennis Clarke <dclarke at blastwave.org> wrote:> I use star a great deal, daily in fact. I have two versions that I am > using because one of them seems to mysteriously create ACL''s when I > perform a copy from one directory to another. > > The two versions that I have are : > > # /opt/csw/bin/star --version > star: star 1.5 (i386-pc-solaris2.8)> # /opt/schily/bin/star --version > star: star 1.5a89 (i386-pc-solaris2.8)> One of them is built by you ( 1.5 ) and the other by me ( 1.5a89 ) with > smake and Studio 11.In star-1.5a89 ACLs don''t work if you use -find.> Recently I had to copy a large collection of files from one dir to another > where the source was on ZFS and the destination was an NFSv4 share. The > end result had ACLs in it that should not be there : > > # ls -lap > total 94 > drwxr-xr-x 6 16411 csw 6 Sep 29 00:51 ./ > drwxr-xr-x 22 16411 csw 38 Sep 25 10:51 ../ > drwxr-xr-x 5 16411 csw 12 Sep 29 02:32 > gcc-4.3.4-SunOS_5.10-corert/ > drwxr-xr-x 5 16411 csw 12 Sep 29 01:20 > gcc-4.3.4-SunOS_5.10-g++rt/ > drwxr-xr-x+ 5 16411 csw 19 Sep 29 03:50 > gcc-4.3.4_SunOS_5.10-pkg/ > drwxr-xr-x+ 22 root root 31 Sep 24 20:58 > gcc-4.3.4_SunOS_5.10-release/If you like to investigate on this problem, you would need to keep the source directory tree and send the commandline you used for copying. In the next step we would need to investigate on the tar archive that was created by star....> The source dir looked like this : > > $ ls -ladE gcc-4.3.4_SunOS_5.10-pkg > drwxr-xr-x 3 root root 5 2009-09-25 10:49:46.081206911 > +0000 gcc-4.3.4_SunOS_5.10-pkg > > The output at the other end of a copy with star looks like this : > > # ls -lVdE gcc-4.3.4_SunOS_5.10-pkg > drwxr-xr-x+ 5 16411 csw 19 2009-09-29 03:50:03.038056700 > +0000 gcc-4.3.4_SunOS_5.10-pkg > owner@:-----DaA--cC-s:------:allow > owner@:--------------:------:deny > group@:------a---c--s:------:allow > group@:-----D-A---C--:------:deny > everyone@:------a---c--s:------:allow > everyone@:-----D-A---C--:------:deny > owner@:--------------:------:deny > owner@:rwxp---A-W-Co-:------:allow > group@:-w-p----------:------:deny > group@:r-x-----------:------:allow > everyone@:-w-p---A-W-Co-:------:deny > everyone@:r-x---a-R-c--s:------:allow > > Both the source and destination were on ZFS but the destination was a NFS > mount on the host that performed the copy.It may be that you are hit by a bug in Sun''s NFS implementation. Sun''s tar implemantation definitely had several critical bugs in 2005 related to ACLs. Given the fact that many critical bugreports I made for Solaris did noit end in a fix, it may be that the related bug still exists. The bug I am currently talking is the one that causes unpacked files to get ACLs although the source file has no ACLs (this is independent from the unterlying file system). The reason is that Sun tar does not _clear_ ACLs on files if the source file has no ACLs. The destination file however may have inherited ACLs from the dierectory. This is a serious security problem in Sun tar. It could be that Sun''s NFS implementation _creates_ ACLs when star sends a request to _clear_ the ACLs by establishing "base ACLs" that just contain the UNIX file permissins. From the Sun documentation, this needs to remove existing ACLs but it may do something else if there is a bug in the NFS implementation.> The command was very typical : > > /opt/csw/bin/star -copy -p -acl -sparse -dump -C src_dir1 . dest_dir > > I am still baffled by the apparent ACL on a dir entry within a ZFS > filesystem when the source never had ACLs at all. > > What I see is this : > > $ $HOME/bin/star_1.5a89 --version > star: star 1.5a89 (i386-pc-solaris2.8)Instead of using an old version with known bugs, you could just omit -acl with a recent version that has no known bugs.> /home/dclarke/bin/star_1.5a89: 0 blocks + 1593968128 bytes (total of > 1593968128 bytes = 1556609.50k). > /home/dclarke/bin/star_1.5a89: Total time 1735.341sec (897 kBytes/sec) > $ > > I should mention really poor performance also.If you don''t care about granted integrity of the copy, you could try add -no-fsync J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Joerg Schilling
2009-Oct-01 09:36 UTC
[zfs-discuss] True in U4? "Tar and cpio...save and restore ZFS File attributes and ACLs"
Joerg.Schilling at fokus.fraunhofer.de (Joerg Schilling) wrote:> Dennis Clarke <dclarke at blastwave.org> wrote:> It could be that Sun''s NFS implementation _creates_ ACLs when star sends a > request to _clear_ the ACLs by establishing "base ACLs" that just contain > the UNIX file permissins. From the Sun documentation, this needs to remove > existing ACLs but it may do something else if there is a bug in the NFS > implementation.I forgot to mention why this may be related to Sun tar....: If you test with buggy tools, you may not see that the OS kernel does not behave correctly. J?rg -- EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin js at cs.tu-berlin.de (uni) joerg.schilling at fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily