zfs discuss - Oct 2006 - directory tree removal issue with zfs on Blade 1500/PC rack server IDE disk

I want to know, if anybody can check/confirm the following issue I observed with
a fully patched Solaris 10 u2 with ZFS running on IDE disks and how the state of
the IDE/ZFS issue is in general in the development of OpenSolaris resp. Nevada:

I have observed the following issue: when I try to remove with 

rm -rf dir

a whole directory tree named dir, only the regular files will be removed on
Solaris 10 u2 (2006/06) when doing so on a ZFS file system residing on an IDE
disk, but not the sub directories --- it is okay, when doing the same command on
a SCSI disk.

Details: the directory traversal is malfunctioning or out of sync with the IDE
disk, I fear, because if I do the following afterwards

find dir -depth -exec rmdir {} \;

the remaining, now aside from sub directories empty directories are removed
silently and successfully. And this is exactly okay when using the -depth option
only, because this guarantees the "right" directory traversal, where
the exec is applied only on the leaves first and afterwards on the higher nodes.

If I rm -rf as root it works too, but as normal user it doesn''t, though
the directory masks are okay (755 for the owning user trying to remove). The ---
too generic --- error message is

rm: cannot determine if this is an ancestor of the current working directory

(in German(auf Deutsch): rm: Es kann nicht ermittelt werden, ob dies ein
Vorg?nger des aktuellen Arbeitsverzeichnisses ist:)

and I fear too, that this is an outcome of the already described IDE/ZFS issue
in Solaris 10. But it is the strangest thing when trying to remove a directory
tree I ever happened to see on Unix in 15 years experience and a truely nasty
bug.

When I understand right, this should only affect PATA=IDE, but neither SCSI (as
observed by myself) nor SATA (this is only assumed).
 
 
This message posted from opensolaris.org

>the remaining, now aside from sub directories empty directories are
r>emoved silently and successfully. And this is exactly okay when using>
the -depth option only, because this guarantees the "right"
director>y traversal, where the exec is applied only on the leaves first and
a>fterwards on the higher nodes.

This is probably caused by the mountpoint (the underlying directory, not
the root of the filesystem) being a mode like 500.


Unmount all the ZFS filesystems and check the permissions on the mount 
points and the paths leading up to them.

Casper

Check the permission of your mountpoint after you unmount the dataset.
 Most likely, you have something like rwx------.

On 10/5/06, Stefan Urbat <sur at compart.net>
wrote:
> I want to know, if anybody can check/confirm the following issue I observed
with a fully patched Solaris 10 u2 with ZFS running on IDE disks and how the
state of the IDE/ZFS issue is in general in the development of OpenSolaris resp.
Nevada:
>
> I have observed the following issue: when I try to remove with
>
> rm -rf dir
>
> a whole directory tree named dir, only the regular files will be removed on
Solaris 10 u2 (2006/06) when doing so on a ZFS file system residing on an IDE
disk, but not the sub directories --- it is okay, when doing the same command on
a SCSI disk.
>
> Details: the directory traversal is malfunctioning or out of sync with the
IDE disk, I fear, because if I do the following afterwards
>
> find dir -depth -exec rmdir {} \;
>
> the remaining, now aside from sub directories empty directories are removed
silently and successfully. And this is exactly okay when using the -depth option
only, because this guarantees the "right" directory traversal, where
the exec is applied only on the leaves first and afterwards on the higher nodes.
>
> If I rm -rf as root it works too, but as normal user it doesn''t,
though the directory masks are okay (755 for the owning user trying to remove).
The --- too generic --- error message is
>
> rm: cannot determine if this is an ancestor of the current working
directory
>
> (in German(auf Deutsch): rm: Es kann nicht ermittelt werden, ob dies ein
Vorg?nger des aktuellen Arbeitsverzeichnisses ist:)
>
> and I fear too, that this is an outcome of the already described IDE/ZFS
issue in Solaris 10. But it is the strangest thing when trying to remove a
directory tree I ever happened to see on Unix in 15 years experience and a
truely nasty bug.
>
> When I understand right, this should only affect PATA=IDE, but neither SCSI
(as observed by myself) nor SATA (this is only assumed).
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

-- 
Just me,
Wire ...

On 10/5/06, Casper.Dik at sun.com <Casper.Dik at sun.com>
wrote:
> Unmount all the ZFS filesystems and check the permissions on the mount
> points and the paths leading up to them.
I experienced the same problem and narrowed it down to that
essentially, chdir("..") in "rm -rf" failed to ascend up the
directory.  The mountpoint was created by ZFS but with the umask set
to 077.

- Should ZFS respect the umask setting even when it is creating the mountpoint?
- Should Solaris (in general) ignore mountpoint permissions?


-- 
Just me,
Wire ...

>On 10/5/06, Casper.Dik at sun.com <Casper.Dik at sun.com> wrote:
>> Unmount all the ZFS filesystems and check the permissions on the mount
>> points and the paths leading up to them.
>
>I experienced the same problem and narrowed it down to that
>essentially, chdir("..") in "rm -rf" failed to ascend up
the
>directory.  The mountpoint was created by ZFS but with the umask set
>to 077.
>
>- Should ZFS respect the umask setting even when it is creating the
mountpoint?

No  (I think this is being fixed).
>- Should Solaris (in general) ignore mountpoint permissions?

Yes, I believe so.

Casper

> 
> >On 10/5/06, Casper.Dik at sun.com <Casper.Dik at sun.com>
> wrote:
> >- Should ZFS respect the umask setting even when it
> is creating the mountpoint?
> 
> 
> No  (I think this is being fixed).
> 
Nice to hear, this is a very strange phenomenon as told.
> >- Should Solaris (in general) ignore mountpoint
> permissions?
> 
> 
> Yes, I believe so.
> 
> Casper
>
I fully agree to this: I never encountered any comparable consequence of ill
mountpoint permissions on any Unix: so I would expect a successful mount should
always hide completely the original mountpoint permissions resp. render them
irrelevant, only using the current, mounted file system permissions.

By the way, I have to wait a few hours to umount and check mountpoint
permissions, because an automated build is currently running on that zfs --- the
performance of zfs at IDE is indeed rather poor (much worse than ufs), but this
is another, already documented and bug entry "honoured" issue.
 
 
This message posted from opensolaris.org

You were completely right in guessing the issue: after umount /export/home the
ZFS residing there the wrong (?), but likely standard Solaris 10, not ufs
affecting attribute mask was visible: 700. After changing this to 755 and
mounting the ZFS thereon again, the issue was resolved completely. Thanks for
the hint!
 
 
This message posted from opensolaris.org

Stefan Urbat wrote:
> By the way, I have to wait a few hours to umount and check mountpoint
> permissions, because an automated build is currently running on that
> zfs --- the performance of zfs at IDE is indeed rather poor (much worse
> than ufs), but this is another, already documented and bug entry
> "honoured" issue.
Really?  Are you allowing ZFS to use the entire disk (and thus turn on 
the disk''s write cache)?  Can you describe your workload and give 
numbers on both ZFS and UFS?  What bug was filed?

--matt

> Stefan Urbat wrote:
> > zfs --- the performance of zfs at IDE is indeed rather
> poor (much worse
> > than ufs), but this is another, already documented
> and bug entry
> > "honoured" issue.
> 
> Really?  Are you allowing ZFS to use the entire disk
> (and thus turn on 
> the disk''s write cache)?
>
No, I couldn''t, because a seemingly correct upgrade trial from Sol 10
u1 to u2 failed: from the original disk I couldn''t boot afterwards any
more (library issues :( ). So I had to use the second, newly inserted IDE disk
opposed to my intention to install Sol10u2 newly, with unavoidable ufs slices
for the root file system and some others, and created just one slice there for
zfs; the data of the first disk (the ufs file systems there are still usable)
must be preserved.
>  Can you describe your
> workload and give 
> numbers on both ZFS and UFS?
>
I can give you rough numbers about it: the nightly build took for the same
amount of data about 7 hours (ufs), now 9,5 hours (zfs). It is running now from
the second but the first disk, but both ar usual 7200 rpm IDE disks, the first
the in-build Barracuda 80 GB, the second a more recent Samsung 200 GB. Due to a
strange phenomenon I can use only 32 GB on the second disk currently (BOOTROM?),
so it could be in principle, that the true reason is a too slow operating second
disk indeed. I can''t compare currently ufs and zfs on the very same
disk in this regard. But I may test it now, being more aware of the potential
issue.
> What bug was filed?
> 
6421427 is nfs related, but another forum member thought, that it is in fact a
general IDE performance bottleneck behind, and was only made visible in this
case. There is a report, that on an also with simple IDE equipped Blade 150 the
same issue with low performance is visible:
http://www.opensolaris.org/jive/thread.jspa?messageID=57201
 
 
This message posted from opensolaris.org

Stefan Urbat wrote:
>> What bug was filed?
>>
> 6421427 is nfs related, but another forum member thought, that it is in
fact a general IDE performance bottleneck behind, and was only made visible in
this case. There is a report, that on an also with simple IDE equipped Blade 150
the same issue with low performance is visible:
http://www.opensolaris.org/jive/thread.jspa?messageID=57201
Ah yes, good old 6421427.  The fix for that should be putback into 
opensolaris any day now.

--matt