zfs discuss - Nov 2005 - GNU find problem on ZFS

I found a really weird bug with the combination of GNU find, Nexenta, and ZFS. I
don''t know exactly whose fault the bug is, but basically GNU find
always fails to descend into the last directory in the root of a ZFS filesystem
or pool. The problem doesn''t occur from non-root directories, only the
last directory fails, and the .zfs directory also exhibits the problem if it
happens to be the last directory. I''m unable to identify any other
program that has this problem.

In case my description of the problem is unclear:

# zfs create export/test
# cd /export/test/
# /usr/bin/find --version
GNU find version 4.2.22
Features enabled: O_NOFOLLOW(enabled)
# /usr/bin/find .
.
./.zfs
# /usr/sun/bin/find .
.
./.zfs
./.zfs/snapshot
# mkdir -p 1/2/3 1/4/5 6/7
# /usr/bin/find .
.
./.zfs
./.zfs/snapshot
./6
./6/7
./1
# /usr/sun/bin/find .
.
./.zfs
./.zfs/snapshot
./6
./6/7
./1
./1/4
./1/4/5 
./1/2 
./1/2/3
# cd 1
# /usr/bin/find .
.
./4
./4/5
./2
./2/3
# /usr/sun/bin/find .
.
./4
./4/5
./2
./2/3
#
This message posted from opensolaris.org

>I found a really weird bug with the combination of GNU find, Nexenta, and
ZFS. I don''t know exactl
y whose fault the bug is, but basically GNU find always fails to descend into
the last directory in
 the root of a ZFS filesystem or pool. The problem doesn''t occur from
non-root directories, only th
e last directory fails, and the .zfs directory also exhibits the problem if it
happens to be the la
st directory. I''m unable to identify any other program that has this
problem.

Tried the -noleaf option?


(GNU find assumes a correlation between the number of links to a directory and
the number of subdirectories it has)

Casper

[ ... ]
> Tried the -noleaf option?
> 
> 
> (GNU find assumes a correlation between the number of links to a directory
and
> the number of subdirectories it has)
> 
> Casper

I thought this behaviour was a "./configure"-time option which
defaulted
to "don''t behave like this" on Solaris ?

ZFS isn''t the only filesystem on Solaris that doesn''t behave
like this.
PCFS doesn''t, HSFS doesn''t. UFS does (I think). Some of the
pseudo fs
(procfs) do, others (objfs/ctfs) don''t.

I remember a discussion a few years ago about this. Fact is that a
meaning for st_nlink (either for files or directories) is not specified
by any standards. stat(2) says it''s an "administrative field"
- blah ...

GNU find overoptimizes here by assuming that it _has_ a particular
meaning for a directory, and therefore misbehaves - it will fail to
descend into subdirectories.

FrankH.

-noleaf makes find work as expected. Interestingly, although find warns of this
behavior in the manpage, even msdos and iso9660 filesystems on Linux
don''t seem to exhibit it, which is probably why I never encountered the
problem before.

If Solaris has a history of in some cases making filesystem root link counts one
less than in other cases, and it doesn''t violate the standards, then
the problem is GNU findutils.

./configure with "--disable-leaf-optimisation" makes GNU find work as
expected, but it isn''t the default. I''ll file a bug with
Nexenta.
This message posted from opensolaris.org

>>>>> "FH" == Frank Hofmann <- Solaris Sustaining
<Frank.Hofmann at sun.com>> writes:
    FH> GNU find overoptimizes here by assuming that it _has_ a particular
    FH> meaning for a directory, and therefore misbehaves - it will fail to
    FH> descend into subdirectories.

OK, I''ll bite.  What optimization can one make if one assumes a meaning
for the
link count?

Matt

-- 
	Matt Simmons - simmonmt at eng.sun.com | Solaris Kernel - New York
    Man does not live by words alone, despite the fact that sometimes he has
			to eat them.  --Adlai Stevenson

>     FH> GNU find overoptimizes here by assuming that it _has_ a
particular
>     FH> meaning for a directory, and therefore misbehaves - it will fail
to
>     FH> descend into subdirectories.
> 
> OK, I''ll bite.  What optimization can one make if one assumes a
meaning for
the
> link count?
> 
> Matt
It stops readdir() once it has found st_nlink subdirectories (to descend
into). If the directory has more than st_nlink subdirs, GNU find will
just not traverse them all.

The problem *IS* GNU findutils. There just is no place at all that
says "filesystems have to return the number of subdirectories as
st_nlink of a directory". If you 100% know your system and all the
filesystems on it behave like that, then ok (Linux does). But you
cannot require anyone to give you what you consider a meaningful
value in st_nlink.

If anyone knows differently I''d like to hear. But as I read the
standards wrt. to stat(2), compliance would hold even if st_nlink
were a completely random number.

In a way, st_nlink exposes implementation details. Not all filesystems
have link counts; for some it may well be impossible to return anything
meaningful. For others (like FAT), determining what a link count for
a directory should be would require parsing the contents of the directory;
one can do that on stat() of course - but why ? Just to satisfy some
overoptimistic assumption ?

FrankH.

>>>>>> "FH" == Frank Hofmann <- Solaris
Sustaining <Frank.Hofmann at sun.com>> writes:
>
>    FH> GNU find overoptimizes here by assuming that it _has_ a
particular
>    FH> meaning for a directory, and therefore misbehaves - it will fail
to
>    FH> descend into subdirectories.
>
>OK, I''ll bite.  What optimization can one make if one assumes a
meaning for the
>link count?


	nsubdirs = stat(dir)->st_nlinks - 2;

	while (readdir(dirp)) {
		if (nsubdirs > 0)
			if (is a directory) {
				nsubdirs --;
				dothedirthing;
			}
	}


So you can save a bunch of stats on leaf directories or as soon
as you''ve seen all subdirs.

Casper

>>>>> "CD" == Casper Dik <Casper.Dik at sun.com>
writes:
    CD> So you can save a bunch of stats on leaf directories or as soon as
    CD> you''ve seen all subdirs.

<smacks head>

Of course.

Matt

-- 
	Matt Simmons - simmonmt at eng.sun.com | Solaris Kernel - New York
		    Save the whales. Collect the whole set.

GNU find is broken and it seems that the maintainer is not willing
to fix it.

A few months ago, we had a discussion with him on the POSIX
maining list. He added -exec + support but he did not fix the 
broken dir code.

The assumptions GNU make does are false for nearly all platforms
and filesystems.

The assumptions are wrong because they violated POSIX directory
semantics.

The assumptions are even wrong on "classical" UNIX systems in case
that
there are hard linked directories.

GNU find works however if you call gfind . -noleaf ...

My proposal: hack the source.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de		(uni)  
       schilling at fokus.fraunhofer.de	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

The reason that the "optimized" find is failing to enter the final
directory in a zfs root file system is that the special .zfs directory
is not currently counted in the nlinks for the root directory.  If
you set the `snapdir'' property to be "hidden", find should
work as
expected.  I think a valid argument could be made that nlinks for the
root should include the .zfs directory when it is in the "visible"
state.

-Mark

Casper.Dik at sun.com wrote:
>>>>>>>"FH" == Frank Hofmann <- Solaris
Sustaining <Frank.Hofmann at sun.com>> writes:
>>
>>   FH> GNU find overoptimizes here by assuming that it _has_ a
particular
>>   FH> meaning for a directory, and therefore misbehaves - it will
fail to
>>   FH> descend into subdirectories.
>>
>>OK, I''ll bite.  What optimization can one make if one assumes a
meaning for the
>>link count?
> 
> 
> 
> 
> 	nsubdirs = stat(dir)->st_nlinks - 2;
> 
> 	while (readdir(dirp)) {
> 		if (nsubdirs > 0)
> 			if (is a directory) {
> 				nsubdirs --;
> 				dothedirthing;
> 			}
> 	}
> 
> 
> So you can save a bunch of stats on leaf directories or as soon
> as you''ve seen all subdirs.
> 
> Casper
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://opensolaris.org/mailman/listinfo/zfs-discuss

Mark Maybee <Mark.Maybee at Sun.COM> wrote:
> The reason that the "optimized" find is failing to enter the
final
> directory in a zfs root file system is that the special .zfs directory
> is not currently counted in the nlinks for the root directory.  If
> you set the `snapdir'' property to be "hidden", find
should work as
> expected.  I think a valid argument could be made that nlinks for the
> root should include the .zfs directory when it is in the
"visible"
> state.
POSIX does not grant anything for directories except that it is a
special file if type S_IFDIR.

There is no grant on the size.

There is no grant on the link count value.

There is no grant that a root dir has ino #2

There is no grant that readdir returns the entries ''.'' &
''..''.


Software that depends on properties that are not granted
needs to be called broken.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de		(uni)  
       schilling at fokus.fraunhofer.de	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily

Matthew Simmons <simmonmt at eng.sun.com> wrote:
> >>>>> "FH" == Frank Hofmann <- Solaris
Sustaining <Frank.Hofmann at sun.com>> writes:
>
>     FH> GNU find overoptimizes here by assuming that it _has_ a
particular
I am nore sure if "overoptimizes" is the right word.
GNU find does one songle optimization and this optimization iw wrong.
>     FH> meaning for a directory, and therefore misbehaves - it will fail
to
>     FH> descend into subdirectories.
>
> OK, I''ll bite.  What optimization can one make if one assumes a
meaning for the
> link count?
You cannot optimize here as this would break POSIX compliance. The link count
of a directory has no meaning and for this reason, any software that makes 
assumptions on the link count is broken.

J?rg

-- 
 EMail:joerg at schily.isdn.cs.tu-berlin.de (home) J?rg Schilling D-13353 Berlin
       js at cs.tu-berlin.de		(uni)  
       schilling at fokus.fraunhofer.de	(work) Blog: http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily