Scott Bennett
2016-Oct-21 13:40 UTC
zfs, a directory that used to hold lot of files and listing pause
On Fri, 21 Oct 2016 16:51:36 +0500 "Eugene M. Zheganin" <emz at norma.perm.ru> wrote:>On 21.10.2016 15:20, Slawa Olhovchenkov wrote: >> >> ZFS prefetch affect performance dpeneds of workload (independed of RAM >> size): for some workloads wins, for some workloads lose (for my >> workload prefetch is lose and manualy disabled with 128GB RAM). >> >> Anyway, this system have only 24MB in ARC by 2.3GB free, this is may >> be too low for this workload. >You mean - "for getting a list of a directory with 20 subdirectories" ? >Why then does only this directory have this issue with pause, not >/usr/ports/..., which has more directories in it ? > >(and yes, /usr/ports/www isn't empty and holds 2410 entities) > >/usr/bin/time -h ls -1 /usr/ports/www >[...] >0.14s real 0.00s user 0.00s sys >Oh, my goodness, how far afield nonsense has gotten! Have all the good folks posting in this thread forgotten how directory blocks are allocated in UNIX? This isn't even a BSD-specific thing; it's really ancient. What Eugene has complained of is exactly what is to be expected-- on really old hardware. The only eyebrow-raiser is that he has created a use case so extreme that a live human can actually notice the delays on modern hardware. I quote from his original posting: "I also have one directory that used to have a lot of (tens of thousands) files." and "But now I have 2 files and a couple of dozens directories in it". A directory with tens of thousands of files in it at one point in time most likely has somewhere well over one thousand blocks allocated. Directories don't shrink. Directory entries do not get moved around within directories when files are added or deleted. Directories can remain the same length or they can grow in length. If a directory once had many tens of thousands of filenames and links to their primary inodes, then the directory is still that big, even if it now only contains two [+ 20 to 30 directory], probably widely separated, entries. To read a file's entry, all blocks must be searched until the desired filename is found. Likewise, to list the contents of a directory, all blocks must be read until the number of files found matches the link count for the directory. IOW, if you want the performance to go back to what it was when the directory was fresh (and still small), you have to create a new directory and then move the remaining entries from the old directory into the new (small) directory. The only real difference here between UFS (or even the early AT&T filesystem) and ZFS is that the two remaining entries in a formerly huge directory are likely to be in different directory blocks that could be at effectively random locations scattered around the space of a partition for one filesystem in UFS or over an entire pool of potentially many filesystems and much more space in ZFS. Scott Bennett, Comm. ASMELG, CFIAG ********************************************************************** * Internet: bennett at sdf.org *xor* bennett at freeshell.org * *--------------------------------------------------------------------* * "A well regulated and disciplined militia, is at all times a good * * objection to the introduction of that bane of all free governments * * -- a standing army." * * -- Gov. John Hancock, New York Journal, 28 January 1790 * **********************************************************************
Pete French
2016-Oct-21 14:04 UTC
zfs, a directory that used to hold lot of files and listing pause
> Oh, my goodness, how far afield nonsense has gotten! Have all the > good folks posting in this thread forgotten how directory blocks are > allocated in UNIX?Not forgotten, just under the impression that ZFS shrinks directories unlike good old UFS. Apparenrly not, and yes, if thats true then the behaviour is not surprising in the slightest. Live and learn... ;-) -pete. [old enough to have used 32V on a Vax, a loooong time ago...]
Jamie Landeg-Jones
2016-Oct-24 17:16 UTC
zfs, a directory that used to hold lot of files and listing pause
Scott Bennett <bennett at sdf.org> wrote:> thousand blocks allocated. Directories don't shrink. Directory entries do > not get moved around within directories when files are added or deleted. > Directories can remain the same length or they can grow in length. If a > directory once had many tens of thousands of filenames and links to their > primary inodes, then the directory is still that big, even if it now only > contains two [+ 20 to 30 directory], probably widely separated, entries.> IOW, if you want the performance to go back to what it was when the directory > was fresh (and still small), you have to create a new directory and then move > the remaining entries from the old directory into the new (small) directory.Not entirely true. FreeBSD on UFS *will* *truncate* directories where possible, each time a new directory entry is made, and there is contiguous unused space to the end of the directory-file. [ As an aside, tmpfs does 'defragment' and rezise directories in real time, when a file from *any* location is deleted. ] Back to UFS (I don't know about ZFS): So, whilst you are right about directory entries not being moved around, consider the following. Note the size of '.' in each directory listing: | 17:05 [2] (1) "~" jamie at lapcat% md dir | dir | | 17:05 [2] (2) "~" jamie at lapcat% cd dir | | 17:05 [2] (3) "~/dir" jamie at lapcat% l | total 8 | 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:05 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:05 [2] (4) "~/dir" jamie at lapcat% jot 999 1 999 | awk '{printf "touch %03d\n", $1}' | sh | | 17:05 [2] (5) "~/dir" jamie at lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:05 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:05 [2] (6) "~/dir" jamie at lapcat% rm [678]* | remove 300 files? y | | 17:06 [2] (7) "~/dir" jamie at lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (8) "~/dir" jamie at lapcat% touch x; rm x | | 17:06 [2] (9) "~/dir" jamie at lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (10) "~/dir" jamie at lapcat% rm 9* | remove 100 files? y | | 17:06 [2] (11) "~/dir" jamie at lapcat% l | head | total 16 | 12 drwxr-xr-x 2 jamie jamie - 12288 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (12) "~/dir" jamie at lapcat% touch x ; rm x | | 17:06 [2] (13) "~/dir" jamie at lapcat% l | head | total 12 | 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 001 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 002 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 003 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 004 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 005 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 006 | 0 -rw-r--r-- 1 jamie jamie - 0 24 Oct 17:05 007 | | *141* 17:06 [2] (14) "~/dir" jamie at lapcat% rm * | remove 599 files? y | | 17:06 [2] (15) "~/dir" jamie at lapcat% l | total 12 | 8 drwxr-xr-x 2 jamie jamie - 7680 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:06 [2] (16) "~/dir" jamie at lapcat% touch x ; rm x | | 17:06 [2] (17) "~/dir" jamie at lapcat% l | total 8 | 4 drwxr-xr-x 2 jamie jamie - 512 24 Oct 17:06 ./ | 4 drwx------ 72 jamie jamie - 3072 24 Oct 17:05 ../ | | 17:07 [2] (18) "~/dir" jamie at lapcat% exit