freebsd stable - Oct 2016 - zfs, a directory that used to hold lot of files and listing pause

On Thu, Oct 20, 2016 at 3:47 PM, Eugene M. Zheganin <emz at norma.perm.ru>
wrote:
> Hi.
>
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation on
> different releases) and a zfs. I also have one directory that used to have
a
> lot of (tens of thousands) files. I surely takes a lot of time to get a
> listing of it. But now I have 2 files and a couple of dozens directories in
> it (I sorted files into directories). Surprisingly, there's still a lag
> between "ls" and an output:
>
>
> ===Cut==>
> # /usr/bin/time -h ls
> .recycle        2016-01         2016-04         2016-07 2016-10
> sort-files.sh
> 2014            2016-02         2016-05         2016-08 ktrace.out
> sort-months.sh
> 2015            2016-03         2016-06         2016-09 old
> sounds
>         5.75s real              0.00s user              0.02s sys
Looks like it's not taking up any processing time, so my guess is
the lag probably comes from stalled I/O ... bad disk?

Does a second "ls" immediately returned (ie. metadata has been
cached) ?
>
> ===Cut==>
>
> I've seen this situation before, on other servers, so it's not the
first
> time I encounter this. However, it's not 100% reproducible (I mean, if
I
> fill the directory with dozens of thousands of files, I will not certainly
> get this lag after the deletion).
>
> Has anyone seen this and does anyone know how to resolve this ? It's
not
> critical issue, but it makes thing uncomfortable here. One method I'm
aware
> of: you can move the contents of this directory to some other place, then
> delete it and create again. But it's kind of a nasty workaround.
>
>
> Thanks.
>
> Eugene.
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

Have ignored this thread untiul now, but I observed the same behaviour
on mysystems over the last week or so. In my case its an exim spool
directory, which was hugely full as some point (thousands of
files) and now takes an awfully long time to open and list. I delet
and remake them and the problem goes away, but I belive it is the same thing.

I am running 10.3-STABLE, r303832

-pete.

Hi.

On 20.10.2016 18:54, Nicolas Gilles wrote:
> Looks like it's not taking up any processing time, so my guess is
> the lag probably comes from stalled I/O ... bad disk?
Well, I cannot rule this out completely, but first time I've seen this 
lag on this particular server about two months ago, and I guess two 
months is enough time for zfs on a redundant pool to ger errors, but as 
you can see:

]# zpool status
   pool: zroot
  state: ONLINE
status: One or more devices are configured to use a non-native block size.
         Expect reduced performance.
action: Replace affected devices with devices that support the
         configured block size, or migrate data to a properly configured
         pool.
   scan: resilvered 5.74G in 0h31m with 0 errors on Wed Jun  8 11:54:14 2016
config:

         NAME            STATE     READ WRITE CKSUM
         zroot           ONLINE       0     0     0
           mirror-0      ONLINE       0     0     0
             gpt/zroot0  ONLINE       0     0     0  block size: 512B 
configured, 4096B native
             gpt/zroot1  ONLINE       0     0     0

errors: No known data errors

there's none. Yup, disks have different sector size, but this issue 
happened with one particular directory, not all of them. So I guess this 
is irrelevant.
> Does a second "ls" immediately returned (ie. metadata has been
> cached) ?
Nope. Although the lag varies slightly:

         4.79s real              0.00s user              0.02s sys
         5.51s real              0.00s user              0.02s sys
         4.78s real              0.00s user              0.02s sys
         6.88s real              0.00s user              0.02s sys

Thanks.
Eugene.