freebsd stable - Oct 2016 - zfs, a directory that used to hold lot of files and listing pause

Hi.

I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation 
on different releases) and a zfs. I also have one directory that used to 
have a lot of (tens of thousands) files. I surely takes a lot of time to 
get a listing of it. But now I have 2 files and a couple of dozens 
directories in it (I sorted files into directories). Surprisingly, 
there's still a lag between "ls" and an output:


===Cut==
# /usr/bin/time -h ls
.recycle        2016-01         2016-04         2016-07 2016-10         
sort-files.sh
2014            2016-02         2016-05         2016-08 ktrace.out      
sort-months.sh
2015            2016-03         2016-06         2016-09 old             
sounds
         5.75s real              0.00s user              0.02s sys

===Cut==

I've seen this situation before, on other servers, so it's not the first
time I encounter this. However, it's not 100% reproducible (I mean, if I 
fill the directory with dozens of thousands of files, I will not 
certainly get this lag after the deletion).

Has anyone seen this and does anyone know how to resolve this ? It's not 
critical issue, but it makes thing uncomfortable here. One method I'm 
aware of: you can move the contents of this directory to some other 
place, then delete it and create again. But it's kind of a nasty workaround.


Thanks.

Eugene.

On Thu, Oct 20, 2016 at 3:47 PM, Eugene M. Zheganin <emz at norma.perm.ru>
wrote:
> Hi.
>
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation on
> different releases) and a zfs. I also have one directory that used to have
a
> lot of (tens of thousands) files. I surely takes a lot of time to get a
> listing of it. But now I have 2 files and a couple of dozens directories in
> it (I sorted files into directories). Surprisingly, there's still a lag
> between "ls" and an output:
>
>
> ===Cut==>
> # /usr/bin/time -h ls
> .recycle        2016-01         2016-04         2016-07 2016-10
> sort-files.sh
> 2014            2016-02         2016-05         2016-08 ktrace.out
> sort-months.sh
> 2015            2016-03         2016-06         2016-09 old
> sounds
>         5.75s real              0.00s user              0.02s sys
Looks like it's not taking up any processing time, so my guess is
the lag probably comes from stalled I/O ... bad disk?

Does a second "ls" immediately returned (ie. metadata has been
cached) ?
>
> ===Cut==>
>
> I've seen this situation before, on other servers, so it's not the
first
> time I encounter this. However, it's not 100% reproducible (I mean, if
I
> fill the directory with dozens of thousands of files, I will not certainly
> get this lag after the deletion).
>
> Has anyone seen this and does anyone know how to resolve this ? It's
not
> critical issue, but it makes thing uncomfortable here. One method I'm
aware
> of: you can move the contents of this directory to some other place, then
> delete it and create again. But it's kind of a nasty workaround.
>
>
> Thanks.
>
> Eugene.
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

Eugene M. Zheganin wrote on 2016/10/20 15:47:
> Hi.
>
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation
> on different releases) and a zfs. I also have one directory that used to
> have a lot of (tens of thousands) files. I surely takes a lot of time to
> get a listing of it. But now I have 2 files and a couple of dozens
> directories in it (I sorted files into directories). Surprisingly,
> there's still a lag between "ls" and an output:
[...]
> I've seen this situation before, on other servers, so it's not the
first
> time I encounter this. However, it's not 100% reproducible (I mean, if
I
> fill the directory with dozens of thousands of files, I will not
> certainly get this lag after the deletion).
What about snapshots? Are there any snapshots on this filesystem?

Or scrub running in the background?

Miroslav Lachman

Am Donnerstag, 20. Oktober 2016 schrieb Eugene M.
Zheganin:
> Hi.
> 
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation 
> on different releases) and a zfs. I also have one directory that used to 
> have a lot of (tens of thousands) files. I surely takes a lot of time to 
> get a listing of it. But now I have 2 files and a couple of dozens 
> directories in it (I sorted files into directories). Surprisingly, 
> there's still a lag between "ls" and an output:
> 
> 
> ===Cut==> 
> # /usr/bin/time -h ls
> .recycle        2016-01         2016-04         2016-07 2016-10         
> sort-files.sh
> 2014            2016-02         2016-05         2016-08 ktrace.out      
> sort-months.sh
> 2015            2016-03         2016-06         2016-09 old             
> sounds
>          5.75s real              0.00s user              0.02s sys
> 
> ===Cut==> 
> 
> I've seen this situation before, on other servers, so it's not the
first
> time I encounter this. However, it's not 100% reproducible (I mean, if
I
> fill the directory with dozens of thousands of files, I will not 
> certainly get this lag after the deletion).
> 
> Has anyone seen this and does anyone know how to resolve this ? It's
not
> critical issue, but it makes thing uncomfortable here. One method I'm 
> aware of: you can move the contents of this directory to some other 
> place, then delete it and create again. But it's kind of a nasty
workaround.

Hi!

I've the same issue, but only if the ZFS resides on a LSI MegaRaid and one
RAID0 for each disk.

Nik


-- 
Please do not email me anything that you are not comfortable also sharing with
the NSA.

Do you have atime enabled for the relevant volume?

If so disable it and see if that helps:
zfs set atime=off <volume>

     Regards
     Steve

On 20/10/2016 14:47, Eugene M. Zheganin wrote:
> Hi.
>
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation 
> on different releases) and a zfs. I also have one directory that used 
> to have a lot of (tens of thousands) files. I surely takes a lot of 
> time to get a listing of it. But now I have 2 files and a couple of 
> dozens directories in it (I sorted files into directories). 
> Surprisingly, there's still a lag between "ls" and an output:
>
>
> ===Cut==>
> # /usr/bin/time -h ls
> .recycle        2016-01         2016-04         2016-07 
> 2016-10         sort-files.sh
> 2014            2016-02         2016-05         2016-08 
> ktrace.out      sort-months.sh
> 2015            2016-03         2016-06         2016-09 
> old             sounds
>         5.75s real              0.00s user              0.02s sys
>
> ===Cut==>
>
> I've seen this situation before, on other servers, so it's not the 
> first time I encounter this. However, it's not 100% reproducible (I 
> mean, if I fill the directory with dozens of thousands of files, I 
> will not certainly get this lag after the deletion).
>
> Has anyone seen this and does anyone know how to resolve this ? It's 
> not critical issue, but it makes thing uncomfortable here. One method 
> I'm aware of: you can move the contents of this directory to some 
> other place, then delete it and create again. But it's kind of a nasty 
> workaround.
>
>
> Thanks.
>
> Eugene.
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"

Eugene M. Zheganin wrote:
> Hi.
>
> I have FreeBSD 10.2-STABLE r289293 (but I have observed this situation
> on different releases) and a zfs. I also have one directory that used to
> have a lot of (tens of thousands) files. I surely takes a lot of time to
> get a listing of it. But now I have 2 files and a couple of dozens
> directories in it (I sorted files into directories). Surprisingly,
> there's still a lag between "ls" and an output:
I see this on my pgsql_tmp dirs (where Postgres stores intermediate
query data that gets too big for mem - usually lots of files) - in
normal operation these dirs are completely empty, but make heavy disk
activity (even writing!) when doing ls.
Seems normal, I dont care as long as the thing is stable. One would need
to check how ZFS stores directories and what kind of fragmentation can
happen there. Or wait for some future feature that would do
housekeeping. ;)