freebsd stable - Oct 2016 - zfs, a directory that used to hold lot of files and listing pause

On 21/10/2016 10:04, Eugene M. Zheganin wrote:
> Hi.
>
> On 21.10.2016 9:22, Steven Hartland wrote:
>> On 21/10/2016 04:52, Eugene M. Zheganin wrote:
>>> Hi.
>>>
>>> On 20.10.2016 21:17, Steven Hartland wrote:
>>>> Do you have atime enabled for the relevant volume?
>>> I do.
>>>>
>>>> If so disable it and see if that helps:
>>>> zfs set atime=off <volume>
>>>>
>>> Nah, it doesn't help at all.
>> As per with Jonathon what does gstat -pd and top -SHz show? 
>
> gstat (while ls'ing):
>
> dT: 1.005s  w: 1.000s
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
> ms/d   %busy Name
>     1     49     49   2948   13.5      0      0    0.0      0 0 0.0   
> 65.0| ada0
>     0     32     32   1798   11.1      0      0    0.0      0 0 0.0   
> 35.3| ada1
>
Averagely busy then on rust.
> gstat (while idling):
>
> dT: 1.003s  w: 1.000s
>  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
> ms/d   %busy Name
>     0      0      0      0    0.0      0      0    0.0      0 0 0.0    
> 0.0| ada0
>     0      2      2    255    0.8      0      0    0.0      0 0 0.0    
> 0.1| ada1
>
> top -SHz output doesn't really differ while ls'ing or idling:
>
> last pid: 12351;  load averages:  0.46,  0.49, 
> 0.46                                           up 39+14:41:02 14:03:05
> 376 processes: 3 running, 354 sleeping, 19 waiting
> CPU:  5.8% user,  0.0% nice, 16.3% system,  0.0% interrupt, 77.9% idle
> Mem: 21M Active, 646M Inact, 931M Wired, 2311M Free
> ARC: 73M Total, 3396K MFU, 21M MRU, 545K Anon, 1292K Header, 47M Other
> Swap: 4096M Total, 4096M Free
>
>   PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
>   600 root        39    0 27564K  5072K nanslp  1 295.0H  24.56% monit
>     0 root       -17    0     0K  2608K -       1  75:24   0.00% 
> kernel{zio_write_issue}
>   767 freeswitch  20    0   139M 31668K uwait   0  48:29   0.00% 
> freeswitch{freeswitch}
>   683 asterisk    20    0   806M   483M uwait   0  41:09   0.00% 
> asterisk{asterisk}
>     0 root        -8    0     0K  2608K -       0  37:43   0.00% 
> kernel{metaslab_group_t}
> [... others lines are just 0% ...]
This looks like you only have ~4Gb ram which is pretty low for ZFS I 
suspect vfs.zfs.prefetch_disable will be 1, which will crash the 
performance.

     Regards
     Steve

On Fri, Oct 21, 2016 at 11:02:57AM +0100, Steven Hartland wrote:
> > Mem: 21M Active, 646M Inact, 931M Wired, 2311M Free
> > ARC: 73M Total, 3396K MFU, 21M MRU, 545K Anon, 1292K Header, 47M Other
> > Swap: 4096M Total, 4096M Free
> >
> >   PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME    WCPU
COMMAND
> >   600 root        39    0 27564K  5072K nanslp  1 295.0H  24.56% monit
> >     0 root       -17    0     0K  2608K -       1  75:24   0.00% 
> > kernel{zio_write_issue}
> >   767 freeswitch  20    0   139M 31668K uwait   0  48:29   0.00% 
> > freeswitch{freeswitch}
> >   683 asterisk    20    0   806M   483M uwait   0  41:09   0.00% 
> > asterisk{asterisk}
> >     0 root        -8    0     0K  2608K -       0  37:43   0.00% 
> > kernel{metaslab_group_t}
> > [... others lines are just 0% ...]
> This looks like you only have ~4Gb ram which is pretty low for ZFS I 
> suspect vfs.zfs.prefetch_disable will be 1, which will crash the 
> performance.
ZFS prefetch affect performance dpeneds of workload (independed of RAM
size): for some workloads wins, for some workloads lose (for my
workload prefetch is lose and manualy disabled with 128GB RAM).

Anyway, this system have only 24MB in ARC by 2.3GB free, this is may
be too low for this workload.