freebsd stable - Aug 2017 - zfs listing and CPU

Hi,


Why does the zfs listing eat so much of the CPU ?


last pid: 47151;  load averages:  3.97,  6.35, 
6.13                                            up 1+23:21:18 09:15:13
146 processes: 3 running, 142 sleeping, 1 waiting
CPU:  0.0% user,  0.0% nice, 30.5% system,  0.3% interrupt, 69.2% idle
Mem: 44M Active, 360M Inact, 37G Wired, 25G Free
ARC: 32G Total, 14G MFU, 17G MRU, 312M Anon, 803M Header, 523M Other
Swap: 32G Total, 185M Used, 32G Free

   PID USERNAME     THR PRI NICE   SIZE    RES STATE   C   TIME WCPU COMMAND
    11 root          32 155 ki31     0K   512K CPU0    0 1104.1 2666.23% 
idle
     0 root        9880 -16    -     0K   154M swapin 11 314.0H 281.29% 
kernel
    13 root           3  -8    -     0K    48K gread   0  20.9H 28.71% geom
47114 root           1  20    0 40432K  3840K db->db  4   0:05 26.84% zfs
47099 root           1  20    0 40432K  3840K zio->i 17   0:05 26.83% zfs
47106 root           1  20    0 40432K  3840K db->db 21   0:05 26.81% zfs
47150 root           1  20    0 40432K  3428K db->db 13   0:03 26.31% zfs
47141 root           1  20    0 40432K  3428K zio->i 28   0:03 26.31% zfs
47135 root           1  20    0 40432K  3312K g_wait  9   0:03 25.51% zfs
     4 root           7 -16    -     0K   112K -      20 975:01 19.73% cam
     5 root        2494  -8    -     0K 39952K arc_re 18  20.2H 18.58% 
zfskern
    12 root          65 -60    -     0K  1040K WAIT    0  17.8H 15.64% intr
    22 root           2 -16    -     0K    32K psleep  3  66:34 7.31% 
pagedaemon
   590 root          10 -16    -     0K   160K -      21 177:02 2.96% ctl

[...]


This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is 
cloning, and all the others are 'zfs list -t all'. I have like 25 gigs 
of free RAM, do I have any chance of speeding this up using may be some 
caching or some sysctl tuning ? We are using a simple ZFS web API that 
may issue concurrent or sequential listing requests, so as you can see 
they sometimes do stack.


Thanks.

Eugene.

> On Aug 11, 2017, at 2:28 AM, Eugene M. Zheganin <emz at
norma.perm.ru> wrote:
> 
> Why does the zfs listing eat so much of the CPU ?
> 47114 root           1  20    0 40432K  3840K db->db  4   0:05 26.84%
zfs
> 47099 root           1  20    0 40432K  3840K zio->i 17   0:05 26.83%
zfs
> 47106 root           1  20    0 40432K  3840K db->db 21   0:05 26.81%
zfs
> 47150 root           1  20    0 40432K  3428K db->db 13   0:03 26.31%
zfs
> 47141 root           1  20    0 40432K  3428K zio->i 28   0:03 26.31%
zfs
> 47135 root           1  20    0 40432K  3312K g_wait  9   0:03 25.51% zfs
> This is from winter 2017 11-STABLE (r310734), one of the 'zfs'es is
cloning, and all the others are 'zfs list -t all'. I have like 25 gigs
of free RAM, do I have any chance of speeding this up using may be some caching
or some sysctl tuning ? We are using a simple ZFS web API that may issue
concurrent or sequential listing requests, so as you can see they sometimes do
stack.
How many snapshots do you have ? I have only seen this behavior with LOTS (not
hundreds, but thousands) of snapshots.

What does your `iostat -x 1` look like ? I expect that you are probably
saturating your drives with random I/O.