Eugene M. Zheganin
2018-Nov-20 09:53 UTC
plenty of memory, but system us intensively swapping
Hello,
I have a recent FreeBSD 11-STABLE which is mainly used as an iSCSI
target. The system has 64G of RAM but is swapping intensively. Yup,
about of half of the memory is used as ZFS ARC (isn't capped in
loader.conf), and another half is eaten by the kernel, but it oly uses
only about half of it (thus 25% of the total amount).
Could this be tweaked by some sysctl oids (I suppose not, but worth asking).
top, vmstat 1 snapshots and zfs-stats -a are listed below.
Thanks.
[root at san01:nginx/vhost.d]# vmstat 1
procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
0 0 38 23G 609M 1544 68 118 64 895 839 0 0 3644 2678 649
0 13 87
0 0 53 23G 601M 1507 185 742 315 1780 33523 651 664 56438 785
476583 0 28 72
0 0 53 23G 548M 1727 330 809 380 2377 33256 758 763 55555 1273
468545 0 26 73
0 0 53 23G 528M 1702 239 660 305 1347 32335 611 631 59962 1025
490365 0 22 78
0 0 52 23G 854M 2409 309 693 203 97943 16944 525 515 64309 1570
540533 0 29 71
3 0 54 23G 1.1G 2756 639 641 149 124049 19531 542 538 64777 1576
553946 0 35 65
0 0 53 23G 982M 1694 236 680 282 2754 35602 597 603 66540 1385
583687 0 28 72
0 0 41 23G 867M 1882 223 767 307 1162 34936 682 638 67284 780
568818 0 33 67
0 0 39 23G 769M 1542 167 673 336 1187 35123 646 610 65925 1176
551623 0 23 77
2 0 41 23G 700M 3602 535 688 327 2192 37109 622 594 65862 4256
518934 0 33 67
0 0 54 23G 650M 2957 219 726 464 4838 36464 852 868 65384 4110
558132 1 37 62
0 0 54 23G 641M 1576 245 730 344 1139 33681 740 679 67216 970
560379 0 31 69
[root at san01:nginx/vhost.d]# top
last pid: 55190; load averages: 11.32, 12.15, 10.76
up 10+16:05:14 14:38:58
101 processes: 1 running, 100 sleeping
CPU: 0.2% user, 0.0% nice, 28.9% system, 1.6% interrupt, 69.3% idle
Mem: 85M Active, 1528K Inact, 12K Laundry, 62G Wired, 540M Free
ARC: 31G Total, 19G MFU, 6935M MRU, 2979M Anon, 556M Header, 1046M Other
25G Compressed, 34G Uncompressed, 1.39:1 Ratio
Swap: 32G Total, 1186M Used, 31G Free, 3% Inuse, 7920K In, 3752K Out
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
40132 root 131 52 0 3152M 75876K uwait 14 36:59 6.10% java
55142 root 1 20 0 7904K 2728K CPU20 20 0:00 0.72% top
20026 root 1 20 0 106M 5676K nanslp 28 1:23 0.60% gstat
53642 root 1 20 0 7904K 2896K select 14 0:03 0.58% top
977 zfsreplica 1 20 0 30300K 3568K kqread 21 4:00 0.42%
uwsgi
968 zfsreplica 1 20 0 30300K 2224K swread 11 2:03 0.21%
uwsgi
973 zfsreplica 1 20 0 30300K 2264K swread 13 12:26 0.13%
uwsgi
53000 www 1 20 0 23376K 1372K kqread 24 0:00 0.05% nginx
1292 root 1 20 0 6584K 2040K select 29 0:23 0.04%
blacklistd
776 zabbix 1 20 0 12408K 4236K nanslp 26 4:42 0.03%
zabbix_agentd
1289 root 1 20 0 67760K 5148K select 13 9:50 0.03%
bsnmpd
777 zabbix 1 20 0 12408K 1408K select 25 5:06 0.03%
zabbix_agentd
785 zfsreplica 1 20 0 27688K 3960K kqread 28 2:04 0.02%
uwsgi
975 zfsreplica 1 20 0 30300K 464K kqread 18 2:33 0.02%
uwsgi
974 zfsreplica 1 20 0 30300K 480K kqread 30 3:39 0.02%
uwsgi
965 zfsreplica 1 20 0 30300K 464K kqread 4 3:23 0.02%
uwsgi
976 zfsreplica 1 20 0 30300K 464K kqread 14 2:59 0.01%
uwsgi
972 zfsreplica 1 20 0 30300K 464K kqread 10 2:57 0.01%
uwsgi
963 zfsreplica 1 20 0 30300K 460K kqread 3 2:45 0.01%
uwsgi
971 zfsreplica 1 20 0 30300K 464K kqread 13 3:16 0.01%
uwsgi
69644 emz 1 20 0 13148K 4596K select 24 0:05 0.01% sshd
18203 vryabov 1 20 0 13148K 4624K select 9 0:02 0.01% sshd
636 root 1 20 0 6412K 1884K select 17 4:10 0.01%
syslogd
51266 emz 1 20 0 13148K 4576K select 5 0:00 0.01% sshd
964 zfsreplica 1 20 0 30300K 460K kqread 18 11:02 0.01%
uwsgi
962 zfsreplica 1 20 0 30300K 460K kqread 28 6:56 0.01%
uwsgi
969 zfsreplica 1 20 0 30300K 464K kqread 12 2:07 0.01%
uwsgi
967 zfsreplica 1 20 0 30300K 464K kqread 27 5:18 0.01%
uwsgi
970 zfsreplica 1 20 0 30300K 464K kqread 0 4:25 0.01%
uwsgi
966 zfsreplica 1 22 0 30300K 468K kqread 14 4:29 0.01%
uwsgi
53001 www 1 20 0 23376K 1256K kqread 10 0:00 0.01% nginx
791 zfsreplica 1 20 0 27664K 4244K kqread 17 1:34 0.01%
uwsgi
52431 root 1 20 0 17132K 4492K select 21 0:00 0.01% mc
70013 root 1 20 0 17132K 4492K select 4 0:03 0.01% mc
870 root 1 20 0 12448K 12544K select 19 0:51 0.01% ntpd
[root at san01:nginx/vhost.d]# zfs-stats -a
------------------------------------------------------------------------
ZFS Subsystem Report Tue Nov 20 14:39:05 2018
------------------------------------------------------------------------
System Information:
Kernel Version: 1102503 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5
FreeBSD 11.2-STABLE #0 r340287M: Fri Nov 9 22:23:22 +05 2018 emz
14:39 up 10 days, 16:05, 5 users, load averages: 10,96 12,05 10,74
------------------------------------------------------------------------
System Memory:
0.14% 90.20 MiB Active, 0.01% 8.62 MiB Inact
98.97% 61.57 GiB Wired, 0.00% 0 Cache
0.88% 560.02 MiB Free, -0.00% -184320 Bytes Gap
Real Installed: 64.00 GiB
Real Available: 99.77% 63.85 GiB
Real Managed: 97.43% 62.21 GiB
Logical Total: 64.00 GiB
Logical Used: 99.13% 63.44 GiB
Logical Free: 0.87% 568.64 MiB
Kernel Memory: 22.77 GiB
Data: 99.84% 22.73 GiB
Text: 0.16% 36.52 MiB
Kernel Memory Map: 62.21 GiB
Size: 46.85% 29.15 GiB
Free: 53.15% 33.06 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 5.61b
Recycle Misses: 0
Mutex Misses: 64.25m
Evict Skips: 98.33m
ARC Size: 50.02% 30.62 GiB
Target Size: (Adaptive) 50.02% 30.62 GiB
Min Size (Hard Limit): 12.50% 7.65 GiB
Max Size (High Water): 8:1 61.21 GiB
ARC Size Breakdown:
Recently Used Cache Size: 50.69% 15.52 GiB
Frequently Used Cache Size: 49.31% 15.10 GiB
ARC Hash Breakdown:
Elements Max: 8.35m
Elements Current: 30.08% 2.51m
Collisions: 2.18b
Chain Max: 10
Chains: 308.03k
------------------------------------------------------------------------
ARC Efficiency: 48.52b
Cache Hit Ratio: 84.49% 40.99b
Cache Miss Ratio: 15.51% 7.53b
Actual Hit Ratio: 84.19% 40.85b
Data Demand Efficiency: 83.84% 13.06b
Data Prefetch Efficiency: 40.66% 1.42b
CACHE HITS BY CACHE LIST:
Most Recently Used: 15.11% 6.19b
Most Frequently Used: 84.54% 34.66b
Most Recently Used Ghost: 0.97% 396.23m
Most Frequently Used Ghost: 0.18% 75.49m
CACHE HITS BY DATA TYPE:
Demand Data: 26.70% 10.95b
Prefetch Data: 1.41% 576.89m
Demand Metadata: 70.83% 29.04b
Prefetch Metadata: 1.06% 434.19m
CACHE MISSES BY DATA TYPE:
Demand Data: 28.04% 2.11b
Prefetch Data: 11.18% 841.81m
Demand Metadata: 60.37% 4.54b
Prefetch Metadata: 0.40% 30.27m
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 9.20b
Hit Ratio: 5.80% 533.41m
Miss Ratio: 94.20% 8.66b
Colinear: 0
Hit Ratio: 100.00% 0
Miss Ratio: 100.00% 0
Stride: 0
Hit Ratio: 100.00% 0
Miss Ratio: 100.00% 0
DMU Misc:
Reclaim: 0
Successes: 100.00% 0
Failures: 100.00% 0
Streams: 0
+Resets: 100.00% 0
-Resets: 100.00% 0
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 4422
vm.kmem_size 66799345664
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 1319413950874
vfs.zfs.trim.max_interval 1
vfs.zfs.trim.timeout 30
vfs.zfs.trim.txg_delay 32
vfs.zfs.trim.enabled 0
vfs.zfs.vol.immediate_write_sz 131072
vfs.zfs.vol.unmap_sync_enabled 0
vfs.zfs.vol.unmap_enabled 1
vfs.zfs.vol.recursive 0
vfs.zfs.vol.mode 1
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 5000
vfs.zfs.version.acl 1
vfs.zfs.version.ioctl 7
vfs.zfs.debug 0
vfs.zfs.super_owner 0
vfs.zfs.immediate_write_sz 32768
vfs.zfs.sync_pass_rewrite 2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.zio.dva_throttle_enabled 1
vfs.zfs.zio.exclude_metadata 0
vfs.zfs.zio.use_uma 1
vfs.zfs.zil_slog_bulk 786432
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.standard_sm_blksz 131072
vfs.zfs.dtl_sm_blksz 4096
vfs.zfs.min_auto_ashift 9
vfs.zfs.max_auto_ashift 13
vfs.zfs.vdev.trim_max_pending 10000
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.vdev.def_queue_depth 32
vfs.zfs.vdev.queue_depth_pct 1000
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 1048576
vfs.zfs.vdev.trim_max_active 64
vfs.zfs.vdev.trim_min_active 1
vfs.zfs.vdev.scrub_max_active 2
vfs.zfs.vdev.scrub_min_active 1
vfs.zfs.vdev.async_write_max_active 10
vfs.zfs.vdev.async_write_min_active 1
vfs.zfs.vdev.async_read_max_active 3
vfs.zfs.vdev.async_read_min_active 1
vfs.zfs.vdev.sync_write_max_active 10
vfs.zfs.vdev.sync_write_min_active 10
vfs.zfs.vdev.sync_read_max_active 10
vfs.zfs.vdev.sync_read_min_active 10
vfs.zfs.vdev.max_active 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent60
vfs.zfs.vdev.async_write_active_min_dirty_percent30
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.mirror.non_rotating_inc 0
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.rotating_seek_inc 5
vfs.zfs.vdev.mirror.rotating_inc 0
vfs.zfs.vdev.trim_on_init 1
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.default_ms_shift 29
vfs.zfs.vdev.min_ms_count 16
vfs.zfs.vdev.max_ms_count 200
vfs.zfs.txg.timeout 5
vfs.zfs.space_map_ibs 14
vfs.zfs.spa_allocators 4
vfs.zfs.spa_min_slop 134217728
vfs.zfs.spa_slop_shift 5
vfs.zfs.spa_asize_inflation 24
vfs.zfs.deadman_enabled 1
vfs.zfs.deadman_checktime_ms 5000
vfs.zfs.deadman_synctime_ms 1000000
vfs.zfs.debug_flags 0
vfs.zfs.debugflags 0
vfs.zfs.recover 0
vfs.zfs.spa_load_verify_data 1
vfs.zfs.spa_load_verify_metadata 1
vfs.zfs.spa_load_verify_maxinflight 10000
vfs.zfs.max_missing_tvds_scan 0
vfs.zfs.max_missing_tvds_cachefile 2
vfs.zfs.max_missing_tvds 0
vfs.zfs.spa_load_print_vdev_tree 0
vfs.zfs.ccw_retry_interval 300
vfs.zfs.check_hostid 1
vfs.zfs.mg_fragmentation_threshold 85
vfs.zfs.mg_noalloc_threshold 0
vfs.zfs.condense_pct 200
vfs.zfs.metaslab_sm_blksz 4096
vfs.zfs.metaslab.bias_enabled 1
vfs.zfs.metaslab.lba_weighting_enabled 1
vfs.zfs.metaslab.fragmentation_factor_enabled1
vfs.zfs.metaslab.preload_enabled 1
vfs.zfs.metaslab.preload_limit 3
vfs.zfs.metaslab.unload_delay 8
vfs.zfs.metaslab.load_pct 50
vfs.zfs.metaslab.min_alloc_size 33554432
vfs.zfs.metaslab.df_free_pct 4
vfs.zfs.metaslab.df_alloc_threshold 131072
vfs.zfs.metaslab.debug_unload 0
vfs.zfs.metaslab.debug_load 0
vfs.zfs.metaslab.fragmentation_threshold70
vfs.zfs.metaslab.force_ganging 16777217
vfs.zfs.free_bpobj_enabled 1
vfs.zfs.free_max_blocks -1
vfs.zfs.zfs_scan_checkpoint_interval 7200
vfs.zfs.zfs_scan_legacy 0
vfs.zfs.no_scrub_prefetch 0
vfs.zfs.no_scrub_io 0
vfs.zfs.resilver_min_time_ms 3000
vfs.zfs.free_min_time_ms 1000
vfs.zfs.scan_min_time_ms 1000
vfs.zfs.scan_idle 50
vfs.zfs.scrub_delay 4
vfs.zfs.resilver_delay 2
vfs.zfs.top_maxinflight 32
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.max_idistance 67108864
vfs.zfs.zfetch.max_distance 8388608
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.delay_scale 500000
vfs.zfs.delay_min_dirty_percent 60
vfs.zfs.dirty_data_sync 67108864
vfs.zfs.dirty_data_max_percent 10
vfs.zfs.dirty_data_max_max 4294967296
vfs.zfs.dirty_data_max 4294967296
vfs.zfs.max_recordsize 1048576
vfs.zfs.default_ibs 17
vfs.zfs.default_bs 9
vfs.zfs.send_holes_without_birth_time 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.per_txg_dirty_frees_percent 30
vfs.zfs.nopwrite_enabled 1
vfs.zfs.dedup.prefetch 1
vfs.zfs.dbuf_cache_lowater_pct 10
vfs.zfs.dbuf_cache_hiwater_pct 10
vfs.zfs.dbuf_metadata_cache_overflow 0
vfs.zfs.dbuf_metadata_cache_shift 6
vfs.zfs.dbuf_cache_shift 5
vfs.zfs.dbuf_metadata_cache_max_bytes 1026962560
vfs.zfs.dbuf_cache_max_bytes 2053925120
vfs.zfs.arc_min_prescient_prefetch_ms 6
vfs.zfs.arc_min_prefetch_ms 1
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_esize 1910587392
vfs.zfs.mfu_ghost_metadata_esize 5158840832
vfs.zfs.mfu_ghost_size 7069428224
vfs.zfs.mfu_data_esize 17620227072
vfs.zfs.mfu_metadata_esize 950300160
vfs.zfs.mfu_size 20773338624
vfs.zfs.mru_ghost_data_esize 6989578240
vfs.zfs.mru_ghost_metadata_esize 18479132160
vfs.zfs.mru_ghost_size 25468710400
vfs.zfs.mru_data_esize 4455460352
vfs.zfs.mru_metadata_esize 70236672
vfs.zfs.mru_size 7413314560
vfs.zfs.anon_data_esize 0
vfs.zfs.anon_metadata_esize 0
vfs.zfs.anon_size 3040037888
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 16431400960
vfs.zfs.arc_free_target 113124
vfs.zfs.arc_kmem_cache_reap_retry_ms 0
vfs.zfs.compressed_arc_enabled 1
vfs.zfs.arc_grow_retry 60
vfs.zfs.arc_shrink_shift 7
vfs.zfs.arc_average_blocksize 8192
vfs.zfs.arc_no_grow_shift 5
vfs.zfs.arc_min 8215700480
vfs.zfs.arc_max 65725603840
vfs.zfs.abd_chunk_size 4096
vfs.zfs.abd_scatter_enabled 1
------------------------------------------------------------------------
Trond Endrestøl
2018-Nov-20 10:12 UTC
plenty of memory, but system us intensively swapping
On Tue, 20 Nov 2018 14:53+0500, Eugene M. Zheganin wrote:> Hello, > > > I have a recent FreeBSD 11-STABLE which is mainly used as an iSCSI target. The > system has 64G of RAM but is swapping intensively. Yup, about of half of the > memory is used as ZFS ARC (isn't capped in loader.conf), and another half is > eaten by the kernel, but it oly uses only about half of it (thus 25% of the > total amount). > > Could this be tweaked by some sysctl oids (I suppose not, but worth asking).On freebsd-hackers the other day, https://lists.freebsd.org/pipermail/freebsd-hackers/2018-November/053575.html, it was suggested to set vm.pageout_update_period=0. This sysctl is at 600 initially. ZFS' ARC needs to be capped, otherwise it will eat most, if not all, of your memory.> top, vmstat 1 snapshots and zfs-stats -a are listed below. > > > Thanks. > > > [root at san01:nginx/vhost.d]# vmstat 1 > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id > 0 0 38 23G 609M 1544 68 118 64 895 839 0 0 3644 2678 649 0 13 > 87 > 0 0 53 23G 601M 1507 185 742 315 1780 33523 651 664 56438 785 476583 0 28 > 72 > 0 0 53 23G 548M 1727 330 809 380 2377 33256 758 763 55555 1273 468545 0 > 26 73 > 0 0 53 23G 528M 1702 239 660 305 1347 32335 611 631 59962 1025 490365 0 > 22 78 > 0 0 52 23G 854M 2409 309 693 203 97943 16944 525 515 64309 1570 540533 0 > 29 71 > 3 0 54 23G 1.1G 2756 639 641 149 124049 19531 542 538 64777 1576 553946 0 > 35 65 > 0 0 53 23G 982M 1694 236 680 282 2754 35602 597 603 66540 1385 583687 0 > 28 72 > 0 0 41 23G 867M 1882 223 767 307 1162 34936 682 638 67284 780 568818 0 33 > 67 > 0 0 39 23G 769M 1542 167 673 336 1187 35123 646 610 65925 1176 551623 0 > 23 77 > 2 0 41 23G 700M 3602 535 688 327 2192 37109 622 594 65862 4256 518934 0 > 33 67 > 0 0 54 23G 650M 2957 219 726 464 4838 36464 852 868 65384 4110 558132 1 > 37 62 > 0 0 54 23G 641M 1576 245 730 344 1139 33681 740 679 67216 970 560379 0 31 > 69 > > > [root at san01:nginx/vhost.d]# top > last pid: 55190; load averages: 11.32, 12.15, 10.76 > up 10+16:05:14 14:38:58 > 101 processes: 1 running, 100 sleeping > CPU: 0.2% user, 0.0% nice, 28.9% system, 1.6% interrupt, 69.3% idle > Mem: 85M Active, 1528K Inact, 12K Laundry, 62G Wired, 540M Free > ARC: 31G Total, 19G MFU, 6935M MRU, 2979M Anon, 556M Header, 1046M Other > 25G Compressed, 34G Uncompressed, 1.39:1 Ratio > Swap: 32G Total, 1186M Used, 31G Free, 3% Inuse, 7920K In, 3752K Out > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 40132 root 131 52 0 3152M 75876K uwait 14 36:59 6.10% java > 55142 root 1 20 0 7904K 2728K CPU20 20 0:00 0.72% top > 20026 root 1 20 0 106M 5676K nanslp 28 1:23 0.60% gstat > 53642 root 1 20 0 7904K 2896K select 14 0:03 0.58% top > 977 zfsreplica 1 20 0 30300K 3568K kqread 21 4:00 0.42% uwsgi > 968 zfsreplica 1 20 0 30300K 2224K swread 11 2:03 0.21% uwsgi > 973 zfsreplica 1 20 0 30300K 2264K swread 13 12:26 0.13% uwsgi > 53000 www 1 20 0 23376K 1372K kqread 24 0:00 0.05% nginx > 1292 root 1 20 0 6584K 2040K select 29 0:23 0.04% > blacklistd > 776 zabbix 1 20 0 12408K 4236K nanslp 26 4:42 0.03% > zabbix_agentd > 1289 root 1 20 0 67760K 5148K select 13 9:50 0.03% bsnmpd > 777 zabbix 1 20 0 12408K 1408K select 25 5:06 0.03% > zabbix_agentd > 785 zfsreplica 1 20 0 27688K 3960K kqread 28 2:04 0.02% uwsgi > 975 zfsreplica 1 20 0 30300K 464K kqread 18 2:33 0.02% uwsgi > 974 zfsreplica 1 20 0 30300K 480K kqread 30 3:39 0.02% uwsgi > 965 zfsreplica 1 20 0 30300K 464K kqread 4 3:23 0.02% uwsgi > 976 zfsreplica 1 20 0 30300K 464K kqread 14 2:59 0.01% uwsgi > 972 zfsreplica 1 20 0 30300K 464K kqread 10 2:57 0.01% uwsgi > 963 zfsreplica 1 20 0 30300K 460K kqread 3 2:45 0.01% uwsgi > 971 zfsreplica 1 20 0 30300K 464K kqread 13 3:16 0.01% uwsgi > 69644 emz 1 20 0 13148K 4596K select 24 0:05 0.01% sshd > 18203 vryabov 1 20 0 13148K 4624K select 9 0:02 0.01% sshd > 636 root 1 20 0 6412K 1884K select 17 4:10 0.01% syslogd > 51266 emz 1 20 0 13148K 4576K select 5 0:00 0.01% sshd > 964 zfsreplica 1 20 0 30300K 460K kqread 18 11:02 0.01% uwsgi > 962 zfsreplica 1 20 0 30300K 460K kqread 28 6:56 0.01% uwsgi > 969 zfsreplica 1 20 0 30300K 464K kqread 12 2:07 0.01% uwsgi > 967 zfsreplica 1 20 0 30300K 464K kqread 27 5:18 0.01% uwsgi > 970 zfsreplica 1 20 0 30300K 464K kqread 0 4:25 0.01% uwsgi > 966 zfsreplica 1 22 0 30300K 468K kqread 14 4:29 0.01% uwsgi > 53001 www 1 20 0 23376K 1256K kqread 10 0:00 0.01% nginx > 791 zfsreplica 1 20 0 27664K 4244K kqread 17 1:34 0.01% uwsgi > 52431 root 1 20 0 17132K 4492K select 21 0:00 0.01% mc > 70013 root 1 20 0 17132K 4492K select 4 0:03 0.01% mc > 870 root 1 20 0 12448K 12544K select 19 0:51 0.01% ntpd > [root at san01:nginx/vhost.d]# zfs-stats -a > ------------------------------------------------------------------------ > ZFS Subsystem Report Tue Nov 20 14:39:05 2018 > ------------------------------------------------------------------------ > System Information: > Kernel Version: 1102503 (osreldate) > Hardware Platform: amd64 > Processor Architecture: amd64 > ZFS Storage pool Version: 5000 > ZFS Filesystem Version: 5 > FreeBSD 11.2-STABLE #0 r340287M: Fri Nov 9 22:23:22 +05 2018 emz > 14:39 up 10 days, 16:05, 5 users, load averages: 10,96 12,05 10,74 > ------------------------------------------------------------------------ > System Memory: > 0.14% 90.20 MiB Active, 0.01% 8.62 MiB Inact > 98.97% 61.57 GiB Wired, 0.00% 0 Cache > 0.88% 560.02 MiB Free, -0.00% -184320 Bytes Gap > Real Installed: 64.00 GiB > Real Available: 99.77% 63.85 GiB > Real Managed: 97.43% 62.21 GiB > Logical Total: 64.00 GiB > Logical Used: 99.13% 63.44 GiB > Logical Free: 0.87% 568.64 MiB > Kernel Memory: 22.77 GiB > Data: 99.84% 22.73 GiB > Text: 0.16% 36.52 MiB > Kernel Memory Map: 62.21 GiB > Size: 46.85% 29.15 GiB > Free: 53.15% 33.06 GiB > ------------------------------------------------------------------------ > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > ARC Misc: > Deleted: 5.61b > Recycle Misses: 0 > Mutex Misses: 64.25m > Evict Skips: 98.33m > ARC Size: 50.02% 30.62 GiB > Target Size: (Adaptive) 50.02% 30.62 GiB > Min Size (Hard Limit): 12.50% 7.65 GiB > Max Size (High Water): 8:1 61.21 GiB > ARC Size Breakdown: > Recently Used Cache Size: 50.69% 15.52 GiB > Frequently Used Cache Size: 49.31% 15.10 GiB > ARC Hash Breakdown: > Elements Max: 8.35m > Elements Current: 30.08% 2.51m > Collisions: 2.18b > Chain Max: 10 > Chains: 308.03k > ------------------------------------------------------------------------ > ARC Efficiency: 48.52b > Cache Hit Ratio: 84.49% 40.99b > Cache Miss Ratio: 15.51% 7.53b > Actual Hit Ratio: 84.19% 40.85b > Data Demand Efficiency: 83.84% 13.06b > Data Prefetch Efficiency: 40.66% 1.42b > CACHE HITS BY CACHE LIST: > Most Recently Used: 15.11% 6.19b > Most Frequently Used: 84.54% 34.66b > Most Recently Used Ghost: 0.97% 396.23m > Most Frequently Used Ghost: 0.18% 75.49m > CACHE HITS BY DATA TYPE: > Demand Data: 26.70% 10.95b > Prefetch Data: 1.41% 576.89m > Demand Metadata: 70.83% 29.04b > Prefetch Metadata: 1.06% 434.19m > CACHE MISSES BY DATA TYPE: > Demand Data: 28.04% 2.11b > Prefetch Data: 11.18% 841.81m > Demand Metadata: 60.37% 4.54b > Prefetch Metadata: 0.40% 30.27m > ------------------------------------------------------------------------ > L2ARC is disabled > ------------------------------------------------------------------------ > File-Level Prefetch: (HEALTHY) > DMU Efficiency: 9.20b > Hit Ratio: 5.80% 533.41m > Miss Ratio: 94.20% 8.66b > Colinear: 0 > Hit Ratio: 100.00% 0 > Miss Ratio: 100.00% 0 > Stride: 0 > Hit Ratio: 100.00% 0 > Miss Ratio: 100.00% 0 > DMU Misc: > Reclaim: 0 > Successes: 100.00% 0 > Failures: 100.00% 0 > Streams: 0 > +Resets: 100.00% 0 > -Resets: 100.00% 0 > Bogus: 0 > ------------------------------------------------------------------------ > VDEV cache is disabled > ------------------------------------------------------------------------ > ZFS Tunables (sysctl): > kern.maxusers 4422 > vm.kmem_size 66799345664 > vm.kmem_size_scale 1 > vm.kmem_size_min 0 > vm.kmem_size_max 1319413950874 > vfs.zfs.trim.max_interval 1 > vfs.zfs.trim.timeout 30 > vfs.zfs.trim.txg_delay 32 > vfs.zfs.trim.enabled 0 > vfs.zfs.vol.immediate_write_sz 131072 > vfs.zfs.vol.unmap_sync_enabled 0 > vfs.zfs.vol.unmap_enabled 1 > vfs.zfs.vol.recursive 0 > vfs.zfs.vol.mode 1 > vfs.zfs.version.zpl 5 > vfs.zfs.version.spa 5000 > vfs.zfs.version.acl 1 > vfs.zfs.version.ioctl 7 > vfs.zfs.debug 0 > vfs.zfs.super_owner 0 > vfs.zfs.immediate_write_sz 32768 > vfs.zfs.sync_pass_rewrite 2 > vfs.zfs.sync_pass_dont_compress 5 > vfs.zfs.sync_pass_deferred_free 2 > vfs.zfs.zio.dva_throttle_enabled 1 > vfs.zfs.zio.exclude_metadata 0 > vfs.zfs.zio.use_uma 1 > vfs.zfs.zil_slog_bulk 786432 > vfs.zfs.cache_flush_disable 0 > vfs.zfs.zil_replay_disable 0 > vfs.zfs.standard_sm_blksz 131072 > vfs.zfs.dtl_sm_blksz 4096 > vfs.zfs.min_auto_ashift 9 > vfs.zfs.max_auto_ashift 13 > vfs.zfs.vdev.trim_max_pending 10000 > vfs.zfs.vdev.bio_delete_disable 0 > vfs.zfs.vdev.bio_flush_disable 0 > vfs.zfs.vdev.def_queue_depth 32 > vfs.zfs.vdev.queue_depth_pct 1000 > vfs.zfs.vdev.write_gap_limit 4096 > vfs.zfs.vdev.read_gap_limit 32768 > vfs.zfs.vdev.aggregation_limit 1048576 > vfs.zfs.vdev.trim_max_active 64 > vfs.zfs.vdev.trim_min_active 1 > vfs.zfs.vdev.scrub_max_active 2 > vfs.zfs.vdev.scrub_min_active 1 > vfs.zfs.vdev.async_write_max_active 10 > vfs.zfs.vdev.async_write_min_active 1 > vfs.zfs.vdev.async_read_max_active 3 > vfs.zfs.vdev.async_read_min_active 1 > vfs.zfs.vdev.sync_write_max_active 10 > vfs.zfs.vdev.sync_write_min_active 10 > vfs.zfs.vdev.sync_read_max_active 10 > vfs.zfs.vdev.sync_read_min_active 10 > vfs.zfs.vdev.max_active 1000 > vfs.zfs.vdev.async_write_active_max_dirty_percent60 > vfs.zfs.vdev.async_write_active_min_dirty_percent30 > vfs.zfs.vdev.mirror.non_rotating_seek_inc1 > vfs.zfs.vdev.mirror.non_rotating_inc 0 > vfs.zfs.vdev.mirror.rotating_seek_offset1048576 > vfs.zfs.vdev.mirror.rotating_seek_inc 5 > vfs.zfs.vdev.mirror.rotating_inc 0 > vfs.zfs.vdev.trim_on_init 1 > vfs.zfs.vdev.cache.bshift 16 > vfs.zfs.vdev.cache.size 0 > vfs.zfs.vdev.cache.max 16384 > vfs.zfs.vdev.default_ms_shift 29 > vfs.zfs.vdev.min_ms_count 16 > vfs.zfs.vdev.max_ms_count 200 > vfs.zfs.txg.timeout 5 > vfs.zfs.space_map_ibs 14 > vfs.zfs.spa_allocators 4 > vfs.zfs.spa_min_slop 134217728 > vfs.zfs.spa_slop_shift 5 > vfs.zfs.spa_asize_inflation 24 > vfs.zfs.deadman_enabled 1 > vfs.zfs.deadman_checktime_ms 5000 > vfs.zfs.deadman_synctime_ms 1000000 > vfs.zfs.debug_flags 0 > vfs.zfs.debugflags 0 > vfs.zfs.recover 0 > vfs.zfs.spa_load_verify_data 1 > vfs.zfs.spa_load_verify_metadata 1 > vfs.zfs.spa_load_verify_maxinflight 10000 > vfs.zfs.max_missing_tvds_scan 0 > vfs.zfs.max_missing_tvds_cachefile 2 > vfs.zfs.max_missing_tvds 0 > vfs.zfs.spa_load_print_vdev_tree 0 > vfs.zfs.ccw_retry_interval 300 > vfs.zfs.check_hostid 1 > vfs.zfs.mg_fragmentation_threshold 85 > vfs.zfs.mg_noalloc_threshold 0 > vfs.zfs.condense_pct 200 > vfs.zfs.metaslab_sm_blksz 4096 > vfs.zfs.metaslab.bias_enabled 1 > vfs.zfs.metaslab.lba_weighting_enabled 1 > vfs.zfs.metaslab.fragmentation_factor_enabled1 > vfs.zfs.metaslab.preload_enabled 1 > vfs.zfs.metaslab.preload_limit 3 > vfs.zfs.metaslab.unload_delay 8 > vfs.zfs.metaslab.load_pct 50 > vfs.zfs.metaslab.min_alloc_size 33554432 > vfs.zfs.metaslab.df_free_pct 4 > vfs.zfs.metaslab.df_alloc_threshold 131072 > vfs.zfs.metaslab.debug_unload 0 > vfs.zfs.metaslab.debug_load 0 > vfs.zfs.metaslab.fragmentation_threshold70 > vfs.zfs.metaslab.force_ganging 16777217 > vfs.zfs.free_bpobj_enabled 1 > vfs.zfs.free_max_blocks -1 > vfs.zfs.zfs_scan_checkpoint_interval 7200 > vfs.zfs.zfs_scan_legacy 0 > vfs.zfs.no_scrub_prefetch 0 > vfs.zfs.no_scrub_io 0 > vfs.zfs.resilver_min_time_ms 3000 > vfs.zfs.free_min_time_ms 1000 > vfs.zfs.scan_min_time_ms 1000 > vfs.zfs.scan_idle 50 > vfs.zfs.scrub_delay 4 > vfs.zfs.resilver_delay 2 > vfs.zfs.top_maxinflight 32 > vfs.zfs.zfetch.array_rd_sz 1048576 > vfs.zfs.zfetch.max_idistance 67108864 > vfs.zfs.zfetch.max_distance 8388608 > vfs.zfs.zfetch.min_sec_reap 2 > vfs.zfs.zfetch.max_streams 8 > vfs.zfs.prefetch_disable 0 > vfs.zfs.delay_scale 500000 > vfs.zfs.delay_min_dirty_percent 60 > vfs.zfs.dirty_data_sync 67108864 > vfs.zfs.dirty_data_max_percent 10 > vfs.zfs.dirty_data_max_max 4294967296 > vfs.zfs.dirty_data_max 4294967296 > vfs.zfs.max_recordsize 1048576 > vfs.zfs.default_ibs 17 > vfs.zfs.default_bs 9 > vfs.zfs.send_holes_without_birth_time 1 > vfs.zfs.mdcomp_disable 0 > vfs.zfs.per_txg_dirty_frees_percent 30 > vfs.zfs.nopwrite_enabled 1 > vfs.zfs.dedup.prefetch 1 > vfs.zfs.dbuf_cache_lowater_pct 10 > vfs.zfs.dbuf_cache_hiwater_pct 10 > vfs.zfs.dbuf_metadata_cache_overflow 0 > vfs.zfs.dbuf_metadata_cache_shift 6 > vfs.zfs.dbuf_cache_shift 5 > vfs.zfs.dbuf_metadata_cache_max_bytes 1026962560 > vfs.zfs.dbuf_cache_max_bytes 2053925120 > vfs.zfs.arc_min_prescient_prefetch_ms 6 > vfs.zfs.arc_min_prefetch_ms 1 > vfs.zfs.l2c_only_size 0 > vfs.zfs.mfu_ghost_data_esize 1910587392 > vfs.zfs.mfu_ghost_metadata_esize 5158840832 > vfs.zfs.mfu_ghost_size 7069428224 > vfs.zfs.mfu_data_esize 17620227072 > vfs.zfs.mfu_metadata_esize 950300160 > vfs.zfs.mfu_size 20773338624 > vfs.zfs.mru_ghost_data_esize 6989578240 > vfs.zfs.mru_ghost_metadata_esize 18479132160 > vfs.zfs.mru_ghost_size 25468710400 > vfs.zfs.mru_data_esize 4455460352 > vfs.zfs.mru_metadata_esize 70236672 > vfs.zfs.mru_size 7413314560 > vfs.zfs.anon_data_esize 0 > vfs.zfs.anon_metadata_esize 0 > vfs.zfs.anon_size 3040037888 > vfs.zfs.l2arc_norw 1 > vfs.zfs.l2arc_feed_again 1 > vfs.zfs.l2arc_noprefetch 1 > vfs.zfs.l2arc_feed_min_ms 200 > vfs.zfs.l2arc_feed_secs 1 > vfs.zfs.l2arc_headroom 2 > vfs.zfs.l2arc_write_boost 8388608 > vfs.zfs.l2arc_write_max 8388608 > vfs.zfs.arc_meta_limit 16431400960 > vfs.zfs.arc_free_target 113124 > vfs.zfs.arc_kmem_cache_reap_retry_ms 0 > vfs.zfs.compressed_arc_enabled 1 > vfs.zfs.arc_grow_retry 60 > vfs.zfs.arc_shrink_shift 7 > vfs.zfs.arc_average_blocksize 8192 > vfs.zfs.arc_no_grow_shift 5 > vfs.zfs.arc_min 8215700480 > vfs.zfs.arc_max 65725603840 > vfs.zfs.abd_chunk_size 4096 > vfs.zfs.abd_scatter_enabled 1 > -------------------------------------------------------------------------- Trond.