On 2/19/2019 2:35 PM, Mike Tancsa wrote:> dd if=/dev/zero of=/tanker/test bs=1m count=100000
The box has 32G of RAM. If I do a
# sysctl -w vfs.zfs.arc_max=12224866304
vfs.zfs.arc_max: 32224866304 -> 12224866304
#
after WIRED memory is at 29G, it doesnt immediately reclaim it and there
is memory pressure.? Booting the box with
vfs.zfs.arc_max=12224866304
keeps WIRED at 15G
last pid:? 1049;? load averages:? 2.02,? 2.30,?
1.07??????????????????????????????????????????????????????????????? up
0+00:04:21? 20:17:43
38 processes:? 2 running, 36 sleeping
CPU:? 3.6% user,? 0.0% nice,? 5.9% system,? 0.0% interrupt, 90.5% idle
Mem: 39M Active, 6212K Inact, 15G Wired, 30M Buf, 16G Free
ARC: 11G Total, 363K MFU, 11G MRU, 117M Anon, 23M Header, 850K Other
???? 11G Compressed, 11G Uncompressed, 1.01:1 Ratio
Swap:
Re-running the test but this time with swap,
# zpool create tanker raidz1 da0p1 da1p1 da2p1 da3p1 da4p1 da5p1 da6p1
da7p1 da8p1 da9p1 da10p1 da11p1 da12p1 da13p1 da14p1
# pstat -T
120/1043936 files
0M/65536M swap space
#
There is still 30G of Wired memory ?
last pid:? 1096;? load averages:? 0.78,? 2.13,?
1.25??????????????????????????????????????????????????????????????? up
0+00:06:42? 20:33:21
35 processes:? 1 running, 34 sleeping
CPU:? 0.0% user,? 0.0% nice,? 0.0% system,? 0.0% interrupt,? 100% idle
Mem: 976K Active, 274M Inact, 88M Laundry, 30G Wired, 1341M Buf, 722M Free
ARC: 1227K Total, 775K MFU, 339K MRU, 32K Anon, 9216 Header, 69K Other
???? 176K Compressed, 1179K Uncompressed, 6.70:1 Ratio
Swap: 64G Total, 64G Free
I start a new dd on a freshly created pool,
CPU:? 0.0% user,? 0.0% nice, 31.4% system,? 0.8% interrupt, 67.8% idle
Mem: 1396K Active, 460K Inact, 18M Laundry, 30G Wired, 1341M Buf, 720M Free
ARC: 27G Total, 54K MFU, 26G MRU, 1380M Anon, 56M Header, 3459K Other
???? 26G Compressed, 26G Uncompressed, 1.00:1 Ratio
Swap: 64G Total, 73M Used, 64G Free
and the box starts to swap
last pid:? 1104;? load averages: 11.61,? 6.10,?
3.04??????????????????????????????????????????????????????????????? up
0+00:09:32? 20:36:11
36 processes:? 2 running, 34 sleeping
CPU:? 0.1% user,? 0.0% nice, 95.3% system,? 2.1% interrupt,? 2.4% idle
Mem: 1212K Active, 124K Inact, 688K Laundry, 30G Wired, 1341M Buf, 819M Free
ARC: 27G Total, 54K MFU, 24G MRU, 1620M Anon, 58M Header, 5572K Other
???? 24G Compressed, 24G Uncompressed, 1.01:1 Ratio
Swap: 64G Total, 90M Used, 64G Free
(avg's oneline script shows)
61,724,160????? arc_buf_hdr_t_full????????? 55,263,744????? 6,460,416
111,783,936???? 512???????????????????????? 109,534,208???? 2,249,728
555,884,000???? UMA_Slabs?????????????????? 553,374,640???? 2,509,360
1,280,966,656?? zio_data_buf_131072???????? 905,576,448???? 375,390,208
28,142,604,288? abd_chunk?????????????????? 27,965,505,536? 177,098,752
30,431,435,848? TOTAL?????????????????????? 29,663,389,468? 768,046,380
# vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",
$2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
?????? 358 zio_data_buf_131072
?? 168.895 abd_chunk
?? 18.5738 zio_cache
???? 14.25 zio_buf_786432
??????? 14 zio_buf_1048576
??? 13.987 BUF TRIE
??? 13.125 zio_buf_655360
???? 9.625 zio_buf_917504
????? 8.75 zio_buf_458752
???????? 8 zio_buf_524288
#
As the box is netbooted over nfs, when it swaps out some of the nfs
helper apps, the box gets a bit unresponsive.? e.g. if I do a find
/usr/src -type f |xargs md5 the box will stall out for periods of time.?
If I limit ARC to say 12G, all is well.
??? ---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada