Lev Serebryakov
2018-Nov-20 12:42 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
I have server which is mostly torrent box. It uses ZFS and equipped
with 16GiB of physical memory. It is running 11-STABLE (r339914 now).
I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago.
I was used to see 13-14GiB of memory in ZFS ARC and it was Ok.
Sometimes it "locks" under heavy disk load due to ARC memory pressure,
but it was bearable, and as ZFS is main reason this server exists, I
didn't limit ARC.
But new revision (r339914) shows very strange behaivor: ARC is no more
than 4GiB, but kernel has 15GiB wired:
Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free
ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other
2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio
It is typical numbers for last week: 15G wired, 237M Free, but only
4252M ARC!
Where is other 11G of memory?!
I've checked USED and FREE in "vmstat -z" output and got this:
$ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n |
tail -20
23001088 9171456 MAP ENTRY
29680800 8404320 VM OBJECT
34417408 10813952 256
36377964 2665656 S VFS Cache
50377392 53856 sa_cache
50593792 622985216 zio_buf_131072
68913152 976896 mbuf_cluster
73543680 7225344 mbuf_jumbo_page
92358552 67848 zfs_znode_cache
95731712 51761152 4096
126962880 159581760 dmu_buf_impl_t
150958080 233920512 mbuf_jumbo_9k
165164600 92040 VNODE
192701120 30350880 UMA Slabs
205520896 291504128 zio_data_buf_1048576
222822400 529530880 zio_data_buf_524288
259143168 293476864 zio_buf_512
352485376 377061376 zio_buf_16384
376109552 346474128 dnode_t
2943016960 5761941504 abd_chunk
$
And total USED/FREE numbers is very strange for me:
$ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}'
5717965420 9328951088
$
So, only ~5.7G is used and 9.3G is free! But why this memory is not
used by ARC anymore and why is it wired and not free?
--
// Lev Serebryakov
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 963 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20181120/ea6352aa/attachment.sig>
Eugene M. Zheganin
2018-Nov-20 19:30 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello, 20.11.2018 15:42, Lev Serebryakov ?????:> I have server which is mostly torrent box. It uses ZFS and equipped > with 16GiB of physical memory. It is running 11-STABLE (r339914 now). > > I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. > > I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. > Sometimes it "locks" under heavy disk load due to ARC memory pressure, > but it was bearable, and as ZFS is main reason this server exists, I > didn't limit ARC. > > But new revision (r339914) shows very strange behaivor: ARC is no more > than 4GiB, but kernel has 15GiB wired: > > Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free > ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other > 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio > > It is typical numbers for last week: 15G wired, 237M Free, but only > 4252M ARC! > > Where is other 11G of memory?! > > [...] > And total USED/FREE numbers is very strange for me: > > $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' > 5717965420 9328951088 > $ > > So, only ~5.7G is used and 9.3G is free! But why this memory is not > used by ARC anymore and why is it wired and not free?I'm getting pretty much same story on recent 11-STABLE from 9th November. Previous versions didn't have that much questions about memory usage (and I run several 11-STABLEs). Eugene.
Mark Johnston
2018-Nov-20 20:45 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote:> > I have server which is mostly torrent box. It uses ZFS and equipped > with 16GiB of physical memory. It is running 11-STABLE (r339914 now). > > I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. > > I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. > Sometimes it "locks" under heavy disk load due to ARC memory pressure, > but it was bearable, and as ZFS is main reason this server exists, I > didn't limit ARC. > > But new revision (r339914) shows very strange behaivor: ARC is no more > than 4GiB, but kernel has 15GiB wired: > > Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free > ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other > 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio > > It is typical numbers for last week: 15G wired, 237M Free, but only > 4252M ARC! > > Where is other 11G of memory?! > > I've checked USED and FREE in "vmstat -z" output and got this: > > $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | > tail -20 > 23001088 9171456 MAP ENTRY > 29680800 8404320 VM OBJECT > 34417408 10813952 256 > 36377964 2665656 S VFS Cache > 50377392 53856 sa_cache > 50593792 622985216 zio_buf_131072 > 68913152 976896 mbuf_cluster > 73543680 7225344 mbuf_jumbo_page > 92358552 67848 zfs_znode_cache > 95731712 51761152 4096 > 126962880 159581760 dmu_buf_impl_t > 150958080 233920512 mbuf_jumbo_9k > 165164600 92040 VNODE > 192701120 30350880 UMA Slabs > 205520896 291504128 zio_data_buf_1048576 > 222822400 529530880 zio_data_buf_524288 > 259143168 293476864 zio_buf_512 > 352485376 377061376 zio_buf_16384 > 376109552 346474128 dnode_t > 2943016960 5761941504 abd_chunk > $ > > And total USED/FREE numbers is very strange for me: > > $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' > 5717965420 9328951088 > $ > > So, only ~5.7G is used and 9.3G is free! But why this memory is not > used by ARC anymore and why is it wired and not free?Could you show the output of "vmstat -s" when in this state?