Mark Johnston
2018-Nov-20 20:45 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
On Tue, Nov 20, 2018 at 03:42:24PM +0300, Lev Serebryakov wrote:> > I have server which is mostly torrent box. It uses ZFS and equipped > with 16GiB of physical memory. It is running 11-STABLE (r339914 now). > > I've updated it to r339914 from some 11.1-STABLE revision 3 weeks ago. > > I was used to see 13-14GiB of memory in ZFS ARC and it was Ok. > Sometimes it "locks" under heavy disk load due to ARC memory pressure, > but it was bearable, and as ZFS is main reason this server exists, I > didn't limit ARC. > > But new revision (r339914) shows very strange behaivor: ARC is no more > than 4GiB, but kernel has 15GiB wired: > > Mem: 22M Active, 656M Inact, 62M Laundry, 15G Wired, 237M Free > ARC: 4252M Total, 2680M MFU, 907M MRU, 3680K Anon, 15M Header, 634M Other > 2789M Compressed, 3126M Uncompressed, 1.12:1 Ratio > > It is typical numbers for last week: 15G wired, 237M Free, but only > 4252M ARC! > > Where is other 11G of memory?! > > I've checked USED and FREE in "vmstat -z" output and got this: > > $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | > tail -20 > 23001088 9171456 MAP ENTRY > 29680800 8404320 VM OBJECT > 34417408 10813952 256 > 36377964 2665656 S VFS Cache > 50377392 53856 sa_cache > 50593792 622985216 zio_buf_131072 > 68913152 976896 mbuf_cluster > 73543680 7225344 mbuf_jumbo_page > 92358552 67848 zfs_znode_cache > 95731712 51761152 4096 > 126962880 159581760 dmu_buf_impl_t > 150958080 233920512 mbuf_jumbo_9k > 165164600 92040 VNODE > 192701120 30350880 UMA Slabs > 205520896 291504128 zio_data_buf_1048576 > 222822400 529530880 zio_data_buf_524288 > 259143168 293476864 zio_buf_512 > 352485376 377061376 zio_buf_16384 > 376109552 346474128 dnode_t > 2943016960 5761941504 abd_chunk > $ > > And total USED/FREE numbers is very strange for me: > > $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' > 5717965420 9328951088 > $ > > So, only ~5.7G is used and 9.3G is free! But why this memory is not > used by ARC anymore and why is it wired and not free?Could you show the output of "vmstat -s" when in this state?
Lev Serebryakov
2018-Nov-21 10:56 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
Hello Mark, Tuesday, November 20, 2018, 11:45:37 PM, you wrote:> Could you show the output of "vmstat -s" when in this state?Now ARC is 1.9GB (!!!) and 15G is still Wired: % vmstat -s 1311134466 cpu context switches 2126950424 device interrupts 45414140 software interrupts 103215017 traps 2910067754 system calls 25 kernel threads created 108756 fork() calls 50914 vfork() calls 111 rfork() calls 0 swap pager pageins 0 swap pager pages paged in 0 swap pager pageouts 0 swap pager pages paged out 4201 vnode pager pageins 51719 vnode pager pages paged in 31597 vnode pager pageouts 63028 vnode pager pages paged out 7 page daemon wakeups 1212387275 pages examined by the page daemon 0 clean page reclamation shortfalls 0 pages reactivated by the page daemon 8281789 copy-on-write faults 7909 copy-on-write optimized faults 44440180 zero fill pages zeroed 24762 zero fill pages prezeroed 2 intransit blocking page faults 63838217 total VM faults taken 3816 page faults requiring I/O 0 pages affected by kernel thread creation 3996512 pages affected by fork() 1842784 pages affected by vfork() 5499 pages affected by rfork() 88771375 pages freed 0 pages freed by daemon 21591789 pages freed by exiting processes 4936 pages active 161234 pages inactive 16674 pages in the laundry queue 3848989 pages wired down 39951 pages free 4096 bytes per page 612681203 total name lookups cache hits (95% pos + 0% neg) system 0% per-directory deletions 0%, falsehits 0%, toolong 0% % -- Best regards, Lev mailto:lev at FreeBSD.org
Lev Serebryakov
2018-Nov-22 13:33 UTC
Where is my memory on 'fresh' 11-STABLE? It should be used by ARC, but it is not used for it anymore.
On 20.11.2018 23:45, Mark Johnston wrote:>> Where is other 11G of memory?! >> >> I've checked USED and FREE in "vmstat -z" output and got this: >> >> $ vmstat -z | tr : , | awk -F , '1{print $2*$4,$2*$5,$1}' | sort -n | >> tail -20 >> 23001088 9171456 MAP ENTRY >> 29680800 8404320 VM OBJECT >> 34417408 10813952 256 >> 36377964 2665656 S VFS Cache >> 50377392 53856 sa_cache >> 50593792 622985216 zio_buf_131072 >> 68913152 976896 mbuf_cluster >> 73543680 7225344 mbuf_jumbo_page >> 92358552 67848 zfs_znode_cache >> 95731712 51761152 4096 >> 126962880 159581760 dmu_buf_impl_t >> 150958080 233920512 mbuf_jumbo_9k >> 165164600 92040 VNODE >> 192701120 30350880 UMA Slabs >> 205520896 291504128 zio_data_buf_1048576 >> 222822400 529530880 zio_data_buf_524288 >> 259143168 293476864 zio_buf_512 >> 352485376 377061376 zio_buf_16384 >> 376109552 346474128 dnode_t >> 2943016960 5761941504 abd_chunk >> $ >> >> And total USED/FREE numbers is very strange for me: >> >> $ vmstat -z | tr : , | awk -F , '1{u+=$2*$4; f+=$2*$5} END{print u,f}' >> 5717965420 9328951088 >> $ One more datapoint: I needed to reboot system and right after rebootARC becomes 14GiB (what was expected) and now it slows defalate, though I didn't get more "Free" memory. So, ARC becomes smaller and smaller after first fill-up, but all memory is "Wired". -- // Lev Serebryakov -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20181122/6c5bd06b/attachment.sig>