On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote:> 12.02.2019 23:34, Mark Johnston wrote: > > > I suspect that the "leaked" memory is simply being used to cache UMA > > items. Note that the values in the FREE column of vmstat -z output are > > quite large. The cached items are reclaimed only when the page daemon > > wakes up to reclaim memory; if there are no memory shortages, large > > amounts of memory may accumulate in UMA caches. In this case, the sum > > of the product of columns 2 and 5 gives a total of roughly 4GB cached. > > > >> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt > >> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt > > It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory > despite of long period of vm thrashing:Depending on the system's workload, it is possible for the caches to grow quite quickly after a reclaim. If you are able to run kgdb on the kernel, you can find the time of the last reclaim by comparing the values of lowmem_uptime and time_uptime.> $ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head > 1892 abd_chunk > 454.629 dnode_t > 351.35 zio_buf_512 > 228.391 zio_buf_16384 > 173.968 dmu_buf_impl_t > 130.25 zio_data_buf_131072 > 93.6887 VNODE > 81.6978 arc_buf_hdr_t_full > 74.9368 256 > 57.4102 4096 >
13.02.2019 1:18, Mark Johnston wrote:> On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote: >> 12.02.2019 23:34, Mark Johnston wrote: >> >>> I suspect that the "leaked" memory is simply being used to cache UMA >>> items. Note that the values in the FREE column of vmstat -z output are >>> quite large. The cached items are reclaimed only when the page daemon >>> wakes up to reclaim memory; if there are no memory shortages, large >>> amounts of memory may accumulate in UMA caches. In this case, the sum >>> of the product of columns 2 and 5 gives a total of roughly 4GB cached. >>> >>>> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt >>>> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt >> >> It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory >> despite of long period of vm thrashing: > > Depending on the system's workload, it is possible for the caches to > grow quite quickly after a reclaim. If you are able to run kgdb on the > kernel, you can find the time of the last reclaim by comparing the > values of lowmem_uptime and time_uptime.Yes, I have debugger compiled into running kernel and have console access. What commands should I use?
13.02.2019 1:18, Mark Johnston wrote:> Depending on the system's workload, it is possible for the caches to > grow quite quickly after a reclaim. If you are able to run kgdb on the > kernel, you can find the time of the last reclaim by comparing the > values of lowmem_uptime and time_uptime.(kgdb) p time_uptime $1 = 7019265 (kgdb) p lowmem_uptime $2 = 7000568 (kgdb) quit # uptime 2:08 up 81 days, 5:48, 5 users, load averages: 0,19 0,13 0,09