On 17/03/2019 21:57, Eugene Grosbein wrote:> I agree. Recently I've found kind-of-workaround for this problem: > increase vm.v_free_min so when "FREE" memory goes low, > page daemon wakes earlier and shrinks UMA (and ZFS ARC too) moving some > memory > from WIRED to FREE quick enough so it can be re-used before bad things > happen. > > But avoid increasing vm.v_free_min too much (e.g. over 1/4 of total RAM) > because kernel may start behaving strange. For 16Gb system it should be > enough > to raise vm.v_free_min upto 262144 (1GB) or 131072 (512M). > > This is not permanent solution in any way but it really helps.Ah, thats very interesting, thankyou for that! I;ve been bitten by this issue too in the past, and it is (as mentioned) much improved on 12, but the act it could still cause issues worries me. -pete.
I suggest caution in raising vm.v_free_min, at least on 11.2-RELEASE systems with less RAM. I tried "65536" (256MB) on a 4GB mini-server, with vfs.zfs.arc_max of 2.5GB. Bad things happened when the cron daemon merely tried to run `periodic daily`. A few more details - ARC was mostly full, and "bad things" was 1: `pagedaemon` seemed to be thrashing memory - using 100% of CPU, with little disk activity, and 2: many normal processes seemed unable to run. The latter is probably explained by `man 3 sysctl` (see entry for "VM_V_FREE_MIN"). On Mon, 18 Mar 2019, Pete French wrote:> On 17/03/2019 21:57, Eugene Grosbein wrote: >> I agree. Recently I've found kind-of-workaround for this problem: >> increase vm.v_free_min so when "FREE" memory goes low, >> page daemon wakes earlier and shrinks UMA (and ZFS ARC too) moving some >> memory >> from WIRED to FREE quick enough so it can be re-used before bad things >> happen. >> >> But avoid increasing vm.v_free_min too much (e.g. over 1/4 of total RAM) >> because kernel may start behaving strange. For 16Gb system it should be >> enough >> to raise vm.v_free_min upto 262144 (1GB) or 131072 (512M). >> >> This is not permanent solution in any way but it really helps. > > Ah, thats very interesting, thankyou for that! I;ve been bitten by this issue > too in the past, and it is (as mentioned) much improved on 12, but the act it > could still cause issues worries me. > > -pete. > _______________________________________________ > freebsd-stable at freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
On 3/18/2019 08:07, Pete French wrote:> > > On 17/03/2019 21:57, Eugene Grosbein wrote: >> I agree. Recently I've found kind-of-workaround for this problem: >> increase vm.v_free_min so when "FREE" memory goes low, >> page daemon wakes earlier and shrinks UMA (and ZFS ARC too) moving >> some memory >> from WIRED to FREE quick enough so it can be re-used before bad >> things happen. >> >> But avoid increasing vm.v_free_min too much (e.g. over 1/4 of total RAM) >> because kernel may start behaving strange. For 16Gb system it should >> be enough >> to raise vm.v_free_min upto 262144 (1GB) or 131072 (512M). >> >> This is not permanent solution in any way but it really helps. > > Ah, thats very interesting, thankyou for that! I;ve been bitten by > this issue too in the past, and it is (as mentioned) much improved on > 12, but the act it could still cause issues worries me. > >The code patch I developed originally essentially sought to have the ARC code pare back before the pager started evicting working set.? A second crack went after clearing allocated-but-not-in-use UMA. v_free_min may not be the right place to do this -- see if bumping up vm.v_free_target also works. I'll stick this on my "to do" list, but it's much less critical in my applications than it was with 10.x and 11.x, both of which suffered from it much more-severely to the point that I was getting "stalls" that in some cases went on for 10 or more seconds due to things like your shell being evicted to swap to make room for arc, which is flat-out nuts.? That, at least, doesn't appear to be a problem with 12. -- Karl Denninger karl at denninger.net <mailto:karl at denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4897 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20190318/feaa54ab/attachment.bin>