(I am not subscribed to -stable, so please CC me, though I doubt I can help in any way/shape/form past this Email) Not the first time this has come up -- and every time it has, all that's heard is crickets in the threads. Recent proof: https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html I sent private mail to Peter Jeremy about his issue. I will not disclose that Email here. However, I will disclose the commits I included in said Email that have touched ZFS ARC-related code: http://www.freshbsd.org/commit/freebsd/r332785 http://www.freshbsd.org/commit/freebsd/r332552 http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights) http://www.freshbsd.org/commit/freebsd/r330061 http://www.freshbsd.org/commit/freebsd/r328235 http://www.freshbsd.org/commit/freebsd/r327491 http://www.freshbsd.org/commit/freebsd/r326619 http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe irrelevant) http://www.freshbsd.org/commit/freebsd/r323667 In short (and nebulous as hell; sorry, I cannot be more specific given the nature of the problem): there have been changes about ZFS's memory allocation/releasing decision-making scheme compared to ZFS on "older" FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x). Recommendations like "limit your ARC" are nothing new in FreeBSD, but are still ridiculous kludges: tech-lists' system clearly has 105GB MRU (MRU = most recently used) in ARC, meaning there is memory that can be released back to the rest of the OS for general use (re: memory contention/pressure situation), but the OS is choosing to use swap instead, eventually exhausting it. That logic sounds broken, IMO. (And yes I did notice the size of bhyve process) ZFS-related kernel folks need to be involved in this conversation. For whatever reason, in the past several years, related committers are no longer participating in these type of discussions. The opposite was true back in the 7.x to 9.x days. The answers have to come from them. I don't know, today, a) how they prefer these problems get reported to them, or b) what exact information they want that can help narrow it down (tech-lists' provided data is, IMO, good and par for the course). -- | Jeremy Chadwick jdc at koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |
Hi,
I have the very same issue on many servers. Mainly on mail servers running
qmail+spamassassin+clamav. I've tuned some variables on loader.conf:
vfs.zfs.vdev.cache.size="2G"
vfs.zfs.arc_min="614400000"
vfs.zfs.arc_max="4915200000"
But after some days, it begins eating swap and the system comes very very
slow then I need to reboot it.
My system config is:
FreeBSD 11.1-STABLE #5 r321625M: Thu Sep 21 16:01:56 -03 2017
root at mail.com:/usr/obj/usr/src/sys/MAIL amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM
4.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Atom(TM) CPU C2518 @ 1.74GHz (1750.04-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d
Stepping=8
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x101<LAHF,Prefetch>
Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 8213245952 (7832 MB)
It's configured with 4GB of swap. Let me know if I can help with any other
information.
Thanks.
On Tue, Jun 19, 2018 at 2:29 PM, Jeremy Chadwick <jdc at koitsu.org>
wrote:
> (I am not subscribed to -stable, so please CC me, though I doubt I can
> help in any way/shape/form past this Email)
>
> Not the first time this has come up -- and every time it has, all
that's
> heard is crickets in the threads. Recent proof:
>
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html
> https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html
>
> I sent private mail to Peter Jeremy about his issue. I will not
> disclose that Email here. However, I will disclose the commits I
> included in said Email that have touched ZFS ARC-related code:
>
> http://www.freshbsd.org/commit/freebsd/r332785
> http://www.freshbsd.org/commit/freebsd/r332552
> http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights)
> http://www.freshbsd.org/commit/freebsd/r330061
> http://www.freshbsd.org/commit/freebsd/r328235
> http://www.freshbsd.org/commit/freebsd/r327491
> http://www.freshbsd.org/commit/freebsd/r326619
> http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe
> irrelevant)
> http://www.freshbsd.org/commit/freebsd/r323667
>
> In short (and nebulous as hell; sorry, I cannot be more specific given
> the nature of the problem): there have been changes about ZFS's memory
> allocation/releasing decision-making scheme compared to ZFS on
"older"
> FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x).
>
> Recommendations like "limit your ARC" are nothing new in FreeBSD,
but
> are still ridiculous kludges: tech-lists' system clearly has 105GB MRU
> (MRU = most recently used) in ARC, meaning there is memory that can be
> released back to the rest of the OS for general use (re: memory
> contention/pressure situation), but the OS is choosing to use swap
> instead, eventually exhausting it. That logic sounds broken, IMO. (And
> yes I did notice the size of bhyve process)
>
> ZFS-related kernel folks need to be involved in this conversation. For
> whatever reason, in the past several years, related committers are no
> longer participating in these type of discussions. The opposite was
> true back in the 7.x to 9.x days. The answers have to come from them.
> I don't know, today, a) how they prefer these problems get reported to
> them, or b) what exact information they want that can help narrow it
> down (tech-lists' provided data is, IMO, good and par for the course).
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
>
19.06.18 20:29, Jeremy Chadwick wrote:> (I am not subscribed to -stable, so please CC me, though I doubt I can > help in any way/shape/form past this Email) > > Not the first time this has come up -- and every time it has, all that's > heard is crickets in the threads. Recent proof:? I may sound lame but I also faced this issues a few month ago. After a few days of load system was trying to push more and more data into the swap up to the point that active window becomes too small to handle all programs so they should be get back from swap and while they are not running something pinches ARC and ARC claims the memory and so on? I wasn't ever a fan of limiting things. If something requires limits it can be easily exploited. Should I switch from normal limits to other limits when I, say, need to test something on 4 VMs? So while paging through documentation I found a rather old memo in tuning(7) about vm.swap_idle_enabled. I played a little with thresholds but that was only making things worse. I left swap_idle_enable on and let machine live. That was near January I suppose. To my amusement swap problems were gone. This doesn't mean swap wasn't used, instead system survived weeks under irregular load without issues. The only other change that I did was bumping up vfs.zfs.arc_free_target a little bit higher then default to make some space between ARC and VM so they wouldn't clash on memory so often. Since then all of my problems with swap was forgotten. I'm not sure what setting fixed that, neither I'm sure that wasn't some recent patches. I'm running 11-STABLE and rebuilding system at least once per month. Hope that can help someone. WBR. -- Sphinx of black quartz judge my vow.
On 20/06/2018 02:59, Jeremy Chadwick wrote:> In short (and nebulous as hell; sorry, I cannot be more specific given > the nature of the problem): there have been changes about ZFS's memory > allocation/releasing decision-making scheme compared to ZFS on "older" > FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x).I would say the issues started with 10.x, I never had memory issues using 9.x with ZFS. I first had all ram marked wired when testing 10.1.> Recommendations like "limit your ARC" are nothing new in FreeBSD, but > are still ridiculous kludges: tech-lists' system clearly has 105GB MRU > (MRU = most recently used) in ARC, meaning there is memory that can be > released back to the rest of the OS for general use (re: memory > contention/pressure situation), but the OS is choosing to use swap > instead, eventually exhausting it. That logic sounds broken, IMO. (And > yes I did notice the size of bhyve process)This review is aiming to fix this - https://reviews.freebsd.org/D7538 I have been running the patch on stable/11 and after eight days uptime I still have zero swap in use, I can't recall a time in the last few years that I have had no swap usage past the first hour or two uptime. As I have commented in that review, the issue I am seeing is that arc_max is not counted when testing max_wired, the two together can add up to more than the physical ram and wiring all physical ram can push ram used by processes out to swap. I know with 8G physical ram having over 7G wired is not recoverable. max_wired seems to default to 30% ram (5G with 16G ram) I have never seen this mentioned in any zfs tuning, it should be subtracted from physical ram when calculating arc_max. arc_max should never be greater than (kmem_size - max_wired - padding) -- FreeBSD - the place to B...Storing Data Shane Ambler