Oliver Brandmueller
2012-Mar-30 08:38 UTC
9-STABLE, ZFS, NFS, ggatec - suspected memory leak
Hi all, Setup: I'm running 2 machines (amd64, 16GB) with FreeBSD 9-STABLE (Mar 14 so far) acting as NFS servers. They each serve 3 zpools (holding a single zfs, hourly snapshots). The zpools each are 3-way mirrors of ggate devices, each 2 TB, so 2 TB per zpool. Compression is "on" (to save bandwith to the backend, compressratio around 1.05 to 1.15), atime is off. There is no special tuning in loader.conf (except I tried to limit ZFS ARC to 8GB lately, which doesn't change a lot). sysctl.conf has: kern.ipc.maxsockbuf=33554432 net.inet.tcp.sendspace=8388608 net.inet.tcp.recvspace=8388608 kern.maxfiles=64000 vfs.nfsd.maxthreads=254 Without the first three zfs+ggate goes bad after a short time (checksum errors, stall), the latter are mainly for NFS and some regular local cleanup stuff. The machines have 4 em and 2 igb network interfaces. 3 of the are dedicated links (with no switches) to the ggate servers, one is a dedicated link to a third machine which gets feeded with incremental snapshots by ZFS send (as backup and fallaback of last resort), one interface for management tasks and one to an internal network with the NFS clients. The NFS clients are mostly FreeBSD 6, 7 and 9 STABLE machines (migration to 9 is running), no NFSv4 (by now), all tcp NFS links, merely no locking. Data consists of a lot of files, this is mainly mailboxes (IMAP: dovecot, incoming mail with exim, some simple web stuff with apache), so lots of small files, only few bigger ones. Directory structures to a reasonable depth. System is on a SSD (ufs, trim), additionally there are 3 (4 actually, 1 unused by now) 120GB SSDs serving as cache devices for the zpools. I first starting using the whole device, but in my hopes to change something limited cache to partitions of 32GB without change in behaviour. Problem: After about a week of runtime in normal workload the systems starts to swap (with about 300 to 500 MB of RAM free). Lots of swapping in and out, but only very few swap space used (30 to 50 MB). ZFS ARC at that point reaches it's minimum (while using up to it's configured maximum before). Most of the RAM is wired. L2ARC headers, accourding to zfs-stats eat about 1GB, ARC is at 1.8GB at this time. No userland processes using lots of RAM. After some time the system becomes unresponsive, the only way to fix this I had found by now is to reboot the machine (which of course gives a service interruption).>From the start of swapping to unresponsiveness I have about 2 to 4 hoursto check several things (if I just knew what!). Workload distribution is not even over the day, from my munin graphs I can see, that wired grows at time of higher workload. At night with lower workload (but far from nothing, let's say about 1/3 to 1/2 in writes, but probably <1/4 in reads from weekday workload) I can barely see any groth of the wired graph. So where is my memory going, any ideas what to change? Kernel is stripped down from GENERIC and then everything I need loaded as modules. Kernel config: http://sysadm.in/zprob/ZSTOR loader.conf : http://sysadm.in/zprob/loader.conf dmesg.boot : http://sysadm.in/zprob/dmesg.boot -- | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | | Ich bin das Internet. Sowahr ich Gott helfe. |
Philip M. Gollucci
2012-Mar-30 21:02 UTC
9-STABLE, ZFS, NFS, ggatec - suspected memory leak
http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239643.html Same issue different thread. Different software. Its not NFS, its ZFS. I don't really have a place to try it on 8.2, but my hunch from things I've done rather similarly which don't cause tell me its a new issue in 9.0, though I won't swear by that. On 03/30/12 08:38, Oliver Brandmueller wrote:> Hi all, > > Setup: > > I'm running 2 machines (amd64, 16GB) with FreeBSD 9-STABLE (Mar 14 so > far) acting as NFS servers. They each serve 3 zpools (holding a single > zfs, hourly snapshots). The zpools each are 3-way mirrors of ggate > devices, each 2 TB, so 2 TB per zpool. Compression is "on" (to save > bandwith to the backend, compressratio around 1.05 to 1.15), atime is > off. > > There is no special tuning in loader.conf (except I tried to limit ZFS > ARC to 8GB lately, which doesn't change a lot). sysctl.conf has: > > kern.ipc.maxsockbuf=33554432 > net.inet.tcp.sendspace=8388608 > net.inet.tcp.recvspace=8388608 > kern.maxfiles=64000 > vfs.nfsd.maxthreads=254 > > Without the first three zfs+ggate goes bad after a short time (checksum > errors, stall), the latter are mainly for NFS and some regular local > cleanup stuff. > > The machines have 4 em and 2 igb network interfaces. 3 of the are > dedicated links (with no switches) to the ggate servers, one is a > dedicated link to a third machine which gets feeded with incremental > snapshots by ZFS send (as backup and fallaback of last resort), one > interface for management tasks and one to an internal network with the > NFS clients. > > The NFS clients are mostly FreeBSD 6, 7 and 9 STABLE machines (migration > to 9 is running), no NFSv4 (by now), all tcp NFS links, merely no > locking. > > Data consists of a lot of files, this is mainly mailboxes (IMAP: > dovecot, incoming mail with exim, some simple web stuff with apache), so > lots of small files, only few bigger ones. Directory structures to a > reasonable depth. > > System is on a SSD (ufs, trim), additionally there are 3 (4 actually, 1 > unused by now) 120GB SSDs serving as cache devices for the zpools. I > first starting using the whole device, but in my hopes to change > something limited cache to partitions of 32GB without change in > behaviour. > > > Problem: > > After about a week of runtime in normal workload the systems starts to > swap (with about 300 to 500 MB of RAM free). Lots of swapping in and > out, but only very few swap space used (30 to 50 MB). ZFS ARC at that > point reaches it's minimum (while using up to it's configured maximum > before). Most of the RAM is wired. L2ARC headers, accourding to > zfs-stats eat about 1GB, ARC is at 1.8GB at this time. No userland > processes using lots of RAM. > > After some time the system becomes unresponsive, the only way to fix > this I had found by now is to reboot the machine (which of course gives > a service interruption). > >>From the start of swapping to unresponsiveness I have about 2 to 4 hours > to check several things (if I just knew what!). > > Workload distribution is not even over the day, from my munin graphs I > can see, that wired grows at time of higher workload. At night with > lower workload (but far from nothing, let's say about 1/3 to 1/2 in > writes, but probably <1/4 in reads from weekday workload) I can barely > see any groth of the wired graph. > > So where is my memory going, any ideas what to change? > > Kernel is stripped down from GENERIC and then everything I need loaded > as modules. > > Kernel config: http://sysadm.in/zprob/ZSTOR > loader.conf : http://sysadm.in/zprob/loader.conf > dmesg.boot : http://sysadm.in/zprob/dmesg.boot > >-- ------------------------------------------------------------------------ 1024D/DB9B8C1C B90B FBC3 A3A1 C71A 8E70 3F8C 75B8 8FFB DB9B 8C1C Philip M. Gollucci (pgollucci@p6m7g8.com) c: 703.336.9354 Member, Apache Software Foundation Committer, FreeBSD Foundation Consultant, P6M7G8 Inc. Director Operations, Ridecharge Inc. Work like you don't need the money, love like you'll never get hurt, and dance like nobody's watching. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120330/0018c545/signature.pgp
Oliver Brandmueller
2012-Apr-01 10:08 UTC
9-STABLE, ZFS, NFS, ggatec - suspected memory leak
Hi, On Fri, Mar 30, 2012 at 09:02:03PM +0000, Philip M. Gollucci wrote:> http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239643.html > > Same issue different thread. Different software. > > Its not NFS, its ZFS. > > I don't really have a place to try it on 8.2, but my hunch from things > I've done rather similarly which don't cause tell me its a new issue in > 9.0, though I won't swear by that.I'll see, if I can setup a machine running 8 to take some of the partitions over - but chances aren't really great. Thanx for the hint to you thread. Also I'll start graphing (or at least logging) some of the higher values from vmstat that seems suspicious and ever growing to me to see if I can match one of them to the growth rate I see for mir wired mem apart from ARC and L2ARC headers. - Oliver -- | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | | Ich bin das Internet. Sowahr ich Gott helfe. | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120401/c12da0c5/attachment.pgp
Oliver Brandmueller
2012-Apr-01 19:31 UTC
9-STABLE, ZFS, NFS, ggatec - suspected memory leak
Hi, On Fri, Mar 30, 2012 at 09:02:03PM +0000, Philip M. Gollucci wrote:> http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239643.html > > Same issue different thread. Different software. > > Its not NFS, its ZFS. > > I don't really have a place to try it on 8.2, but my hunch from things > I've done rather similarly which don't cause tell me its a new issue in > 9.0, though I won't swear by that.Just digging around I find one value ever-growing in vmstat -z and being by far the biggest "USED" value: virgin# while ( 1 ) while? echo -n `date`: ; vmstat -z | fgrep NAMEI | sed 's/ */ /g' while? sleep 10 while? end Sun Apr 1 21:26:33 CEST 2012:NAMEI: 1024, 0, 9150656, 1152,288542469, 0, 0 Sun Apr 1 21:26:43 CEST 2012:NAMEI: 1024, 0, 9150877, 1187,288546408, 0, 0 Sun Apr 1 21:26:53 CEST 2012:NAMEI: 1024, 0, 9150966, 1226,288549358, 0, 0 Sun Apr 1 21:27:03 CEST 2012:NAMEI: 1024, 0, 9151079, 1241,288551944, 0, 0 Sun Apr 1 21:27:13 CEST 2012:NAMEI: 1024, 0, 9151264, 1184,288555091, 0, 0 Sun Apr 1 21:27:23 CEST 2012:NAMEI: 1024, 0, 9151338, 1238,288557632, 0, 0 Sun Apr 1 21:27:33 CEST 2012:NAMEI: 1024, 0, 9151488, 1344,288560345, 0, 0 Sun Apr 1 21:27:43 CEST 2012:NAMEI: 1024, 0, 9151610, 1350,288563532, 0, 0 Sun Apr 1 21:27:53 CEST 2012:NAMEI: 1024, 0, 9151765, 1451,288567086, 0, 0 Sun Apr 1 21:28:03 CEST 2012:NAMEI: 1024, 0, 9151935, 1409,288570645, 0, 0 Sun Apr 1 21:28:13 CEST 2012:NAMEI: 1024, 0, 9152086, 1258,288573371, 0, 0 Sun Apr 1 21:28:23 CEST 2012:NAMEI: 1024, 0, 9152244, 1100,288576862, 0, 0 Sun Apr 1 21:28:33 CEST 2012:NAMEI: 1024, 0, 9152429, 1171,288580235, 0, 0 Sun Apr 1 21:28:43 CEST 2012:NAMEI: 1024, 0, 9152615, 1113,288583356, 0, 0 I'll recheck that in more busy times again. Philip: Are you using: - another file system (like UFS/FFS for the system or similar?) - ZFS snapshots - ZFS send on the system? Can you also see the NAMEI growth? - Oliver -- | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | | Ich bin das Internet. Sowahr ich Gott helfe. | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120401/eb41f488/attachment.pgp
Oliver Brandmueller
2012-Apr-03 13:34 UTC
9-STABLE, ZFS, NFS, ggatec - suspected memory leak
Hi, On Fri, Mar 30, 2012 at 09:02:03PM +0000, Philip M. Gollucci wrote:> http://lists.freebsd.org/pipermail/freebsd-questions/2012-March/239643.html > > Same issue different thread. Different software. > > Its not NFS, its ZFS. > > I don't really have a place to try it on 8.2, but my hunch from things > I've done rather similarly which don't cause tell me its a new issue in > 9.0, though I won't swear by that.So we're having different issues. I could boil it down to NFS definitely: If I change back to the old NFS server the problem immediately stops. I'm currently checking, where in the new NFS server might be the problem (together with people who have a deeper understanding of what's happening). - Oliver -- | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | | Ich bin das Internet. Sowahr ich Gott helfe. | -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120403/897c8e62/attachment.pgp
Oliver Brandmueller wrote:> Hi, > > After figuring an easy way to repeat the behaviour and hunting it down > to the combination of ZFS+newNFS and removal of files or directories I > opened PR kern/167266 >Oops, the patch for this is at: http://people.freebsd.org/~rmacklem/namei-leak.patch rick
----- Original Message ----- From: "Rick Macklem" <rmacklem@uoguelph.ca> To: "Oliver Brandmueller" <ob@e-Gitt.NET> Cc: <freebsd-stable@freebsd.org> Sent: Thursday, April 26, 2012 1:24 AM Subject: Re: 9-STABLE, ZFS, NFS, ggatec - suspected memory leak> Oliver Brandmueller wrote: >> Hi, >> >> After figuring an easy way to repeat the behaviour and hunting it down >> to the combination of ZFS+newNFS and removal of files or directories I >> opened PR kern/167266 >> > Oops, the patch for this is at: > http://people.freebsd.org/~rmacklem/namei-leak.patchIs this specific to 9.x or is 8.x effected? Regards Steve ===============================================This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.