Benjamin Lutz
2013-Dec-10 10:41 UTC
Swap on ZFS - bad idea? (swap_pager: indefinite wait buffer)
We have storage server that I've found unresponsive this morning. The console was filled with these messages: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 17344, size: 61440 The lines were all exactly the same, no variation in the numbers. Ctrl-Alt-Delete had no effect. Now, the only thing related to swap that's special about this machine, is that we use swap on ZFS, because we figured, silent disk corruption is just as bad when it happens to swapped memory as it is when it happens to stored data. However, might this be the problem? One thought I was having was: ZFS hits its memory bounds, a process makes the kernel want to allocate a swap buffer which causes more ZFS operations which never complete because ZFS would need to do some housework to free memory, but the kernel is waiting for swap to become available, some sort of deadlock? I'm just speculating here, not knowing the internals of this subsystem. The machine is a quad core system with 12GiB RAM and a RAIDZ3 consisting of 12x2TB disks. It's used as a storage system, with accesses mainly through netatalk (an AFP server, what OS X uses) and a bit of iSCSI. I've set the following sysctls: # http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 vfs.zfs.vdev.max_pending=20 # limit kernel size to 10GB to leave some RAM for processes vm.kmem_size_max="10737418240" # increase number of mbuf clusters, see igb(4). The default is 25600 kern.ipc.nmbclusters="102400" This sort of hang seems to happen every couple of months. I'd really like some pointers on how to increase the stability of this system. Maybe there are some more tunables I need to adjust? Or is swap on ZFS maybe just not a great idea? Cheers, Benjamin -- Benjamin Lutz | Software Engineer | BIOLAB Technology AG Dufourstr. 80 | CH-8008 Zurich | www.biolab.ch | benjamin.lutz at biolab.ch PHONE +41 44 295 97 13 | MOBILE +41 79 558 57 13 | FAX +41 44 295 97 19 This e-mail and the information it contains including attachments are confidential and meant only for use by the intended recipient(s); disclosure or copying is strictly prohibited. If you are not addressed, but in the possession of this e-mail, please notify the sender immediately.
Oscar Prieto
2013-Dec-10 17:36 UTC
Swap on ZFS - bad idea? (swap_pager: indefinite wait buffer)
Personally I still have to read an announcement telling that swap on zfs is relliable. I'm still using a dedicated swap partition on every install due to that kind of behaviour. On Tue, Dec 10, 2013 at 11:41 AM, Benjamin Lutz <benjamin.lutz at biolab.ch>wrote:> We have storage server that I've found unresponsive this morning. The > console was filled with these messages: > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 17344, size: 61440 > > The lines were all exactly the same, no variation in the numbers. > Ctrl-Alt-Delete had no effect. > > Now, the only thing related to swap that's special about this machine, is > that we use swap on ZFS, because we figured, silent disk corruption is > just as bad when it happens to swapped memory as it is when it happens to > stored data. However, might this be the problem? One thought I was having > was: ZFS hits its memory bounds, a process makes the kernel want to > allocate a swap buffer which causes more ZFS operations which never > complete because ZFS would need to do some housework to free memory, but > the kernel is waiting for swap to become available, some sort of deadlock? > I'm just speculating here, not knowing the internals of this subsystem. > > The machine is a quad core system with 12GiB RAM and a RAIDZ3 consisting > of 12x2TB disks. It's used as a storage system, with accesses mainly > through netatalk (an AFP server, what OS X uses) and a bit of iSCSI. I've > set the following sysctls: > > # > > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 > vfs.zfs.vdev.max_pending=20 > # limit kernel size to 10GB to leave some RAM for processes > vm.kmem_size_max="10737418240" > # increase number of mbuf clusters, see igb(4). The default is 25600 > kern.ipc.nmbclusters="102400" > > This sort of hang seems to happen every couple of months. I'd really like > some pointers on how to increase the stability of this system. Maybe there > are some more tunables I need to adjust? Or is swap on ZFS maybe just not > a great idea? > > Cheers, > Benjamin > -- > Benjamin Lutz | Software Engineer | BIOLAB Technology AG > Dufourstr. 80 | CH-8008 Zurich | www.biolab.ch | benjamin.lutz at biolab.ch > PHONE +41 44 295 97 13 | MOBILE +41 79 558 57 13 | FAX +41 44 295 97 19 > > > > This e-mail and the information it contains including attachments are > confidential and meant > only for use by the intended recipient(s); disclosure or copying is > strictly prohibited. If you > are not addressed, but in the possession of this e-mail, please notify the > sender immediately. > > _______________________________________________ > freebsd-stable at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org" >