John Brunelle
2013-Jun-13 19:38 UTC
[Gluster-users] incomplete listing of a directory, sometimes getdents loops until out of memory
Hello, We're having an issue with our distributed gluster filesystem: * gluster 3.3.1 servers and clients * distributed volume -- 69 bricks (4.6T each) split evenly across 3 nodes * xfs backend * nfs clients * nfs.enable-ino32: On * servers: CentOS 6.3, 2.6.32-279.14.1.el6.centos.plus.x86_64 * cleints: CentOS 5.7, 2.6.18-274.12.1.el5 We have a directory containing 3,343 subdirectories. On some clients, ls lists only a subset of the directories (a different amount on different clients). On others, ls gets stuck in a getdents loop and consumes more and more memory until it hits ENOMEM. On yet others, it works fine. Having the bad clients remount or drop caches makes the problem temporarily go away, but eventually it comes back. The issue sounds a lot like bug #838784, but we are using xfs on the backend, and this seems like more of a client issue. But we are also getting some page allocation failures on the server side, e.g. the stack strace below. These are nearly identical to bug #842206 and bug #767127. I'm trying to sort out if these are related to the above issue or just recoverable nic driver GFP_ATOMIC kmalloc failures as suggested in the comments. Slab allocations for dentry, xfs_inode, fuse_inode, fuse_request, etc. are all at ~100% active, and the total number appears to be monotonically growing. Overall memory looks healthy (2/3 is buffers/cache, almost no swap is used). I'd need some help to determine if the memory is overly fragmented or not, but looking at pagetypeinfo and zoneinfo It doesn't appear so to me, and the failures are order:1 anyways. Any suggestions for what might be the problem here? Thanks, John Jun 13 09:41:18 myhost kernel: glusterfsd: page allocation failure. order:1, mode:0x20 Jun 13 09:41:18 myhost kernel: Pid: 20498, comm: glusterfsd Not tainted 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 Jun 13 09:41:18 myhost kernel: Call Trace: Jun 13 09:41:18 myhost kernel: <IRQ> [<ffffffff8112790f>] ? __alloc_pages_nodemask+0x77f/0x940 Jun 13 09:41:18 myhost kernel: [<ffffffff81162382>] ? kmem_getpages+0x62/0x170 Jun 13 09:41:18 myhost kernel: [<ffffffff81162f9a>] ? fallback_alloc+0x1ba/0x270 Jun 13 09:41:18 myhost kernel: [<ffffffff811629ef>] ? cache_grow+0x2cf/0x320 Jun 13 09:41:18 myhost kernel: [<ffffffff81162d19>] ? ____cache_alloc_node+0x99/0x160 Jun 13 09:41:18 myhost kernel: [<ffffffff81163afb>] ? kmem_cache_alloc+0x11b/0x190 Jun 13 09:41:18 myhost kernel: [<ffffffff81435298>] ? sk_prot_alloc+0x48/0x1c0 Jun 13 09:41:18 myhost kernel: [<ffffffff81435562>] ? sk_clone+0x22/0x2e0 Jun 13 09:41:18 myhost kernel: [<ffffffff814833a6>] ? inet_csk_clone+0x16/0xd0 Jun 13 09:41:18 myhost kernel: [<ffffffff8149c383>] ? tcp_create_openreq_child+0x23/0x450 Jun 13 09:41:18 myhost kernel: [<ffffffff81499bed>] ? tcp_v4_syn_recv_sock+0x4d/0x310 Jun 13 09:41:18 myhost kernel: [<ffffffff8149c126>] ? tcp_check_req+0x226/0x460 Jun 13 09:41:18 myhost kernel: [<ffffffff81437087>] ? __kfree_skb+0x47/0xa0 Jun 13 09:41:18 myhost kernel: [<ffffffff8149960b>] ? tcp_v4_do_rcv+0x35b/0x430 Jun 13 09:41:18 myhost kernel: [<ffffffff8149ae4e>] ? tcp_v4_rcv+0x4fe/0x8d0 Jun 13 09:41:18 myhost kernel: [<ffffffff81432f6c>] ? sk_reset_timer+0x1c/0x30 Jun 13 09:41:18 myhost kernel: [<ffffffff81478add>] ? ip_local_deliver_finish+0xdd/0x2d0 Jun 13 09:41:18 myhost kernel: [<ffffffff81478d68>] ? ip_local_deliver+0x98/0xa0 Jun 13 09:41:18 myhost kernel: [<ffffffff8147822d>] ? ip_rcv_finish+0x12d/0x440 Jun 13 09:41:18 myhost kernel: [<ffffffff814787b5>] ? ip_rcv+0x275/0x350 Jun 13 09:41:18 myhost kernel: [<ffffffff81441deb>] ? __netif_receive_skb+0x49b/0x6f0 Jun 13 09:41:18 myhost kernel: [<ffffffff8149813a>] ? tcp4_gro_receive+0x5a/0xd0 Jun 13 09:41:18 myhost kernel: [<ffffffff81444068>] ? netif_receive_skb+0x58/0x60 Jun 13 09:41:18 myhost kernel: [<ffffffff81444170>] ? napi_skb_finish+0x50/0x70 Jun 13 09:41:18 myhost kernel: [<ffffffff814466a9>] ? napi_gro_receive+0x39/0x50 Jun 13 09:41:18 myhost kernel: [<ffffffffa01303b4>] ? igb_poll+0x864/0xb00 [igb] Jun 13 09:41:18 myhost kernel: [<ffffffff810606ec>] ? rebalance_domains+0x3cc/0x5a0 Jun 13 09:41:18 myhost kernel: [<ffffffff814467c3>] ? net_rx_action+0x103/0x2f0 Jun 13 09:41:18 myhost kernel: [<ffffffff81096523>] ? hrtimer_get_next_event+0xc3/0x100 Jun 13 09:41:18 myhost kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0 Jun 13 09:41:18 myhost kernel: [<ffffffff810dbb70>] ? handle_IRQ_event+0x60/0x170 Jun 13 09:41:18 myhost kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30 Jun 13 09:41:18 myhost kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0 Jun 13 09:41:18 myhost kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90 Jun 13 09:41:18 myhost kernel: [<ffffffff8150d505>] ? do_IRQ+0x75/0xf0 Jun 13 09:41:18 myhost kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Vijay Bellur
2013-Jun-14 14:35 UTC
[Gluster-users] incomplete listing of a directory, sometimes getdents loops until out of memory
On 06/13/2013 03:38 PM, John Brunelle wrote:> Hello, > > We're having an issue with our distributed gluster filesystem: > > * gluster 3.3.1 servers and clients > * distributed volume -- 69 bricks (4.6T each) split evenly across 3 nodes > * xfs backend > * nfs clients > * nfs.enable-ino32: On > > * servers: CentOS 6.3, 2.6.32-279.14.1.el6.centos.plus.x86_64 > * cleints: CentOS 5.7, 2.6.18-274.12.1.el5 > > We have a directory containing 3,343 subdirectories. On some clients, > ls lists only a subset of the directories (a different amount on > different clients). On others, ls gets stuck in a getdents loop and > consumes more and more memory until it hits ENOMEM. On yet others, it > works fine. Having the bad clients remount or drop caches makes the > problem temporarily go away, but eventually it comes back. The issue > sounds a lot like bug #838784, but we are using xfs on the backend, > and this seems like more of a client issue.Turning on "cluster.readdir-optimize" can help readdir when a directory contains a number of sub-directories and there are more bricks in the volume. Do you observe any change with this option enabled? -Vijay
Jeff Darcy
2013-Jun-14 16:01 UTC
[Gluster-users] incomplete listing of a directory, sometimes getdents loops until out of memory
On 06/13/2013 03:38 PM, John Brunelle wrote:> We have a directory containing 3,343 subdirectories. On some > clients, ls lists only a subset of the directories (a different > amount on different clients). On others, ls gets stuck in a getdents > loop and consumes more and more memory until it hits ENOMEM. On yet > others, it works fine. Having the bad clients remount or drop caches > makes the problem temporarily go away, but eventually it comes back. > The issue sounds a lot like bug #838784, but we are using xfs on the > backend, and this seems like more of a client issue.The fact that drop_caches makes it go away temporarily suggests to me that something's going on in FUSE. The reference to #838784 might also be significant even though you're not using ext4. Even the fix for that still makes some assumptions about how certain directory-entry fields are used and might still be sensitive to changes in that usage by the local FS or by FUSE. That might explain both skipping and looping, as you say you've seen. Would it be possible for you to compile and run the attached program on one of the affected directories so we can see what d_off values are involved?> But we are also getting some page allocation failures on the server > side, e.g. the stack strace below. These are nearly identical to > bug #842206 and bug #767127. I'm trying to sort out if these are > related to the above issue or just recoverable nic driver GFP_ATOMIC > kmalloc failures as suggested in the comments. Slab allocations for > dentry, xfs_inode, fuse_inode, fuse_request, etc. are all at ~100% > active, and the total number appears to be monotonically growing. > Overall memory looks healthy (2/3 is buffers/cache, almost no swap is > used). I'd need some help to determine if the memory is overly > fragmented or not, but looking at pagetypeinfo and zoneinfo It > doesn't appear so to me, and the failures are order:1 anyways.This one definitely seems like one of those "innocent victim" kind of things where the real problem is in the network code and we just happen to be the app that's running. -------------- next part -------------- A non-text attachment was scrubbed... Name: readdir.c Type: text/x-csrc Size: 702 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130614/378f2e83/attachment.bin>