> I am trying to use qgroups > functionality & with a basic random-write workload, it constantly > keeps leaking memory & within few minutes of IO, there is either > out-of-memory killer trying to kill some tasks or there are > page-allocation failures that btrfs or other kernel module > experiences.FYI, I just saw something similar with 3.10 on a server with 32 GB RAM: The result was a frozen server and a need to hard reset. Aug 4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020 Aug 4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3 Aug 4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 Aug 4 21:51:14 bkp010 kernel: [661831.436943] 0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8 Aug 4 21:51:14 bkp010 kernel: [661831.437033] ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70 Aug 4 21:51:14 bkp010 kernel: [661831.437123] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 Aug 4 21:51:14 bkp010 kernel: [661831.437213] Call Trace: Aug 4 21:51:14 bkp010 kernel: [661831.437263] [<ffffffff81371348>] dump_stack+0x19/0x1b Aug 4 21:51:14 bkp010 kernel: [661831.437312] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 Aug 4 21:51:14 bkp010 kernel: [661831.437360] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 Aug 4 21:51:14 bkp010 kernel: [661831.437419] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 Aug 4 21:51:14 bkp010 kernel: [661831.437477] [<ffffffff810d3fb9>] new_slab+0xb5/0x269 Aug 4 21:51:14 bkp010 kernel: [661831.437525] [<ffffffff813701b2>] __slab_alloc+0x30e/0x493 Aug 4 21:51:14 bkp010 kernel: [661831.437583] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.437637] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.437686] [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110 Aug 4 21:51:14 bkp010 kernel: [661831.437737] [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20 Aug 4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 Aug 4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 Aug 4 21:51:14 bkp010 kernel: [661831.437772] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 Aug 4 21:51:14 bkp010 kernel: [661831.437774] ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8 Aug 4 21:51:14 bkp010 kernel: [661831.437775] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 Aug 4 21:51:14 bkp010 kernel: [661831.437775] Call Trace: Aug 4 21:51:14 bkp010 kernel: [661831.437778] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b Aug 4 21:51:14 bkp010 kernel: [661831.437780] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 Aug 4 21:51:14 bkp010 kernel: [661831.437782] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 Aug 4 21:51:14 bkp010 kernel: [661831.437785] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 Aug 4 21:51:14 bkp010 kernel: [661831.437788] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3 Aug 4 21:51:14 bkp010 kernel: [661831.437789] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb Aug 4 21:51:14 bkp010 kernel: [661831.437798] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169] Aug 4 21:51:14 bkp010 kernel: [661831.437800] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182 Aug 4 21:51:14 bkp010 kernel: [661831.437803] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d Aug 4 21:51:14 bkp010 kernel: [661831.437804] [<ffffffff81034dde>] irq_exit+0x3e/0x83 Aug 4 21:51:14 bkp010 kernel: [661831.437807] [<ffffffff81003c35>] do_IRQ+0x98/0xaf Aug 4 21:51:14 bkp010 kernel: [661831.437809] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a Aug 4 21:51:14 bkp010 kernel: [661831.437811] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac Aug 4 21:51:14 bkp010 kernel: [661831.437812] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac Aug 4 21:51:14 bkp010 kernel: [661831.437813] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f Aug 4 21:51:14 bkp010 kernel: [661831.437815] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18 Aug 4 21:51:14 bkp010 kernel: [661831.437816] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104 Aug 4 21:51:14 bkp010 kernel: [661831.437819] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3 Aug 4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info: Aug 4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 1: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 2: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 3: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 4: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 5: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437824] CPU 6: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437825] CPU 7: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 0: hi: 186, btch: 31 usd: 176 Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 1: hi: 186, btch: 31 usd: 37 Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 2: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 3: hi: 186, btch: 31 usd: 41 Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 4: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 5: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437829] CPU 6: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.437830] CPU 7: hi: 186, btch: 31 usd: 172 Aug 4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.437831] CPU 0: hi: 186, btch: 31 usd: 180 Aug 4 21:51:14 bkp010 kernel: [661831.437832] CPU 1: hi: 186, btch: 31 usd: 64 Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 2: hi: 186, btch: 31 usd: 60 Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 3: hi: 186, btch: 31 usd: 84 Aug 4 21:51:14 bkp010 kernel: [661831.437834] CPU 4: hi: 186, btch: 31 usd: 90 Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 5: hi: 186, btch: 31 usd: 66 Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 6: hi: 186, btch: 31 usd: 20 Aug 4 21:51:14 bkp010 kernel: [661831.437836] CPU 7: hi: 186, btch: 31 usd: 209 Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0 Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_file:12576 inactive_file:14674 isolated_file:0 Aug 4 21:51:14 bkp010 kernel: [661831.437839] unevictable:0 dirty:4574 writeback:17253 unstable:0 Aug 4 21:51:14 bkp010 kernel: [661831.437839] free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894 Aug 4 21:51:14 bkp010 kernel: [661831.437839] mapped:1456 shmem:169 pagetables:3293 bounce:0 Aug 4 21:51:14 bkp010 kernel: [661831.437839] free_cma:0 Aug 4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846 Aug 4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672 Aug 4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0 Aug 4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB Aug 4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB Aug 4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB Aug 4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages Aug 4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache Aug 4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659 Aug 4 21:51:14 bkp010 kernel: [661831.437870] Free swap = 16634240kB Aug 4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB Aug 4 21:51:14 bkp010 kernel: [661831.443483] [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443574] [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443626] [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443680] [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443772] [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443825] [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.443874] [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a Aug 4 21:51:14 bkp010 kernel: [661831.443925] [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444018] [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444069] [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444163] [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444215] [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444307] [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444360] [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444448] [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14 Aug 4 21:51:14 bkp010 kernel: [661831.444496] [<ffffffff810a3717>] ? mempool_free+0x72/0x7a Aug 4 21:51:14 bkp010 kernel: [661831.444547] [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444639] [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444692] [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444744] [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs] Aug 4 21:51:14 bkp010 kernel: [661831.444793] [<ffffffff81048ab2>] kthread+0xb5/0xbd Aug 4 21:51:14 bkp010 kernel: [661831.444840] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43 Aug 4 21:51:14 bkp010 kernel: [661831.444889] [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0 Aug 4 21:51:14 bkp010 kernel: [661831.444936] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43 Aug 4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info: Aug 4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.445070] CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445115] CPU 1: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445161] CPU 2: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445206] CPU 3: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20 Aug 4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 Aug 4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 Aug 4 21:51:14 bkp010 kernel: [661831.445238] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 Aug 4 21:51:14 bkp010 kernel: [661831.445240] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8 Aug 4 21:51:14 bkp010 kernel: [661831.445241] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 Aug 4 21:51:14 bkp010 kernel: [661831.445241] Call Trace: Aug 4 21:51:14 bkp010 kernel: [661831.445244] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b Aug 4 21:51:14 bkp010 kernel: [661831.445245] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 Aug 4 21:51:14 bkp010 kernel: [661831.445247] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 Aug 4 21:51:14 bkp010 kernel: [661831.445250] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 Aug 4 21:51:14 bkp010 kernel: [661831.445252] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3 Aug 4 21:51:14 bkp010 kernel: [661831.445254] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb Aug 4 21:51:14 bkp010 kernel: [661831.445260] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169] Aug 4 21:51:14 bkp010 kernel: [661831.445261] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182 Aug 4 21:51:14 bkp010 kernel: [661831.445264] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d Aug 4 21:51:14 bkp010 kernel: [661831.445265] [<ffffffff81034dde>] irq_exit+0x3e/0x83 Aug 4 21:51:14 bkp010 kernel: [661831.445267] [<ffffffff81003c35>] do_IRQ+0x98/0xaf Aug 4 21:51:14 bkp010 kernel: [661831.445268] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a Aug 4 21:51:14 bkp010 kernel: [661831.445270] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac Aug 4 21:51:14 bkp010 kernel: [661831.445271] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac Aug 4 21:51:14 bkp010 kernel: [661831.445272] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f Aug 4 21:51:14 bkp010 kernel: [661831.445274] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18 Aug 4 21:51:14 bkp010 kernel: [661831.445275] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104 Aug 4 21:51:14 bkp010 kernel: [661831.445277] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3 Aug 4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info: Aug 4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 0: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 1: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 2: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 3: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 4: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 5: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 6: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445282] CPU 7: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.445283] CPU 0: hi: 186, btch: 31 usd: 165 Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 1: hi: 186, btch: 31 usd: 37 Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 2: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 3: hi: 186, btch: 31 usd: 41 Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 4: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 5: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 6: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.445287] CPU 7: hi: 186, btch: 31 usd: 172 Aug 4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 0: hi: 186, btch: 31 usd: 181 Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 1: hi: 186, btch: 31 usd: 64 Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 2: hi: 186, btch: 31 usd: 60 Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 3: hi: 186, btch: 31 usd: 84 Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 4: hi: 186, btch: 31 usd: 90 Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 5: hi: 186, btch: 31 usd: 66 Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 6: hi: 186, btch: 31 usd: 20 Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 7: hi: 186, btch: 31 usd: 209 Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0 Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_file:12576 inactive_file:14654 isolated_file:0 Aug 4 21:51:14 bkp010 kernel: [661831.445293] unevictable:0 dirty:4574 writeback:17362 unstable:0 Aug 4 21:51:14 bkp010 kernel: [661831.445293] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894 Aug 4 21:51:14 bkp010 kernel: [661831.445293] mapped:1456 shmem:169 pagetables:3293 bounce:0 Aug 4 21:51:14 bkp010 kernel: [661831.445293] free_cma:0 Aug 4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846 Aug 4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672 Aug 4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0 Aug 4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB Aug 4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB Aug 4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB Aug 4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages Aug 4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache Aug 4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659 Aug 4 21:51:14 bkp010 kernel: [661831.445327] Free swap = 16633976kB Aug 4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB Aug 4 21:51:14 bkp010 kernel: [661831.450878] CPU 4: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.450924] CPU 5: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.450969] CPU 6: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451014] CPU 7: hi: 0, btch: 1 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.451104] CPU 0: hi: 186, btch: 31 usd: 165 Aug 4 21:51:14 bkp010 kernel: [661831.451149] CPU 1: hi: 186, btch: 31 usd: 37 Aug 4 21:51:14 bkp010 kernel: [661831.451194] CPU 2: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451240] CPU 3: hi: 186, btch: 31 usd: 41 Aug 4 21:51:14 bkp010 kernel: [661831.451285] CPU 4: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451330] CPU 5: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451376] CPU 6: hi: 186, btch: 31 usd: 0 Aug 4 21:51:14 bkp010 kernel: [661831.451421] CPU 7: hi: 186, btch: 31 usd: 172 Aug 4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu: Aug 4 21:51:14 bkp010 kernel: [661831.451510] CPU 0: hi: 186, btch: 31 usd: 181 Aug 4 21:51:14 bkp010 kernel: [661831.451556] CPU 1: hi: 186, btch: 31 usd: 64 Aug 4 21:51:14 bkp010 kernel: [661831.451601] CPU 2: hi: 186, btch: 31 usd: 60 Aug 4 21:51:14 bkp010 kernel: [661831.451647] CPU 3: hi: 186, btch: 31 usd: 84 Aug 4 21:51:14 bkp010 kernel: [661831.451692] CPU 4: hi: 186, btch: 31 usd: 90 Aug 4 21:51:14 bkp010 kernel: [661831.451737] CPU 5: hi: 186, btch: 31 usd: 66 Aug 4 21:51:14 bkp010 kernel: [661831.451783] CPU 6: hi: 186, btch: 31 usd: 20 Aug 4 21:51:14 bkp010 kernel: [661831.451828] CPU 7: hi: 186, btch: 31 usd: 209 Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0 Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_file:12576 inactive_file:14654 isolated_file:0 Aug 4 21:51:14 bkp010 kernel: [661831.451875] unevictable:0 dirty:4574 writeback:17362 unstable:0 Aug 4 21:51:14 bkp010 kernel: [661831.451875] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894 Aug 4 21:51:14 bkp010 kernel: [661831.451875] mapped:1456 shmem:169 pagetables:3293 bounce:0 Aug 4 21:51:14 bkp010 kernel: [661831.451875] free_cma:0 Aug 4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Aug 4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846 Aug 4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672 Aug 4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no Aug 4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0 Aug 4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20 Aug 4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 Aug 4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 Aug 4 21:51:14 bkp010 kernel: [661831.453233] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 Aug 4 21:51:14 bkp010 kernel: [661831.453235] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8 Aug 4 21:51:14 bkp010 kernel: [661831.453236] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hello.>> I am trying to use qgroups >> functionality & with a basic random-write workload, it constantly >> keeps leaking memory & within few minutes of IO, there is either >> out-of-memory killer trying to kill some tasks or there are >> page-allocation failures that btrfs or other kernel module >> experiences. > > FYI, I just saw something similar with 3.10 on a server with 32 GB RAM:Would please show how many qgroups are there in the system. btrfs qgroup show <mnt>> > The result was a frozen server and a need to hard reset. > > Aug 4 21:51:14 bkp010 kernel: [661831.436751] btrfs-endio-wri: page allocation failure: order:0, mode:0x284020 > Aug 4 21:51:14 bkp010 kernel: [661831.436805] CPU: 3 PID: 19598 Comm: btrfs-endio-wri Not tainted 3.10.0 #3 > Aug 4 21:51:14 bkp010 kernel: [661831.436853] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > Aug 4 21:51:14 bkp010 kernel: [661831.436943] 0000000000284020 ffff88047e3fd518 ffffffff81371348 ffff88047e3fd5a8 > Aug 4 21:51:14 bkp010 kernel: [661831.437033] ffffffff810a5629 0000000000000000 ffffea0002adab20 ffff88081fad4a70 > Aug 4 21:51:14 bkp010 kernel: [661831.437123] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 > Aug 4 21:51:14 bkp010 kernel: [661831.437213] Call Trace: > Aug 4 21:51:14 bkp010 kernel: [661831.437263] [<ffffffff81371348>] dump_stack+0x19/0x1b > Aug 4 21:51:14 bkp010 kernel: [661831.437312] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 > Aug 4 21:51:14 bkp010 kernel: [661831.437360] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 > Aug 4 21:51:14 bkp010 kernel: [661831.437419] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 > Aug 4 21:51:14 bkp010 kernel: [661831.437477] [<ffffffff810d3fb9>] new_slab+0xb5/0x269 > Aug 4 21:51:14 bkp010 kernel: [661831.437525] [<ffffffff813701b2>] __slab_alloc+0x30e/0x493 > Aug 4 21:51:14 bkp010 kernel: [661831.437583] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.437637] [<ffffffffa071b6a6>] ? __alloc_extent_buffer+0x25/0xd8 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.437686] [<ffffffff810d5ddb>] kmem_cache_alloc+0x95/0x110 > Aug 4 21:51:14 bkp010 kernel: [661831.437737] [<ffffffffa071b6a6>] __alloc_extent_buffer+0x25/0xd8 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.437769] swapper/7: page allocation failure: order:0, mode:0x20 > Aug 4 21:51:14 bkp010 kernel: [661831.437770] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 > Aug 4 21:51:14 bkp010 kernel: [661831.437771] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > Aug 4 21:51:14 bkp010 kernel: [661831.437772] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 > Aug 4 21:51:14 bkp010 kernel: [661831.437774] ffffffff810a5629 ffff880400000000 0000000000011780 ffff88081fbc3bd8 > Aug 4 21:51:14 bkp010 kernel: [661831.437775] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 > Aug 4 21:51:14 bkp010 kernel: [661831.437775] Call Trace: > Aug 4 21:51:14 bkp010 kernel: [661831.437778] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b > Aug 4 21:51:14 bkp010 kernel: [661831.437780] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 > Aug 4 21:51:14 bkp010 kernel: [661831.437782] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 > Aug 4 21:51:14 bkp010 kernel: [661831.437785] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 > Aug 4 21:51:14 bkp010 kernel: [661831.437788] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3 > Aug 4 21:51:14 bkp010 kernel: [661831.437789] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb > Aug 4 21:51:14 bkp010 kernel: [661831.437798] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169] > Aug 4 21:51:14 bkp010 kernel: [661831.437800] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182 > Aug 4 21:51:14 bkp010 kernel: [661831.437803] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d > Aug 4 21:51:14 bkp010 kernel: [661831.437804] [<ffffffff81034dde>] irq_exit+0x3e/0x83 > Aug 4 21:51:14 bkp010 kernel: [661831.437807] [<ffffffff81003c35>] do_IRQ+0x98/0xaf > Aug 4 21:51:14 bkp010 kernel: [661831.437809] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a > Aug 4 21:51:14 bkp010 kernel: [661831.437811] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac > Aug 4 21:51:14 bkp010 kernel: [661831.437812] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac > Aug 4 21:51:14 bkp010 kernel: [661831.437813] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f > Aug 4 21:51:14 bkp010 kernel: [661831.437815] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18 > Aug 4 21:51:14 bkp010 kernel: [661831.437816] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104 > Aug 4 21:51:14 bkp010 kernel: [661831.437819] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3 > Aug 4 21:51:14 bkp010 kernel: [661831.437819] Mem-Info: > Aug 4 21:51:14 bkp010 kernel: [661831.437820] Node 0 DMA per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 0: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437821] CPU 1: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 2: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437822] CPU 3: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 4: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437823] CPU 5: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437824] CPU 6: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437825] CPU 7: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437825] Node 0 DMA32 per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 0: hi: 186, btch: 31 usd: 176 > Aug 4 21:51:14 bkp010 kernel: [661831.437826] CPU 1: hi: 186, btch: 31 usd: 37 > Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 2: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437827] CPU 3: hi: 186, btch: 31 usd: 41 > Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 4: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437828] CPU 5: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437829] CPU 6: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437830] CPU 7: hi: 186, btch: 31 usd: 172 > Aug 4 21:51:14 bkp010 kernel: [661831.437830] Node 0 Normal per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.437831] CPU 0: hi: 186, btch: 31 usd: 180 > Aug 4 21:51:14 bkp010 kernel: [661831.437832] CPU 1: hi: 186, btch: 31 usd: 64 > Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 2: hi: 186, btch: 31 usd: 60 > Aug 4 21:51:14 bkp010 kernel: [661831.437833] CPU 3: hi: 186, btch: 31 usd: 84 > Aug 4 21:51:14 bkp010 kernel: [661831.437834] CPU 4: hi: 186, btch: 31 usd: 90 > Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 5: hi: 186, btch: 31 usd: 66 > Aug 4 21:51:14 bkp010 kernel: [661831.437835] CPU 6: hi: 186, btch: 31 usd: 20 > Aug 4 21:51:14 bkp010 kernel: [661831.437836] CPU 7: hi: 186, btch: 31 usd: 209 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_anon:49535 inactive_anon:56858 isolated_anon:0 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] active_file:12576 inactive_file:14674 isolated_file:0 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] unevictable:0 dirty:4574 writeback:17253 unstable:0 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] free:34679 slab_reclaimable:1154786 slab_unreclaimable:18894 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] mapped:1456 shmem:169 pagetables:3293 bounce:0 > Aug 4 21:51:14 bkp010 kernel: [661831.437839] free_cma:0 > Aug 4 21:51:14 bkp010 kernel: [661831.437843] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > Aug 4 21:51:14 bkp010 kernel: [661831.437845] lowmem_reserve[]: 0 3174 31846 31846 > Aug 4 21:51:14 bkp010 kernel: [661831.437849] Node 0 DMA32 free:115440kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32616kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:68 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.437850] lowmem_reserve[]: 0 0 28672 28672 > Aug 4 21:51:14 bkp010 kernel: [661831.437853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69012kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:101 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.437854] lowmem_reserve[]: 0 0 0 0 > Aug 4 21:51:14 bkp010 kernel: [661831.437859] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > Aug 4 21:51:14 bkp010 kernel: [661831.437863] Node 0 DMA32: 28844*4kB (EM) 0*8kB 0*16kB 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115408kB > Aug 4 21:51:14 bkp010 kernel: [661831.437867] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB > Aug 4 21:51:14 bkp010 kernel: [661831.437868] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > Aug 4 21:51:14 bkp010 kernel: [661831.437869] 44196 total pagecache pages > Aug 4 21:51:14 bkp010 kernel: [661831.437870] 16756 pages in swap cache > Aug 4 21:51:14 bkp010 kernel: [661831.437870] Swap cache stats: add 828307, delete 811551, find 615437/665659 > Aug 4 21:51:14 bkp010 kernel: [661831.437870] Free swap = 16634240kB > Aug 4 21:51:14 bkp010 kernel: [661831.437871] Total swap = 16776116kB > Aug 4 21:51:14 bkp010 kernel: [661831.443483] [<ffffffffa072128d>] btrfs_clone_extent_buffer+0x3a/0xc3 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443574] [<ffffffffa06ebefb>] btrfs_search_old_slot+0x680/0x78b [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443626] [<ffffffffa06edd71>] btrfs_next_old_leaf+0xc9/0x3a6 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443680] [<ffffffffa074da50>] __resolve_indirect_refs+0x331/0x467 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443772] [<ffffffffa074e205>] find_parent_nodes+0x357/0x5be [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443825] [<ffffffffa074e4e8>] btrfs_find_all_roots+0x7c/0xd7 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.443874] [<ffffffff810d58b9>] ? kmem_cache_free+0x164/0x17a > Aug 4 21:51:14 bkp010 kernel: [661831.443925] [<ffffffffa07514bd>] btrfs_qgroup_account_ref+0x10c/0x44f [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444018] [<ffffffffa071c49f>] ? clear_state_bit+0x106/0x115 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444069] [<ffffffffa06f316f>] btrfs_delayed_refs_qgroup_accounting+0xab/0xdf [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444163] [<ffffffffa0706896>] __btrfs_end_transaction+0x49/0x281 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444215] [<ffffffffa06f4514>] ? btrfs_block_rsv_release+0x49/0x4b [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444307] [<ffffffffa0706af9>] btrfs_end_transaction+0xb/0xd [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444360] [<ffffffffa0711224>] btrfs_finish_ordered_io+0x683/0x84e [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444448] [<ffffffff810a3683>] ? mempool_free_slab+0x12/0x14 > Aug 4 21:51:14 bkp010 kernel: [661831.444496] [<ffffffff810a3717>] ? mempool_free+0x72/0x7a > Aug 4 21:51:14 bkp010 kernel: [661831.444547] [<ffffffffa073c357>] ? end_compressed_bio_write+0xc0/0xc8 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444639] [<ffffffffa07113ff>] finish_ordered_fn+0x10/0x12 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444692] [<ffffffffa072ab5c>] worker_loop+0x15e/0x48e [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444744] [<ffffffffa072a9fe>] ? btrfs_queue_worker+0x267/0x267 [btrfs] > Aug 4 21:51:14 bkp010 kernel: [661831.444793] [<ffffffff81048ab2>] kthread+0xb5/0xbd > Aug 4 21:51:14 bkp010 kernel: [661831.444840] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43 > Aug 4 21:51:14 bkp010 kernel: [661831.444889] [<ffffffff8137576c>] ret_from_fork+0x7c/0xb0 > Aug 4 21:51:14 bkp010 kernel: [661831.444936] [<ffffffff810489fd>] ? kthread_freezable_should_stop+0x43/0x43 > Aug 4 21:51:14 bkp010 kernel: [661831.444984] Mem-Info: > Aug 4 21:51:14 bkp010 kernel: [661831.445026] Node 0 DMA per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.445070] CPU 0: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445115] CPU 1: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445161] CPU 2: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445206] CPU 3: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445235] swapper/7: page allocation failure: order:0, mode:0x20 > Aug 4 21:51:14 bkp010 kernel: [661831.445237] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 > Aug 4 21:51:14 bkp010 kernel: [661831.445237] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > Aug 4 21:51:14 bkp010 kernel: [661831.445238] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 > Aug 4 21:51:14 bkp010 kernel: [661831.445240] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8 > Aug 4 21:51:14 bkp010 kernel: [661831.445241] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 > Aug 4 21:51:14 bkp010 kernel: [661831.445241] Call Trace: > Aug 4 21:51:14 bkp010 kernel: [661831.445244] <IRQ> [<ffffffff81371348>] dump_stack+0x19/0x1b > Aug 4 21:51:14 bkp010 kernel: [661831.445245] [<ffffffff810a5629>] warn_alloc_failed+0x110/0x124 > Aug 4 21:51:14 bkp010 kernel: [661831.445247] [<ffffffff810a7be3>] __alloc_pages_nodemask+0x6a4/0x793 > Aug 4 21:51:14 bkp010 kernel: [661831.445250] [<ffffffff810d07fc>] alloc_pages_current+0xc8/0xe5 > Aug 4 21:51:14 bkp010 kernel: [661831.445252] [<ffffffff812c9a91>] __netdev_alloc_frag+0x52/0xf3 > Aug 4 21:51:14 bkp010 kernel: [661831.445254] [<ffffffff812cb019>] __netdev_alloc_skb+0x43/0xdb > Aug 4 21:51:14 bkp010 kernel: [661831.445260] [<ffffffffa0007375>] rtl8169_poll+0x241/0x4e2 [r8169] > Aug 4 21:51:14 bkp010 kernel: [661831.445261] [<ffffffff812d47f2>] net_rx_action+0xa3/0x182 > Aug 4 21:51:14 bkp010 kernel: [661831.445264] [<ffffffff81034c55>] __do_softirq+0xb7/0x16d > Aug 4 21:51:14 bkp010 kernel: [661831.445265] [<ffffffff81034dde>] irq_exit+0x3e/0x83 > Aug 4 21:51:14 bkp010 kernel: [661831.445267] [<ffffffff81003c35>] do_IRQ+0x98/0xaf > Aug 4 21:51:14 bkp010 kernel: [661831.445268] [<ffffffff813750ea>] common_interrupt+0x6a/0x6a > Aug 4 21:51:14 bkp010 kernel: [661831.445270] <EOI> [<ffffffff812a3a67>] ? cpuidle_enter_state+0x49/0xac > Aug 4 21:51:14 bkp010 kernel: [661831.445271] [<ffffffff812a3a60>] ? cpuidle_enter_state+0x42/0xac > Aug 4 21:51:14 bkp010 kernel: [661831.445272] [<ffffffff812a3b8d>] cpuidle_idle_call+0xc3/0x10f > Aug 4 21:51:14 bkp010 kernel: [661831.445274] [<ffffffff810097de>] arch_cpu_idle+0x9/0x18 > Aug 4 21:51:14 bkp010 kernel: [661831.445275] [<ffffffff81062a7c>] cpu_startup_entry+0xa8/0x104 > Aug 4 21:51:14 bkp010 kernel: [661831.445277] [<ffffffff8136581f>] start_secondary+0x1af/0x1b3 > Aug 4 21:51:14 bkp010 kernel: [661831.445278] Mem-Info: > Aug 4 21:51:14 bkp010 kernel: [661831.445278] Node 0 DMA per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 0: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445279] CPU 1: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 2: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 3: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445280] CPU 4: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 5: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445281] CPU 6: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445282] CPU 7: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445282] Node 0 DMA32 per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.445283] CPU 0: hi: 186, btch: 31 usd: 165 > Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 1: hi: 186, btch: 31 usd: 37 > Aug 4 21:51:14 bkp010 kernel: [661831.445284] CPU 2: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 3: hi: 186, btch: 31 usd: 41 > Aug 4 21:51:14 bkp010 kernel: [661831.445285] CPU 4: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 5: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445286] CPU 6: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445287] CPU 7: hi: 186, btch: 31 usd: 172 > Aug 4 21:51:14 bkp010 kernel: [661831.445287] Node 0 Normal per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 0: hi: 186, btch: 31 usd: 181 > Aug 4 21:51:14 bkp010 kernel: [661831.445288] CPU 1: hi: 186, btch: 31 usd: 64 > Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 2: hi: 186, btch: 31 usd: 60 > Aug 4 21:51:14 bkp010 kernel: [661831.445289] CPU 3: hi: 186, btch: 31 usd: 84 > Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 4: hi: 186, btch: 31 usd: 90 > Aug 4 21:51:14 bkp010 kernel: [661831.445290] CPU 5: hi: 186, btch: 31 usd: 66 > Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 6: hi: 186, btch: 31 usd: 20 > Aug 4 21:51:14 bkp010 kernel: [661831.445291] CPU 7: hi: 186, btch: 31 usd: 209 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_anon:49535 inactive_anon:56858 isolated_anon:0 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] active_file:12576 inactive_file:14654 isolated_file:0 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] unevictable:0 dirty:4574 writeback:17362 unstable:0 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] mapped:1456 shmem:169 pagetables:3293 bounce:0 > Aug 4 21:51:14 bkp010 kernel: [661831.445293] free_cma:0 > Aug 4 21:51:14 bkp010 kernel: [661831.445298] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > Aug 4 21:51:14 bkp010 kernel: [661831.445299] lowmem_reserve[]: 0 3174 31846 31846 > Aug 4 21:51:14 bkp010 kernel: [661831.445303] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.445305] lowmem_reserve[]: 0 0 28672 28672 > Aug 4 21:51:14 bkp010 kernel: [661831.445309] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.445310] lowmem_reserve[]: 0 0 0 0 > Aug 4 21:51:14 bkp010 kernel: [661831.445316] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB > Aug 4 21:51:14 bkp010 kernel: [661831.445320] Node 0 DMA32: 28841*4kB (UEM) 15*8kB (UEM) 1*16kB (M) 1*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 115532kB > Aug 4 21:51:14 bkp010 kernel: [661831.445324] Node 0 Normal: 1367*4kB (M) 0*8kB 0*16kB 7*32kB (R) 1*64kB (R) 1*128kB (R) 1*256kB (R) 0*512kB 1*1024kB (R) 0*2048kB 0*4096kB = 7164kB > Aug 4 21:51:14 bkp010 kernel: [661831.445325] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > Aug 4 21:51:14 bkp010 kernel: [661831.445325] 44279 total pagecache pages > Aug 4 21:51:14 bkp010 kernel: [661831.445326] 16820 pages in swap cache > Aug 4 21:51:14 bkp010 kernel: [661831.445326] Swap cache stats: add 828373, delete 811553, find 615437/665659 > Aug 4 21:51:14 bkp010 kernel: [661831.445327] Free swap = 16633976kB > Aug 4 21:51:14 bkp010 kernel: [661831.445327] Total swap = 16776116kB > Aug 4 21:51:14 bkp010 kernel: [661831.450878] CPU 4: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.450924] CPU 5: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.450969] CPU 6: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451014] CPU 7: hi: 0, btch: 1 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451059] Node 0 DMA32 per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.451104] CPU 0: hi: 186, btch: 31 usd: 165 > Aug 4 21:51:14 bkp010 kernel: [661831.451149] CPU 1: hi: 186, btch: 31 usd: 37 > Aug 4 21:51:14 bkp010 kernel: [661831.451194] CPU 2: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451240] CPU 3: hi: 186, btch: 31 usd: 41 > Aug 4 21:51:14 bkp010 kernel: [661831.451285] CPU 4: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451330] CPU 5: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451376] CPU 6: hi: 186, btch: 31 usd: 0 > Aug 4 21:51:14 bkp010 kernel: [661831.451421] CPU 7: hi: 186, btch: 31 usd: 172 > Aug 4 21:51:14 bkp010 kernel: [661831.451466] Node 0 Normal per-cpu: > Aug 4 21:51:14 bkp010 kernel: [661831.451510] CPU 0: hi: 186, btch: 31 usd: 181 > Aug 4 21:51:14 bkp010 kernel: [661831.451556] CPU 1: hi: 186, btch: 31 usd: 64 > Aug 4 21:51:14 bkp010 kernel: [661831.451601] CPU 2: hi: 186, btch: 31 usd: 60 > Aug 4 21:51:14 bkp010 kernel: [661831.451647] CPU 3: hi: 186, btch: 31 usd: 84 > Aug 4 21:51:14 bkp010 kernel: [661831.451692] CPU 4: hi: 186, btch: 31 usd: 90 > Aug 4 21:51:14 bkp010 kernel: [661831.451737] CPU 5: hi: 186, btch: 31 usd: 66 > Aug 4 21:51:14 bkp010 kernel: [661831.451783] CPU 6: hi: 186, btch: 31 usd: 20 > Aug 4 21:51:14 bkp010 kernel: [661831.451828] CPU 7: hi: 186, btch: 31 usd: 209 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_anon:49535 inactive_anon:56858 isolated_anon:0 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] active_file:12576 inactive_file:14654 isolated_file:0 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] unevictable:0 dirty:4574 writeback:17362 unstable:0 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] free:34697 slab_reclaimable:1154786 slab_unreclaimable:18894 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] mapped:1456 shmem:169 pagetables:3293 bounce:0 > Aug 4 21:51:14 bkp010 kernel: [661831.451875] free_cma:0 > Aug 4 21:51:14 bkp010 kernel: [661831.452149] Node 0 DMA free:15900kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > Aug 4 21:51:14 bkp010 kernel: [661831.452432] lowmem_reserve[]: 0 3174 31846 31846 > Aug 4 21:51:14 bkp010 kernel: [661831.452479] Node 0 DMA32 free:115512kB min:2276kB low:2844kB high:3412kB active_anon:56kB inactive_anon:28672kB active_file:31172kB inactive_file:32536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3334376kB managed:3250672kB mlocked:0kB dirty:144kB writeback:0kB mapped:1184kB shmem:0kB slab_reclaimable:1883292kB slab_unreclaimable:14256kB kernel_stack:80kB pagetables:20kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.452806] lowmem_reserve[]: 0 0 28672 28672 > Aug 4 21:51:14 bkp010 kernel: [661831.452853] Node 0 Normal free:7376kB min:20560kB low:25700kB high:30840kB active_anon:198084kB inactive_anon:198760kB active_file:19132kB inactive_file:26080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29882368kB managed:29360436kB mlocked:0kB dirty:18152kB writeback:69448kB mapped:4640kB shmem:676kB slab_reclaimable:2735852kB slab_unreclaimable:61320kB kernel_stack:3984kB pagetables:13152kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:311 all_unreclaimable? no > Aug 4 21:51:14 bkp010 kernel: [661831.453182] lowmem_reserve[]: 0 0 0 0 > Aug 4 21:51:14 bkp010 kernel: [661831.453230] swapper/7: page allocation failure: order:0, mode:0x20 > Aug 4 21:51:14 bkp010 kernel: [661831.453231] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 3.10.0 #3 > Aug 4 21:51:14 bkp010 kernel: [661831.453231] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013 > Aug 4 21:51:14 bkp010 kernel: [661831.453233] 0000000000000020 ffff88081fbc3b88 ffffffff81371348 ffff88081fbc3c18 > Aug 4 21:51:14 bkp010 kernel: [661831.453235] ffffffff810a5629 ffff880700000000 ffff88081fa51780 ffff88081fbc3bc8 > Aug 4 21:51:14 bkp010 kernel: [661831.453236] ffff88081fdd9a00 ffff88081fdd9a38 0000000100000000 0000000000000000 > > > -- > Tomasz Chmielewski > http://wpkg.org > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 6 Aug 2013 07:29:03 +0800 Wang Shilong <wangshilong1991@gmail.com> wrote:> Hello. > > >> I am trying to use qgroups > >> functionality & with a basic random-write workload, it constantly > >> keeps leaking memory & within few minutes of IO, there is either > >> out-of-memory killer trying to kill some tasks or there are > >> page-allocation failures that btrfs or other kernel module > >> experiences. > > > > FYI, I just saw something similar with 3.10 on a server with 32 GB > > RAM: > > Would please show how many qgroups are there in the system. > > btrfs qgroup show <mnt>There are two different btrfs filesystems here; notice there are quite a bit of qgroups with negative numbers. # btrfs qgroup show /mnt/lxc1 0/258 2068160512 2068160512 0/1601 0 0 0/2359 1361963708416 1361963708416 0/3870 12996984832 4096 0/3871 12936032256 275820544 0/3872 40094982144 -3129344 0/3873 1238417408 -286720 0/3900 40061427712 2592768 0/3901 0 4096 0/3902 -4096 0 0/3903 895385600 4096 0/3904 -8192 -4096 0/3905 40075735040 3133440 0/3906 -4096 0 0/3908 891162624 114688 0/3909 40076042240 372736 0/3910 -4096 0 0/3911 891224064 139264 0/3912 40076103680 708608 0/3913 7718686720 4096 0/3918 7631826944 3203072 0/3922 891256832 278528 0/3923 5142913024 4096 0/3924 0 0 0/3928 4530425856 21856256 0/3973 891744256 4096 0/3974 854753280 4096 0/3975 41452056576 4096 0/3976 72792285184 4096 0/3977 43131879424 4096 0/3978 882757632 0 0/3980 882757632 0 0/3981 852537344 0 0/3983 2410848256 4096 0/3984 852537344 0 0/3985 109482221568 4096 0/3987 -4096 0 0/3989 34457268224 4096 0/3990 -4096 0 0/4005 2454089728 0 0/4009 2454089728 0 0/4014 10458886144 4096 0/4020 1543729152 4096 0/4022 1015640064 4096 0/4039 55234445312 4096 0/4059 1014833152 0 0/4061 1014833152 0 0/4074 1457680384 0 0/4077 1457680384 0 0/4083 24318590976 0 0/4087 24318590976 0 0/4162 10207858688 0 0/4164 10207858688 0 0/4243 1598795776 1347584 0/4245 1598795776 0 0/4277 43304882176 0 0/4280 43304882176 0 0/4328 0 0 0/4339 108909649920 0 0/4341 108909649920 0 0/4342 72641134592 0 0/4343 72641134592 0 0/4346 54204522496 0 0/4347 54204522496 0 0/4348 5137539072 2932736 0/4349 7714144256 729088 0/4350 894935040 618496 0/4351 12996780032 210608128 0/4352 40108421120 8347648 0/4353 41457135616 99024896 0/4354 852701184 17911808 0/4355 888950784 23416832 0/4356 72641462272 45056 0/4357 43208957952 54571008 0/4358 2400776192 41017344 0/4359 109428551680 117022720 0/4360 24396820480 11505664 0/4361 10275590144 12472320 0/4362 1543102464 22564864 0/4363 1014951936 425984 0/4364 54707920896 183185408 0/4365 5142007808 5021696 0/4366 7714271232 774144 0/4367 895250432 733184 0/4368 78995476480 4096 0/4369 12999450624 209784832 0/4370 40116916224 7626752 0/4371 41466302464 11272192 0/4372 853426176 18030592 0/4373 889573376 23425024 0/4374 72641757184 57344 0/4375 79529275392 1884889088 0/4376 43121913856 58900480 0/4377 2397597696 37494784 0/4378 109448671232 149102592 0/4379 24440803328 27103232 0/4380 10342871040 8065024 0/4381 1543413760 557056 0/4382 1015025664 466944 0/4383 54857736192 180170752 0/4384 77708091392 10326016 0/4385 5139664896 4096 0/4386 5139664896 4096 0/4387 7718678528 4096 0/4388 7718678528 4096 0/4389 895197184 4096 0/4390 895197184 4096 0/4391 12951941120 4096 0/4392 12951941120 4096 0/4393 40108793856 4096 0/4394 40108793856 4096 0/4396 41437810688 4096 0/4397 41437810688 4096 0/4398 854634496 4096 0/4399 854634496 4096 0/4400 890630144 4096 0/4401 890630144 4096 0/4402 72791916544 4096 0/4403 72791916544 4096 0/4404 43130384384 4096 0/4405 43130384384 4096 0/4406 2395500544 4096 0/4407 2395500544 4096 0/4408 109446926336 4096 0/4409 109446926336 4096 0/4410 24507891712 4096 0/4411 24507891712 4096 0/4412 10401660928 4096 0/4413 10401660928 4096 0/4414 1543028736 4096 0/4415 1543028736 4096 0/4416 1014980608 4096 0/4417 1014980608 4096 0/4418 54934056960 4096 0/4419 54934056960 4096 0/4420 5142913024 4096 0/4421 7718686720 4096 0/4422 895385600 4096 0/4423 12996984832 4096 0/4424 40112762880 4096 0/4425 41452056576 4096 0/4426 854753280 4096 0/4427 891744256 4096 0/4428 72792285184 4096 0/4429 43131879424 4096 0/4430 2410848256 4096 0/4431 109482221568 4096 0/4432 34457268224 4096 0/4433 10458886144 4096 0/4434 1543729152 4096 0/4435 1015640064 4096 0/4436 55234445312 4096 0/4437 78995476480 4096 # btrfs qgroup show /mnt/lxc2 0/260 161639960576 161639960576 0/261 181149696 181149696 0/262 155557888 1007616 0/377 180408320 25899008 0/378 336346464256 336346464256 0/535 572301312 417751040 0/536 70203207680 70203207680 0/642 247451648 92934144 0/1276 631365632 4096 0/1277 38598176768 0 0/1278 603922432 -12591104 0/1279 38268366848 -21348352 0/1284 38299160576 0 0/1285 38299160576 0 0/1286 607039488 -12697600 0/1287 620396544 241664 0/1293 38301650944 21417984 0/1294 623255552 13103104 0/1303 38449602560 10883072 0/1304 624873472 4096 0/1305 624873472 4096 0/1307 38449627136 106496 0/1319 38450397184 94208 0/1321 628367360 192512 0/1322 0 0 0/1323 0 0 0/1324 0 -1051197440 0/1325 1051197440 0 0/1326 1073741824 0 0/1331 38597541888 61440 0/1332 630042624 118784 0/1335 38598545408 4096 0/1336 38598545408 4096 0/1337 631840768 4096 0/1338 631840768 4096 1/1 375578464256 375578464256 2/1 -4096 22732800 -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 05.08.2013 18:35, Tomasz Chmielewski wrote:>> I am trying to use qgroups >> functionality & with a basic random-write workload, it constantly >> keeps leaking memory & within few minutes of IO, there is either >> out-of-memory killer trying to kill some tasks or there are >> page-allocation failures that btrfs or other kernel module >> experiences. > > FYI, I just saw something similar with 3.10 on a server with 32 GB RAM: > > The result was a frozen server and a need to hard reset. >What do I have to do to reproduce it here? How do you generate the load? What is the disk setup, what the qgroups setup? Thanks, Arne -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 06 Aug 2013 07:24:26 +0200 Arne Jansen <sensille@gmx.net> wrote:> On 05.08.2013 18:35, Tomasz Chmielewski wrote: > >> I am trying to use qgroups > >> functionality & with a basic random-write workload, it constantly > >> keeps leaking memory & within few minutes of IO, there is either > >> out-of-memory killer trying to kill some tasks or there are > >> page-allocation failures that btrfs or other kernel module > >> experiences. > > > > FYI, I just saw something similar with 3.10 on a server with 32 GB > > RAM: > > > > The result was a frozen server and a need to hard reset. > > > > What do I have to do to reproduce it here? How do you generate the > load? What is the disk setup, what the qgroups setup?Unfortunately I don''t have a way to reproduce, as the issue happened to me only once. Server uptime was about 2 weeks, and there were millions of files with multiple hardlinks on disk (backuppc archive copied from ext4 to btrfs). Then, "rm" command was running for many days, about 5 times in parallel, and many other rsync commands, with snapshots being added and removed. When I saw the issue I looked up in btrfs list archives that there was a similar report in the past - but I doubt I will be able to find a reliable way to reproduce this. -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 7 Aug 2013 00:42:27 +0800 Tomasz Chmielewski <mangoo@wpkg.org> wrote:> > What do I have to do to reproduce it here? How do you generate the > > load? What is the disk setup, what the qgroups setup? > > Unfortunately I don''t have a way to reproduce, as the issue > happened to me only once.I keep removing the files, and memory usage has grown to 28 GB (out of 32 GB). Is there anything I can debug at this point? # free total used free shared buffers cached Mem: 32638512 32476992 161520 0 732 3531152 -/+ buffers/cache: 28945108 3693404 Swap: 16776116 173392 16602724 # cat /proc/meminfo MemTotal: 32638512 kB MemFree: 163364 kB Buffers: 864 kB Cached: 3531940 kB SwapCached: 60992 kB Active: 3599804 kB Inactive: 2422176 kB Active(anon): 1860412 kB Inactive(anon): 629848 kB Active(file): 1739392 kB Inactive(file): 1792328 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 16776116 kB SwapFree: 16602916 kB Dirty: 85432 kB Writeback: 10320 kB AnonPages: 2458792 kB Mapped: 11068 kB Shmem: 796 kB Slab: 3168820 kB SReclaimable: 3099388 kB SUnreclaim: 69432 kB KernelStack: 3408 kB PageTables: 14764 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 33095372 kB Committed_AS: 4301988 kB VmallocTotal: 34359738367 kB VmallocUsed: 130036 kB VmallocChunk: 34359595900 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 12520 kB DirectMap2M: 33220608 kB -- Tomasz Chmielewski http://wpkg.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
I''m pretty I encountered the same leak over here, 15 days uptime and I had consistent out of memory problems, I got angry and did go in single user mode and noticed that with only systemd->screen->bash(2) I had 2 GB of memory used on 4 GB total. This box mostly has random reads and just before the out of memory condition had 12 GB written to a lot of files. # btrfs qgroup show /mnt/btrfs 0/337 13953818624 9494970368 0/338 0 0 0/353 20934205440 20934205440 0/373 36420952064 36420952064 0/668 0 19161088 0/669 21279420416 16839733248 0/890 -97652736 -97652736 0/907 65536 65536 0/910 0 0 Screenshot of top/processes/memory: http://i39.tinypic.com/2rrrjtl.jpg On Wed, Aug 7, 2013 at 4:00 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:> On Wed, 7 Aug 2013 00:42:27 +0800 > Tomasz Chmielewski <mangoo@wpkg.org> wrote: > >> > What do I have to do to reproduce it here? How do you generate the >> > load? What is the disk setup, what the qgroups setup? >> >> Unfortunately I don''t have a way to reproduce, as the issue >> happened to me only once. > > I keep removing the files, and memory usage has grown to 28 GB (out of > 32 GB). > > Is there anything I can debug at this point? > > # free > total used free shared buffers cached > Mem: 32638512 32476992 161520 0 732 3531152 > -/+ buffers/cache: 28945108 3693404 > Swap: 16776116 173392 16602724 > > > # cat /proc/meminfo > MemTotal: 32638512 kB > MemFree: 163364 kB > Buffers: 864 kB > Cached: 3531940 kB > SwapCached: 60992 kB > Active: 3599804 kB > Inactive: 2422176 kB > Active(anon): 1860412 kB > Inactive(anon): 629848 kB > Active(file): 1739392 kB > Inactive(file): 1792328 kB > Unevictable: 0 kB > Mlocked: 0 kB > SwapTotal: 16776116 kB > SwapFree: 16602916 kB > Dirty: 85432 kB > Writeback: 10320 kB > AnonPages: 2458792 kB > Mapped: 11068 kB > Shmem: 796 kB > Slab: 3168820 kB > SReclaimable: 3099388 kB > SUnreclaim: 69432 kB > KernelStack: 3408 kB > PageTables: 14764 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 33095372 kB > Committed_AS: 4301988 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 130036 kB > VmallocChunk: 34359595900 kB > HardwareCorrupted: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 12520 kB > DirectMap2M: 33220608 kB > > -- > Tomasz Chmielewski > http://wpkg.org > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html