Christopher S. Aker
2011-May-24 02:13 UTC
[Xen-devel] 2.6.39 domU - page allocation failure
Moments after deploying 2.6.39 32 bit this evening, got a report of the following. Recent pv_ops kernels seem to suffer from a too-low vm/min_free_kbytes value. Raising it from the default of ~2700 to 16k (on a 512M instance) seems to relieve some of these symptoms, but it''s not a scientific analysis. Noteworthy is the "HighMem free:0kB min:128kB low:136kB high:148kB" line from the spew. HighMem ratio calculations seem to be insufficient ... maybe I''ll try just setting it to 0. Any thoughts? Thanks! index.scgi: page allocation failure. order:5, mode:0x20 Pid: 1997, comm: index.scgi Not tainted 2.6.39-linode33 #1 Call Trace: [<c0189576>] ? __alloc_pages_nodemask+0x516/0x6d0 [<c01af4a0>] ? T.819+0x30/0xc0 [<c01af5c5>] ? T.816+0x95/0x250 [<c01af9c6>] ? cache_alloc_refill+0x246/0x290 [<c01afac4>] ? __kmalloc+0xb4/0xe0 [<c0509ffe>] ? pskb_expand_head+0x12e/0x200 [<c050a54d>] ? __pskb_pull_tail+0x4d/0x2b0 [<c05cfd73>] ? ipv4_confirm+0xd3/0x180 [<c0512ecd>] ? dev_hard_start_xmit+0x1dd/0x3e0 [<c0591270>] ? ip_finish_output2+0x250/0x250 [<c0591270>] ? ip_finish_output2+0x250/0x250 [<c0526bf2>] ? sch_direct_xmit+0xb2/0x170 [<c05131c9>] ? dev_queue_xmit+0xf9/0x320 [<c05913a7>] ? ip_finish_output+0x137/0x300 [<c059161a>] ? ip_output+0xaa/0xe0 [<c0591270>] ? ip_finish_output2+0x250/0x250 [<c05907f8>] ? ip_local_out+0x18/0x20 [<c0590bd7>] ? ip_queue_xmit+0x117/0x3d0 [<c0188d89>] ? get_page_from_freelist+0x119/0x340 [<c01c2d60>] ? __pollwait+0xf0/0xf0 [<c01062eb>] ? xen_restore_fl_direct_reloc+0x4/0x4 [<c05a420f>] ? tcp_transmit_skb+0x37f/0x660 [<c05a6c0b>] ? tcp_write_xmit+0x18b/0x4a0 [<c01062f4>] ? check_events+0x8/0xc [<c05a6f4d>] ? tcp_push_one+0x2d/0x40 [<c0599dc4>] ? tcp_sendmsg+0x9d4/0xab0 [<c05b7f27>] ? inet_sendmsg+0x47/0xb0 [<c05025e1>] ? sock_sendmsg+0xf1/0x110 [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 [<c0106258>] ? xen_vcpuop_set_next_event+0x48/0x80 [<c015a4a0>] ? clockevents_program_event+0xa0/0x160 [<c0502ece>] ? sys_sendto+0x11e/0x170 [<c010614a>] ? do_stolen_accounting+0x21a/0x250 [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 [<c0502f57>] ? sys_send+0x37/0x40 [<c05043d4>] ? sys_socketcall+0x1f4/0x2c0 [<c0683931>] ? syscall_call+0x7/0xb [<c0680000>] ? migrate_timers+0x80/0xbe Mem-Info: DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 CPU 2: hi: 0, btch: 1 usd: 0 CPU 3: hi: 0, btch: 1 usd: 0 Normal per-cpu: CPU 0: hi: 186, btch: 31 usd: 130 CPU 1: hi: 186, btch: 31 usd: 168 CPU 2: hi: 186, btch: 31 usd: 159 CPU 3: hi: 186, btch: 31 usd: 193 HighMem per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 CPU 1: hi: 0, btch: 1 usd: 0 CPU 2: hi: 0, btch: 1 usd: 0 CPU 3: hi: 0, btch: 1 usd: 0 active_anon:32680 inactive_anon:32950 isolated_anon:43 active_file:22048 inactive_file:23990 isolated_file:19 unevictable:0 dirty:122 writeback:8 unstable:0 free:2486 slab_reclaimable:1069 slab_unreclaimable:1605 mapped:5588 shmem:33178 pagetables:255 bounce:0 DMA free:2092kB min:84kB low:104kB high:124kB active_anon:588kB inactive_anon:632kB active_file:32kB inactive_file:2996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:1220kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no lowmem_reserve[]: 0 490 498 498 Normal free:7852kB min:2788kB low:3484kB high:4180kB active_anon:130132kB inactive_anon:131168kB active_file:88160kB inactive_file:92964kB unevictable:0kB isolated(anon):172kB isolated(file):76kB present:502240kB mlocked:0kB dirty:488kB writeback:32kB mapped:22332kB shmem:131492kB slab_reclaimable:4276kB slab_unreclaimable:6416kB kernel_stack:568kB pagetables:1020kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 all_unreclaimable? no lowmem_reserve[]: 0 0 64 64 HighMem free:0kB min:128kB low:136kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 DMA: 3*4kB 4*8kB 4*16kB 2*32kB 4*64kB 3*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2092kB Normal: 507*4kB 197*8kB 110*16kB 31*32kB 23*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7828kB HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB 79628 total pagecache pages 460 pages in swap cache Swap cache stats: add 1791, delete 1330, find 1257/1321 Free swap = 259640kB Total swap = 262140kB 1050608 pages RAM 865282 pages HighMem 930420 pages reserved 18088 pages shared 104397 pages non-shared -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-May-24 16:13 UTC
Re: [Xen-devel] 2.6.39 domU - page allocation failure
On Mon, May 23, 2011 at 10:13:21PM -0400, Christopher S. Aker wrote:> Moments after deploying 2.6.39 32 bit this evening, got a report of the following. Recent pv_ops kernels seem to suffer from a too-low vm/min_free_kbytes value. Raising it from the default of ~2700 to 16k (on a 512M instance) seems to relieve some of these symptoms, but it''s not a scientific analysis. > > Noteworthy is the "HighMem free:0kB min:128kB low:136kB high:148kB" line from the spew. HighMem ratio calculations seem to be insufficient ... maybe I''ll try just setting it to 0. > > Any thoughts? Thanks! > > index.scgi: page allocation failure. order:5, mode:0x20 > Pid: 1997, comm: index.scgi Not tainted 2.6.39-linode33 #1 > Call Trace: > [<c0189576>] ? __alloc_pages_nodemask+0x516/0x6d0 > [<c01af4a0>] ? T.819+0x30/0xc0 > [<c01af5c5>] ? T.816+0x95/0x250 > [<c01af9c6>] ? cache_alloc_refill+0x246/0x290 > [<c01afac4>] ? __kmalloc+0xb4/0xe0 > [<c0509ffe>] ? pskb_expand_head+0x12e/0x200 > [<c050a54d>] ? __pskb_pull_tail+0x4d/0x2b0 > [<c05cfd73>] ? ipv4_confirm+0xd3/0x180 > [<c0512ecd>] ? dev_hard_start_xmit+0x1dd/0x3e0 > [<c0591270>] ? ip_finish_output2+0x250/0x250 > [<c0591270>] ? ip_finish_output2+0x250/0x250 > [<c0526bf2>] ? sch_direct_xmit+0xb2/0x170 > [<c05131c9>] ? dev_queue_xmit+0xf9/0x320 > [<c05913a7>] ? ip_finish_output+0x137/0x300 > [<c059161a>] ? ip_output+0xaa/0xe0 > [<c0591270>] ? ip_finish_output2+0x250/0x250 > [<c05907f8>] ? ip_local_out+0x18/0x20 > [<c0590bd7>] ? ip_queue_xmit+0x117/0x3d0 > [<c0188d89>] ? get_page_from_freelist+0x119/0x340 > [<c01c2d60>] ? __pollwait+0xf0/0xf0 > [<c01062eb>] ? xen_restore_fl_direct_reloc+0x4/0x4 > [<c05a420f>] ? tcp_transmit_skb+0x37f/0x660 > [<c05a6c0b>] ? tcp_write_xmit+0x18b/0x4a0 > [<c01062f4>] ? check_events+0x8/0xc > [<c05a6f4d>] ? tcp_push_one+0x2d/0x40 > [<c0599dc4>] ? tcp_sendmsg+0x9d4/0xab0 > [<c05b7f27>] ? inet_sendmsg+0x47/0xb0 > [<c05025e1>] ? sock_sendmsg+0xf1/0x110 > [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 > [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 > [<c0106258>] ? xen_vcpuop_set_next_event+0x48/0x80 > [<c015a4a0>] ? clockevents_program_event+0xa0/0x160 > [<c0502ece>] ? sys_sendto+0x11e/0x170 > [<c010614a>] ? do_stolen_accounting+0x21a/0x250 > [<c011e0d6>] ? pvclock_clocksource_read+0x116/0x180 > [<c0502f57>] ? sys_send+0x37/0x40 > [<c05043d4>] ? sys_socketcall+0x1f4/0x2c0 > [<c0683931>] ? syscall_call+0x7/0xb > [<c0680000>] ? migrate_timers+0x80/0xbe > Mem-Info: > DMA per-cpu: > CPU 0: hi: 0, btch: 1 usd: 0 > CPU 1: hi: 0, btch: 1 usd: 0 > CPU 2: hi: 0, btch: 1 usd: 0 > CPU 3: hi: 0, btch: 1 usd: 0 > Normal per-cpu: > CPU 0: hi: 186, btch: 31 usd: 130 > CPU 1: hi: 186, btch: 31 usd: 168 > CPU 2: hi: 186, btch: 31 usd: 159 > CPU 3: hi: 186, btch: 31 usd: 193 > HighMem per-cpu: > CPU 0: hi: 0, btch: 1 usd: 0 > CPU 1: hi: 0, btch: 1 usd: 0 > CPU 2: hi: 0, btch: 1 usd: 0 > CPU 3: hi: 0, btch: 1 usd: 0 > active_anon:32680 inactive_anon:32950 isolated_anon:43 > active_file:22048 inactive_file:23990 isolated_file:19 > unevictable:0 dirty:122 writeback:8 unstable:0 > free:2486 slab_reclaimable:1069 slab_unreclaimable:1605 > mapped:5588 shmem:33178 pagetables:255 bounce:0 > DMA free:2092kB min:84kB low:104kB high:124kB active_anon:588kB inactive_anon:632kB active_file:32kB inactive_file:2996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:1220kB slab_reclaimable:0kB slab_unreclaimable:4kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no > lowmem_reserve[]: 0 490 498 498 > Normal free:7852kB min:2788kB low:3484kB high:4180kB active_anon:130132kB inactive_anon:131168kB active_file:88160kB inactive_file:92964kB unevictable:0kB isolated(anon):172kB isolated(file):76kB present:502240kB mlocked:0kB dirty:488kB writeback:32kB mapped:22332kB shmem:131492kB slab_reclaimable:4276kB slab_unreclaimable:6416kB kernel_stack:568kB pagetables:1020kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:62 all_unreclaimable? no > lowmem_reserve[]: 0 0 64 64 > HighMem free:0kB min:128kB low:136kB high:148kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8192kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > lowmem_reserve[]: 0 0 0 0 > DMA: 3*4kB 4*8kB 4*16kB 2*32kB 4*64kB 3*128kB 3*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 2092kB > Normal: 507*4kB 197*8kB 110*16kB 31*32kB 23*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7828kB > HighMem: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB > 79628 total pagecache pages > 460 pages in swap cache > Swap cache stats: add 1791, delete 1330, find 1257/1321 > Free swap = 259640kB > Total swap = 262140kB > 1050608 pages RAMOk, so 4G> 865282 pages HighMem > 930420 pages reservedThat is a large amount of reserved.. Did the previous guest (2.6.38) have less of a reserve amount? Can you for fun try to merge #stable/e820.bugfixes ?> 18088 pages shared > 104397 pages non-shared > > -Chris > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel