Daniel Musketa
2011-Aug-01 22:10 UTC
[Xen-devel] 2.6.32 pvops dom0 oom when creating a domU
Hello, when running Jeremy''s 2.6.32.43 kernel I get OOM in dom0 when trying to create a domU. This happened both on xen-4.0.2.gz hypervisor and Debian''s 4.0.1 xen tools and on xen-4.1.1.gz and compiled 4.1.1 xen tools. Full (serial console) output is attached. The OOM part looks like this: -------- 8< -------- [ 654.208337] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 654.215814] cron cpuset=/ mems_allowed=0 [ 654.220276] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 654.226477] Call Trace: [ 654.229295] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 654.234914] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 654.241604] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 654.248101] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 654.255469] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 654.263033] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 654.270304] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 654.276217] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 654.282714] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 654.288822] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 654.296094] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 654.303946] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 654.310637] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 654.316939] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 654.323143] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 654.329639] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 654.335649] Mem-Info: [ 654.338266] Node 0 DMA per-cpu: [ 654.341854] CPU 0: hi: 0, btch: 1 usd: 0 [ 654.347284] CPU 1: hi: 0, btch: 1 usd: 0 [ 654.352713] CPU 2: hi: 0, btch: 1 usd: 0 [ 654.358143] CPU 3: hi: 0, btch: 1 usd: 0 [ 654.363572] CPU 4: hi: 0, btch: 1 usd: 0 [ 654.369001] CPU 5: hi: 0, btch: 1 usd: 0 [ 654.374431] CPU 6: hi: 0, btch: 1 usd: 0 [ 654.379860] CPU 7: hi: 0, btch: 1 usd: 0 [ 654.385290] CPU 8: hi: 0, btch: 1 usd: 0 [ 654.390719] CPU 9: hi: 0, btch: 1 usd: 0 [ 654.396148] CPU 10: hi: 0, btch: 1 usd: 0 [ 654.401579] CPU 11: hi: 0, btch: 1 usd: 0 [ 654.407007] Node 0 DMA32 per-cpu: [ 654.410789] CPU 0: hi: 186, btch: 31 usd: 30 [ 654.416219] CPU 1: hi: 186, btch: 31 usd: 30 [ 654.421647] CPU 2: hi: 186, btch: 31 usd: 0 [ 654.427077] CPU 3: hi: 186, btch: 31 usd: 0 [ 654.432506] CPU 4: hi: 186, btch: 31 usd: 0 [ 654.437936] CPU 5: hi: 186, btch: 31 usd: 0 [ 654.443366] CPU 6: hi: 186, btch: 31 usd: 0 [ 654.448796] CPU 7: hi: 186, btch: 31 usd: 0 [ 654.454225] CPU 8: hi: 186, btch: 31 usd: 0 [ 654.459654] CPU 9: hi: 186, btch: 31 usd: 0 [ 654.465084] CPU 10: hi: 186, btch: 31 usd: 2 [ 654.470513] CPU 11: hi: 186, btch: 31 usd: 0 [ 654.475942] Node 0 Normal per-cpu: [ 654.479821] CPU 0: hi: 186, btch: 31 usd: 29 [ 654.485250] CPU 1: hi: 186, btch: 31 usd: 30 [ 654.490679] CPU 2: hi: 186, btch: 31 usd: 0 [ 654.496109] CPU 3: hi: 186, btch: 31 usd: 30 [ 654.501539] CPU 4: hi: 186, btch: 31 usd: 0 [ 654.506968] CPU 5: hi: 186, btch: 31 usd: 0 [ 654.512398] CPU 6: hi: 186, btch: 31 usd: 0 [ 654.517827] CPU 7: hi: 186, btch: 31 usd: 0 [ 654.523256] CPU 8: hi: 186, btch: 31 usd: 0 [ 654.528686] CPU 9: hi: 186, btch: 31 usd: 0 [ 654.534116] CPU 10: hi: 186, btch: 31 usd: 0 [ 654.539545] CPU 11: hi: 186, btch: 31 usd: 0 [ 654.544976] active_anon:13004 inactive_anon:3325 isolated_anon:160 [ 654.544977] active_file:308 inactive_file:1114 isolated_file:64 [ 654.544978] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 654.544979] free:30116 slab_reclaimable:1654 slab_unreclaimable:10667 [ 654.544980] mapped:193 shmem:26 pagetables:639 bounce:0 [ 654.578715] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 654.619339] lowmem_reserve[]: 0 2171 25559 25559 [ 654.624574] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52016kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6616kB slab_unreclaimable:41608kB kernel_stack:2384kB pagetables:2556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 654.671016] lowmem_reserve[]: 0 0 23387 23387 [ 654.675960] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 654.718523] lowmem_reserve[]: 0 0 0 0 [ 654.722692] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 654.734618] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 654.747804] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 654.760021] 2124 total pagecache pages [ 654.764286] 601 pages in swap cache [ 654.768261] Swap cache stats: add 2432, delete 1831, find 0/0 [ 654.774757] Free swap = 514552kB [ 654.778541] Total swap = 524280kB [ 654.827020] 6660096 pages RAM [ 654.830229] 6207063 pages reserved [ 654.834108] 3196 pages shared [ 654.837500] 420123 pages non-shared [ 654.841481] Out of memory: kill process 2768 (exim4) score 11037 or a child [ 654.849332] Killed process 2768 (exim4) [ 654.854024] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 654.861504] cron cpuset=/ mems_allowed=0 [ 654.865961] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 654.872165] Call Trace: [ 654.874983] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 654.880605] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 654.887290] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 654.893788] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 654.901156] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 654.908718] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 654.915988] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 654.921903] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 654.928400] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 654.934507] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 654.941779] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 654.949633] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 654.956324] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 654.962626] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 654.968832] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 654.975326] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 654.981336] Mem-Info: [ 654.983952] Node 0 DMA per-cpu: [ 654.987540] CPU 0: hi: 0, btch: 1 usd: 0 [ 654.992970] CPU 1: hi: 0, btch: 1 usd: 0 [ 654.998400] CPU 2: hi: 0, btch: 1 usd: 0 [ 655.003828] CPU 3: hi: 0, btch: 1 usd: 0 [ 655.009306] CPU 4: hi: 0, btch: 1 usd: 0 [ 655.014689] CPU 5: hi: 0, btch: 1 usd: 0 [ 655.020120] CPU 6: hi: 0, btch: 1 usd: 0 [ 655.025546] CPU 7: hi: 0, btch: 1 usd: 0 [ 655.030977] CPU 8: hi: 0, btch: 1 usd: 0 [ 655.036405] CPU 9: hi: 0, btch: 1 usd: 0 [ 655.041836] CPU 10: hi: 0, btch: 1 usd: 0 [ 655.047265] CPU 11: hi: 0, btch: 1 usd: 0 [ 655.052696] Node 0 DMA32 per-cpu: [ 655.056476] CPU 0: hi: 186, btch: 31 usd: 30 [ 655.061905] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.067334] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.072764] CPU 3: hi: 186, btch: 31 usd: 0 [ 655.078193] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.083622] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.089052] CPU 6: hi: 186, btch: 31 usd: 35 [ 655.094481] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.099911] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.105376] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.110806] CPU 10: hi: 186, btch: 31 usd: 2 [ 655.116235] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.121665] Node 0 Normal per-cpu: [ 655.125546] CPU 0: hi: 186, btch: 31 usd: 29 [ 655.130972] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.136402] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.141832] CPU 3: hi: 186, btch: 31 usd: 30 [ 655.147260] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.152691] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.158120] CPU 6: hi: 186, btch: 31 usd: 0 [ 655.163548] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.168978] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.174409] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.179837] CPU 10: hi: 186, btch: 31 usd: 0 [ 655.185267] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.190698] active_anon:13004 inactive_anon:3325 isolated_anon:160 [ 655.190699] active_file:308 inactive_file:1114 isolated_file:64 [ 655.190700] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 655.190701] free:30116 slab_reclaimable:1654 slab_unreclaimable:10667 [ 655.190702] mapped:193 shmem:26 pagetables:639 bounce:0 [ 655.224437] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.265063] lowmem_reserve[]: 0 2171 25559 25559 [ 655.270297] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52016kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6616kB slab_unreclaimable:41608kB kernel_stack:2384kB pagetables:2556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 655.316737] lowmem_reserve[]: 0 0 23387 23387 [ 655.321682] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.364246] lowmem_reserve[]: 0 0 0 0 [ 655.368414] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 655.380340] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 655.393526] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 655.405743] 2124 total pagecache pages [ 655.410009] 601 pages in swap cache [ 655.413984] Swap cache stats: add 2432, delete 1831, find 0/0 [ 655.420479] Free swap = 514948kB [ 655.424261] Total swap = 524280kB [ 655.476216] 6660096 pages RAM [ 655.479425] 6207063 pages reserved [ 655.483303] 3175 pages shared [ 655.486697] 420088 pages non-shared [ 655.490673] Out of memory: kill process 2974 (xend) score 9860 or a child [ 655.498333] Killed process 2975 (xend) [ 655.506939] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 655.514415] cron cpuset=/ mems_allowed=0 [ 655.518875] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 655.525079] Call Trace: [ 655.527896] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 655.533516] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 655.540206] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 655.546702] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 655.554071] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 655.561634] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 655.568905] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 655.574819] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 655.581315] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 655.587422] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 655.594694] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 655.602548] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 655.609238] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 655.615541] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 655.621745] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 655.628240] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 655.634250] Mem-Info: [ 655.636868] Node 0 DMA per-cpu: [ 655.640458] CPU 0: hi: 0, btch: 1 usd: 0 [ 655.645884] CPU 1: hi: 0, btch: 1 usd: 0 [ 655.651315] CPU 2: hi: 0, btch: 1 usd: 0 [ 655.656743] CPU 3: hi: 0, btch: 1 usd: 0 [ 655.662173] CPU 4: hi: 0, btch: 1 usd: 0 [ 655.667603] CPU 5: hi: 0, btch: 1 usd: 0 [ 655.673032] CPU 6: hi: 0, btch: 1 usd: 0 [ 655.678461] CPU 7: hi: 0, btch: 1 usd: 0 [ 655.683890] CPU 8: hi: 0, btch: 1 usd: 0 [ 655.689320] CPU 9: hi: 0, btch: 1 usd: 0 [ 655.694750] CPU 10: hi: 0, btch: 1 usd: 0 [ 655.700180] CPU 11: hi: 0, btch: 1 usd: 0 [ 655.705609] Node 0 DMA32 per-cpu: [ 655.709389] CPU 0: hi: 186, btch: 31 usd: 30 [ 655.714820] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.720250] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.725679] CPU 3: hi: 186, btch: 31 usd: 0 [ 655.731109] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.736538] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.741968] CPU 6: hi: 186, btch: 31 usd: 35 [ 655.747397] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.752826] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.758255] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.763685] CPU 10: hi: 186, btch: 31 usd: 2 [ 655.769114] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.774543] Node 0 Normal per-cpu: [ 655.778422] CPU 0: hi: 186, btch: 31 usd: 29 [ 655.783852] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.789281] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.794710] CPU 3: hi: 186, btch: 31 usd: 30 [ 655.800141] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.805570] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.810999] CPU 6: hi: 186, btch: 31 usd: 0 [ 655.816428] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.821858] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.827288] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.832716] CPU 10: hi: 186, btch: 31 usd: 0 [ 655.838147] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.843578] active_anon:13003 inactive_anon:3325 isolated_anon:160 [ 655.843579] active_file:308 inactive_file:1114 isolated_file:64 [ 655.843580] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 655.843581] free:30116 slab_reclaimable:1653 slab_unreclaimable:10665 [ 655.843581] mapped:193 shmem:26 pagetables:616 bounce:0 [ 655.877316] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.917940] lowmem_reserve[]: 0 2171 25559 25559 [ 655.923175] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52012kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6612kB slab_unreclaimable:41600kB kernel_stack:2384kB pagetables:2464kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 655.969617] lowmem_reserve[]: 0 0 23387 23387 [ 655.974561] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 656.017126] lowmem_reserve[]: 0 0 0 0 [ 656.021293] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 656.033219] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 656.046404] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 656.058623] 2124 total pagecache pages [ 656.062887] 601 pages in swap cache [ 656.066863] Swap cache stats: add 2432, delete 1831, find 0/0 [ 656.073358] Free swap = 514948kB [ 656.077139] Total swap = 524280kB [ 656.124934] 6660096 pages RAM [ 656.128145] 6207063 pages reserved [ 656.132023] 3176 pages shared [ 656.135417] 420088 pages non-shared [ 656.139393] Out of memory: kill process 2792 (ntpd) score 9615 or a child [ 656.147051] Killed process 2792 (ntpd) -------- >8 -------- Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Musketa
2011-Aug-01 22:25 UTC
[Xen-devel] 2.6.32 pvops dom0 oom when creating a domU
Hello, when running Jeremy''s 2.6.32.43 kernel I get OOM in dom0 when trying to create a domU. This happened both on xen-4.0.2.gz hypervisor and Debian''s 4.0.1 xen tools and on xen-4.1.1.gz and compiled 4.1.1 xen tools. Full (serial console) output is attached. The OOM part looks like this: -------- 8< -------- [ 654.208337] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 654.215814] cron cpuset=/ mems_allowed=0 [ 654.220276] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 654.226477] Call Trace: [ 654.229295] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 654.234914] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 654.241604] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 654.248101] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 654.255469] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 654.263033] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 654.270304] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 654.276217] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 654.282714] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 654.288822] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 654.296094] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 654.303946] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 654.310637] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 654.316939] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 654.323143] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 654.329639] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 654.335649] Mem-Info: [ 654.338266] Node 0 DMA per-cpu: [ 654.341854] CPU 0: hi: 0, btch: 1 usd: 0 [ 654.347284] CPU 1: hi: 0, btch: 1 usd: 0 [ 654.352713] CPU 2: hi: 0, btch: 1 usd: 0 [ 654.358143] CPU 3: hi: 0, btch: 1 usd: 0 [ 654.363572] CPU 4: hi: 0, btch: 1 usd: 0 [ 654.369001] CPU 5: hi: 0, btch: 1 usd: 0 [ 654.374431] CPU 6: hi: 0, btch: 1 usd: 0 [ 654.379860] CPU 7: hi: 0, btch: 1 usd: 0 [ 654.385290] CPU 8: hi: 0, btch: 1 usd: 0 [ 654.390719] CPU 9: hi: 0, btch: 1 usd: 0 [ 654.396148] CPU 10: hi: 0, btch: 1 usd: 0 [ 654.401579] CPU 11: hi: 0, btch: 1 usd: 0 [ 654.407007] Node 0 DMA32 per-cpu: [ 654.410789] CPU 0: hi: 186, btch: 31 usd: 30 [ 654.416219] CPU 1: hi: 186, btch: 31 usd: 30 [ 654.421647] CPU 2: hi: 186, btch: 31 usd: 0 [ 654.427077] CPU 3: hi: 186, btch: 31 usd: 0 [ 654.432506] CPU 4: hi: 186, btch: 31 usd: 0 [ 654.437936] CPU 5: hi: 186, btch: 31 usd: 0 [ 654.443366] CPU 6: hi: 186, btch: 31 usd: 0 [ 654.448796] CPU 7: hi: 186, btch: 31 usd: 0 [ 654.454225] CPU 8: hi: 186, btch: 31 usd: 0 [ 654.459654] CPU 9: hi: 186, btch: 31 usd: 0 [ 654.465084] CPU 10: hi: 186, btch: 31 usd: 2 [ 654.470513] CPU 11: hi: 186, btch: 31 usd: 0 [ 654.475942] Node 0 Normal per-cpu: [ 654.479821] CPU 0: hi: 186, btch: 31 usd: 29 [ 654.485250] CPU 1: hi: 186, btch: 31 usd: 30 [ 654.490679] CPU 2: hi: 186, btch: 31 usd: 0 [ 654.496109] CPU 3: hi: 186, btch: 31 usd: 30 [ 654.501539] CPU 4: hi: 186, btch: 31 usd: 0 [ 654.506968] CPU 5: hi: 186, btch: 31 usd: 0 [ 654.512398] CPU 6: hi: 186, btch: 31 usd: 0 [ 654.517827] CPU 7: hi: 186, btch: 31 usd: 0 [ 654.523256] CPU 8: hi: 186, btch: 31 usd: 0 [ 654.528686] CPU 9: hi: 186, btch: 31 usd: 0 [ 654.534116] CPU 10: hi: 186, btch: 31 usd: 0 [ 654.539545] CPU 11: hi: 186, btch: 31 usd: 0 [ 654.544976] active_anon:13004 inactive_anon:3325 isolated_anon:160 [ 654.544977] active_file:308 inactive_file:1114 isolated_file:64 [ 654.544978] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 654.544979] free:30116 slab_reclaimable:1654 slab_unreclaimable:10667 [ 654.544980] mapped:193 shmem:26 pagetables:639 bounce:0 [ 654.578715] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 654.619339] lowmem_reserve[]: 0 2171 25559 25559 [ 654.624574] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52016kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6616kB slab_unreclaimable:41608kB kernel_stack:2384kB pagetables:2556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 654.671016] lowmem_reserve[]: 0 0 23387 23387 [ 654.675960] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 654.718523] lowmem_reserve[]: 0 0 0 0 [ 654.722692] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 654.734618] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 654.747804] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 654.760021] 2124 total pagecache pages [ 654.764286] 601 pages in swap cache [ 654.768261] Swap cache stats: add 2432, delete 1831, find 0/0 [ 654.774757] Free swap = 514552kB [ 654.778541] Total swap = 524280kB [ 654.827020] 6660096 pages RAM [ 654.830229] 6207063 pages reserved [ 654.834108] 3196 pages shared [ 654.837500] 420123 pages non-shared [ 654.841481] Out of memory: kill process 2768 (exim4) score 11037 or a child [ 654.849332] Killed process 2768 (exim4) [ 654.854024] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 654.861504] cron cpuset=/ mems_allowed=0 [ 654.865961] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 654.872165] Call Trace: [ 654.874983] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 654.880605] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 654.887290] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 654.893788] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 654.901156] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 654.908718] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 654.915988] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 654.921903] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 654.928400] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 654.934507] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 654.941779] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 654.949633] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 654.956324] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 654.962626] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 654.968832] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 654.975326] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 654.981336] Mem-Info: [ 654.983952] Node 0 DMA per-cpu: [ 654.987540] CPU 0: hi: 0, btch: 1 usd: 0 [ 654.992970] CPU 1: hi: 0, btch: 1 usd: 0 [ 654.998400] CPU 2: hi: 0, btch: 1 usd: 0 [ 655.003828] CPU 3: hi: 0, btch: 1 usd: 0 [ 655.009306] CPU 4: hi: 0, btch: 1 usd: 0 [ 655.014689] CPU 5: hi: 0, btch: 1 usd: 0 [ 655.020120] CPU 6: hi: 0, btch: 1 usd: 0 [ 655.025546] CPU 7: hi: 0, btch: 1 usd: 0 [ 655.030977] CPU 8: hi: 0, btch: 1 usd: 0 [ 655.036405] CPU 9: hi: 0, btch: 1 usd: 0 [ 655.041836] CPU 10: hi: 0, btch: 1 usd: 0 [ 655.047265] CPU 11: hi: 0, btch: 1 usd: 0 [ 655.052696] Node 0 DMA32 per-cpu: [ 655.056476] CPU 0: hi: 186, btch: 31 usd: 30 [ 655.061905] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.067334] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.072764] CPU 3: hi: 186, btch: 31 usd: 0 [ 655.078193] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.083622] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.089052] CPU 6: hi: 186, btch: 31 usd: 35 [ 655.094481] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.099911] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.105376] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.110806] CPU 10: hi: 186, btch: 31 usd: 2 [ 655.116235] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.121665] Node 0 Normal per-cpu: [ 655.125546] CPU 0: hi: 186, btch: 31 usd: 29 [ 655.130972] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.136402] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.141832] CPU 3: hi: 186, btch: 31 usd: 30 [ 655.147260] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.152691] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.158120] CPU 6: hi: 186, btch: 31 usd: 0 [ 655.163548] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.168978] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.174409] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.179837] CPU 10: hi: 186, btch: 31 usd: 0 [ 655.185267] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.190698] active_anon:13004 inactive_anon:3325 isolated_anon:160 [ 655.190699] active_file:308 inactive_file:1114 isolated_file:64 [ 655.190700] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 655.190701] free:30116 slab_reclaimable:1654 slab_unreclaimable:10667 [ 655.190702] mapped:193 shmem:26 pagetables:639 bounce:0 [ 655.224437] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.265063] lowmem_reserve[]: 0 2171 25559 25559 [ 655.270297] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52016kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6616kB slab_unreclaimable:41608kB kernel_stack:2384kB pagetables:2556kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 655.316737] lowmem_reserve[]: 0 0 23387 23387 [ 655.321682] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.364246] lowmem_reserve[]: 0 0 0 0 [ 655.368414] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 655.380340] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 655.393526] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 655.405743] 2124 total pagecache pages [ 655.410009] 601 pages in swap cache [ 655.413984] Swap cache stats: add 2432, delete 1831, find 0/0 [ 655.420479] Free swap = 514948kB [ 655.424261] Total swap = 524280kB [ 655.476216] 6660096 pages RAM [ 655.479425] 6207063 pages reserved [ 655.483303] 3175 pages shared [ 655.486697] 420088 pages non-shared [ 655.490673] Out of memory: kill process 2974 (xend) score 9860 or a child [ 655.498333] Killed process 2975 (xend) [ 655.506939] cron invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 [ 655.514415] cron cpuset=/ mems_allowed=0 [ 655.518875] Pid: 2504, comm: cron Not tainted 2.6.32.43 #3 [ 655.525079] Call Trace: [ 655.527896] [<ffffffff810b5605>] ? T.463+0x7c/0x239 [ 655.533516] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b [ 655.540206] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e [ 655.546702] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 [ 655.554071] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 [ 655.561634] [<ffffffff8100edbf>] ? xen_restore_fl_direct_end+0x0/0x1 [ 655.568905] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 [ 655.574819] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 [ 655.581315] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 [ 655.587422] [<ffffffff8100e6f9>] ? xen_force_evtchn_callback+0x9/0xa [ 655.594694] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e [ 655.602548] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b [ 655.609238] [<ffffffff810673b0>] ? hrtimer_cancel+0xc/0x16 [ 655.615541] [<ffffffff81308b2e>] ? do_nanosleep+0x6a/0xa2 [ 655.621745] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c [ 655.628240] [<ffffffff813097c5>] ? page_fault+0x25/0x30 [ 655.634250] Mem-Info: [ 655.636868] Node 0 DMA per-cpu: [ 655.640458] CPU 0: hi: 0, btch: 1 usd: 0 [ 655.645884] CPU 1: hi: 0, btch: 1 usd: 0 [ 655.651315] CPU 2: hi: 0, btch: 1 usd: 0 [ 655.656743] CPU 3: hi: 0, btch: 1 usd: 0 [ 655.662173] CPU 4: hi: 0, btch: 1 usd: 0 [ 655.667603] CPU 5: hi: 0, btch: 1 usd: 0 [ 655.673032] CPU 6: hi: 0, btch: 1 usd: 0 [ 655.678461] CPU 7: hi: 0, btch: 1 usd: 0 [ 655.683890] CPU 8: hi: 0, btch: 1 usd: 0 [ 655.689320] CPU 9: hi: 0, btch: 1 usd: 0 [ 655.694750] CPU 10: hi: 0, btch: 1 usd: 0 [ 655.700180] CPU 11: hi: 0, btch: 1 usd: 0 [ 655.705609] Node 0 DMA32 per-cpu: [ 655.709389] CPU 0: hi: 186, btch: 31 usd: 30 [ 655.714820] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.720250] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.725679] CPU 3: hi: 186, btch: 31 usd: 0 [ 655.731109] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.736538] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.741968] CPU 6: hi: 186, btch: 31 usd: 35 [ 655.747397] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.752826] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.758255] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.763685] CPU 10: hi: 186, btch: 31 usd: 2 [ 655.769114] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.774543] Node 0 Normal per-cpu: [ 655.778422] CPU 0: hi: 186, btch: 31 usd: 29 [ 655.783852] CPU 1: hi: 186, btch: 31 usd: 30 [ 655.789281] CPU 2: hi: 186, btch: 31 usd: 0 [ 655.794710] CPU 3: hi: 186, btch: 31 usd: 30 [ 655.800141] CPU 4: hi: 186, btch: 31 usd: 0 [ 655.805570] CPU 5: hi: 186, btch: 31 usd: 0 [ 655.810999] CPU 6: hi: 186, btch: 31 usd: 0 [ 655.816428] CPU 7: hi: 186, btch: 31 usd: 0 [ 655.821858] CPU 8: hi: 186, btch: 31 usd: 0 [ 655.827288] CPU 9: hi: 186, btch: 31 usd: 0 [ 655.832716] CPU 10: hi: 186, btch: 31 usd: 0 [ 655.838147] CPU 11: hi: 186, btch: 31 usd: 0 [ 655.843578] active_anon:13003 inactive_anon:3325 isolated_anon:160 [ 655.843579] active_file:308 inactive_file:1114 isolated_file:64 [ 655.843580] unevictable:6 dirty:577 writeback:1196 unstable:0 [ 655.843581] free:30116 slab_reclaimable:1653 slab_unreclaimable:10665 [ 655.843581] mapped:193 shmem:26 pagetables:616 bounce:0 [ 655.877316] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 655.917940] lowmem_reserve[]: 0 2171 25559 25559 [ 655.923175] Node 0 DMA32 free:94700kB min:1736kB low:2168kB high:2604kB active_anon:52012kB inactive_anon:13300kB active_file:1232kB inactive_file:4456kB unevictable:24kB isolated(anon):640kB isolated(file):256kB present:2223968kB mlocked:24kB dirty:2308kB writeback:4784kB mapped:772kB shmem:104kB slab_reclaimable:6612kB slab_unreclaimable:41600kB kernel_stack:2384kB pagetables:2464kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:112320 all_unreclaimable? yes [ 655.969617] lowmem_reserve[]: 0 0 23387 23387 [ 655.974561] Node 0 Normal free:13640kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:1060kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [ 656.017126] lowmem_reserve[]: 0 0 0 0 [ 656.021293] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB [ 656.033219] Node 0 DMA32: 39*4kB 186*8kB 314*16kB 287*32kB 94*64kB 29*128kB 18*256kB 6*512kB 4*1024kB 2*2048kB 13*4096kB = 94700kB [ 656.046404] Node 0 Normal: 0*4kB 1*8kB 0*16kB 0*32kB 1*64kB 0*128kB 1*256kB 2*512kB 0*1024kB 2*2048kB 2*4096kB = 13640kB [ 656.058623] 2124 total pagecache pages [ 656.062887] 601 pages in swap cache [ 656.066863] Swap cache stats: add 2432, delete 1831, find 0/0 [ 656.073358] Free swap = 514948kB [ 656.077139] Total swap = 524280kB [ 656.124934] 6660096 pages RAM [ 656.128145] 6207063 pages reserved [ 656.132023] 3176 pages shared [ 656.135417] 420088 pages non-shared [ 656.139393] Out of memory: kill process 2792 (ntpd) score 9615 or a child [ 656.147051] Killed process 2792 (ntpd) -------- >8 -------- Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Musketa
2011-Aug-01 23:13 UTC
[Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>:> when running Jeremy''s 2.6.32.43 kernel I get OOM in dom0 when trying > to create a domU.Maybe I should say more about memory usage: Total memory on hypervisor is 24 GB. dom0_mem=4000M, but dom0 got only about 2,500 MB. No domU was running, so lots of memory was free. domU only wanted 1,500 MB of RAM, so no ballooning nessessary. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Musketa
2011-Aug-01 23:57 UTC
[Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>:> Maybe I should say more about memory usage:I got even more detailled logs for you in the attached file: line 3: dom0 memory: 2,560 M line 20: 21,637 M free line 34: dom0_mem=4000M line 47: domU memory = ''2800'' line 72: OOM begins line 157: key ''m'' on (XEN) console Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Aug-02 03:19 UTC
Re: [Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
On Tue, Aug 02, 2011 at 01:57:53AM +0200, Daniel Musketa wrote:> 2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>: > > Maybe I should say more about memory usage: > > I got even more detailled logs for you in the attached file: > > line 3: dom0 memory: 2,560 M > line 20: 21,637 M free > line 34: dom0_mem=4000M > line 47: domU memory = ''2800'' > line 72: OOM begins > line 157: key ''m'' on (XEN) consoleTry the ''mem=4G'' in your Linux line (along with the one in your hypervisor). The problem is that Dom0 sets up pagetables for the 32GB even thought it is never going to use it - the ''mem=4G'' should fix that for you. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Aug-02 04:59 UTC
Re: [Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
On Mon, Aug 01, 2011 at 11:19:25PM -0400, Konrad Rzeszutek Wilk wrote:> On Tue, Aug 02, 2011 at 01:57:53AM +0200, Daniel Musketa wrote: > > 2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>: > > > Maybe I should say more about memory usage: > > > > I got even more detailled logs for you in the attached file: > > > > line 3: dom0 memory: 2,560 M > > line 20: 21,637 M free > > line 34: dom0_mem=4000M > > line 47: domU memory = ''2800'' > > line 72: OOM begins > > line 157: key ''m'' on (XEN) console > > Try the ''mem=4G'' in your Linux line (along with the one in your hypervisor). > The problem is that Dom0 sets up pagetables for the 32GB even thought it is never > going to use it - the ''mem=4G'' should fix that for you. >Hmm.. Does that explain the dom0 OOM ? Why does dom0 OOM when creating a domU ? In this case dom0 didn''t have to be ballooned down, it shouldn''t see any changes to its memory? Just trying to understand it better :) -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Aug-02 16:38 UTC
Re: [Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
On Tue, Aug 02, 2011 at 07:59:03AM +0300, Pasi Kärkkäinen wrote:> On Mon, Aug 01, 2011 at 11:19:25PM -0400, Konrad Rzeszutek Wilk wrote: > > On Tue, Aug 02, 2011 at 01:57:53AM +0200, Daniel Musketa wrote: > > > 2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>: > > > > Maybe I should say more about memory usage: > > > > > > I got even more detailled logs for you in the attached file: > > > > > > line 3: dom0 memory: 2,560 M > > > line 20: 21,637 M free > > > line 34: dom0_mem=4000M > > > line 47: domU memory = ''2800'' > > > line 72: OOM begins > > > line 157: key ''m'' on (XEN) console > > > > Try the ''mem=4G'' in your Linux line (along with the one in your hypervisor). > > The problem is that Dom0 sets up pagetables for the 32GB even thought it is never > > going to use it - the ''mem=4G'' should fix that for you. > > > > Hmm.. Does that explain the dom0 OOM ? Why does dom0 OOM when creating a domU ? > In this case dom0 didn''t have to be ballooned down, it shouldn''t see any changes to its memory?So Linux kernel looks at the E820 and sees 30GB of "reserved" memory (that memory is what dom0 "thinks" it might balloon too - even thought it won''t ever do). So Linux thinks to itself: "OK, we _might_ use that in the future, so lets create the pagetables for it". And we end up with 800MBs (or more) of pagetables that never end up using. That 800MB does not get released or what not. So your dom0 ends up with 300MB or so (say with dom0_mem=max:1G) and we end up with the OOM happily killing things. The idea, now that Daniel Kipper''s patches are in - to use the memory hotplug API to balloon pages in and out. That should make this much easier.> > Just trying to understand it better :)Of course. Thanks for asking!> > -- Pasi > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Aug-03 16:57 UTC
Re: [Xen-devel] Re: 2.6.32 pvops dom0 oom when creating a domU
On Tue, Aug 02, 2011 at 01:57:53AM +0200, Daniel Musketa wrote:> 2011/8/2 Daniel Musketa <daniel.musketa@googlemail.com>: > > Maybe I should say more about memory usage: > > I got even more detailled logs for you in the attached file: > > line 3: dom0 memory: 2,560 M > line 20: 21,637 M free > line 34: dom0_mem=4000M > line 47: domU memory = ''2800'' > line 72: OOM begins > line 157: key ''m'' on (XEN) console >Please try editing /etc/xen/xend-config.sxp and disable memory ballooning totally. (ie. set option "dom0-min-mem" to match dom0 memory, and "enable-dom0-ballooning" to no). Then try again with both dom0_mem=4096M for xen.gz and mem=4096M for Linux dom0. Does that work better? Konrad was guessing that xend might be ballooning dom0 down even when there''s free memory in xen, and thus causing that problem.. so better make sure ballooning is disabled. -- Pasi> > Daniel> root@2ven:/etc/xen# xm li > Name ID Mem VCPUs State Time(s) > Domain-0 0 2560 12 r----- 111.1 > joel 1 1500 12 -b---- 5.3 > > > > root@2ven:/etc/xen# xm info > host : 2ven > release : 2.6.32.43 > version : #3 SMP Mon Jul 25 23:39:42 CEST 2011 > machine : x86_64 > nr_cpus : 12 > nr_nodes : 1 > cores_per_socket : 6 > threads_per_core : 2 > cpu_mhz : 2400 > hw_caps : bfebfbff:2c100800:00000000:00001f40:029ee3ff:00000000:00000001:00000000 > virt_caps : hvm > total_memory : 24515 > free_memory : 21637 > node_to_cpu : node0:0-11 > node_to_memory : node0:21637 > node_to_dma32_mem : node0:2174 > max_node_id : 0 > xen_major : 4 > xen_minor : 0 > xen_extra : .2 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : unavailable > xen_commandline : dom0_mem=4000M loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 sync_console console_to_ring lapic=debug apic_verbosity=debug apic=debug iommu=off > cc_compiler : gcc version 4.4.5 (Debian 4.4.5-8) > cc_compile_by : root > cc_compile_domain : (none) > cc_compile_date : Mon Aug 1 11:16:18 CEST 2011 > xend_config_format : 4 > > > > root@2ven:/etc/xen# cat clapton.cfg > name = ''clapton'' > kernel = ''/root/domU-boot/clapton/vmlinuz'' > ramdisk = ''/root/domU-boot/clapton/initrd.img'' > memory = ''2800'' > vcpus = 12 > cpu_weight = 512 > root = ''LABEL=clapton_root ro'' > disk = [ > ''phy:/dev/zven/clapton_root,sda1,w'', > ''phy:/dev/zven/clapton_samba,sda2,w'', > ''phy:/dev/zven/clapton_scan,sda3,w'', > ''phy:/dev/zven/clapton_mail,sda4,w'', > ''phy:/dev/zven/clapton_tmp,sda6,w'', > ''phy:/dev/zven/clapton_swap,sda7,w'', > ''phy:/dev/mapper/zven-ISO_sage2,sdc,r'', > ] > vif = [ ''mac=00:16:3E:88:88:88'' ] > extra = ''xencons=tty'' > > > > root@2ven:/etc/xen# xm cre clapton.cfg > Using config file "./clapton.cfg". > > > ==> OOM > > > [ 566.419826] rs:main Q:Reg invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0 > [ 566.428172] rs:main Q:Reg cpuset=/ mems_allowed=0 > [ 566.433503] Pid: 3418, comm: rs:main Q:Reg Not tainted 2.6.32.43 #3 > [ 566.440582] Call Trace: > [ 566.443396] [<ffffffff810b5605>] ? T.463+0x7c/0x239 > [ 566.449015] [<ffffffff810b58f6>] ? __out_of_memory+0x134/0x14b > [ 566.455704] [<ffffffff810b5a49>] ? out_of_memory+0x13c/0x16e > [ 566.462201] [<ffffffff810b9397>] ? __alloc_pages_nodemask+0x4b3/0x5d0 > [ 566.469570] [<ffffffff810bacbb>] ? __do_page_cache_readahead+0x93/0x1b3 > [ 566.477132] [<ffffffff810badf7>] ? ra_submit+0x1c/0x20 > [ 566.483045] [<ffffffff810b4b12>] ? filemap_fault+0x17e/0x300 > [ 566.489542] [<ffffffff8100ca36>] ? xen_make_pte+0x7d/0x7f > [ 566.495748] [<ffffffff810cc26a>] ? __do_fault+0x52/0x3b3 > [ 566.501855] [<ffffffff8100c2e5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e > [ 566.509708] [<ffffffff810ccff6>] ? handle_mm_fault+0x393/0x81b > [ 566.516400] [<ffffffff8130b873>] ? do_page_fault+0x2f0/0x30c > [ 566.522893] [<ffffffff813097c5>] ? page_fault+0x25/0x30 > [ 566.528905] Mem-Info: > [ 566.531522] Node 0 DMA per-cpu: > [ 566.535108] CPU 0: hi: 0, btch: 1 usd: 0 > [ 566.540539] CPU 1: hi: 0, btch: 1 usd: 0 > [ 566.545967] CPU 2: hi: 0, btch: 1 usd: 0 > [ 566.551397] CPU 3: hi: 0, btch: 1 usd: 0 > [ 566.556826] CPU 4: hi: 0, btch: 1 usd: 0 > [ 566.562256] CPU 5: hi: 0, btch: 1 usd: 0 > [ 566.567685] CPU 6: hi: 0, btch: 1 usd: 0 > [ 566.573115] CPU 7: hi: 0, btch: 1 usd: 0 > [ 566.578544] CPU 8: hi: 0, btch: 1 usd: 0 > [ 566.583974] CPU 9: hi: 0, btch: 1 usd: 0 > [ 566.589403] CPU 10: hi: 0, btch: 1 usd: 0 > [ 566.594833] CPU 11: hi: 0, btch: 1 usd: 0 > [ 566.600262] Node 0 DMA32 per-cpu: > [ 566.604043] CPU 0: hi: 186, btch: 31 usd: 30 > [ 566.609473] CPU 1: hi: 186, btch: 31 usd: 0 > [ 566.614902] CPU 2: hi: 186, btch: 31 usd: 0 > [ 566.620332] CPU 3: hi: 186, btch: 31 usd: 37 > [ 566.625760] CPU 4: hi: 186, btch: 31 usd: 0 > [ 566.631191] CPU 5: hi: 186, btch: 31 usd: 0 > [ 566.636620] CPU 6: hi: 186, btch: 31 usd: 0 > [ 566.642050] CPU 7: hi: 186, btch: 31 usd: 30 > [ 566.647479] CPU 8: hi: 186, btch: 31 usd: 0 > [ 566.652908] CPU 9: hi: 186, btch: 31 usd: 0 > [ 566.658337] CPU 10: hi: 186, btch: 31 usd: 0 > [ 566.663767] CPU 11: hi: 186, btch: 31 usd: 0 > [ 566.669197] Node 0 Normal per-cpu: > [ 566.673075] CPU 0: hi: 186, btch: 31 usd: 30 > [ 566.678504] CPU 1: hi: 186, btch: 31 usd: 0 > [ 566.683934] CPU 2: hi: 186, btch: 31 usd: 0 > [ 566.689362] CPU 3: hi: 186, btch: 31 usd: 0 > [ 566.694793] CPU 4: hi: 186, btch: 31 usd: 0 > [ 566.700222] CPU 5: hi: 186, btch: 31 usd: 0 > [ 566.705651] CPU 6: hi: 186, btch: 31 usd: 0 > [ 566.711080] CPU 7: hi: 186, btch: 31 usd: 0 > [ 566.716509] CPU 8: hi: 186, btch: 31 usd: 0 > [ 566.721939] CPU 9: hi: 186, btch: 31 usd: 0 > [ 566.727369] CPU 10: hi: 186, btch: 31 usd: 0 > [ 566.732799] CPU 11: hi: 186, btch: 31 usd: 0 > [ 566.738230] active_anon:8857 inactive_anon:2282 isolated_anon:448 > [ 566.738231] active_file:1368 inactive_file:1661 isolated_file:0 > [ 566.738232] unevictable:6 dirty:0 writeback:3724 unstable:0 > [ 566.738232] free:30171 slab_reclaimable:1883 slab_unreclaimable:12433 > [ 566.738233] mapped:483 shmem:2 pagetables:921 bounce:0 > [ 566.771581] Node 0 DMA free:12124kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11968kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > [ 566.812205] lowmem_reserve[]: 0 2171 25559 25559 > [ 566.817439] Node 0 DMA32 free:94800kB min:1736kB low:2168kB high:2604kB active_anon:35428kB inactive_anon:9128kB active_file:5472kB inactive_file:6644kB unevictable:24kB isolated(anon):1792kB isolated(file):0kB present:2223968kB mlocked:24kB dirty:0kB writeback:14896kB mapped:1932kB shmem:8kB slab_reclaimable:7516kB slab_unreclaimable:48524kB kernel_stack:2648kB pagetables:3684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:86560 all_unreclaimable? yes > [ 566.863299] lowmem_reserve[]: 0 0 23387 23387 > [ 566.868243] Node 0 Normal free:13760kB min:18720kB low:23400kB high:28080kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:23949120kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:1208kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes > [ 566.910903] lowmem_reserve[]: 0 0 0 0 > [ 566.915115] Node 0 DMA: 1*4kB 1*8kB 1*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12124kB > [ 566.927040] Node 0 DMA32: 660*4kB 222*8kB 171*16kB 205*32kB 157*64kB 103*128kB 44*256kB 15*512kB 16*1024kB 3*2048kB 4*4096kB = 94800kB > [ 566.940614] Node 0 Normal: 0*4kB 0*8kB 2*16kB 1*32kB 2*64kB 2*128kB 2*256kB 3*512kB 1*1024kB 3*2048kB 1*4096kB = 13760kB > [ 566.952829] 4750 total pagecache pages > [ 566.957096] 1719 pages in swap cache > [ 566.961169] Swap cache stats: add 9971, delete 8252, find 497/815 > [ 566.968051] Free swap = 494992kB > [ 566.971833] Total swap = 524280kB > [ 567.019196] 6660096 pages RAM > [ 567.022406] 6207063 pages reserved > [ 567.026284] 7958 pages shared > [ 567.029678] 418013 pages non-shared > [ 567.033654] Out of memory: kill process 2999 (xend) score 14111 or a child > [ 567.041410] Killed process 3000 (xend) > > > > (XEN) Physical memory information: > (XEN) Xen heap: 0kB free > (XEN) heap[14]: 64480kB free > (XEN) heap[15]: 131072kB free > (XEN) heap[16]: 262144kB free > (XEN) heap[17]: 524288kB free > (XEN) heap[18]: 1048576kB free > (XEN) heap[19]: 196292kB free > (XEN) heap[20]: 4194304kB free > (XEN) heap[21]: 8388608kB free > (XEN) heap[22]: 4479588kB free > (XEN) Dom heap: 19289352kB free >> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel