Arnd Hannemann
2010-Jun-14 07:53 UTC
[Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Hi, we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest "xen/stable-2.6.32.x" git tree. Any idea, anyone? Best regards, Arnd Jun 11 20:03:16 vmhost1 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif861.1, bridge br0. Jun 11 20:03:16 vmhost1 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/861/1/hotplug-status connected to xenstore. Jun 11 20:03:16 vmhost1 [21717.809194] ------------[ cut here ]------------ Jun 11 20:03:16 vmhost1 [21717.809227] kernel BUG at drivers/xen/grant-table.c:583! Jun 11 20:03:16 vmhost1 [21717.809251] invalid opcode: 0000 [#1] Jun 11 20:03:16 vmhost1 SMP Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.809295] last sysfs file: /sys/devices/virtual/net/br0/bridge/topology_change_detected Jun 11 20:03:16 vmhost1 [21717.809324] Modules linked in: Jun 11 20:03:16 vmhost1 netconsole Jun 11 20:03:16 vmhost1 raid0 Jun 11 20:03:16 vmhost1 md_mod Jun 11 20:03:16 vmhost1 rtc_cmos Jun 11 20:03:16 vmhost1 rtc_core Jun 11 20:03:16 vmhost1 rtc_lib Jun 11 20:03:16 vmhost1 thermal Jun 11 20:03:16 vmhost1 processor Jun 11 20:03:16 vmhost1 pl2303 Jun 11 20:03:16 vmhost1 thermal_sys Jun 11 20:03:16 vmhost1 button Jun 11 20:03:16 vmhost1 usbserial Jun 11 20:03:16 vmhost1 acpi_processor Jun 11 20:03:16 vmhost1 hwmon Jun 11 20:03:16 vmhost1 sr_mod Jun 11 20:03:16 vmhost1 cdrom Jun 11 20:03:16 vmhost1 ipv6 Jun 11 20:03:16 vmhost1 evdev Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.809575] Jun 11 20:03:16 vmhost1 [21717.809593] Pid: 0, comm: swapper Not tainted (2.6.32.15-xen4.0.0-dom0 #1) System Product Name Jun 11 20:03:16 vmhost1 [21717.809624] EIP: 0061:[<c120f170>] EFLAGS: 00010282 CPU: 0 Jun 11 20:03:16 vmhost1 [21717.809652] EIP is at gnttab_copy_grant_page+0x1f0/0x260 Jun 11 20:03:16 vmhost1 [21717.809675] EAX: ffffffea EBX: c153be84 ECX: 00000001 EDX: 00000000 Jun 11 20:03:16 vmhost1 [21717.809700] ESI: 00007ff0 EDI: 0000003e EBP: c290dde0 ESP: c153be50 Jun 11 20:03:16 vmhost1 [21717.809726] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Jun 11 20:03:16 vmhost1 [21717.809750] Process swapper (pid: 0, ti=c153a000 task=c1543760 task.ti=c153a000) Jun 11 20:03:16 vmhost1 [21717.809777] Stack: Jun 11 20:03:16 vmhost1 [21717.809793] 00000000 Jun 11 20:03:16 vmhost1 0021412d Jun 11 20:03:16 vmhost1 c28f02e0 Jun 11 20:03:16 vmhost1 0002c1ef Jun 11 20:03:16 vmhost1 ec1ef000 Jun 11 20:03:16 vmhost1 ecf85adc Jun 11 20:03:16 vmhost1 0000003e Jun 11 20:03:16 vmhost1 ec1ef000 Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.809917] <0> Jun 11 20:03:16 vmhost1 00000000 Jun 11 20:03:16 vmhost1 eb317000 Jun 11 20:03:16 vmhost1 00000000 Jun 11 20:03:16 vmhost1 0000003e Jun 11 20:03:16 vmhost1 eb2a0000 Jun 11 20:03:16 vmhost1 1412d001 Jun 11 20:03:16 vmhost1 00000000 Jun 11 20:03:16 vmhost1 0002c1ef Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.810059] <0> Jun 11 20:03:16 vmhost1 00000000 Jun 11 20:03:16 vmhost1 c1627f54 Jun 11 20:03:16 vmhost1 c155b97 Jun 11 20:03:16 vmhost1 8 Jun 11 20:03:16 vmhost1 c1627f54 Jun 11 20:03:16 vmhost1 01c5b765 Jun 11 20:03:16 vmhost1 c121fff1 Jun 11 20:03:16 vmhost1 c1006a98 Jun 11 20:03:16 vmhost1 c3b90240 Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.810146] Call Trace: Jun 11 20:03:16 vmhost1 [21717.810168] [<c121fff1>] ? net_tx_action+0x1d1/0x9b0 Jun 11 20:03:16 vmhost1 [21717.810197] [<c1006a98>] ? check_events+0x8/0xc Jun 11 20:03:16 vmhost1 [21717.810224] [<c1040c1d>] ? run_timer_softirq+0x15d/0x1e0 Jun 11 20:03:16 vmhost1 [21717.810252] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 11 20:03:16 vmhost1 [21717.810278] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 11 20:03:16 vmhost1 [21717.810304] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 11 20:03:16 vmhost1 [21717.810332] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 11 20:03:16 vmhost1 [21717.810357] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 11 20:03:16 vmhost1 [21717.810385] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 11 20:03:16 vmhost1 [21717.810411] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 11 20:03:16 vmhost1 [21717.810442] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 11 20:03:16 vmhost1 [21717.810474] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 11 20:03:16 vmhost1 [21717.810501] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 11 20:03:16 vmhost1 [21717.810527] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 11 20:03:16 vmhost1 [21717.810559] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 11 20:03:16 vmhost1 [21717.810594] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c Jun 11 20:03:16 vmhost1 [21717.810620] Code: Jun 11 20:03:16 vmhost1 8d Jun 11 20:03:16 vmhost1 5c Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 34 Jun 11 20:03:16 vmhost1 c1 Jun 11 20:03:16 vmhost1 e0 Jun 11 20:03:16 vmhost1 0c Jun 11 20:03:16 vmhost1 83 Jun 11 20:03:16 vmhost1 c8 Jun 11 20:03:16 vmhost1 01 Jun 11 20:03:16 vmhost1 89 Jun 11 20:03:16 vmhost1 44 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 34 Jun 11 20:03:16 vmhost1 8b Jun 11 20:03:16 vmhost1 44 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 0c Jun 11 20:03:16 vmhost1 c7 Jun 11 20:03:16 vmhost1 44 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 40 Jun 11 20:03:16 vmhost1 00 Jun 11 20:03:16 vmhost1 00 Jun 11 20:03:16 vmhost1 00 Jun 11 20:03:16 vmhost1 00 Jun 11 20:03:16 vmhost1 89 Jun 11 20:03:16 vmhost1 44 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 3c Jun 11 20:03:16 vmhost1 e8 Jun 11 20:03:16 vmhost1 b8 Jun 11 20:03:16 vmhost1 1e Jun 11 20:03:16 vmhost1 df Jun 11 20:03:16 vmhost1 ff Jun 11 20:03:16 vmhost1 85 Jun 11 20:03:16 vmhost1 c0 Jun 11 20:03:16 vmhost1 0f Jun 11 20:03:16 vmhost1 84 Jun 11 20:03:16 vmhost1 2c Jun 11 20:03:16 vmhost1 ff Jun 11 20:03:16 vmhost1 ff Jun 11 20:03:16 vmhost1 ff Jun 11 18:03:15 vmhost1 unparseable log message: "<0f> " Jun 11 20:03:16 vmhost1 0b Jun 11 20:03:16 vmhost1 eb Jun 11 20:03:16 vmhost1 fe Jun 11 20:03:16 vmhost1 0f Jun 11 20:03:16 vmhost1 0b Jun 11 20:03:16 vmhost1 eb Jun 11 20:03:16 vmhost1 fe Jun 11 20:03:16 vmhost1 0f Jun 11 20:03:16 vmhost1 0b Jun 11 20:03:16 vmhost1 eb Jun 11 20:03:16 vmhost1 fe Jun 11 20:03:16 vmhost1 8b Jun 11 20:03:16 vmhost1 54 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 04 Jun 11 20:03:16 vmhost1 8b Jun 11 20:03:16 vmhost1 44 Jun 11 20:03:16 vmhost1 24 Jun 11 20:03:16 vmhost1 0c Jun 11 20:03:16 vmhost1 e8 Jun 11 20:03:16 vmhost1 Jun 11 20:03:16 vmhost1 [21717.811364] EIP: [<c120f170>] Jun 11 20:03:16 vmhost1 gnttab_copy_grant_page+0x1f0/0x260 Jun 11 20:03:16 vmhost1 SS:ESP 0069:c153be50 Jun 11 20:03:16 vmhost1 [21717.811821] ---[ end trace 763de4792c00b5c0 ]--- Jun 11 20:03:16 vmhost1 [21717.811962] Kernel panic - not syncing: Fatal exception in interrupt Jun 11 20:03:16 vmhost1 [21717.812043] Pid: 0, comm: swapper Tainted: G D 2.6.32.15-xen4.0.0-dom0 #1 Jun 11 20:03:16 vmhost1 [21717.812192] Call Trace: Jun 11 20:03:16 vmhost1 [21717.812276] [<c141d2e5>] ? panic+0x42/0xe1 Jun 11 20:03:16 vmhost1 [21717.812350] [<c100cc56>] ? oops_end+0x96/0xa0 Jun 11 20:03:16 vmhost1 [21717.812422] [<c100a73f>] ? do_invalid_op+0x7f/0x90 Jun 11 20:03:16 vmhost1 [21717.812496] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 11 20:03:16 vmhost1 [21717.812574] [<c10741e4>] ? __alloc_pages_nodemask+0xe4/0x5b0 Jun 11 20:03:16 vmhost1 [21717.812650] [<c1006197>] ? xen_force_evtchn_callback+0x17/0x30 Jun 11 20:03:16 vmhost1 [21717.812726] [<c1006a98>] ? check_events+0x8/0xc Jun 11 20:03:16 vmhost1 [21717.812799] [<c141f9d6>] ? error_code+0x66/0x6c Jun 11 20:03:16 vmhost1 [21717.812872] [<c100a6c0>] ? do_invalid_op+0x0/0x90 Jun 11 20:03:16 vmhost1 [21717.812945] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 11 20:03:16 vmhost1 [21717.813022] [<c121fff1>] ? net_tx_action+0x1d1/0x9b0 Jun 11 20:03:16 vmhost1 [21717.813108] [<c1006a98>] ? check_events+0x8/0xc Jun 11 20:03:16 vmhost1 [21717.813185] [<c1040c1d>] ? run_timer_softirq+0x15d/0x1e0 Jun 11 20:03:16 vmhost1 [21717.813260] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 11 20:03:16 vmhost1 [21717.813333] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 11 20:03:16 vmhost1 [21717.813407] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 11 20:03:16 vmhost1 [21717.813483] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 11 20:03:16 vmhost1 [21717.813556] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 11 20:03:16 vmhost1 [21717.813631] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 11 20:03:16 vmhost1 [21717.813704] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 11 20:03:16 vmhost1 [21717.813779] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 11 20:03:16 vmhost1 [21717.813852] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 11 20:03:16 vmhost1 [21717.813939] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 11 20:03:16 vmhost1 [21717.814024] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 11 20:03:16 vmhost1 [21717.814101] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 11 20:03:16 vmhost1 [21717.814181] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Jun-14 10:56 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
On 06/14/2010 08:53 AM, Arnd Hannemann wrote:> Hi, > > we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest > "xen/stable-2.6.32.x" git tree. >Are you sure its the latest? There were some problems with the tasklet-based netback patches which I since reverted. (And I now have fixes for, so I''ll reapply them soon.) J> Any idea, anyone? > > Best regards, > Arnd > > Jun 11 20:03:16 vmhost1 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif861.1, bridge br0. > Jun 11 20:03:16 vmhost1 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/861/1/hotplug-status connected to xenstore. > Jun 11 20:03:16 vmhost1 [21717.809194] ------------[ cut here ]------------ > Jun 11 20:03:16 vmhost1 [21717.809227] kernel BUG at drivers/xen/grant-table.c:583! > Jun 11 20:03:16 vmhost1 [21717.809251] invalid opcode: 0000 [#1] > Jun 11 20:03:16 vmhost1 SMP > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.809295] last sysfs file: /sys/devices/virtual/net/br0/bridge/topology_change_detected > Jun 11 20:03:16 vmhost1 [21717.809324] Modules linked in: > Jun 11 20:03:16 vmhost1 netconsole > Jun 11 20:03:16 vmhost1 raid0 > Jun 11 20:03:16 vmhost1 md_mod > Jun 11 20:03:16 vmhost1 rtc_cmos > Jun 11 20:03:16 vmhost1 rtc_core > Jun 11 20:03:16 vmhost1 rtc_lib > Jun 11 20:03:16 vmhost1 thermal > Jun 11 20:03:16 vmhost1 processor > Jun 11 20:03:16 vmhost1 pl2303 > Jun 11 20:03:16 vmhost1 thermal_sys > Jun 11 20:03:16 vmhost1 button > Jun 11 20:03:16 vmhost1 usbserial > Jun 11 20:03:16 vmhost1 acpi_processor > Jun 11 20:03:16 vmhost1 hwmon > Jun 11 20:03:16 vmhost1 sr_mod > Jun 11 20:03:16 vmhost1 cdrom > Jun 11 20:03:16 vmhost1 ipv6 > Jun 11 20:03:16 vmhost1 evdev > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.809575] > Jun 11 20:03:16 vmhost1 [21717.809593] Pid: 0, comm: swapper Not tainted (2.6.32.15-xen4.0.0-dom0 #1) System Product Name > Jun 11 20:03:16 vmhost1 [21717.809624] EIP: 0061:[<c120f170>] EFLAGS: 00010282 CPU: 0 > Jun 11 20:03:16 vmhost1 [21717.809652] EIP is at gnttab_copy_grant_page+0x1f0/0x260 > Jun 11 20:03:16 vmhost1 [21717.809675] EAX: ffffffea EBX: c153be84 ECX: 00000001 EDX: 00000000 > Jun 11 20:03:16 vmhost1 [21717.809700] ESI: 00007ff0 EDI: 0000003e EBP: c290dde0 ESP: c153be50 > Jun 11 20:03:16 vmhost1 [21717.809726] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 > Jun 11 20:03:16 vmhost1 [21717.809750] Process swapper (pid: 0, ti=c153a000 task=c1543760 task.ti=c153a000) > Jun 11 20:03:16 vmhost1 [21717.809777] Stack: > Jun 11 20:03:16 vmhost1 [21717.809793] 00000000 > Jun 11 20:03:16 vmhost1 0021412d > Jun 11 20:03:16 vmhost1 c28f02e0 > Jun 11 20:03:16 vmhost1 0002c1ef > Jun 11 20:03:16 vmhost1 ec1ef000 > Jun 11 20:03:16 vmhost1 ecf85adc > Jun 11 20:03:16 vmhost1 0000003e > Jun 11 20:03:16 vmhost1 ec1ef000 > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.809917] <0> > Jun 11 20:03:16 vmhost1 00000000 > Jun 11 20:03:16 vmhost1 eb317000 > Jun 11 20:03:16 vmhost1 00000000 > Jun 11 20:03:16 vmhost1 0000003e > Jun 11 20:03:16 vmhost1 eb2a0000 > Jun 11 20:03:16 vmhost1 1412d001 > Jun 11 20:03:16 vmhost1 00000000 > Jun 11 20:03:16 vmhost1 0002c1ef > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.810059] <0> > Jun 11 20:03:16 vmhost1 00000000 > Jun 11 20:03:16 vmhost1 c1627f54 > Jun 11 20:03:16 vmhost1 c155b97 > Jun 11 20:03:16 vmhost1 8 > Jun 11 20:03:16 vmhost1 c1627f54 > Jun 11 20:03:16 vmhost1 01c5b765 > Jun 11 20:03:16 vmhost1 c121fff1 > Jun 11 20:03:16 vmhost1 c1006a98 > Jun 11 20:03:16 vmhost1 c3b90240 > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.810146] Call Trace: > Jun 11 20:03:16 vmhost1 [21717.810168] [<c121fff1>] ? net_tx_action+0x1d1/0x9b0 > Jun 11 20:03:16 vmhost1 [21717.810197] [<c1006a98>] ? check_events+0x8/0xc > Jun 11 20:03:16 vmhost1 [21717.810224] [<c1040c1d>] ? run_timer_softirq+0x15d/0x1e0 > Jun 11 20:03:16 vmhost1 [21717.810252] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 > Jun 11 20:03:16 vmhost1 [21717.810278] [<c103c378>] ? __do_softirq+0x88/0x110 > Jun 11 20:03:16 vmhost1 [21717.810304] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 > Jun 11 20:03:16 vmhost1 [21717.810332] [<c103c43d>] ? do_softirq+0x3d/0x40 > Jun 11 20:03:16 vmhost1 [21717.810357] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 > Jun 11 20:03:16 vmhost1 [21717.810385] [<c1009da7>] ? xen_do_upcall+0x7/0xc > Jun 11 20:03:16 vmhost1 [21717.810411] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 > Jun 11 20:03:16 vmhost1 [21717.810442] [<c10061ef>] ? xen_safe_halt+0xf/0x20 > Jun 11 20:03:16 vmhost1 [21717.810474] [<c100382c>] ? xen_idle+0x1c/0x30 > Jun 11 20:03:16 vmhost1 [21717.810501] [<c10081fa>] ? cpu_idle+0x3a/0x60 > Jun 11 20:03:16 vmhost1 [21717.810527] [<c15787ef>] ? start_kernel+0x2c6/0x2cb > Jun 11 20:03:16 vmhost1 [21717.810559] [<c1578367>] ? unknown_bootoption+0x0/0x190 > Jun 11 20:03:16 vmhost1 [21717.810594] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c > Jun 11 20:03:16 vmhost1 [21717.810620] Code: > Jun 11 20:03:16 vmhost1 8d > Jun 11 20:03:16 vmhost1 5c > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 34 > Jun 11 20:03:16 vmhost1 c1 > Jun 11 20:03:16 vmhost1 e0 > Jun 11 20:03:16 vmhost1 0c > Jun 11 20:03:16 vmhost1 83 > Jun 11 20:03:16 vmhost1 c8 > Jun 11 20:03:16 vmhost1 01 > Jun 11 20:03:16 vmhost1 89 > Jun 11 20:03:16 vmhost1 44 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 34 > Jun 11 20:03:16 vmhost1 8b > Jun 11 20:03:16 vmhost1 44 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 0c > Jun 11 20:03:16 vmhost1 c7 > Jun 11 20:03:16 vmhost1 44 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 40 > Jun 11 20:03:16 vmhost1 00 > Jun 11 20:03:16 vmhost1 00 > Jun 11 20:03:16 vmhost1 00 > Jun 11 20:03:16 vmhost1 00 > Jun 11 20:03:16 vmhost1 89 > Jun 11 20:03:16 vmhost1 44 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 3c > Jun 11 20:03:16 vmhost1 e8 > Jun 11 20:03:16 vmhost1 b8 > Jun 11 20:03:16 vmhost1 1e > Jun 11 20:03:16 vmhost1 df > Jun 11 20:03:16 vmhost1 ff > Jun 11 20:03:16 vmhost1 85 > Jun 11 20:03:16 vmhost1 c0 > Jun 11 20:03:16 vmhost1 0f > Jun 11 20:03:16 vmhost1 84 > Jun 11 20:03:16 vmhost1 2c > Jun 11 20:03:16 vmhost1 ff > Jun 11 20:03:16 vmhost1 ff > Jun 11 20:03:16 vmhost1 ff > Jun 11 18:03:15 vmhost1 unparseable log message: "<0f> " > Jun 11 20:03:16 vmhost1 0b > Jun 11 20:03:16 vmhost1 eb > Jun 11 20:03:16 vmhost1 fe > Jun 11 20:03:16 vmhost1 0f > Jun 11 20:03:16 vmhost1 0b > Jun 11 20:03:16 vmhost1 eb > Jun 11 20:03:16 vmhost1 fe > Jun 11 20:03:16 vmhost1 0f > Jun 11 20:03:16 vmhost1 0b > Jun 11 20:03:16 vmhost1 eb > Jun 11 20:03:16 vmhost1 fe > Jun 11 20:03:16 vmhost1 8b > Jun 11 20:03:16 vmhost1 54 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 04 > Jun 11 20:03:16 vmhost1 8b > Jun 11 20:03:16 vmhost1 44 > Jun 11 20:03:16 vmhost1 24 > Jun 11 20:03:16 vmhost1 0c > Jun 11 20:03:16 vmhost1 e8 > Jun 11 20:03:16 vmhost1 > Jun 11 20:03:16 vmhost1 [21717.811364] EIP: [<c120f170>] > Jun 11 20:03:16 vmhost1 gnttab_copy_grant_page+0x1f0/0x260 > Jun 11 20:03:16 vmhost1 SS:ESP 0069:c153be50 > Jun 11 20:03:16 vmhost1 [21717.811821] ---[ end trace 763de4792c00b5c0 ]--- > Jun 11 20:03:16 vmhost1 [21717.811962] Kernel panic - not syncing: Fatal exception in interrupt > Jun 11 20:03:16 vmhost1 [21717.812043] Pid: 0, comm: swapper Tainted: G D 2.6.32.15-xen4.0.0-dom0 #1 > Jun 11 20:03:16 vmhost1 [21717.812192] Call Trace: > Jun 11 20:03:16 vmhost1 [21717.812276] [<c141d2e5>] ? panic+0x42/0xe1 > Jun 11 20:03:16 vmhost1 [21717.812350] [<c100cc56>] ? oops_end+0x96/0xa0 > Jun 11 20:03:16 vmhost1 [21717.812422] [<c100a73f>] ? do_invalid_op+0x7f/0x90 > Jun 11 20:03:16 vmhost1 [21717.812496] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 > Jun 11 20:03:16 vmhost1 [21717.812574] [<c10741e4>] ? __alloc_pages_nodemask+0xe4/0x5b0 > Jun 11 20:03:16 vmhost1 [21717.812650] [<c1006197>] ? xen_force_evtchn_callback+0x17/0x30 > Jun 11 20:03:16 vmhost1 [21717.812726] [<c1006a98>] ? check_events+0x8/0xc > Jun 11 20:03:16 vmhost1 [21717.812799] [<c141f9d6>] ? error_code+0x66/0x6c > Jun 11 20:03:16 vmhost1 [21717.812872] [<c100a6c0>] ? do_invalid_op+0x0/0x90 > Jun 11 20:03:16 vmhost1 [21717.812945] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 > Jun 11 20:03:16 vmhost1 [21717.813022] [<c121fff1>] ? net_tx_action+0x1d1/0x9b0 > Jun 11 20:03:16 vmhost1 [21717.813108] [<c1006a98>] ? check_events+0x8/0xc > Jun 11 20:03:16 vmhost1 [21717.813185] [<c1040c1d>] ? run_timer_softirq+0x15d/0x1e0 > Jun 11 20:03:16 vmhost1 [21717.813260] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 > Jun 11 20:03:16 vmhost1 [21717.813333] [<c103c378>] ? __do_softirq+0x88/0x110 > Jun 11 20:03:16 vmhost1 [21717.813407] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 > Jun 11 20:03:16 vmhost1 [21717.813483] [<c103c43d>] ? do_softirq+0x3d/0x40 > Jun 11 20:03:16 vmhost1 [21717.813556] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 > Jun 11 20:03:16 vmhost1 [21717.813631] [<c1009da7>] ? xen_do_upcall+0x7/0xc > Jun 11 20:03:16 vmhost1 [21717.813704] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 > Jun 11 20:03:16 vmhost1 [21717.813779] [<c10061ef>] ? xen_safe_halt+0xf/0x20 > Jun 11 20:03:16 vmhost1 [21717.813852] [<c100382c>] ? xen_idle+0x1c/0x30 > Jun 11 20:03:16 vmhost1 [21717.813939] [<c10081fa>] ? cpu_idle+0x3a/0x60 > Jun 11 20:03:16 vmhost1 [21717.814024] [<c15787ef>] ? start_kernel+0x2c6/0x2cb > Jun 11 20:03:16 vmhost1 [21717.814101] [<c1578367>] ? unknown_bootoption+0x0/0x190 > Jun 11 20:03:16 vmhost1 [21717.814181] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Jun-14 10:57 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
On Mon, 14 Jun 2010, Arnd Hannemann wrote:> Hi, > > we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest > "xen/stable-2.6.32.x" git tree. > > Any idea, anyone? >this CS from origin/xen/dom0/gntdev should fix your problem: sstabellini@kaball-desktop:~/xensource/linux-pvops-latest$ git show ad469f0da31bc16b945f9a06710b9d45434d0091 commit ad469f0da31bc16b945f9a06710b9d45434d0091 Author: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> Date: Wed Jun 9 12:34:02 2010 -0700 xen/gntdev: use spinlocks rather than rwsem for locking The mmu notifier mechanism calls its callbacks with an rcu lock, which disables preemption. This means we cannot use any blocking synchronization for locking. Convert all the rwsemas to plain spinlocks. This requires that the memory allocation and copying to/from userspace be split from the actual datastructure updates since they can''t be done under spinlock. Signed-off-by: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Arnd Hannemann
2010-Jun-14 11:04 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Am 14.06.2010 12:56, schrieb Jeremy Fitzhardinge:> On 06/14/2010 08:53 AM, Arnd Hannemann wrote: >> Hi, >> >> we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest >> "xen/stable-2.6.32.x" git tree. >> > > Are you sure its the latest? There were some problems with the > tasklet-based netback patches which I since reverted. (And I now have > fixes for, so I''ll reapply them soon.)Yes its latest. git log shows this commit as top: commit 57028f01dcd263bcee41e00ddcda167a082ae075 Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> Date: Wed Jun 9 11:29:42 2010 -0700 Revert "Merge branch ''xen/dom0/backend/netback-tasklet'' into xen/next" This reverts commit 528ea798f02ccb03023e72cbda665ed5c3eec6b1, reversing changes made to 6c004dff353a3588684bff1c0eac8b14a1b6f67e. The Panic seems to originate from this code snippet: if (!xen_feature(XENFEAT_auto_translated_physmap)) { set_phys_to_machine(page_to_pfn(new_page), INVALID_P2M_ENTRY); mmu.ptr = (new_mfn << PAGE_SHIFT) | MMU_MACHPHYS_UPDATE; mmu.val = pfn; err = HYPERVISOR_mmu_update(&mmu, 1, NULL, DOMID_SELF); BUG_ON(err); } I will try if the patch Stefano Stabellini mentioned fixes the problem... Thanks, Arnd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Arnd Hannemann
2010-Jun-14 11:09 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Am 14.06.2010 12:57, schrieb Stefano Stabellini:> On Mon, 14 Jun 2010, Arnd Hannemann wrote: >> Hi, >> >> we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest >> "xen/stable-2.6.32.x" git tree. >> >> Any idea, anyone? >> > > this CS from origin/xen/dom0/gntdev should fix your problem:Thanks a lot. We will try this, and post results...> > sstabellini@kaball-desktop:~/xensource/linux-pvops-latest$ git show ad469f0da31bc16b945f9a06710b9d45434d0091 > commit ad469f0da31bc16b945f9a06710b9d45434d0091 > Author: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> > Date: Wed Jun 9 12:34:02 2010 -0700 > > xen/gntdev: use spinlocks rather than rwsem for locking > > The mmu notifier mechanism calls its callbacks with an rcu lock, > which disables preemption. This means we cannot use any blocking > synchronization for locking. > > Convert all the rwsemas to plain spinlocks. This requires that > the memory allocation and copying to/from userspace be split > from the actual datastructure updates since they can''t be done > under spinlock. > > Signed-off-by: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> > Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> >Best regards, Arnd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Arnd Hannemann
2010-Jun-14 12:26 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Hi, Am 14.06.2010 12:57, schrieb Stefano Stabellini:> On Mon, 14 Jun 2010, Arnd Hannemann wrote: >> Hi, >> >> we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest >> "xen/stable-2.6.32.x" git tree. >> >> Any idea, anyone? >> > > this CS from origin/xen/dom0/gntdev should fix your problem: > > sstabellini@kaball-desktop:~/xensource/linux-pvops-latest$ git show ad469f0da31bc16b945f9a06710b9d45434d0091 > commit ad469f0da31bc16b945f9a06710b9d45434d0091 > Author: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> > Date: Wed Jun 9 12:34:02 2010 -0700 > > xen/gntdev: use spinlocks rather than rwsem for locking > > The mmu notifier mechanism calls its callbacks with an rcu lock, > which disables preemption. This means we cannot use any blocking > synchronization for locking. > > Convert all the rwsemas to plain spinlocks. This requires that > the memory allocation and copying to/from userspace be split > from the actual datastructure updates since they can''t be done > under spinlock. > > Signed-off-by: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> > Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> >Unfortunately, this patch does not seem to help. We get a very similar backtrace after one hour stress testing with a script starting and stopping domUs in a loop. Maybe the problem is the hypervisor itself? We are currently using 4.0.1-rc2-pre (we updated from 4.0.0 because of what we believed was the same problem, we had no working netconsole back then though). Jun 14 14:07:22 vmhost2 [ 2418.542425] ------------[ cut here ]------------ Jun 14 14:07:22 vmhost2 [ 2418.542475] kernel BUG at drivers/xen/grant-table.c:583! Jun 14 14:07:22 vmhost2 [ 2418.542515] invalid opcode: 0000 [#1] Jun 14 14:07:22 vmhost2 SMP Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.542574] last sysfs file: /sys/devices/virtual/net/br0/bridge/topology_change_detected Jun 14 14:07:22 vmhost2 [ 2418.542640] Modules linked in: Jun 14 14:07:22 vmhost2 netconsole Jun 14 14:07:22 vmhost2 raid0 Jun 14 14:07:22 vmhost2 md_mod Jun 14 14:07:22 vmhost2 rtc_cmos Jun 14 14:07:22 vmhost2 rtc_core Jun 14 14:07:22 vmhost2 rtc_lib Jun 14 14:07:22 vmhost2 ipv6 Jun 14 14:07:22 vmhost2 thermal Jun 14 14:07:22 vmhost2 processor Jun 14 14:07:22 vmhost2 thermal_sys Jun 14 14:07:22 vmhost2 hwmon Jun 14 14:07:22 vmhost2 pl2303 Jun 14 14:07:22 vmhost2 button Jun 14 14:07:22 vmhost2 acpi_processor Jun 14 14:07:22 vmhost2 usbserial Jun 14 14:07:22 vmhost2 sr_mod Jun 14 14:07:22 vmhost2 evdev Jun 14 14:07:22 vmhost2 cdrom Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.542937] Jun 14 14:07:22 vmhost2 [ 2418.542970] Pid: 0, comm: swapper Not tainted (2.6.32.15-xen4.0.0-dom0-stefano #2) System Product Name Jun 14 14:07:22 vmhost2 [ 2418.543034] EIP: 0061:[<c120f170>] EFLAGS: 00010282 CPU: 0 Jun 14 14:07:22 vmhost2 [ 2418.543077] EIP is at gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:07:22 vmhost2 [ 2418.543117] EAX: ffffffea EBX: c153be84 ECX: 00000001 EDX: 00000000 Jun 14 14:07:22 vmhost2 [ 2418.543158] ESI: 00007ff0 EDI: 00000013 EBP: c290e660 ESP: c153be50 Jun 14 14:07:22 vmhost2 [ 2418.543199] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Jun 14 14:07:22 vmhost2 [ 2418.543239] Process swapper (pid: 0, ti=c153a000 task=c1543760 task.ti=c153a000) Jun 14 14:07:22 vmhost2 [ 2418.543297] Stack: Jun 14 14:07:22 vmhost2 [ 2418.543329] 00000000 Jun 14 14:07:22 vmhost2 00213784 Jun 14 14:07:22 vmhost2 c2904dc0 Jun 14 14:07:22 vmhost2 0002c233 Jun 14 14:07:22 vmhost2 ec233000 Jun 14 14:07:22 vmhost2 ecf85bec Jun 14 14:07:22 vmhost2 00000013 Jun 14 14:07:22 vmhost2 ec233000 Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.543461] <0> Jun 14 14:07:22 vmhost2 00000000 Jun 14 14:07:22 vmhost2 ebd6e000 Jun 14 14:07:22 vmhost2 00000000 Jun 14 14:07:22 vmhost2 00000013 Jun 14 14:07:22 vmhost2 c1350000 Jun 14 14:07:22 vmhost2 13784001 Jun 14 14:07:22 vmhost2 00000000 Jun 14 14:07:22 vmhost2 0002c233 Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.543616] <0> Jun 14 14:07:22 vmhost2 00000000 Jun 14 14:07:22 vmhost2 c1628284 Jun 14 14:07:22 vmhost2 c155b978 Jun 14 14:07:22 vmhost2 c1628284 Jun 14 14:07:22 vmhost2 00560014 Jun 14 14:07:22 vmhost2 c12200c1 Jun 14 14:07:22 vmhost2 00000001 Jun 14 14:07:22 vmhost2 00000000 Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.543797] Call Trace: Jun 14 14:07:22 vmhost2 [ 2418.543838] [<c1350000>] ? sock_release+0x10/0x80 Jun 14 14:07:22 vmhost2 [ 2418.543882] [<c12200c1>] ? net_tx_action+0x1d1/0x9b0 Jun 14 14:07:22 vmhost2 [ 2418.543925] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 14 14:07:22 vmhost2 [ 2418.543967] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 14 14:07:22 vmhost2 [ 2418.544009] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 14 14:07:22 vmhost2 [ 2418.544053] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 14 14:07:22 vmhost2 [ 2418.544094] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 14 14:07:22 vmhost2 [ 2418.544147] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 14 14:07:22 vmhost2 [ 2418.544190] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 14 14:07:22 vmhost2 [ 2418.544234] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 14 14:07:22 vmhost2 [ 2418.544275] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 14 14:07:22 vmhost2 [ 2418.544316] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 14 14:07:22 vmhost2 [ 2418.544359] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 14 14:07:22 vmhost2 [ 2418.544401] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 14 14:07:22 vmhost2 [ 2418.544444] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c Jun 14 14:07:22 vmhost2 [ 2418.544483] Code: Jun 14 14:07:22 vmhost2 8d Jun 14 14:07:22 vmhost2 5c Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 34 Jun 14 14:07:22 vmhost2 c1 Jun 14 14:07:22 vmhost2 e0 Jun 14 14:07:22 vmhost2 0c Jun 14 14:07:22 vmhost2 83 Jun 14 14:07:22 vmhost2 c8 Jun 14 14:07:22 vmhost2 01 Jun 14 14:07:22 vmhost2 89 Jun 14 14:07:22 vmhost2 44 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 34 Jun 14 14:07:22 vmhost2 8b Jun 14 14:07:22 vmhost2 44 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 0c Jun 14 14:07:22 vmhost2 c7 Jun 14 14:07:22 vmhost2 44 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 40 Jun 14 14:07:22 vmhost2 00 Jun 14 14:07:22 vmhost2 00 Jun 14 14:07:22 vmhost2 00 Jun 14 14:07:22 vmhost2 00 Jun 14 14:07:22 vmhost2 89 Jun 14 14:07:22 vmhost2 44 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 3c Jun 14 14:07:22 vmhost2 e8 Jun 14 14:07:22 vmhost2 b8 Jun 14 14:07:22 vmhost2 1e Jun 14 14:07:22 vmhost2 df Jun 14 14:07:22 vmhost2 ff Jun 14 14:07:22 vmhost2 85 Jun 14 14:07:22 vmhost2 c0 Jun 14 14:07:22 vmhost2 0f Jun 14 14:07:22 vmhost2 84 Jun 14 14:07:22 vmhost2 2c Jun 14 14:07:22 vmhost2 ff Jun 14 14:07:22 vmhost2 ff Jun 14 14:07:22 vmhost2 ff Jun 14 12:07:21 vmhost2 unparseable log message: "<0f> " Jun 14 14:07:22 vmhost2 0b Jun 14 14:07:22 vmhost2 eb Jun 14 14:07:22 vmhost2 fe Jun 14 14:07:22 vmhost2 0f Jun 14 14:07:22 vmhost2 0b Jun 14 14:07:22 vmhost2 eb Jun 14 14:07:22 vmhost2 fe Jun 14 14:07:22 vmhost2 0f Jun 14 14:07:22 vmhost2 0b Jun 14 14:07:22 vmhost2 eb Jun 14 14:07:22 vmhost2 fe Jun 14 14:07:22 vmhost2 8b Jun 14 14:07:22 vmhost2 54 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 04 Jun 14 14:07:22 vmhost2 8b Jun 14 14:07:22 vmhost2 44 Jun 14 14:07:22 vmhost2 24 Jun 14 14:07:22 vmhost2 0c Jun 14 14:07:22 vmhost2 e8 Jun 14 14:07:22 vmhost2 Jun 14 14:07:22 vmhost2 [ 2418.545277] EIP: [<c120f170>] Jun 14 14:07:22 vmhost2 gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:07:22 vmhost2 SS:ESP 0069:c153be50 Jun 14 14:07:22 vmhost2 [ 2418.545597] ---[ end trace f877a40240218318 ]--- Jun 14 14:07:22 vmhost2 [ 2418.545669] Kernel panic - not syncing: Fatal exception in interrupt Jun 14 14:07:22 vmhost2 [ 2418.545746] Pid: 0, comm: swapper Tainted: G D 2.6.32.15-xen4.0.0-dom0-stefano #2 Jun 14 14:07:22 vmhost2 [ 2418.545840] Call Trace: Jun 14 14:07:22 vmhost2 [ 2418.545912] [<c141d3b5>] ? panic+0x42/0xe1 Jun 14 14:07:22 vmhost2 [ 2418.545986] [<c100cc56>] ? oops_end+0x96/0xa0 Jun 14 14:07:22 vmhost2 [ 2418.546060] [<c100a73f>] ? do_invalid_op+0x7f/0x90 Jun 14 14:07:22 vmhost2 [ 2418.546135] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:07:22 vmhost2 [ 2418.546223] [<c10741e4>] ? __alloc_pages_nodemask+0xe4/0x5b0 Jun 14 14:07:22 vmhost2 [ 2418.546303] [<c1006197>] ? xen_force_evtchn_callback+0x17/0x30 Jun 14 14:07:22 vmhost2 [ 2418.546380] [<c1006a98>] ? check_events+0x8/0xc Jun 14 14:07:22 vmhost2 [ 2418.546455] [<c141faa6>] ? error_code+0x66/0x6c Jun 14 14:07:22 vmhost2 [ 2418.546530] [<c100a6c0>] ? do_invalid_op+0x0/0x90 Jun 14 14:07:22 vmhost2 [ 2418.546606] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:07:22 vmhost2 [ 2418.546687] [<c1350000>] ? sock_release+0x10/0x80 Jun 14 14:07:22 vmhost2 [ 2418.546763] [<c12200c1>] ? net_tx_action+0x1d1/0x9b0 Jun 14 14:07:22 vmhost2 [ 2418.546839] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 14 14:07:22 vmhost2 [ 2418.546915] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 14 14:07:22 vmhost2 [ 2418.546993] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 14 14:07:22 vmhost2 [ 2418.547070] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 14 14:07:22 vmhost2 [ 2418.547145] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 14 14:07:22 vmhost2 [ 2418.547222] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 14 14:07:22 vmhost2 [ 2418.547299] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 14 14:07:22 vmhost2 [ 2418.547385] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 14 14:07:22 vmhost2 [ 2418.547463] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 14 14:07:22 vmhost2 [ 2418.547537] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 14 14:07:22 vmhost2 [ 2418.547615] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 14 14:07:22 vmhost2 [ 2418.547690] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 14 14:07:22 vmhost2 [ 2418.547766] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c Best regards, Arnd _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Arnd Hannemann
2010-Jun-14 12:44 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Am 14.06.2010 14:26, schrieb Arnd Hannemann:> Hi, > > Am 14.06.2010 12:57, schrieb Stefano Stabellini: >> On Mon, 14 Jun 2010, Arnd Hannemann wrote: >>> Hi, >>> >>> we have regular but hard to reproduce (wait for a day or two starting domUs) kernel panics (see below) with latest >>> "xen/stable-2.6.32.x" git tree. >>> >>> Any idea, anyone? >>> >> >> this CS from origin/xen/dom0/gntdev should fix your problem: >> >> sstabellini@kaball-desktop:~/xensource/linux-pvops-latest$ git show ad469f0da31bc16b945f9a06710b9d45434d0091 >> commit ad469f0da31bc16b945f9a06710b9d45434d0091 >> Author: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> >> Date: Wed Jun 9 12:34:02 2010 -0700 >> >> xen/gntdev: use spinlocks rather than rwsem for locking >> >> The mmu notifier mechanism calls its callbacks with an rcu lock, >> which disables preemption. This means we cannot use any blocking >> synchronization for locking. >> >> Convert all the rwsemas to plain spinlocks. This requires that >> the memory allocation and copying to/from userspace be split >> from the actual datastructure updates since they can''t be done >> under spinlock. >> >> Signed-off-by: Stefano Stabellini <Stefano.Stabellini@eu.citrix.com> >> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> >> > > Unfortunately, this patch does not seem to help. We get a very similar > backtrace after one hour stress testing with a script starting and stopping > domUs in a loop. > > Maybe the problem is the hypervisor itself? > We are currently using 4.0.1-rc2-pre (we updated from 4.0.0 because of what we believed was the same > problem, we had no working netconsole back then though).FYI: I got lucky and reproduced the error within only 15 minutes and hypervisor version: (XEN) Xen version 4.0.1-rc3-pre (samsel@umic-mesh.net) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) Mon Jun 14 12:43:49 CEST 2010 (XEN) Latest ChangeSet: Fri Jun 11 14:04:36 2010 +0100 21203:3903d95733f7 traceback below Jun 14 14:38:14 vmhost2 [ 201.636188] ------------[ cut here ]------------ Jun 14 14:38:14 vmhost2 [ 201.636272] kernel BUG at drivers/xen/grant-table.c:583! Jun 14 14:38:14 vmhost2 [ 201.636345] invalid opcode: 0000 [#1] Jun 14 14:38:14 vmhost2 SMP Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.636503] last sysfs file: /sys/devices/virtual/net/br0/bridge/topology_change_detected Jun 14 14:38:14 vmhost2 [ 201.636596] Modules linked in: Jun 14 14:38:14 vmhost2 netconsole Jun 14 14:38:14 vmhost2 raid0 Jun 14 14:38:14 vmhost2 md_mod Jun 14 14:38:14 vmhost2 rtc_cmos Jun 14 14:38:14 vmhost2 rtc_core Jun 14 14:38:14 vmhost2 rtc_lib Jun 14 14:38:14 vmhost2 thermal Jun 14 14:38:14 vmhost2 processor Jun 14 14:38:14 vmhost2 ipv6 Jun 14 14:38:14 vmhost2 thermal_sys Jun 14 14:38:14 vmhost2 hwmon Jun 14 14:38:14 vmhost2 button Jun 14 14:38:14 vmhost2 acpi_processor Jun 14 14:38:14 vmhost2 sr_mod Jun 14 14:38:14 vmhost2 pl2303 Jun 14 14:38:14 vmhost2 cdrom Jun 14 14:38:14 vmhost2 usbserial Jun 14 14:38:14 vmhost2 evdev Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.637553] Jun 14 14:38:14 vmhost2 [ 201.637619] Pid: 0, comm: swapper Not tainted (2.6.32.15-xen4.0.0-dom0-stefano #2) System Product Name Jun 14 14:38:14 vmhost2 [ 201.637715] EIP: 0061:[<c120f170>] EFLAGS: 00010282 CPU: 0 Jun 14 14:38:14 vmhost2 [ 201.637792] EIP is at gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:38:14 vmhost2 [ 201.637864] EAX: ffffffea EBX: c153be84 ECX: 00000001 EDX: 00000000 Jun 14 14:38:14 vmhost2 [ 201.637937] ESI: 00007ff0 EDI: 0000000f EBP: c290d120 ESP: c153be50 Jun 14 14:38:14 vmhost2 [ 201.638022] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Jun 14 14:38:14 vmhost2 [ 201.638096] Process swapper (pid: 0, ti=c153a000 task=c1543760 task.ti=c153a000) Jun 14 14:38:14 vmhost2 [ 201.638187] Stack: Jun 14 14:38:14 vmhost2 [ 201.638251] 00000000 Jun 14 14:38:14 vmhost2 00213e1c Jun 14 14:38:14 vmhost2 c28f20c0 Jun 14 14:38:14 vmhost2 0002c189 Jun 14 14:38:14 vmhost2 ec189000 Jun 14 14:38:14 vmhost2 ecd95944 Jun 14 14:38:14 vmhost2 0000000f Jun 14 14:38:14 vmhost2 ec189000 Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.638634] <0> Jun 14 14:38:14 vmhost2 00000000 Jun 14 14:38:14 vmhost2 eb406000 Jun 14 14:38:14 vmhost2 00000000 Jun 14 14:38:14 vmhost2 0000000f Jun 14 14:38:14 vmhost2 ece40000 Jun 14 14:38:14 vmhost2 13e1c001 Jun 14 14:38:14 vmhost2 00000000 Jun 14 14:38:14 vmhost2 0002c189 Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.639115] <0> Jun 14 14:38:14 vmhost2 00000000 Jun 14 14:38:14 vmhost2 c1627a8c Jun 14 14:38:14 vmhost2 c16277c8 Jun 14 14:38:14 vmhost2 c1627a8c Jun 14 14:38:14 vmhost2 000068c4 Jun 14 14:38:14 vmhost2 c12200c1 Jun 14 14:38:14 vmhost2 00000000 Jun 14 14:38:14 vmhost2 ebce8000 Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.639655] Call Trace: Jun 14 14:38:14 vmhost2 [ 201.639729] [<c12200c1>] ? net_tx_action+0x1d1/0x9b0 Jun 14 14:38:14 vmhost2 [ 201.639805] [<c135e4e0>] ? process_backlog+0x90/0xa0 Jun 14 14:38:14 vmhost2 [ 201.639882] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 14 14:38:14 vmhost2 [ 201.639956] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 14 14:38:14 vmhost2 [ 201.640032] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 14 14:38:14 vmhost2 [ 201.640108] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 14 14:38:14 vmhost2 [ 201.640184] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 14 14:38:14 vmhost2 [ 201.640261] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 14 14:38:14 vmhost2 [ 201.640336] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 14 14:38:14 vmhost2 [ 201.640411] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 14 14:38:14 vmhost2 [ 201.640486] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 14 14:38:14 vmhost2 [ 201.640560] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 14 14:38:14 vmhost2 [ 201.640635] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 14 14:38:14 vmhost2 [ 201.640710] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 14 14:38:14 vmhost2 [ 201.640786] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c Jun 14 14:38:14 vmhost2 [ 201.640857] Code: Jun 14 14:38:14 vmhost2 8d Jun 14 14:38:14 vmhost2 5c Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 34 Jun 14 14:38:14 vmhost2 c1 Jun 14 14:38:14 vmhost2 e0 Jun 14 14:38:14 vmhost2 0c Jun 14 14:38:14 vmhost2 83 Jun 14 14:38:14 vmhost2 c8 Jun 14 14:38:14 vmhost2 01 Jun 14 14:38:14 vmhost2 89 Jun 14 14:38:14 vmhost2 44 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 34 Jun 14 14:38:14 vmhost2 8b Jun 14 14:38:14 vmhost2 44 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 0c Jun 14 14:38:14 vmhost2 c7 Jun 14 14:38:14 vmhost2 44 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 40 Jun 14 14:38:14 vmhost2 00 Jun 14 14:38:14 vmhost2 00 Jun 14 14:38:14 vmhost2 00 Jun 14 14:38:14 vmhost2 00 Jun 14 14:38:14 vmhost2 89 Jun 14 14:38:14 vmhost2 44 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 3c Jun 14 14:38:14 vmhost2 e8 Jun 14 14:38:14 vmhost2 b8 Jun 14 14:38:14 vmhost2 1e Jun 14 14:38:14 vmhost2 df Jun 14 14:38:14 vmhost2 ff Jun 14 14:38:14 vmhost2 85 Jun 14 14:38:14 vmhost2 c0 Jun 14 14:38:14 vmhost2 0f Jun 14 14:38:14 vmhost2 84 Jun 14 14:38:14 vmhost2 2c Jun 14 14:38:14 vmhost2 ff Jun 14 14:38:14 vmhost2 ff Jun 14 14:38:14 vmhost2 ff Jun 14 12:38:13 vmhost2 unparseable log message: "<0f> " Jun 14 14:38:14 vmhost2 0b Jun 14 14:38:14 vmhost2 eb Jun 14 14:38:14 vmhost2 fe Jun 14 14:38:14 vmhost2 0f Jun 14 14:38:14 vmhost2 0b Jun 14 14:38:14 vmhost2 eb Jun 14 14:38:14 vmhost2 fe Jun 14 14:38:14 vmhost2 0f Jun 14 14:38:14 vmhost2 0b Jun 14 14:38:14 vmhost2 eb Jun 14 14:38:14 vmhost2 fe Jun 14 14:38:14 vmhost2 0f Jun 14 14:38:14 vmhost2 0b Jun 14 14:38:14 vmhost2 eb Jun 14 14:38:14 vmhost2 fe Jun 14 14:38:14 vmhost2 8b Jun 14 14:38:14 vmhost2 54 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 04 Jun 14 14:38:14 vmhost2 8b Jun 14 14:38:14 vmhost2 44 Jun 14 14:38:14 vmhost2 24 Jun 14 14:38:14 vmhost2 0c Jun 14 14:38:14 vmhost2 e8 Jun 14 14:38:14 vmhost2 Jun 14 14:38:14 vmhost2 [ 201.643843] EIP: [<c120f170>] Jun 14 14:38:14 vmhost2 gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:38:14 vmhost2 SS:ESP 0069:c153be50 Jun 14 14:38:14 vmhost2 [ 201.644028] ---[ end trace af6399fb7ba91a18 ]--- Jun 14 14:38:14 vmhost2 [ 201.644098] Kernel panic - not syncing: Fatal exception in interrupt Jun 14 14:38:14 vmhost2 [ 201.644173] Pid: 0, comm: swapper Tainted: G D 2.6.32.15-xen4.0.0-dom0-stefano #2 Jun 14 14:38:14 vmhost2 [ 201.644265] Call Trace: Jun 14 14:38:14 vmhost2 [ 201.644336] [<c141d3b5>] ? panic+0x42/0xe1 Jun 14 14:38:14 vmhost2 [ 201.644408] [<c100cc56>] ? oops_end+0x96/0xa0 Jun 14 14:38:14 vmhost2 [ 201.644481] [<c100a73f>] ? do_invalid_op+0x7f/0x90 Jun 14 14:38:14 vmhost2 [ 201.644555] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:38:14 vmhost2 [ 201.644632] [<c13de9b0>] ? br_nf_pre_routing_finish+0x0/0x310 Jun 14 14:38:14 vmhost2 [ 201.644709] [<c137ae82>] ? nf_hook_slow+0x62/0xe0 Jun 14 14:38:14 vmhost2 [ 201.644784] [<c10741e4>] ? __alloc_pages_nodemask+0xe4/0x5b0 Jun 14 14:38:14 vmhost2 [ 201.644860] [<c106271d>] ? handle_IRQ_event+0x5d/0xc0 Jun 14 14:38:14 vmhost2 [ 201.644935] [<c141faa6>] ? error_code+0x66/0x6c Jun 14 14:38:14 vmhost2 [ 201.645009] [<c137007b>] ? dev_graft_qdisc+0x5b/0x70 Jun 14 14:38:14 vmhost2 [ 201.645083] [<c100a6c0>] ? do_invalid_op+0x0/0x90 Jun 14 14:38:14 vmhost2 [ 201.645157] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 14 14:38:14 vmhost2 [ 201.645234] [<c12200c1>] ? net_tx_action+0x1d1/0x9b0 Jun 14 14:38:14 vmhost2 [ 201.645308] [<c135e4e0>] ? process_backlog+0x90/0xa0 Jun 14 14:38:14 vmhost2 [ 201.645382] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 14 14:38:14 vmhost2 [ 201.645455] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 14 14:38:14 vmhost2 [ 201.645529] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 14 14:38:14 vmhost2 [ 201.645604] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 14 14:38:14 vmhost2 [ 201.645677] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 14 14:38:14 vmhost2 [ 201.645754] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 14 14:38:14 vmhost2 [ 201.645830] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 14 14:38:14 vmhost2 [ 201.645904] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 14 14:38:14 vmhost2 [ 201.645989] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 14 14:38:14 vmhost2 [ 201.646063] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 14 14:38:14 vmhost2 [ 201.646139] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 14 14:38:14 vmhost2 [ 201.646213] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 14 14:38:14 vmhost2 [ 201.646288] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian Samsel
2010-Jun-21 08:37 UTC
Re: [Xen-devel] xen dom0 2.6.32.15 kernel BUG at drivers/xen/grant-table.c:583
Am Montag, 14. Juni 2010, 14:44:36 schrieb Arnd Hannemann:> > FYI: I got lucky and reproduced the error within only 15 minutes and > hypervisor version: >problem still exists. maybe something of the following information rings a bell. if even more infos are needed just ask. vmhost2 runs small (128 mb ram) domUs which simulate wireless mesh network nodes (network bridged, no blk - they boot over nfs). it runs ubuntu 10.4 over nfs. after some hours of starting and shutting down domUs it hangs with the following (or similiar) traceback. The issue might have gotten worse over the last kernel updates, some weeks ago we managed to start and shutdown over 500 dom0s, but now i hangs quicker, often after 50-60 iterations. Our "old" setup runs ubuntu 8.4 with xen 3.4 and runs stable for weeks doing the same thing. Jun 17 13:13:51 vmhost2 [16259.938609] ------------[ cut here ]------------ Jun 17 13:13:51 vmhost2 [16259.938658] kernel BUG at drivers/xen/grant-table.c:583! Jun 17 13:13:51 vmhost2 [16259.938698] invalid opcode: 0000 [#1] Jun 17 13:13:51 vmhost2 SMP Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.938764] last sysfs file: /sys/devices/virtual/net/br0/bridge/topology_change_detected Jun 17 13:13:51 vmhost2 [16259.938824] Modules linked in: Jun 17 13:13:51 vmhost2 nf_conntrack_ipv4 Jun 17 13:13:51 vmhost2 nf_defrag_ipv4 Jun 17 13:13:51 vmhost2 xt_state Jun 17 13:13:51 vmhost2 nf_conntrack Jun 17 13:13:51 vmhost2 xt_physdev Jun 17 13:13:51 vmhost2 iptable_filter Jun 17 13:13:51 vmhost2 ip_tables Jun 17 13:13:51 vmhost2 x_tables Jun 17 13:13:51 vmhost2 netconsole Jun 17 13:13:51 vmhost2 raid0 Jun 17 13:13:51 vmhost2 md_mod Jun 17 13:13:51 vmhost2 rtc_cmos Jun 17 13:13:51 vmhost2 rtc_core Jun 17 13:13:51 vmhost2 rtc_lib Jun 17 13:13:51 vmhost2 pl2303 Jun 17 13:13:51 vmhost2 thermal Jun 17 13:13:51 vmhost2 usbserial Jun 17 13:13:51 vmhost2 processor Jun 17 13:13:51 vmhost2 thermal_sys Jun 17 13:13:51 vmhost2 button Jun 17 13:13:51 vmhost2 hwmon Jun 17 13:13:51 vmhost2 acpi_processor Jun 17 13:13:51 vmhost2 sr_mod Jun 17 13:13:51 vmhost2 cdrom Jun 17 13:13:51 vmhost2 evdev Jun 17 13:13:51 vmhost2 ipv6 Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.939240] Jun 17 13:13:51 vmhost2 [16259.939273] Pid: 0, comm: swapper Not tainted (2.6.32.15-xen4.0.0-dom0 #2) System Product Name Jun 17 13:13:51 vmhost2 [16259.939335] EIP: 0061:[<c120f170>] EFLAGS: 00010282 CPU: 0 Jun 17 13:13:51 vmhost2 [16259.939385] EIP is at gnttab_copy_grant_page+0x1f0/0x260 Jun 17 13:13:51 vmhost2 [16259.939428] EAX: ffffffea EBX: c153be74 ECX: 00000001 EDX: 00000000 Jun 17 13:13:51 vmhost2 [16259.939469] ESI: 00007ff0 EDI: 0000001c EBP: c28f9ae0 ESP: c153be40 Jun 17 13:13:51 vmhost2 [16259.939510] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Jun 17 13:13:51 vmhost2 [16259.939550] Process swapper (pid: 0, ti=c153a000 task=c1543760 task.ti=c153a000) Jun 17 13:13:51 vmhost2 [16259.939608] Stack: Jun 17 13:13:51 vmhost2 [16259.939640] 00000000 Jun 17 13:13:51 vmhost2 00214815 Jun 17 13:13:51 vmhost2 c28cefe0 Jun 17 13:13:51 vmhost2 0002bf57 Jun 17 13:13:51 vmhost2 ebf57000 Jun 17 13:13:51 vmhost2 ebc42078 Jun 17 13:13:51 vmhost2 0000001c Jun 17 13:13:51 vmhost2 ebf57000 Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.939762] <0> Jun 17 13:13:51 vmhost2 00000000 Jun 17 13:13:51 vmhost2 ea9ff000 Jun 17 13:13:51 vmhost2 00000000 Jun 17 13:13:51 vmhost2 0000001c Jun 17 13:13:51 vmhost2 00000000 Jun 17 13:13:51 vmhost2 14815001 Jun 17 13:13:51 vmhost2 00000000 Jun 17 13:13:51 vmhost2 0002bf57 Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.939914] <0> Jun 17 13:13:51 vmhost2 00000000 Jun 17 13:13:51 vmhost2 edab8a24 Jun 17 13:13:51 vmhost2 edab78a4 Jun 17 13:13:51 vmhost2 edab8a24 Jun 17 13:13:51 vmhost2 edab7818 Jun 17 13:13:51 vmhost2 c1220dc2 Jun 17 13:13:51 vmhost2 00000100 Jun 17 13:13:51 vmhost2 00000001 Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.940094] Call Trace: Jun 17 13:13:51 vmhost2 [16259.940135] [<c1220dc2>] ? net_tx_action+0x1d2/0x9f0 Jun 17 13:13:51 vmhost2 [16259.940179] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 17 13:13:51 vmhost2 [16259.940221] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 17 13:13:51 vmhost2 [16259.940263] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 17 13:13:51 vmhost2 [16259.940307] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 17 13:13:51 vmhost2 [16259.940348] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 17 13:13:51 vmhost2 [16259.940400] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 17 13:13:51 vmhost2 [16259.940443] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 17 13:13:51 vmhost2 [16259.940486] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 17 13:13:51 vmhost2 [16259.940528] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 17 13:13:51 vmhost2 [16259.940569] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 17 13:13:51 vmhost2 [16259.940611] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 17 13:13:51 vmhost2 [16259.940653] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 17 13:13:51 vmhost2 [16259.940696] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c Jun 17 13:13:51 vmhost2 [16259.940735] Code: Jun 17 13:13:51 vmhost2 8d Jun 17 13:13:51 vmhost2 5c Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 34 Jun 17 13:13:51 vmhost2 c1 Jun 17 13:13:51 vmhost2 e0 Jun 17 13:13:51 vmhost2 0c Jun 17 13:13:51 vmhost2 83 Jun 17 13:13:51 vmhost2 c8 Jun 17 13:13:51 vmhost2 01 Jun 17 13:13:51 vmhost2 89 Jun 17 13:13:51 vmhost2 44 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 34 Jun 17 13:13:51 vmhost2 8b Jun 17 13:13:51 vmhost2 44 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 0c Jun 17 13:13:51 vmhost2 c7 Jun 17 13:13:51 vmhost2 44 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 40 Jun 17 13:13:51 vmhost2 00 Jun 17 13:13:51 vmhost2 00 Jun 17 13:13:51 vmhost2 00 Jun 17 13:13:51 vmhost2 00 Jun 17 13:13:51 vmhost2 89 Jun 17 13:13:51 vmhost2 44 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 3c Jun 17 13:13:51 vmhost2 e8 Jun 17 13:13:51 vmhost2 b8 Jun 17 13:13:51 vmhost2 1e Jun 17 13:13:51 vmhost2 df Jun 17 13:13:51 vmhost2 ff Jun 17 13:13:51 vmhost2 85 Jun 17 13:13:51 vmhost2 c0 Jun 17 13:13:51 vmhost2 0f Jun 17 13:13:51 vmhost2 84 Jun 17 13:13:51 vmhost2 2c Jun 17 13:13:51 vmhost2 ff Jun 17 13:13:51 vmhost2 ff Jun 17 13:13:51 vmhost2 ff Jun 17 11:13:50 vmhost2 unparseable log message: "<0f> " Jun 17 13:13:51 vmhost2 0b Jun 17 13:13:51 vmhost2 eb Jun 17 13:13:51 vmhost2 fe Jun 17 13:13:51 vmhost2 0f Jun 17 13:13:51 vmhost2 0b Jun 17 13:13:51 vmhost2 eb Jun 17 13:13:51 vmhost2 fe Jun 17 13:13:51 vmhost2 0f Jun 17 13:13:51 vmhost2 0b Jun 17 13:13:51 vmhost2 eb Jun 17 13:13:51 vmhost2 fe Jun 17 13:13:51 vmhost2 8b Jun 17 13:13:51 vmhost2 54 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 04 Jun 17 13:13:51 vmhost2 8b Jun 17 13:13:51 vmhost2 44 Jun 17 13:13:51 vmhost2 24 Jun 17 13:13:51 vmhost2 0c Jun 17 13:13:51 vmhost2 e8 Jun 17 13:13:51 vmhost2 Jun 17 13:13:51 vmhost2 [16259.941540] EIP: [<c120f170>] Jun 17 13:13:51 vmhost2 gnttab_copy_grant_page+0x1f0/0x260 Jun 17 13:13:51 vmhost2 SS:ESP 0069:c153be40 Jun 17 13:13:51 vmhost2 [16259.941864] ---[ end trace 0514df71b6948a8c ]--- Jun 17 13:13:51 vmhost2 [16259.941934] Kernel panic - not syncing: Fatal exception in interrupt Jun 17 13:13:51 vmhost2 [16259.942009] Pid: 0, comm: swapper Tainted: G D 2.6.32.15-xen4.0.0-dom0 #2 Jun 17 13:13:51 vmhost2 [16259.942100] Call Trace: Jun 17 13:13:51 vmhost2 [16259.942170] [<c141d9d5>] ? panic+0x42/0xe1 Jun 17 13:13:51 vmhost2 [16259.942243] [<c100cc56>] ? oops_end+0x96/0xa0 Jun 17 13:13:51 vmhost2 [16259.942316] [<c100a73f>] ? do_invalid_op+0x7f/0x90 Jun 17 13:13:51 vmhost2 [16259.942390] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 17 13:13:51 vmhost2 [16259.942468] [<c10741e4>] ? __alloc_pages_nodemask+0xe4/0x5b0 Jun 17 13:13:51 vmhost2 [16259.942543] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 17 13:13:51 vmhost2 [16259.942621] [<c14200c6>] ? error_code+0x66/0x6c Jun 17 13:13:51 vmhost2 [16259.942694] [<c100a6c0>] ? do_invalid_op+0x0/0x90 Jun 17 13:13:51 vmhost2 [16259.942767] [<c120f170>] ? gnttab_copy_grant_page+0x1f0/0x260 Jun 17 13:13:51 vmhost2 [16259.942844] [<c1220dc2>] ? net_tx_action+0x1d2/0x9f0 Jun 17 13:13:51 vmhost2 [16259.942919] [<c103bc2e>] ? tasklet_action+0x9e/0xb0 Jun 17 13:13:51 vmhost2 [16259.942992] [<c103c378>] ? __do_softirq+0x88/0x110 Jun 17 13:13:51 vmhost2 [16259.943066] [<c1210057>] ? __xen_evtchn_do_upcall+0xd7/0x160 Jun 17 13:13:51 vmhost2 [16259.943142] [<c103c43d>] ? do_softirq+0x3d/0x40 Jun 17 13:13:51 vmhost2 [16259.943215] [<c121063a>] ? xen_evtchn_do_upcall+0x2a/0x40 Jun 17 13:13:51 vmhost2 [16259.943289] [<c1009da7>] ? xen_do_upcall+0x7/0xc Jun 17 13:13:51 vmhost2 [16259.943369] [<c10013a7>] ? hypercall_page+0x3a7/0x1010 Jun 17 13:13:51 vmhost2 [16259.943451] [<c10061ef>] ? xen_safe_halt+0xf/0x20 Jun 17 13:13:51 vmhost2 [16259.943525] [<c100382c>] ? xen_idle+0x1c/0x30 Jun 17 13:13:51 vmhost2 [16259.943598] [<c10081fa>] ? cpu_idle+0x3a/0x60 Jun 17 13:13:51 vmhost2 [16259.943671] [<c15787ef>] ? start_kernel+0x2c6/0x2cb Jun 17 13:13:51 vmhost2 [16259.943744] [<c1578367>] ? unknown_bootoption+0x0/0x190 Jun 17 13:13:51 vmhost2 [16259.943819] [<c157b0e6>] ? xen_start_kernel+0x624/0x62c unfortunately our test machine has no native serial port, so we have no access to the hypervisor output atm. although we suspect the hypervisor to be the problem. Kernel latest xen/stable-2.6.32.x (config attached): samsel@vmhost2:~$ uname -a Linux vmhost2 2.6.32.15-xen4.0-dom0 #2 SMP Wed Jun 16 14:02:14 CEST 2010 i686 GNU/Linux samsel@vmhost2:~/build/linux-2.6.32-xen$ git log | head commit 01d9fbca207ec232c758d991d66466fc6e38349e Merge: cfce2d4 0a904db Author: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com> Date: Tue Jun 15 14:09:48 2010 +0100 Merge branch ''xen/next'' into xen/stable-2.6.32.x * xen/next: netback: minor code formatting fixup Netback: Set allocated memory to zero from vmalloc. $ sudo xm info host : vmhost2 release : 2.6.32.15-xen4.0.0-dom0 version : #2 SMP Wed Jun 16 14:02:14 CEST 2010 machine : i686 nr_cpus : 8 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2808 hw_caps : bfebfbff:28100000:00000000:00001b40:0098e3fd:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 8183 free_memory : 5032 node_to_cpu : node0:0-7 node_to_memory : node0:5032 node_to_dma32_mem : node0:2925 max_node_id : 0 xen_major : 4 xen_minor : 0 xen_extra : .1-rc3-pre xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : Fri Jun 11 14:04:36 2010 +0100 21203:3903d95733f7 xen_commandline : dom0_mem=4G cc_compiler : gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) cc_compile_by : samsel cc_compile_domain : umic-mesh.net cc_compile_date : Mon Jun 14 12:43:49 CEST 2010 xend_config_format : 4 pxe boot config: KERNEL /linux/default/vmeshhost-mboot.c32 APPEND /initrd/vmeshhost-xen-testing dom0_mem=4G --- /linux/default/vmeshhost-vmlinuz-testing id=default image=vmeshhost/testing nodetype=vmeshhost rw root=/dev/ram0 initrd=/initrd/vmeshhost-dom0-initrd-testing init=/linuxrc --- /initrd/vmeshhost-dom0-initrd-testing samsel@vmhost2:~$ lspci 00:00.0 Host bridge: Intel Corporation Device d131 (rev 11) 00:03.0 PCI bridge: Intel Corporation Device d138 (rev 11) 00:08.0 System peripheral: Intel Corporation Device d155 (rev 11) 00:08.1 System peripheral: Intel Corporation Device d156 (rev 11) 00:08.2 System peripheral: Intel Corporation Device d157 (rev 11) 00:08.3 System peripheral: Intel Corporation Device d158 (rev 11) 00:10.0 System peripheral: Intel Corporation Device d150 (rev 11) 00:10.1 System peripheral: Intel Corporation Device d151 (rev 11) 00:1a.0 USB Controller: Intel Corporation Device 3b3c (rev 05) 00:1b.0 Audio device: Intel Corporation Device 3b56 (rev 05) 00:1c.0 PCI bridge: Intel Corporation Device 3b42 (rev 05) 00:1c.4 PCI bridge: Intel Corporation Device 3b4a (rev 05) 00:1c.5 PCI bridge: Intel Corporation Device 3b4c (rev 05) 00:1c.6 PCI bridge: Intel Corporation Device 3b4e (rev 05) 00:1c.7 PCI bridge: Intel Corporation Device 3b50 (rev 05) 00:1d.0 USB Controller: Intel Corporation Device 3b34 (rev 05) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation Device 3b02 (rev 05) 00:1f.2 SATA controller: Intel Corporation Device 3b22 (rev 05) 00:1f.3 SMBus: Intel Corporation Device 3b30 (rev 05) 01:00.0 VGA compatible controller: ATI Technologies Inc Device 954f 01:00.1 Audio device: ATI Technologies Inc Device aa38 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03) <- not used 03:00.0 SATA controller: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 03) 03:00.1 IDE interface: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller (rev 03) 07:01.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller <- used 07:04.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev c0) samsel@vmhost2:/proc$ cat cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 30 model name : Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz stepping : 5 cpu MHz : 2808.822 cache size : 8192 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu de tsc msr pae mce cx8 apic mtrr mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht nx constant_tsc nonstop_tsc aperfmperf pni est ssse3 sse4_1 sse4_2 popcnt hypervisor ida bogomips : 5617.64 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: DomUs (example, we run various kernels): samsel@vmrouter313:~$ uname -a Linux vmrouter313 2.6.24.7-pae-um #7 SMP Thu Apr 9 15:35:55 CEST 2009 i686 GNU/Linux _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel