Elena Ufimtseva
2013-Dec-01 06:03 UTC
pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
Hi pv kernel crashes when starting guest with vcpus above 150 with latest linux kernel. WIth increase of max_event_channel it boots (I tried 4096). [ 0.387911] ------------[ cut here ]------------ [ 0.387918] kernel BUG at drivers/xen/events.c:909! [ 0.387922] invalid opcode: 0000 [#1] SMP [ 0.387928] Modules linked in: [ 0.387934] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.12.0+ #95 [ 0.387940] task: ffff880088ecb080 ti: ffff880088ecc000 task.ti: ffff880088ecc000 [ 0.387945] RIP: e030:[<ffffffff8126b310>] [<ffffffff8126b310>] bind_ipi_to_irqhandler+0x9c/0x15d [ 0.387959] RSP: e02b:ffff880088ecdd18 EFLAGS: 00010286 [ 0.387963] RAX: ffffffffffffffe4 RBX: 000000000000040d RCX: 0000000000000007 [ 0.387968] RDX: 0000000000000001 RSI: 00000000deadbeef RDI: 00000000deadbeef [ 0.387972] RBP: 0000000000000091 R08: ffff880089003d70 R09: ffff880088ecdd50 [ 0.387977] R10: 0000000000001f00 R11: 0000000000000001 R12: 0000000000000091 [ 0.387982] R13: 0000000000000091 R14: 0000000000010b20 R15: 0000000000000004 [ 0.387990] FS: 0000000000000000(0000) GS:ffff880089400000(0000) knlGS:0000000000000000 [ 0.387996] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.388000] CR2: 0000000000000000 CR3: 000000000160c000 CR4: 0000000000042660 [ 0.388000] Stack: [ 0.388000] 0000000000000091 0000000000000091 ffffffff8100bc33 0000000000000c20 [ 0.388000] ffff880168c88760 0000000000000000 ffff880088ecdd98 ffff880000000091 [ 0.388000] ffff880088ecddb4 ffff88008c468000 0000000000000091 0000000000000091 [ 0.388000] Call Trace: [ 0.388000] [<ffffffff8100bc33>] ? xen_hvm_cpu_up+0x53/0x53 [ 0.388000] [<ffffffff8100bba0>] ? xen_smp_intr_init+0x1be/0x1fe [ 0.388000] [<ffffffff8100c17c>] ? xen_cpu_up+0x308/0x35d [ 0.388000] [<ffffffff81047532>] ? _cpu_up+0xc1/0x120 [ 0.388000] [<ffffffff8104765f>] ? cpu_up+0xce/0xe2 [ 0.388000] [<ffffffff816e79f5>] ? smp_init+0x56/0x70 [ 0.388000] [<ffffffff816c8e22>] ? kernel_init_freeable+0xc4/0x1eb [ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70 [ 0.388000] [<ffffffff813b34d2>] ? kernel_init+0x6/0xd3 [ 0.388000] [<ffffffff813c860c>] ? ret_from_fork+0x7c/0xb0 [ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70 [ 0.388000] Code: 81 48 c7 c2 bb 4c 08 81 48 c7 c6 20 d6 6a 81 e8 ad 96 e1 ff 48 8d 74 24 38 bf 07 00 00 00 89 6c 24 38 e8 c9 ea ff ff 85 c0 74 02 <0f> 0b 44 8b 64 24 3c 89 df e8 80 ea ff ff 89 de 48 89 c7 ba 03 [ 0.388000] RIP [<ffffffff8126b310>] bind_ipi_to_irqhandler+0x9c/0x15d [ 0.388000] RSP <ffff880088ecdd18> [ 0.388005] ---[ end trace 8347cf354b5857c5 ]--- [ 0.388026] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 0.388026] -- Elena
Dario Faggioli
2013-Dec-01 08:03 UTC
Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote:> Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. >Wow... >150 vcpus for a single guest? What is it that you are up to here?!? :-P :-P> WIth increase of max_event_channel it boots (I tried > 4096). >I don''t remember the exact numbers. This: http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements says you should be able to reach 750 vcpus, but it''s probably _in_total_, not necessarily within a single guest. Also, these talks should have some numbers (although the latter is about XenServer), but again, it''s mostly about total number, I think: http://www.youtube.com/watch?v=uz52hxKbv5I http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1 Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Wei Liu
2013-Dec-02 12:38 UTC
Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote:> Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. WIth increase of max_event_channel it boots (I tried > 4096). > > > [ 0.387911] ------------[ cut here ]------------ > [ 0.387918] kernel BUG at drivers/xen/events.c:909!This is due to hypervisor failing to bind IPI for that guest. You''re probably running out of resources. Try ''wc -l /proc/interrupts'' ''grep ipi /proc/interrputs | wc -l'' when the guest is running to have an idea how many it needs. ;-) Wei.
Elena Ufimtseva
2013-Dec-02 16:41 UTC
Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
On Sun, Dec 1, 2013 at 3:03 AM, Dario Faggioli <dario.faggioli@citrix.com> wrote:> On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote: >> Hi >> >> pv kernel crashes when starting guest with vcpus above 150 with latest >> linux kernel. >> > Wow... >150 vcpus for a single guest? What is it that you are up to > here?!? :-P :-PWas experimenting with 150 nodes :P> >> WIth increase of max_event_channel it boots (I tried >> 4096). >> > I don''t remember the exact numbers. This: > http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements > says you should be able to reach 750 vcpus, but it''s probably > _in_total_, not necessarily within a single guest. > > Also, these talks should have some numbers (although the latter is about > XenServer), but again, it''s mostly about total number, I think: > http://www.youtube.com/watch?v=uz52hxKbv5I > http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1Yes, I wanted to see if it should behave this way.> > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) >-- Elena
Elena Ufimtseva
2013-Dec-02 16:42 UTC
Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
On Mon, Dec 2, 2013 at 7:38 AM, Wei Liu <wei.liu2@citrix.com> wrote:> On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote: >> Hi >> >> pv kernel crashes when starting guest with vcpus above 150 with latest >> linux kernel. WIth increase of max_event_channel it boots (I tried >> 4096). >> >> >> [ 0.387911] ------------[ cut here ]------------ >> [ 0.387918] kernel BUG at drivers/xen/events.c:909! > > This is due to hypervisor failing to bind IPI for that guest. You''re > probably running out of resources. > > Try ''wc -l /proc/interrupts'' ''grep ipi /proc/interrputs | wc -l'' when > the guest is running to have an idea how many it needs. ;-) >Yes, I am sure it is ) Thank you, will check it out.> Wei.-- Elena
David Vrabel
2013-Dec-02 17:06 UTC
Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
On 01/12/13 06:03, Elena Ufimtseva wrote:> Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. WIth increase of max_event_channel it boots (I tried > 4096).The default for max_event_channels is 1023 which should be enough for a typical guest. 150 VCPUs is very much not a typical guest. We''ll need to be sure to include a release note on this for the 4.4 release. David