4.1.2-rc3-pre @ 23159 3.0.4 + vmalloc_sync_all() patch While running my Xen test suite: BUG: unable to handle kernel paging request at bffffd68 IP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 *pdpt = 0000000027d71027 *pde = 0000000000000000 Oops: 0000 [#1] SMP Modules linked in: ebt_arp ip6t_rt ebt_mark ebt_limit ip6table_mangle xt_mark ebtable_nat ebtable_filter Pid: 39, comm: xenwatch Not tainted 3.0.4-1 #1 Supermicro X8DTU/X8DTU EIP: 0061:[<c102bb51>] EFLAGS: 00010283 CPU: 3 EIP is at vmalloc_sync_all+0x141/0x1e0 EAX: bffffd68 EBX: efe5a720 ECX: 00000008 EDX: 00000001 ESI: ea082a34 EDI: c1a73d68 EBP: eb1d9e80 ESP: eb1d9e4c DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 Process xenwatch (pid: 39, ti=eb1d8000 task=eb11b0c0 task.ti=eb1d8000) Stack: 2b048000 00000000 2b048000 00000000 fffff001 00000008 c0000d68 7bf85067 f5a00000 00000018 ea254480 eb1d9f14 eb1d9f54 eb1d9e94 c10e5212 c10e35a0 00000000 e7d3d600 eb1d9ee4 c1446275 eb1d9ea8 c13932c0 e7d3d600 00000011 Call Trace: [<c10e5212>] alloc_vm_area+0x42/0x60 [<c10e35a0>] ? is_vmalloc_or_module_addr+0x50/0x50 [<c1446275>] xen_blkif_map+0x35/0x1e0 [<c13932c0>] ? xen_evtchn_do_upcall+0x20/0x30 [<c1446a61>] frontend_changed+0x261/0x2d0 [<c139852d>] xenbus_otherend_changed+0x7d/0x90 [<c1398732>] frontend_changed+0x12/0x20 [<c1396aa5>] xenwatch_thread+0x85/0x130 [<c10625d0>] ? wake_up_bit+0x60/0x60 [<c1396a20>] ? split+0xd0/0xd0 [<c10621e4>] kthread+0x74/0x80 [<c1062170>] ? kthread_worker_fn+0x160/0x160 [<c16ca2b6>] kernel_thread_helper+0x6/0x10 Code: c1 89 c7 81 e7 00 f0 ff ff 03 7d e4 8b 17 89 55 e8 8b 4f 04 83 e2 01 89 4d e0 0f 84 68 ff ff ff 8b 45 dc 25 00 f0 ff ff 03 45 e4 <8b> 08 8b 50 04 f6 c1 01 89 55 dc 74 7a 8b 55 dc 89 c8 ff 15 14 EIP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 SS:ESP 0069:eb1d9e4c CR2: 00000000bffffd68 ---[ end trace 486c192808e46938 ]--- INFO: rcu_sched_state detected stall on CPU 7 (t=60000 jiffies) INFO: rcu_sched_state detected stall on CPU 7 (t=240030 jiffies) ... and so on -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28/09/11 15:28, Christopher S. Aker wrote:> 4.1.2-rc3-pre @ 23159 > 3.0.4 + vmalloc_sync_all() patchThis is surprising as that vmalloc_sync_all() patch is a revert of a recent change only present in 3.0 and others have reported the patch works for them. You''re going to have to provide more information on your system and tests I think. David> While running my Xen test suite: > > BUG: unable to handle kernel paging request at bffffd68 > IP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 > *pdpt = 0000000027d71027 *pde = 0000000000000000 > Oops: 0000 [#1] SMP > Modules linked in: ebt_arp ip6t_rt ebt_mark ebt_limit ip6table_mangle > xt_mark ebtable_nat ebtable_filter > Pid: 39, comm: xenwatch Not tainted 3.0.4-1 #1 Supermicro X8DTU/X8DTU > EIP: 0061:[<c102bb51>] EFLAGS: 00010283 CPU: 3 > EIP is at vmalloc_sync_all+0x141/0x1e0 > EAX: bffffd68 EBX: efe5a720 ECX: 00000008 EDX: 00000001 > ESI: ea082a34 EDI: c1a73d68 EBP: eb1d9e80 ESP: eb1d9e4c > DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069 > Process xenwatch (pid: 39, ti=eb1d8000 task=eb11b0c0 task.ti=eb1d8000) > Stack: > 2b048000 00000000 2b048000 00000000 fffff001 00000008 c0000d68 7bf85067 > f5a00000 00000018 ea254480 eb1d9f14 eb1d9f54 eb1d9e94 c10e5212 c10e35a0 > 00000000 e7d3d600 eb1d9ee4 c1446275 eb1d9ea8 c13932c0 e7d3d600 00000011 > Call Trace: > [<c10e5212>] alloc_vm_area+0x42/0x60 > [<c10e35a0>] ? is_vmalloc_or_module_addr+0x50/0x50 > [<c1446275>] xen_blkif_map+0x35/0x1e0 > [<c13932c0>] ? xen_evtchn_do_upcall+0x20/0x30 > [<c1446a61>] frontend_changed+0x261/0x2d0 > [<c139852d>] xenbus_otherend_changed+0x7d/0x90 > [<c1398732>] frontend_changed+0x12/0x20 > [<c1396aa5>] xenwatch_thread+0x85/0x130 > [<c10625d0>] ? wake_up_bit+0x60/0x60 > [<c1396a20>] ? split+0xd0/0xd0 > [<c10621e4>] kthread+0x74/0x80 > [<c1062170>] ? kthread_worker_fn+0x160/0x160 > [<c16ca2b6>] kernel_thread_helper+0x6/0x10 > Code: c1 89 c7 81 e7 00 f0 ff ff 03 7d e4 8b 17 89 55 e8 8b 4f 04 83 e2 > 01 89 4d e0 0f 84 68 ff ff ff 8b 45 dc 25 00 f0 ff ff 03 45 e4 <8b> 08 > 8b 50 04 f6 c1 01 89 55 dc 74 7a 8b 55 dc 89 c8 ff 15 14 > EIP: [<c102bb51>] vmalloc_sync_all+0x141/0x1e0 SS:ESP 0069:eb1d9e4c > CR2: 00000000bffffd68 > ---[ end trace 486c192808e46938 ]--- > INFO: rcu_sched_state detected stall on CPU 7 (t=60000 jiffies) > INFO: rcu_sched_state detected stall on CPU 7 (t=240030 jiffies) > ... and so on > > -Chris > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Sep-28 16:02 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On 9/28/11 10:45 AM, David Vrabel wrote:> You''re going to have to provide more information on your system and > tests I think.Nothing crazy. 64 bit Xen, 32 bit dom0, my test suite creates many domUs (in this case about 40) each with a root image and swap image. Some swap thrash, some spin cpu, some are repeatedly shut down or xm destroyed. No networking. This particular box has about 32G in it, and itself and many other boxes identical to it have no problem with our old stack (xen 3.4, 2.6.18 dom0). I''ve restarted the tests to see if I can reproduce, but I''m certain that if it happened once, it''ll happen again. Can I provide anything else? -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28/09/11 17:02, Christopher S. Aker wrote:> On 9/28/11 10:45 AM, David Vrabel wrote: >> You''re going to have to provide more information on your system and >> tests I think. > > Nothing crazy. 64 bit Xen, 32 bit dom0, my test suite creates many > domUs (in this case about 40) each with a root image and swap image. > Some swap thrash, some spin cpu, some are repeatedly shut down or xm > destroyed. No networking. This particular box has about 32G in it, and > itself and many other boxes identical to it have no problem with our old > stack (xen 3.4, 2.6.18 dom0). > > I''ve restarted the tests to see if I can reproduce, but I''m certain that > if it happened once, it''ll happen again.Instead of the vmalloc_sync_all() patch you could try this series instead. http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html David _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Sep-30 15:38 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On 9/28/11 1:04 PM, David Vrabel wrote:> On 28/09/11 17:02, Christopher S. Aker wrote: >> I''ve restarted the tests to see if I can reproduce, but I''m certain that >> if it happened once, it''ll happen again.I was able to reproduce it.> Instead of the vmalloc_sync_all() patch you could try this series instead. > > http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.htmlI''ll rebuild using this series and reset the tests. Will let you know! Thanks, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-10 15:51 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote:> On 9/28/11 1:04 PM, David Vrabel wrote: > >On 28/09/11 17:02, Christopher S. Aker wrote: > >>I''ve restarted the tests to see if I can reproduce, but I''m certain that > >>if it happened once, it''ll happen again. > > I was able to reproduce it. > > >Instead of the vmalloc_sync_all() patch you could try this series instead. > > > >http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html > > I''ll rebuild using this series and reset the tests. Will let you know!ping? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Oct-11 15:18 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On Oct 10, 2011, at 11:51 AM, Konrad Rzeszutek Wilk wrote:> On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote: >> On 9/28/11 1:04 PM, David Vrabel wrote: >>> On 28/09/11 17:02, Christopher S. Aker wrote: >>>> I''ve restarted the tests to see if I can reproduce, but I''m certain that >>>> if it happened once, it''ll happen again. >> >> I was able to reproduce it. >> >>> Instead of the vmalloc_sync_all() patch you could try this series instead. >>> >>> http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html >> >> I''ll rebuild using this series and reset the tests. Will let you know! > > ping?With that patchset applied: 6 days of thrashing, 30000 domains cycled, and no crashes. I''d say it worked. Cheers, -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Oct-11 17:28 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On Tue, Oct 11, 2011 at 11:18:12AM -0400, Christopher S. Aker wrote:> On Oct 10, 2011, at 11:51 AM, Konrad Rzeszutek Wilk wrote: > > On Fri, Sep 30, 2011 at 11:38:23AM -0400, Christopher S. Aker wrote: > >> On 9/28/11 1:04 PM, David Vrabel wrote: > >>> On 28/09/11 17:02, Christopher S. Aker wrote: > >>>> I''ve restarted the tests to see if I can reproduce, but I''m certain that > >>>> if it happened once, it''ll happen again. > >> > >> I was able to reproduce it. > >> > >>> Instead of the vmalloc_sync_all() patch you could try this series instead. > >>> > >>> http://lists.xensource.com/archives/html/xen-devel/2011-09/msg01343.html > >> > >> I''ll rebuild using this series and reset the tests. Will let you know! > > > > ping? > > With that patchset applied: 6 days of thrashing, 30000 domains cycled, and no crashes. I''d say it worked.Excellent. You got any more bugs that we should be aware off ? (hopefully not). _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christopher S. Aker
2011-Oct-12 19:37 UTC
Re: [Xen-devel] vmalloc_sync_all() patch problems?
On 10/11/11 1:28 PM, Konrad Rzeszutek Wilk wrote:>> With that patchset applied: 6 days of thrashing, 30000 domains >> cycled, and no crashes. I''d say it worked. > > Excellent. You got any more bugs that we should be aware off ? > (hopefully not).Always! I''ll gather the info and author posts over the next couple of days. -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel