Konrad, et al. I''m having an issue with my guests that, when I am running wireless drivers in dom0, and the PV-on-HVM guest tries to access the network - I get a hung dom0. When I turn on Kernel Hacking-> Spinlock and rw-lock debugging: basic checks The hang goes away, but does not print out anything terribly useful in the serial log. The config file changes made by this menuconfig setting can be found below: While I know from other conversations, that Konrad is aware of my setup, for other''s benefit - this is a 64bit dom0 3.2.2 kernel, with a few of his branches pulled in. Any thoughts, or suggestions on how best to debug these deadlocks would be greatly appreciated. -Ben Guthro diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64 index 205b5c6..f7a233f 100644 --- a/configs/config.ubuntu.amd64 +++ b/configs/config.ubuntu.amd64 @@ -257,27 +257,27 @@ CONFIG_PADATA=y # CONFIG_INLINE_SPIN_LOCK_BH is not set # CONFIG_INLINE_SPIN_LOCK_IRQ is not set # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set -CONFIG_INLINE_SPIN_UNLOCK=y +# CONFIG_INLINE_SPIN_UNLOCK is not set # CONFIG_INLINE_SPIN_UNLOCK_BH is not set -CONFIG_INLINE_SPIN_UNLOCK_IRQ=y +# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_READ_TRYLOCK is not set # CONFIG_INLINE_READ_LOCK is not set # CONFIG_INLINE_READ_LOCK_BH is not set # CONFIG_INLINE_READ_LOCK_IRQ is not set # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set -CONFIG_INLINE_READ_UNLOCK=y +# CONFIG_INLINE_READ_UNLOCK is not set # CONFIG_INLINE_READ_UNLOCK_BH is not set -CONFIG_INLINE_READ_UNLOCK_IRQ=y +# CONFIG_INLINE_READ_UNLOCK_IRQ is not set # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set # CONFIG_INLINE_WRITE_TRYLOCK is not set # CONFIG_INLINE_WRITE_LOCK is not set # CONFIG_INLINE_WRITE_LOCK_BH is not set # CONFIG_INLINE_WRITE_LOCK_IRQ is not set # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set -CONFIG_INLINE_WRITE_UNLOCK=y +# CONFIG_INLINE_WRITE_UNLOCK is not set # CONFIG_INLINE_WRITE_UNLOCK_BH is not set -CONFIG_INLINE_WRITE_UNLOCK_IRQ=y +# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set CONFIG_MUTEX_SPIN_ON_OWNER=y CONFIG_FREEZER=y @@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y CONFIG_ARCH_DISCARD_MEMBLOCK=y # CONFIG_MEMORY_HOTPLUG is not set CONFIG_PAGEFLAGS_EXTENDED=y -CONFIG_SPLIT_PTLOCK_CPUS=4 +CONFIG_SPLIT_PTLOCK_CPUS=999999 # CONFIG_COMPACTION is not set CONFIG_MIGRATION=y CONFIG_PHYS_ADDR_T_64BIT=y @@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y # CONFIG_DEBUG_KMEMLEAK is not set # CONFIG_DEBUG_RT_MUTEXES is not set # CONFIG_RT_MUTEX_TESTER is not set -# CONFIG_DEBUG_SPINLOCK is not set +CONFIG_DEBUG_SPINLOCK=y # CONFIG_DEBUG_MUTEXES is not set # CONFIG_DEBUG_LOCK_ALLOC is not set # CONFIG_PROVE_LOCKING is not set
On 30/01/12 16:27, Ben Guthro wrote:> Konrad, et al. > > I''m having an issue with my guests that, when I am running wireless > drivers in dom0, and the PV-on-HVM guest tries to access the network - > I get a hung dom0. > > When I turn on > Kernel Hacking-> > Spinlock and rw-lock debugging: basic checks > > The hang goes away, but does not print out anything terribly useful in > the serial log. > > > The config file changes made by this menuconfig setting can be found below: > > > While I know from other conversations, that Konrad is aware of my > setup, for other''s benefit - this is a 64bit dom0 3.2.2 kernel, with a > few of his branches pulled in. > > Any thoughts, or suggestions on how best to debug these deadlocks > would be greatly appreciated.It looks like http://lists.xen.org/archives/html/xen-devel/2012-01/msg01917.html might apply to you. Does that patch help? ~Andrew> > -Ben Guthro > > > diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64 > index 205b5c6..f7a233f 100644 > --- a/configs/config.ubuntu.amd64 > +++ b/configs/config.ubuntu.amd64 > @@ -257,27 +257,27 @@ CONFIG_PADATA=y > # CONFIG_INLINE_SPIN_LOCK_BH is not set > # CONFIG_INLINE_SPIN_LOCK_IRQ is not set > # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set > -CONFIG_INLINE_SPIN_UNLOCK=y > +# CONFIG_INLINE_SPIN_UNLOCK is not set > # CONFIG_INLINE_SPIN_UNLOCK_BH is not set > -CONFIG_INLINE_SPIN_UNLOCK_IRQ=y > +# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set > # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set > # CONFIG_INLINE_READ_TRYLOCK is not set > # CONFIG_INLINE_READ_LOCK is not set > # CONFIG_INLINE_READ_LOCK_BH is not set > # CONFIG_INLINE_READ_LOCK_IRQ is not set > # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set > -CONFIG_INLINE_READ_UNLOCK=y > +# CONFIG_INLINE_READ_UNLOCK is not set > # CONFIG_INLINE_READ_UNLOCK_BH is not set > -CONFIG_INLINE_READ_UNLOCK_IRQ=y > +# CONFIG_INLINE_READ_UNLOCK_IRQ is not set > # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set > # CONFIG_INLINE_WRITE_TRYLOCK is not set > # CONFIG_INLINE_WRITE_LOCK is not set > # CONFIG_INLINE_WRITE_LOCK_BH is not set > # CONFIG_INLINE_WRITE_LOCK_IRQ is not set > # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set > -CONFIG_INLINE_WRITE_UNLOCK=y > +# CONFIG_INLINE_WRITE_UNLOCK is not set > # CONFIG_INLINE_WRITE_UNLOCK_BH is not set > -CONFIG_INLINE_WRITE_UNLOCK_IRQ=y > +# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set > # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set > CONFIG_MUTEX_SPIN_ON_OWNER=y > CONFIG_FREEZER=y > @@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y > CONFIG_ARCH_DISCARD_MEMBLOCK=y > # CONFIG_MEMORY_HOTPLUG is not set > CONFIG_PAGEFLAGS_EXTENDED=y > -CONFIG_SPLIT_PTLOCK_CPUS=4 > +CONFIG_SPLIT_PTLOCK_CPUS=999999 > # CONFIG_COMPACTION is not set > CONFIG_MIGRATION=y > CONFIG_PHYS_ADDR_T_64BIT=y > @@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y > # CONFIG_DEBUG_KMEMLEAK is not set > # CONFIG_DEBUG_RT_MUTEXES is not set > # CONFIG_RT_MUTEX_TESTER is not set > -# CONFIG_DEBUG_SPINLOCK is not set > +CONFIG_DEBUG_SPINLOCK=y > # CONFIG_DEBUG_MUTEXES is not set > # CONFIG_DEBUG_LOCK_ALLOC is not set > # CONFIG_PROVE_LOCKING is not set > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer T: +44 (0)1223 225 900, http://www.citrix.com
>>> On 30.01.12 at 17:27, Ben Guthro <ben@guthro.net> wrote: > Any thoughts, or suggestions on how best to debug these deadlocks > would be greatly appreciated.For Dom0 deadlocks all I generally need are the ''e'' and ''0'' debugkeys (with, for the latter, stack printing depth being set sufficiently high). If that turns out insufficient, custom debugging changes to hypervisor or kernel are the usual next (and final) step. Jan
Yes, it helped a lot! This seems to have resolved the hang - thanks for pointing it my way! So - if I''m reading the git log correctly, I should just have to cherry-pick 7a7546b377bdaa25ac77f33d9433c59f259b9688 from mainline? It seems like this may be something to CC the -stable queue on, IMHO. On Mon, Jan 30, 2012 at 11:34 AM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:> On 30/01/12 16:27, Ben Guthro wrote: >> Konrad, et al. >> >> I''m having an issue with my guests that, when I am running wireless >> drivers in dom0, and the PV-on-HVM guest tries to access the network - >> I get a hung dom0. >> >> When I turn on >> Kernel Hacking-> >> Spinlock and rw-lock debugging: basic checks >> >> The hang goes away, but does not print out anything terribly useful in >> the serial log. >> >> >> The config file changes made by this menuconfig setting can be found below: >> >> >> While I know from other conversations, that Konrad is aware of my >> setup, for other''s benefit - this is a 64bit dom0 3.2.2 kernel, with a >> few of his branches pulled in. >> >> Any thoughts, or suggestions on how best to debug these deadlocks >> would be greatly appreciated. > > It looks like > http://lists.xen.org/archives/html/xen-devel/2012-01/msg01917.html might > apply to you. > > Does that patch help? > > ~Andrew > >> >> -Ben Guthro >> >> >> diff --git a/configs/config.ubuntu.amd64 b/configs/config.ubuntu.amd64 >> index 205b5c6..f7a233f 100644 >> --- a/configs/config.ubuntu.amd64 >> +++ b/configs/config.ubuntu.amd64 >> @@ -257,27 +257,27 @@ CONFIG_PADATA=y >> # CONFIG_INLINE_SPIN_LOCK_BH is not set >> # CONFIG_INLINE_SPIN_LOCK_IRQ is not set >> # CONFIG_INLINE_SPIN_LOCK_IRQSAVE is not set >> -CONFIG_INLINE_SPIN_UNLOCK=y >> +# CONFIG_INLINE_SPIN_UNLOCK is not set >> # CONFIG_INLINE_SPIN_UNLOCK_BH is not set >> -CONFIG_INLINE_SPIN_UNLOCK_IRQ=y >> +# CONFIG_INLINE_SPIN_UNLOCK_IRQ is not set >> # CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE is not set >> # CONFIG_INLINE_READ_TRYLOCK is not set >> # CONFIG_INLINE_READ_LOCK is not set >> # CONFIG_INLINE_READ_LOCK_BH is not set >> # CONFIG_INLINE_READ_LOCK_IRQ is not set >> # CONFIG_INLINE_READ_LOCK_IRQSAVE is not set >> -CONFIG_INLINE_READ_UNLOCK=y >> +# CONFIG_INLINE_READ_UNLOCK is not set >> # CONFIG_INLINE_READ_UNLOCK_BH is not set >> -CONFIG_INLINE_READ_UNLOCK_IRQ=y >> +# CONFIG_INLINE_READ_UNLOCK_IRQ is not set >> # CONFIG_INLINE_READ_UNLOCK_IRQRESTORE is not set >> # CONFIG_INLINE_WRITE_TRYLOCK is not set >> # CONFIG_INLINE_WRITE_LOCK is not set >> # CONFIG_INLINE_WRITE_LOCK_BH is not set >> # CONFIG_INLINE_WRITE_LOCK_IRQ is not set >> # CONFIG_INLINE_WRITE_LOCK_IRQSAVE is not set >> -CONFIG_INLINE_WRITE_UNLOCK=y >> +# CONFIG_INLINE_WRITE_UNLOCK is not set >> # CONFIG_INLINE_WRITE_UNLOCK_BH is not set >> -CONFIG_INLINE_WRITE_UNLOCK_IRQ=y >> +# CONFIG_INLINE_WRITE_UNLOCK_IRQ is not set >> # CONFIG_INLINE_WRITE_UNLOCK_IRQRESTORE is not set >> CONFIG_MUTEX_SPIN_ON_OWNER=y >> CONFIG_FREEZER=y >> @@ -398,7 +398,7 @@ CONFIG_HAVE_MEMBLOCK_NODE_MAP=y >> CONFIG_ARCH_DISCARD_MEMBLOCK=y >> # CONFIG_MEMORY_HOTPLUG is not set >> CONFIG_PAGEFLAGS_EXTENDED=y >> -CONFIG_SPLIT_PTLOCK_CPUS=4 >> +CONFIG_SPLIT_PTLOCK_CPUS=999999 >> # CONFIG_COMPACTION is not set >> CONFIG_MIGRATION=y >> CONFIG_PHYS_ADDR_T_64BIT=y >> @@ -4255,7 +4255,7 @@ CONFIG_TIMER_STATS=y >> # CONFIG_DEBUG_KMEMLEAK is not set >> # CONFIG_DEBUG_RT_MUTEXES is not set >> # CONFIG_RT_MUTEX_TESTER is not set >> -# CONFIG_DEBUG_SPINLOCK is not set >> +CONFIG_DEBUG_SPINLOCK=y >> # CONFIG_DEBUG_MUTEXES is not set >> # CONFIG_DEBUG_LOCK_ALLOC is not set >> # CONFIG_PROVE_LOCKING is not set >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > -- > Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer > T: +44 (0)1223 225 900, http://www.citrix.com >
On 30/01/12 17:32, Ben Guthro wrote:> Yes, it helped a lot! > This seems to have resolved the hang - thanks for pointing it my way! > > So - if I''m reading the git log correctly, I should just have to > cherry-pick 7a7546b377bdaa25ac77f33d9433c59f259b9688 from mainline?Yes.> It seems like this may be something to CC the -stable queue on, IMHO.It''s already queued. David