thr3ads.net - Xen users - [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32 [Apr 2010]

If this information is useful, please help other people find it:
Share via:

Pim van Riezen

2010-Apr-07 23:34 UTC

[Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Good day,

We''re trying to get 2.6.31 and 2.6.32 rolled out on our clusters to
offer newer features like FUSE fo our customers, but we''re ran into a
couple of showstopper issues when deploying these kernels on busier guests,
showing a lot of errors like this:

  BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
  Modules linked in:
  CPU 0:
  Modules linked in:
  Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2 
  RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>]
hypercall_page+0x3aa/0x1001
  RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
  RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
  RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
  RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
  R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
  R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
  FS:  00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000) knlGS:0000000000000000
  CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
  CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  Call Trace:
   [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
   [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
   [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
   [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f

in our hope to get rid of this issue we upgraded from Xen 3.3 to Xen 3.4.1.7 out
of the gitco repos. The issue persisted. Is there a magic version of Xen,
preferably one that can be found in an rpm repository for CentOS 5, that *does*
properly support pvops kernels without these issues?

I''m also seing this one:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=543 where the bug is
still open but no activity since 2008. I don''t know if that bugzilla is
still being actively maintained?

Cheers,
Pim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-08 14:28 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Thu, Apr 08, 2010 at 01:34:24AM +0200, Pim van Riezen
wrote:> Good day,
> 
> We''re trying to get 2.6.31 and 2.6.32 rolled out on our clusters
to offer newer features like FUSE fo our customers, but we''re ran into
a couple of showstopper issues when deploying these kernels on busier guests,
showing a lot of errors like this:
> 
Which exact kernel versions are you using? From which kernel tree/branch? 

>   BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
>   Modules linked in:
>   CPU 0:
>   Modules linked in:
>   Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2 
>   RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>]
hypercall_page+0x3aa/0x1001
>   RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>   RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>   RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>   RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>   R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>   R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>   FS:  00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000)
knlGS:0000000000000000
>   CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>   CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
>   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>   DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>   Call Trace:
>    [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>    [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>    [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>    [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
> 
> in our hope to get rid of this issue we upgraded from Xen 3.3 to Xen
3.4.1.7 out of the gitco repos. The issue persisted. Is there a magic version of
Xen, preferably one that can be found in an rpm repository for CentOS 5, that
*does* properly support pvops kernels without these issues?
> 
I''ve used Xen 3.4.2 and Xen 4.0.0 with pvops dom0 kernels. 
pvops dom0 2.6.32 kernels _require_ Xen 4.0.0 or Xen 3.4.3.

> I''m also seing this one:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=543 where the bug is
still open but no activity since 2008. I don''t know if that bugzilla is
still being actively maintained?
> 
xen bugzilla is not actively used/maintained.
Usually the bugs are discussed/fixed on xen-devel mailinglist.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pim van Riezen

2010-Apr-08 22:38 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Apr 8, 2010, at 16:28 , Pasi Kärkkäinen wrote:
> On Thu, Apr 08, 2010 at 01:34:24AM +0200, Pim van Riezen wrote:
>> We''re trying to get 2.6.31 and 2.6.32 rolled out on our
clusters to offer newer features like FUSE fo our customers, but we''re
ran into a couple of showstopper issues when deploying these kernels on busier
guests, showing a lot of errors like this:
>> 
> 
> Which exact kernel versions are you using? From which kernel tree/branch? 
The last we tried was 2.6.32.9 from kernel.org. Had the same results with
kernel.org 2.6.31.5.
>> in our hope to get rid of this issue we upgraded from Xen 3.3 to Xen
3.4.1.7 out of the gitco repos. The issue persisted. Is there a magic version of
Xen, preferably one that can be found in an rpm repository for CentOS 5, that
*does* properly support pvops kernels without these issues?
> 
> I''ve used Xen 3.4.2 and Xen 4.0.0 with pvops dom0 kernels. 
> pvops dom0 2.6.32 kernels _require_ Xen 4.0.0 or Xen 3.4.3.
That''s what I understood, too. But then I ran into this:

  [root@sausage ~]# xm dmesg | grep Xen- | head -1
  (XEN) ----[ Xen-3.3.1  x86_64  debug=n  Not tainted ]----
  [root@sausage ~]# xm create -c vps3510
  Using config file "/etc/xen/vps3510".
  Started domain vps3510
  Linux version 2.6.32.9xls-domU (root@pablo) (gcc version 4.3.3 (Ubuntu
4.3.3-5ubuntu4) ) #2 SMP Mon Apr 5 21:43:30 CEST 2010
  (...)
  CentOS release 5.4 (Final)
  Kernel 2.6.32.9xls-domU on an x86_64
  
  vps3510.xlshosting.net login: 

Am I going mad? This is a vanilla kernel.

Are there any reliable RPM sources for 3.4.2?
> xen bugzilla is not actively used/maintained.
> Usually the bugs are discussed/fixed on xen-devel mailinglist.
Clear.

Cheers,
Pi



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-09 08:13 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Fri, Apr 09, 2010 at 12:38:19AM +0200, Pim van Riezen
wrote:>    On Apr 8, 2010, at 16:28 , Pasi Kärkkäinen wrote:
> 
>      On Thu, Apr 08, 2010 at 01:34:24AM +0200, Pim van Riezen wrote:
> 
>        We''re trying to get 2.6.31 and 2.6.32 rolled out on our
clusters to
>        offer newer features like FUSE fo our customers, but we''re
ran into a
>        couple of showstopper issues when deploying these kernels on busier
>        guests, showing a lot of errors like this:
> 
>      Which exact kernel versions are you using? From which kernel
>      tree/branch?
> 
>    The last we tried was 2.6.32.9 from [1]kernel.org. Had the same results
>    with [2]kernel.org 2.6.31.5.
>
OK. Are your guests 32bit or 64bit? 
 
I think 2.6.32.10 has the CONFIG_HIGHPTE fix.. so earlier versions will crash
if CONFIG_HIGHPTE is enabled for 32bit kernels.
>        in our hope to get rid of this issue we upgraded from Xen 3.3 to Xen
>        3.4.1.7 out of the gitco repos. The issue persisted. Is there a
magic
>        version of Xen, preferably one that can be found in an rpm
repository
>        for CentOS 5, that *does* properly support pvops kernels without
these
>        issues?
> 
>      I''ve used Xen 3.4.2 and Xen 4.0.0 with pvops dom0 kernels.
>      pvops dom0 2.6.32 kernels _require_ Xen 4.0.0 or Xen 3.4.3.
> 
>    That''s what I understood, too. But then I ran into this:
>      [root@sausage ~]# xm dmesg | grep Xen- | head -1
>      (XEN) ----[ Xen-3.3.1  x86_64  debug=n  Not tainted ]----
>      [root@sausage ~]# xm create -c vps3510
>      Using config file "/etc/xen/vps3510".
>      Started domain vps3510
>      Linux version 2.6.32.9xls-domU (root@pablo) (gcc version 4.3.3 (Ubuntu
>    4.3.3-5ubuntu4) ) #2 SMP Mon Apr 5 21:43:30 CEST 2010
>      (...)
>      CentOS release 5.4 (Final)
>      Kernel 2.6.32.9xls-domU on an x86_64
> 
>      [3]vps3510.xlshosting.net login:
>    Am I going mad? This is a vanilla kernel.
No, you''re not mad.
pvops *dom0* 2.6.32 kernels require Xen 3.4.3 or Xen 4.0.0.

Vanilla pvops domU kernels from kernel.org don''t require any specific
new Xen version.

>    Are there any reliable RPM sources for 3.4.2?
> 
Gitco has Xen 3.4.2 rpms.

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pim van Riezen

2010-Apr-09 09:13 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Sorry,

Last reply went all kinds of places but where they should go. Attempt #2:

On Apr 9, 2010, at 10:13 , Pasi Kärkkäinen wrote:
> On Fri, Apr 09, 2010 at 12:38:19AM +0200, Pim van Riezen wrote:
>>   On Apr 8, 2010, at 16:28 , Pasi Kärkkäinen wrote:
>> 
>>     Which exact kernel versions are you using? From which kernel
>>     tree/branch?
>> 
>>   The last we tried was 2.6.32.9 from [1]kernel.org. Had the same
results
>>   with [2]kernel.org 2.6.31.5.
>> 
> 
> OK. Are your guests 32bit or 64bit? 
64 bits.
> I think 2.6.32.10 has the CONFIG_HIGHPTE fix.. so earlier versions will
crash
> if CONFIG_HIGHPTE is enabled for 32bit kernels.
That shouldn''t be an issue then. Should I move the soft lockup issue to
xen-devel?
>>   Am I going mad? This is a vanilla kernel.
> 
> No, you''re not mad.
> pvops *dom0* 2.6.32 kernels require Xen 3.4.3 or Xen 4.0.0.
Ah, that ''splains :-)
> Gitco has Xen 3.4.2 rpms.
Must have missed those, will give it a try. Although I don''t expect
further updates to the hypervisor to solve this.

Cheers,
Pim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-09 14:21 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Fri, Apr 09, 2010 at 11:13:16AM +0200, Pim van Riezen
wrote:> Sorry,
> 
> Last reply went all kinds of places but where they should go. Attempt #2:
> 
> On Apr 9, 2010, at 10:13 , Pasi Kärkkäinen wrote:
> 
> > On Fri, Apr 09, 2010 at 12:38:19AM +0200, Pim van Riezen wrote:
> >>   On Apr 8, 2010, at 16:28 , Pasi Kärkkäinen wrote:
> >> 
> >>     Which exact kernel versions are you using? From which kernel
> >>     tree/branch?
> >> 
> >>   The last we tried was 2.6.32.9 from [1]kernel.org. Had the same
results
> >>   with [2]kernel.org 2.6.31.5.
> >> 
> > 
> > OK. Are your guests 32bit or 64bit? 
> 
> 64 bits.
> 
Ok. Then it''s not that.
> > I think 2.6.32.10 has the CONFIG_HIGHPTE fix.. so earlier versions
will crash
> > if CONFIG_HIGHPTE is enabled for 32bit kernels.
> 
> That shouldn''t be an issue then. Should I move the soft lockup
issue to xen-devel?
> 
> >>   Am I going mad? This is a vanilla kernel.
> > 
> > No, you''re not mad.
> > pvops *dom0* 2.6.32 kernels require Xen 3.4.3 or Xen 4.0.0.
> 
> Ah, that ''splains :-)
> 
:)
> > Gitco has Xen 3.4.2 rpms.
> 
> Must have missed those, will give it a try. Although I don''t
expect further updates to the hypervisor to solve this.
> 
Can you please post the soft lockup messages again..

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pim van Riezen

2010-Apr-09 14:34 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Apr 9, 2010, at 16:21 , Pasi Kärkkäinen wrote:
> Can you please post the soft lockup messages again..
Here''s a metric truckload of them. The lockups don''t seem to
happen consistently in the same code path. It may also be worth mentioning that
it only seems to happen with guests that have more than 1 vcpu, but I''m
not sure of that distinction.

Apr  9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s!
[gzip:27658]
Apr  9 08:21:01 host kernel: Modules linked in:
Apr  9 08:21:01 host kernel: CPU 1:
Apr  9 08:21:01 host kernel: Modules linked in:
Apr  9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:21:01 host kernel: RIP: e033:[<0000000000401958>] 
[<0000000000401958>] 0x401958
Apr  9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38  EFLAGS: 00000297
Apr  9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002 RCX:
0000000000657ce8
Apr  9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002 RDI:
000000000000da48
Apr  9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034 R09:
000000000000da48
Apr  9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080 R12:
0000000000658c06
Apr  9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06 R15:
0000000000000080
Apr  9 08:21:01 host kernel: FS:  00007fa6088946e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 08:21:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000 CR4:
0000000000002660
Apr  9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:21:02 host kernel: Call Trace:
Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s!
[swapper:0]
Apr  9 08:29:39 host kernel: Modules linked in:
Apr  9 08:29:39 host kernel: CPU 1:
Apr  9 08:29:39 host kernel: Modules linked in:
Apr  9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 08:29:39 host kernel: FS:  00007fa913f42720(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 08:29:39 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000 CR4:
0000000000002660
Apr  9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:29:39 host kernel: Call Trace:
Apr  9 08:29:39 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 08:29:39 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 08:29:39 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s!
[swapper:0]
Apr  9 08:29:39 host kernel: Modules linked in:
Apr  9 08:29:39 host kernel: CPU 0:
Apr  9 08:29:40 host kernel: Modules linked in:
Apr  9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 08:29:40 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 08:29:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000 CR4:
0000000000002660
Apr  9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:29:40 host kernel: Call Trace:
Apr  9 08:29:40 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 08:29:40 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 08:29:40 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 08:29:40 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s! [ssh:2324]
Apr  9 08:34:34 host kernel: Modules linked in:
Apr  9 08:34:34 host kernel: CPU 1:
Apr  9 08:34:34 host kernel: Modules linked in:
Apr  9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>] 
[<00007f1b0fcac76a>] 0x7f1b0fcac76a
Apr  9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480  EFLAGS: 00010202
Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000000000181
Apr  9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013 RDI:
000000000000000a
Apr  9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004 R09:
0101010101010101
Apr  9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690 R12:
0000000000000010
Apr  9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001 R15:
0000000000000004
Apr  9 08:34:34 host kernel: FS:  00007f1b10528710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000 CR4:
0000000000002660
Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:34:34 host kernel: Call Trace:
Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s!
[swapper:0]
Apr  9 08:34:34 host kernel: Modules linked in:
Apr  9 08:34:34 host kernel: CPU 0:
Apr  9 08:34:34 host kernel: Modules linked in:
Apr  9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 08:34:34 host kernel: FS:  00007fa913f42720(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000 CR4:
0000000000002660
Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:34:34 host kernel: Call Trace:
Apr  9 08:34:34 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 08:34:34 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 08:34:34 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 08:34:34 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s!
[gzip:30581]
Apr  9 08:39:49 host kernel: Modules linked in:
Apr  9 08:39:49 host kernel: CPU 1:
Apr  9 08:39:49 host kernel: Modules linked in:
Apr  9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>] 
[<ffffffff811ede38>] fuse_request_send+0x60/0x22b
Apr  9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28  EFLAGS: 00000202
Apr  9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0 RCX:
fffffffffffdfb5e
Apr  9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d RDI:
ffff88007eb1f710
Apr  9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40 R09:
ffff8800593170d8
Apr  9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f R12:
ffff88007eb1f400
Apr  9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000 R15:
0000000000001000
Apr  9 08:39:49 host kernel: FS:  00007f7b12f606e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000 CR4:
0000000000002660
Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:39:49 host kernel: Call Trace:
Apr  9 08:39:49 host kernel:  [<ffffffff811ede34>] ?
fuse_request_send+0x5c/0x22b
Apr  9 08:39:49 host kernel:  [<ffffffff811f1655>] ?
fuse_wait_on_page_writeback+0x31/0x168
Apr  9 08:39:49 host kernel:  [<ffffffff810904b4>] ?
add_to_page_cache_lru+0x41/0x58
Apr  9 08:39:49 host kernel:  [<ffffffff811f186a>] ?
fuse_send_write+0xb8/0xc2
Apr  9 08:39:49 host kernel:  [<ffffffff811f29d7>] ?
fuse_file_aio_write+0x497/0x62f
Apr  9 08:39:49 host kernel:  [<ffffffff810bab7e>] ?
do_sync_write+0xce/0x113
Apr  9 08:39:49 host kernel:  [<ffffffff8100f6af>] ?
__switch_to+0x1ad/0x297
Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
Apr  9 08:39:49 host kernel:  [<ffffffff810bb4b2>] ? vfs_write+0xad/0x149
Apr  9 08:39:49 host kernel:  [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s!
[sshfs:2332]
Apr  9 08:39:49 host kernel: Modules linked in:
Apr  9 08:39:49 host kernel: CPU 0:
Apr  9 08:39:49 host kernel: Modules linked in:
Apr  9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted 2.6.32.9xls-domU
#2
Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>] 
[<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
Apr  9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0  EFLAGS: 00000206
Apr  9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080 RCX:
0000000101b312ce
Apr  9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0 RDI:
ffff88007d1dc1c0
Apr  9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f R09:
0000000101b312ce
Apr  9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a R12:
ffff880068b8f1c0
Apr  9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020 R15:
ffff880068b8f1f8
Apr  9 08:39:49 host kernel: FS:  00007f1b10528710(0063)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000 CR4:
0000000000002660
Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 08:39:49 host kernel: Call Trace:
Apr  9 08:39:49 host kernel:  <IRQ>  [<ffffffff81419140>] ?
tcp_rcv_established+0x587/0x87b
Apr  9 08:39:49 host kernel:  [<ffffffff8141f7f1>] ?
tcp_v4_do_rcv+0x2c/0x1c9
Apr  9 08:39:49 host kernel:  [<ffffffff81046156>] ?
local_bh_enable+0x22/0x8f
Apr  9 08:39:49 host kernel:  [<ffffffff8141fe05>] ?
tcp_v4_rcv+0x477/0x6f8
Apr  9 08:39:49 host kernel:  [<ffffffff814064f7>] ?
ip_local_deliver_finish+0xf2/0x1c7
Apr  9 08:39:49 host kernel:  [<ffffffff81406126>] ?
ip_rcv_finish+0x326/0x340
Apr  9 08:39:49 host kernel:  [<ffffffff814063c0>] ? ip_rcv+0x280/0x2c5
Apr  9 08:39:49 host kernel:  [<ffffffff8135d39d>] ?
xennet_poll+0xb06/0xc68
Apr  9 08:39:49 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
Apr  9 08:39:49 host kernel:  [<ffffffff81334afc>] ?
do_blkif_request+0x341/0x350
Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff813c469b>] ?
net_rx_action+0x77/0x179
Apr  9 08:39:49 host kernel:  [<ffffffff810462b4>] ?
__do_softirq+0xdd/0x1a1
Apr  9 08:39:49 host kernel:  [<ffffffff81011bec>] ?
call_softirq+0x1c/0x30
Apr  9 08:39:49 host kernel:  [<ffffffff81012edf>] ? do_softirq+0x3f/0x7c
Apr  9 08:39:49 host kernel:  [<ffffffff812ff975>] ?
xen_evtchn_do_upcall+0x173/0x190
Apr  9 08:39:49 host kernel:  [<ffffffff81011c3e>] ?
xen_do_hypervisor_callback+0x1e/0x30
Apr  9 08:39:49 host kernel:  <EOI>  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
Apr  9 08:39:49 host kernel:  [<ffffffff8103d336>] ?
default_wake_function+0x0/0x9
Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
Apr  9 08:39:49 host kernel:  [<ffffffff814d6e59>] ?
schedule_timeout+0x23/0x1d9
Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ?
_spin_unlock_irqrestore+0xc/0xd
Apr  9 08:39:49 host kernel:  [<ffffffff81456ebe>] ?
unix_stream_recvmsg+0x2c4/0x644
Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
Apr  9 08:39:49 host kernel:  [<ffffffff81457231>] ?
unix_stream_recvmsg+0x637/0x644
Apr  9 08:39:49 host kernel:  [<ffffffff813b6303>] ?
sock_aio_read+0x103/0x117
Apr  9 08:39:49 host kernel:  [<ffffffff810bac91>] ?
do_sync_read+0xce/0x113
Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ?
xen_hypervisor_callback+0x1b/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ?
_spin_unlock_irqrestore+0xc/0xd
Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ?
xen_hypervisor_callback+0x1b/0x20
Apr  9 08:39:49 host kernel:  [<ffffffff810bb6f0>] ? vfs_read+0xbd/0x146
Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
Apr  9 08:39:49 host kernel:  [<ffffffff810bb835>] ? sys_read+0x45/0x6e
Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s!
[swapper:0]
Apr  9 09:11:36 host kernel: Modules linked in:
Apr  9 09:11:36 host kernel: CPU 1:
Apr  9 09:11:36 host kernel: Modules linked in:
Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 09:11:36 host kernel: FS:  00007f44bc51f710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 09:11:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:11:36 host kernel: Call Trace:
Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s!
[swapper:0]
Apr  9 09:11:36 host kernel: Modules linked in:
Apr  9 09:11:36 host kernel: CPU 0:
Apr  9 09:11:36 host kernel: Modules linked in:
Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 09:11:36 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 09:11:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:11:36 host kernel: Call Trace:
Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 09:11:36 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s! [ssh:2324]
Apr  9 09:20:24 host kernel: Modules linked in:
Apr  9 09:20:24 host kernel: CPU 1:
Apr  9 09:20:24 host kernel: Modules linked in:
Apr  9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>] 
[<00007f1b0e259f20>] 0x7f1b0e259f20
Apr  9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338  EFLAGS: 00000206
Apr  9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0 RCX:
00000000000000a0
Apr  9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004 RDI:
00007f1b0e539c30
Apr  9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001 R09:
0000000000000001
Apr  9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0 R12:
0000000000000000
Apr  9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111 R15:
0000000000000071
Apr  9 09:20:24 host kernel: FS:  00007f1b10528710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000 CR4:
0000000000002660
Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:20:24 host kernel: Call Trace:
Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s!
[swapper:0]
Apr  9 09:20:24 host kernel: Modules linked in:
Apr  9 09:20:24 host kernel: CPU 0:
Apr  9 09:20:24 host kernel: Modules linked in:
Apr  9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 09:20:24 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000 CR4:
0000000000002660
Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:20:24 host kernel: Call Trace:
Apr  9 09:20:24 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 09:20:24 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 09:20:24 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 09:20:24 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s!
[gzip:4496]
Apr  9 09:24:36 host kernel: Modules linked in:
Apr  9 09:24:36 host kernel: CPU 1:
Apr  9 09:24:36 host kernel: Modules linked in:
Apr  9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:24:36 host kernel: RIP: e033:[<0000000000409804>] 
[<0000000000409804>] 0x409804
Apr  9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8  EFLAGS: 00000202
Apr  9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000 RCX:
00007f529598c610
Apr  9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6 RDI:
00000000006534fb
Apr  9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000 R09:
7464697720223333
Apr  9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246 R12:
00000000006522a0
Apr  9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 09:24:36 host kernel: FS:  00007f5295e2a6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 09:24:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000 CR4:
0000000000002660
Apr  9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:24:37 host kernel: Call Trace:
Apr  9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s!
[swapper:0]
Apr  9 09:24:37 host kernel: Modules linked in:
Apr  9 09:24:37 host kernel: CPU 0:
Apr  9 09:24:37 host kernel: Modules linked in:
Apr  9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 09:24:38 host kernel: FS:  00007f2f6a31a710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 09:24:38 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000 CR4:
0000000000002660
Apr  9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 09:24:38 host kernel: Call Trace:
Apr  9 09:24:38 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 09:24:38 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 09:24:38 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 09:24:38 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s!
[swapper:0]
Apr  9 10:05:51 host kernel: Modules linked in:
Apr  9 10:05:51 host kernel: CPU 1:
Apr  9 10:05:51 host kernel: Modules linked in:
Apr  9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 10:05:51 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:05:51 host kernel: Call Trace:
Apr  9 10:05:51 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 10:05:51 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 10:05:51 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s!
[httpd:9922]
Apr  9 10:05:51 host kernel: Modules linked in:
Apr  9 10:05:51 host kernel: CPU 0:
Apr  9 10:05:51 host kernel: Modules linked in:
Apr  9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted 2.6.32.9xls-domU
#2
Apr  9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>] 
[<00007ff39a44a36d>] 0x7ff39a44a36d
Apr  9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118  EFLAGS: 00000a86
Apr  9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60 RCX:
c0c176e403831574
Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004 RDI:
000000000420e43b
Apr  9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004 R09:
00007fff4b9ad270
Apr  9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188 R12:
000000000320e6d0
Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368 R15:
0000000000000000
Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 10:05:51 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000 CR4:
0000000000002660
Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:05:51 host kernel: Call Trace:
Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s!
[swapper:0]
Apr  9 10:50:04 host kernel: Modules linked in:
Apr  9 10:50:04 host kernel: CPU 1:
Apr  9 10:50:04 host kernel: Modules linked in:
Apr  9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 10:50:04 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 10:50:04 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000 CR4:
0000000000002660
Apr  9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:50:04 host kernel: Call Trace:
Apr  9 10:50:04 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 10:50:04 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 10:50:04 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s!
[httpd:16797]
Apr  9 10:50:04 host kernel: Modules linked in:
Apr  9 10:50:04 host kernel: CPU 0:
Apr  9 10:50:04 host kernel: Modules linked in:
Apr  9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted
2.6.32.9xls-domU #2
Apr  9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>] 
[<00007ff39a49ae55>] 0x7ff39a49ae55
Apr  9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8  EFLAGS: 00000246
Apr  9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0 RCX:
222879726575713e
Apr  9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c RDI:
00007fff4b9b1d00
Apr  9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001 R09:
0101010101010101
Apr  9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be R12:
00007fff4b9b1d00
Apr  9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0 R15:
0000000003695df0
Apr  9 10:50:05 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 10:50:05 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000 CR4:
0000000000002660
Apr  9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:50:05 host kernel: Call Trace:
Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s!
[exim:17977]
Apr  9 10:57:01 host kernel: Modules linked in:
Apr  9 10:57:01 host kernel: CPU 1:
Apr  9 10:57:01 host kernel: Modules linked in:
Apr  9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted 2.6.32.9xls-domU
#2
Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
Apr  9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0  EFLAGS: 00000246
Apr  9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX:
ffffffff8100922a
Apr  9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000 RDI:
0000000000000000
Apr  9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710 R09:
0000000000000003
Apr  9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246 R12:
0000000000000004
Apr  9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0 R15:
ffff88005a51b3c0
Apr  9 10:57:01 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000 CR4:
0000000000002660
Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:57:01 host kernel: Call Trace:
Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 10:57:01 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
Apr  9 10:57:01 host kernel:  [<ffffffff814da477>] ?
do_page_fault+0xfb/0x28d
Apr  9 10:57:01 host kernel:  [<ffffffff814d85b5>] ? page_fault+0x25/0x30
Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s!
[netstat:17966]
Apr  9 10:57:01 host kernel: Modules linked in:
Apr  9 10:57:01 host kernel: CPU 0:
Apr  9 10:57:01 host kernel: Modules linked in:
Apr  9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted
2.6.32.9xls-domU #2
Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>] 
[<ffffffff814d8226>] _spin_lock+0xa/0x15
Apr  9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0  EFLAGS: 00000202
Apr  9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180 RCX:
0000000000000012
Apr  9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8 RDI:
ffff88007ff77188
Apr  9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000 R09:
0000000000300000
Apr  9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e R12:
ffff88000ff859c0
Apr  9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38 R15:
0000000000000002
Apr  9 10:57:01 host kernel: FS:  00007fb435545710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000 CR4:
0000000000002660
Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 10:57:01 host kernel: Call Trace:
Apr  9 10:57:01 host kernel:  [<ffffffff810cb01b>] ? __d_lookup+0x7a/0x105
Apr  9 10:57:01 host kernel:  [<ffffffff810c251e>] ? do_lookup+0x2a/0x1b3
Apr  9 10:57:01 host kernel:  [<ffffffff810c4498>] ?
__link_path_walk+0x348/0xdcd
Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 10:57:01 host kernel:  [<ffffffff810c514b>] ? path_walk+0x66/0xca
Apr  9 10:57:01 host kernel:  [<ffffffff810c51cf>] ?
do_path_lookup+0x20/0x77
Apr  9 10:57:01 host kernel:  [<ffffffff810c5ae9>] ?
user_path_at+0x48/0x79
Apr  9 10:57:01 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ?
mntput_no_expire+0x23/0xed
Apr  9 10:57:01 host kernel:  [<ffffffff8129b703>] ?
_atomic_dec_and_lock+0x33/0x50
Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ?
mntput_no_expire+0x23/0xed
Apr  9 10:57:01 host kernel:  [<ffffffff810bdf4d>] ?
sys_readlinkat+0x25/0x8d
Apr  9 10:57:01 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
Apr  9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s!
[swapper:0]
Apr  9 11:00:36 host kernel: Modules linked in:
Apr  9 11:00:36 host kernel: CPU 1:
Apr  9 11:00:36 host kernel: Modules linked in:
Apr  9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 11:00:36 host kernel: FS:  00007f719f5fa6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 11:00:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000 CR4:
0000000000002660
Apr  9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 11:00:37 host kernel: Call Trace:
Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s!
[swapper:0]
Apr  9 11:00:37 host kernel: Modules linked in:
Apr  9 11:00:37 host kernel: CPU 0:
Apr  9 11:00:37 host kernel: Modules linked in:
Apr  9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 11:00:37 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 11:00:37 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000 CR4:
0000000000002660
Apr  9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 11:00:37 host kernel: Call Trace:
Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 11:00:37 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s!
[httpd:19360]
Apr  9 11:08:50 host kernel: Modules linked in:
Apr  9 11:08:50 host kernel: CPU 0:
Apr  9 11:08:50 host kernel: Modules linked in:
Apr  9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted
2.6.32.9xls-domU #2
Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>] 
[<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
Apr  9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80  EFLAGS: 00000246
Apr  9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8 RCX:
ffff88005675dce4
Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057 RDI:
ffff88002678c900
Apr  9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4 R09:
75646f72705f6d76
Apr  9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683 R12:
0000000000000057
Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80 R15:
ffff88005675dd60
Apr  9 11:08:50 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 11:08:50 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000 CR4:
0000000000002660
Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 11:08:50 host kernel: Call Trace:
Apr  9 11:08:50 host kernel:  [<ffffffff814573c8>] ?
unix_stream_sendmsg+0x18a/0x31f
Apr  9 11:08:50 host kernel:  [<ffffffff813b61ec>] ?
sock_aio_write+0xf6/0x10a
Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 11:08:50 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
Apr  9 11:08:50 host kernel:  [<ffffffff810bab7e>] ?
do_sync_write+0xce/0x113
Apr  9 11:08:50 host kernel:  [<ffffffff810c5af3>] ?
user_path_at+0x52/0x79
Apr  9 11:08:50 host kernel:  [<ffffffff810be0b0>] ? cp_new_stat+0xe9/0xfc
Apr  9 11:08:50 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 11:08:50 host kernel:  [<ffffffff810bb4c5>] ? vfs_write+0xc0/0x149
Apr  9 11:08:50 host kernel:  [<ffffffff810bb60a>] ? sys_write+0x45/0x6e
Apr  9 11:08:50 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s!
[swapper:0]
Apr  9 11:08:50 host kernel: Modules linked in:
Apr  9 11:08:50 host kernel: CPU 1:
Apr  9 11:08:50 host kernel: Modules linked in:
Apr  9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 11:08:50 host kernel: FS:  00007f0e429cb6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 11:08:50 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 11:08:50 host kernel: Call Trace:
Apr  9 11:08:50 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 11:08:50 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 11:08:50 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s!
[httpd:26903]
Apr  9 12:00:40 host kernel: Modules linked in:
Apr  9 12:00:40 host kernel: CPU 1:
Apr  9 12:00:40 host kernel: Modules linked in:
Apr  9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted
2.6.32.9xls-domU #2
Apr  9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>] 
[<00007ff39a427cd2>] 0x7ff39a427cd2
Apr  9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20  EFLAGS: 00000202
Apr  9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30 RCX:
0000000000000013
Apr  9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50 RDI:
0000000003bd6418
Apr  9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60 R09:
00007ff39a783d80
Apr  9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12:
00007ff39aaaa3e0
Apr  9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50 R15:
000000000000015e
Apr  9 12:00:40 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000 CR4:
0000000000002660
Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:00:40 host kernel: Call Trace:
Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s!
[swapper:0]
Apr  9 12:00:40 host kernel: Modules linked in:
Apr  9 12:00:40 host kernel: CPU 0:
Apr  9 12:00:40 host kernel: Modules linked in:
Apr  9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 12:00:40 host kernel: FS:  00007f74fde746e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:00:40 host kernel: Call Trace:
Apr  9 12:00:40 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 12:00:40 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 12:00:40 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 12:00:40 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s!
[swapper:0]
Apr  9 12:19:45 host kernel: Modules linked in:
Apr  9 12:19:45 host kernel: CPU 1:
Apr  9 12:19:45 host kernel: Modules linked in:
Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 12:19:45 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000 CR4:
0000000000002660
Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:19:45 host kernel: Call Trace:
Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s!
[swapper:0]
Apr  9 12:19:45 host kernel: Modules linked in:
Apr  9 12:19:45 host kernel: CPU 0:
Apr  9 12:19:45 host kernel: Modules linked in:
Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 12:19:45 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:19:45 host kernel: Call Trace:
Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 12:19:45 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s!
[swapper:0]
Apr  9 12:37:26 host kernel: Modules linked in:
Apr  9 12:37:26 host kernel: CPU 1:
Apr  9 12:37:26 host kernel: Modules linked in:
Apr  9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 12:37:26 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:37:26 host kernel: Call Trace:
Apr  9 12:37:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 12:37:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 12:37:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s!
[httpd:31684]
Apr  9 12:37:26 host kernel: Modules linked in:
Apr  9 12:37:26 host kernel: CPU 0:
Apr  9 12:37:26 host kernel: Modules linked in:
Apr  9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted
2.6.32.9xls-domU #2
Apr  9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>] 
[<00007ff39a41f7d3>] 0x7ff39a41f7d3
Apr  9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330  EFLAGS: 00000246
Apr  9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0 RCX:
0000000000000001
Apr  9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e RDI:
0000000000000000
Apr  9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60 R09:
0000000000000003
Apr  9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2 R12:
0000000003a2b297
Apr  9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0 R15:
00007ff39a783e80
Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 12:37:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000 CR4:
0000000000002660
Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 12:37:26 host kernel: Call Trace:
Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s!
[swapper:0]
Apr  9 13:44:52 host kernel: Modules linked in:
Apr  9 13:44:52 host kernel: CPU 1:
Apr  9 13:44:52 host kernel: Modules linked in:
Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 13:44:52 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 13:44:52 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 13:44:52 host kernel: Call Trace:
Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s!
[swapper:0]
Apr  9 13:44:52 host kernel: Modules linked in:
Apr  9 13:44:52 host kernel: CPU 0:
Apr  9 13:44:52 host kernel: Modules linked in:
Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 13:44:52 host kernel: FS:  00007f0e429cb6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 13:44:52 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000 CR4:
0000000000002660
Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 13:44:52 host kernel: Call Trace:
Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 13:44:52 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s!
[swapper:0]
Apr  9 13:55:43 host kernel: Modules linked in:
Apr  9 13:55:43 host kernel: CPU 1:
Apr  9 13:55:43 host kernel: Modules linked in:
Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 13:55:43 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000 CR4:
0000000000002660
Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 13:55:43 host kernel: Call Trace:
Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s!
[swapper:0]
Apr  9 13:55:43 host kernel: Modules linked in:
Apr  9 13:55:43 host kernel: CPU 0:
Apr  9 13:55:43 host kernel: Modules linked in:
Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 13:55:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 13:55:43 host kernel: Call Trace:
Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 13:55:43 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s!
[swapper:0]
Apr  9 14:16:21 host kernel: Modules linked in:
Apr  9 14:16:21 host kernel: CPU 1:
Apr  9 14:16:21 host kernel: Modules linked in:
Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
Apr  9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8  EFLAGS: 00000246
Apr  9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810 RCX:
ffffffff8100922a
Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
Apr  9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000 R09:
ffffffff8100e3b2
Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000 R15:
0000000000001000
Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 14:16:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 14:16:21 host kernel: Call Trace:
Apr  9 14:16:21 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 14:16:21 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
Apr  9 14:16:21 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
Apr  9 14:16:21 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
Apr  9 14:16:21 host kernel:  [<ffffffff814d87ba>] ? error_exit+0x2a/0x60
Apr  9 14:16:21 host kernel:  [<ffffffff8101149d>] ?
retint_restore_args+0x5/0x6
Apr  9 14:16:21 host kernel:  [<ffffffff8100fe5f>] ? cpu_idle+0x80/0x82
Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s!
[swapper:0]
Apr  9 14:16:21 host kernel: Modules linked in:
Apr  9 14:16:21 host kernel: CPU 0:
Apr  9 14:16:21 host kernel: Modules linked in:
Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 14:16:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000 CR4:
0000000000002660
Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 14:16:21 host kernel: Call Trace:
Apr  9 14:16:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 14:16:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 14:16:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 14:16:21 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s!
[swapper:0]
Apr  9 15:40:21 host kernel: Modules linked in:
Apr  9 15:40:21 host kernel: CPU 1:
Apr  9 15:40:21 host kernel: Modules linked in:
Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 15:40:21 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 15:40:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 15:40:21 host kernel: Call Trace:
Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s!
[swapper:0]
Apr  9 15:40:21 host kernel: Modules linked in:
Apr  9 15:40:21 host kernel: CPU 0:
Apr  9 15:40:21 host kernel: Modules linked in:
Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 15:40:21 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 15:40:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 15:40:21 host kernel: Call Trace:
Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 15:40:21 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s!
[swapper:0]
Apr  9 16:06:21 host kernel: Modules linked in:
Apr  9 16:06:21 host kernel: CPU 1:
Apr  9 16:06:21 host kernel: Modules linked in:
Apr  9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 16:06:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 16:06:25 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:06:26 host kernel: Call Trace:
Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s!
[swapper:0]
Apr  9 16:06:26 host kernel: Modules linked in:
Apr  9 16:06:26 host kernel: CPU 0:
Apr  9 16:06:26 host kernel: Modules linked in:
Apr  9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 16:06:26 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 16:06:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000 CR4:
0000000000002660
Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:06:26 host kernel: Call Trace:
Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:06:26 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s!
[httpd:32342]
Apr  9 16:17:43 host kernel: Modules linked in:
Apr  9 16:17:43 host kernel: CPU 1:
Apr  9 16:17:43 host kernel: Modules linked in:
Apr  9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted
2.6.32.9xls-domU #2
Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
Apr  9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0  EFLAGS: 00000246
Apr  9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000 RCX:
ffffffff8100922a
Apr  9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000 RDI:
0000000000000000
Apr  9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240 R09:
000000310f6de059
Apr  9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000006
Apr  9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0 R15:
ffff88007d093b40
Apr  9 16:17:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 16:17:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000 CR4:
0000000000002660
Apr  9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:17:43 host kernel: Call Trace:
Apr  9 16:17:43 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
Apr  9 16:17:43 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
Apr  9 16:17:43 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
Apr  9 16:17:43 host kernel:  [<ffffffff814da477>] ?
do_page_fault+0xfb/0x28d
Apr  9 16:17:43 host kernel:  [<ffffffff814d85b5>] ? page_fault+0x25/0x30
Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s!
[swapper:0]
Apr  9 16:17:43 host kernel: Modules linked in:
Apr  9 16:17:43 host kernel: CPU 0:
Apr  9 16:17:43 host kernel: Modules linked in:
Apr  9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 16:17:44 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 16:17:44 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000 CR4:
0000000000002660
Apr  9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:17:44 host kernel: Call Trace:
Apr  9 16:17:44 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:17:44 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:17:44 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:17:44 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s!
[swapper:0]
Apr  9 16:20:22 host kernel: Modules linked in:
Apr  9 16:20:22 host kernel: CPU 1:
Apr  9 16:20:22 host kernel: Modules linked in:
Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 16:20:22 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 16:20:22 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:20:22 host kernel: Call Trace:
Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s!
[swapper:0]
Apr  9 16:20:22 host kernel: Modules linked in:
Apr  9 16:20:22 host kernel: CPU 0:
Apr  9 16:20:22 host kernel: Modules linked in:
Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000 RCX:
ffffffff810093aa
Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
ffffffffffffffff
Apr  9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70 R15:
0000000000000000
Apr  9 16:20:22 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 16:20:22 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000 CR4:
0000000000002660
Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:20:22 host kernel: Call Trace:
Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:20:22 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s!
[swapper:0]
Apr  9 16:22:29 host kernel: Modules linked in:
Apr  9 16:22:29 host kernel: CPU 1:
Apr  9 16:22:29 host kernel: Modules linked in:
Apr  9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU
#2
Apr  9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
Apr  9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
Apr  9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000 RCX:
ffffffff810093aa
Apr  9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000001
Apr  9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000 R09:
ffffffff8100e3b2
Apr  9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246 R12:
0000000000000000
Apr  9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
Apr  9 16:22:29 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
Apr  9 16:22:29 host kernel: CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
Apr  9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000 CR4:
0000000000002660
Apr  9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:22:29 host kernel: Call Trace:
Apr  9 16:22:29 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
Apr  9 16:22:29 host kernel:  [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
Apr  9 16:22:29 host kernel:  [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s!
[syslogd:1848]
Apr  9 16:22:29 host kernel: Modules linked in:
Apr  9 16:22:29 host kernel: CPU 0:
Apr  9 16:22:29 host kernel: Modules linked in:
Apr  9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted
2.6.32.9xls-domU #2
Apr  9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>] 
[<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
Apr  9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0  EFLAGS: 00000246
Apr  9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000 RCX:
ffffffff810092eb
Apr  9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0 RDI:
0000000000000000
Apr  9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000 R09:
0000000000000000
Apr  9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246 R12:
00007fff22fa01b0
Apr  9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120 R15:
00007f74fde74690
Apr  9 16:22:30 host kernel: FS:  00007f74fde746e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
Apr  9 16:22:30 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000 CR4:
0000000000002660
Apr  9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
Apr  9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
Apr  9 16:22:30 host kernel: Call Trace:


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-09 16:48 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

On Fri, Apr 09, 2010 at 04:34:08PM +0200, Pim van Riezen
wrote:> 
> On Apr 9, 2010, at 16:21 , Pasi Kärkkäinen wrote:
> 
> > Can you please post the soft lockup messages again..
> 
> Here''s a metric truckload of them. The lockups don''t seem
to happen consistently in the same code path. It may also be worth mentioning
that it only seems to happen with guests that have more than 1 vcpu, but
I''m not sure of that distinction.
>
I guess you already posted these to xen-devel ? 

-- Pasi

 > Apr  9 08:21:01 host kernel: BUG: soft lockup - CPU#1 stuck for 628s!
[gzip:27658]
> Apr  9 08:21:01 host kernel: Modules linked in:
> Apr  9 08:21:01 host kernel: CPU 1:
> Apr  9 08:21:01 host kernel: Modules linked in:
> Apr  9 08:21:01 host kernel: Pid: 27658, comm: gzip Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:21:01 host kernel: RIP: e033:[<0000000000401958>] 
[<0000000000401958>] 0x401958
> Apr  9 08:21:01 host kernel: RSP: e02b:00007fff026d7e38  EFLAGS: 00000297
> Apr  9 08:21:01 host kernel: RAX: 0000000000000020 RBX: 0000000000000002
RCX: 0000000000657ce8
> Apr  9 08:21:01 host kernel: RDX: 0000000000000048 RSI: 0000000000000002
RDI: 000000000000da48
> Apr  9 08:21:01 host kernel: RBP: 0000000000006a6c R08: 0000000000000034
R09: 000000000000da48
> Apr  9 08:21:01 host kernel: R10: 0000000000000048 R11: 0000000000000080
R12: 0000000000658c06
> Apr  9 08:21:01 host kernel: R13: 0000000000658d08 R14: 0000000000658c06
R15: 0000000000000080
> Apr  9 08:21:01 host kernel: FS:  00007fa6088946e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:21:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:21:01 host kernel: CR2: 00007f7addede020 CR3: 00000000267db000
CR4: 0000000000002660
> Apr  9 08:21:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:21:02 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:21:02 host kernel: Call Trace:
> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#1 stuck for 483s!
[swapper:0]
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: CPU 1:
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:29:39 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:29:39 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 08:29:39 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 08:29:39 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 08:29:39 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 08:29:39 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 08:29:39 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 08:29:39 host kernel: FS:  00007fa913f42720(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:29:39 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 08:29:39 host kernel: CR2: 00007ff39862b1f0 CR3: 0000000052c5e000
CR4: 0000000000002660
> Apr  9 08:29:39 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:29:39 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:29:39 host kernel: Call Trace:
> Apr  9 08:29:39 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 08:29:39 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 08:29:39 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 08:29:39 host kernel: BUG: soft lockup - CPU#0 stuck for 483s!
[swapper:0]
> Apr  9 08:29:39 host kernel: Modules linked in:
> Apr  9 08:29:39 host kernel: CPU 0:
> Apr  9 08:29:40 host kernel: Modules linked in:
> Apr  9 08:29:40 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:29:40 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:29:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 08:29:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 08:29:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 08:29:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 08:29:40 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 08:29:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 08:29:40 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:29:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:29:40 host kernel: CR2: 00007ff39e4a8000 CR3: 000000007ea99000
CR4: 0000000000002660
> Apr  9 08:29:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:29:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:29:40 host kernel: Call Trace:
> Apr  9 08:29:40 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 08:29:40 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 08:29:40 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 08:29:40 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#1 stuck for 275s!
[ssh:2324]
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: CPU 1:
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: Pid: 2324, comm: ssh Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:34:34 host kernel: RIP: e033:[<00007f1b0fcac76a>] 
[<00007f1b0fcac76a>] 0x7f1b0fcac76a
> Apr  9 08:34:34 host kernel: RSP: e02b:00007fffa45a7480  EFLAGS: 00010202
> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: 0000000000000000
RCX: 0000000000000181
> Apr  9 08:34:34 host kernel: RDX: 00007f1b0fcdca28 RSI: 0000000000000013
RDI: 000000000000000a
> Apr  9 08:34:34 host kernel: RBP: 0000000000000010 R08: 0000000000000004
R09: 0101010101010101
> Apr  9 08:34:34 host kernel: R10: 737361702e707466 R11: 00007f1b0fcac690
R12: 0000000000000010
> Apr  9 08:34:34 host kernel: R13: 00007f1b11f825e0 R14: 0000000000000001
R15: 0000000000000004
> Apr  9 08:34:34 host kernel: FS:  00007f1b10528710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006ad44000
CR4: 0000000000002660
> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:34:34 host kernel: Call Trace:
> Apr  9 08:34:34 host kernel: BUG: soft lockup - CPU#0 stuck for 275s!
[swapper:0]
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: CPU 0:
> Apr  9 08:34:34 host kernel: Modules linked in:
> Apr  9 08:34:34 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:34:34 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 08:34:34 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 08:34:34 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 08:34:34 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 08:34:34 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 08:34:34 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 08:34:34 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 08:34:34 host kernel: FS:  00007fa913f42720(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:34:34 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:34:34 host kernel: CR2: 00007fa913f54000 CR3: 000000006acb1000
CR4: 0000000000002660
> Apr  9 08:34:34 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:34:34 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:34:34 host kernel: Call Trace:
> Apr  9 08:34:34 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 08:34:34 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 08:34:34 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 08:34:34 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#1 stuck for 293s!
[gzip:30581]
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: CPU 1:
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: Pid: 30581, comm: gzip Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff811ede38>] 
[<ffffffff811ede38>] fuse_request_send+0x60/0x22b
> Apr  9 08:39:49 host kernel: RSP: e02b:ffff88007952bc28  EFLAGS: 00000202
> Apr  9 08:39:49 host kernel: RAX: ffff880079652df0 RBX: ffff88007e07d2c0
RCX: fffffffffffdfb5e
> Apr  9 08:39:49 host kernel: RDX: 0000000000020001 RSI: 000000000000001d
RDI: ffff88007eb1f710
> Apr  9 08:39:49 host kernel: RBP: ffff88007e07d2c0 R08: 000000037ede1c40
R09: ffff8800593170d8
> Apr  9 08:39:49 host kernel: R10: 0000000000000000 R11: 000000000000000f
R12: ffff88007eb1f400
> Apr  9 08:39:49 host kernel: R13: ffff88007e07d2e0 R14: 0000000000001000
R15: 0000000000001000
> Apr  9 08:39:49 host kernel: FS:  00007f7b12f606e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:39:49 host kernel: CR2: 00000000006d9138 CR3: 00000000264f7000
CR4: 0000000000002660
> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:39:49 host kernel: Call Trace:
> Apr  9 08:39:49 host kernel:  [<ffffffff811ede34>] ?
fuse_request_send+0x5c/0x22b
> Apr  9 08:39:49 host kernel:  [<ffffffff811f1655>] ?
fuse_wait_on_page_writeback+0x31/0x168
> Apr  9 08:39:49 host kernel:  [<ffffffff810904b4>] ?
add_to_page_cache_lru+0x41/0x58
> Apr  9 08:39:49 host kernel:  [<ffffffff811f186a>] ?
fuse_send_write+0xb8/0xc2
> Apr  9 08:39:49 host kernel:  [<ffffffff811f29d7>] ?
fuse_file_aio_write+0x497/0x62f
> Apr  9 08:39:49 host kernel:  [<ffffffff810bab7e>] ?
do_sync_write+0xce/0x113
> Apr  9 08:39:49 host kernel:  [<ffffffff8100f6af>] ?
__switch_to+0x1ad/0x297
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb4b2>] ?
vfs_write+0xad/0x149
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb60a>] ?
sys_write+0x45/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
> Apr  9 08:39:49 host kernel: BUG: soft lockup - CPU#0 stuck for 293s!
[sshfs:2332]
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: CPU 0:
> Apr  9 08:39:49 host kernel: Modules linked in:
> Apr  9 08:39:49 host kernel: Pid: 2332, comm: sshfs Not tainted
2.6.32.9xls-domU #2
> Apr  9 08:39:49 host kernel: RIP: e030:[<ffffffff813b93bd>] 
[<ffffffff813b93bd>] sk_reset_timer+0x1/0x1d
> Apr  9 08:39:49 host kernel: RSP: e02b:ffff8800022afbc0  EFLAGS: 00000206
> Apr  9 08:39:49 host kernel: RAX: 0000000101b312c6 RBX: 0000000000000080
RCX: 0000000101b312ce
> Apr  9 08:39:49 host kernel: RDX: 0000000101b312ce RSI: ffff88007d1dc4d0
RDI: ffff88007d1dc1c0
> Apr  9 08:39:49 host kernel: RBP: ffff88007d1dc1c0 R08: 000000007d1dc10f
R09: 0000000101b312ce
> Apr  9 08:39:49 host kernel: R10: ffff880068b8f1c0 R11: ffffffff8126167a
R12: ffff880068b8f1c0
> Apr  9 08:39:49 host kernel: R13: 0000000000000000 R14: 0000000000000020
R15: ffff880068b8f1f8
> Apr  9 08:39:49 host kernel: FS:  00007f1b10528710(0063)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 08:39:49 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 08:39:49 host kernel: CR2: 00007f66a4ede000 CR3: 000000006acb1000
CR4: 0000000000002660
> Apr  9 08:39:49 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 08:39:49 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 08:39:49 host kernel: Call Trace:
> Apr  9 08:39:49 host kernel:  <IRQ>  [<ffffffff81419140>] ?
tcp_rcv_established+0x587/0x87b
> Apr  9 08:39:49 host kernel:  [<ffffffff8141f7f1>] ?
tcp_v4_do_rcv+0x2c/0x1c9
> Apr  9 08:39:49 host kernel:  [<ffffffff81046156>] ?
local_bh_enable+0x22/0x8f
> Apr  9 08:39:49 host kernel:  [<ffffffff8141fe05>] ?
tcp_v4_rcv+0x477/0x6f8
> Apr  9 08:39:49 host kernel:  [<ffffffff814064f7>] ?
ip_local_deliver_finish+0xf2/0x1c7
> Apr  9 08:39:49 host kernel:  [<ffffffff81406126>] ?
ip_rcv_finish+0x326/0x340
> Apr  9 08:39:49 host kernel:  [<ffffffff814063c0>] ?
ip_rcv+0x280/0x2c5
> Apr  9 08:39:49 host kernel:  [<ffffffff8135d39d>] ?
xennet_poll+0xb06/0xc68
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 08:39:49 host kernel:  [<ffffffff81334afc>] ?
do_blkif_request+0x341/0x350
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff813c469b>] ?
net_rx_action+0x77/0x179
> Apr  9 08:39:49 host kernel:  [<ffffffff810462b4>] ?
__do_softirq+0xdd/0x1a1
> Apr  9 08:39:49 host kernel:  [<ffffffff81011bec>] ?
call_softirq+0x1c/0x30
> Apr  9 08:39:49 host kernel:  [<ffffffff81012edf>] ?
do_softirq+0x3f/0x7c
> Apr  9 08:39:49 host kernel:  [<ffffffff812ff975>] ?
xen_evtchn_do_upcall+0x173/0x190
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c3e>] ?
xen_do_hypervisor_callback+0x1e/0x30
> Apr  9 08:39:49 host kernel:  <EOI>  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 08:39:49 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
> Apr  9 08:39:49 host kernel:  [<ffffffff8103d336>] ?
default_wake_function+0x0/0x9
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d6e59>] ?
schedule_timeout+0x23/0x1d9
> Apr  9 08:39:49 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 08:39:49 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ?
_spin_unlock_irqrestore+0xc/0xd
> Apr  9 08:39:49 host kernel:  [<ffffffff81456ebe>] ?
unix_stream_recvmsg+0x2c4/0x644
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff81457231>] ?
unix_stream_recvmsg+0x637/0x644
> Apr  9 08:39:49 host kernel:  [<ffffffff813b6303>] ?
sock_aio_read+0x103/0x117
> Apr  9 08:39:49 host kernel:  [<ffffffff810bac91>] ?
do_sync_read+0xce/0x113
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ?
xen_hypervisor_callback+0x1b/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
> Apr  9 08:39:49 host kernel:  [<ffffffff81261668>] ?
cap_socket_recvmsg+0x0/0x3
> Apr  9 08:39:49 host kernel:  [<ffffffff814d824c>] ?
_spin_unlock_irqrestore+0xc/0xd
> Apr  9 08:39:49 host kernel:  [<ffffffff81011c1b>] ?
xen_hypervisor_callback+0x1b/0x20
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb6f0>] ?
vfs_read+0xbd/0x146
> Apr  9 08:39:49 host kernel:  [<ffffffff8100922a>] ?
hypercall_page+0x22a/0x1001
> Apr  9 08:39:49 host kernel:  [<ffffffff810bb835>] ?
sys_read+0x45/0x6e
> Apr  9 08:39:49 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#1 stuck for 1776s!
[swapper:0]
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: CPU 1:
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:11:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 09:11:36 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 09:11:36 host kernel: FS:  00007f44bc51f710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:11:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 09:11:36 host kernel: CR2: 00007f44bc52c000 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:11:36 host kernel: Call Trace:
> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 09:11:36 host kernel: BUG: soft lockup - CPU#0 stuck for 1776s!
[swapper:0]
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: CPU 0:
> Apr  9 09:11:36 host kernel: Modules linked in:
> Apr  9 09:11:36 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:11:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:11:36 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 09:11:36 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 09:11:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 09:11:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 09:11:36 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 09:11:36 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 09:11:36 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:11:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 09:11:36 host kernel: CR2: 00000000004884b2 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 09:11:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:11:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:11:36 host kernel: Call Trace:
> Apr  9 09:11:36 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 09:11:36 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 09:11:36 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 09:11:36 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#1 stuck for 492s!
[ssh:2324]
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: CPU 1:
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: Pid: 2324, comm: ssh Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:20:24 host kernel: RIP: e033:[<00007f1b0e259f20>] 
[<00007f1b0e259f20>] 0x7f1b0e259f20
> Apr  9 09:20:24 host kernel: RSP: e02b:00007fffa45a7338  EFLAGS: 00000206
> Apr  9 09:20:24 host kernel: RAX: 00007f1b11f32b20 RBX: 00007f1b0e5399c0
RCX: 00000000000000a0
> Apr  9 09:20:24 host kernel: RDX: 00007f1b11f32b80 RSI: 0000000000000004
RDI: 00007f1b0e539c30
> Apr  9 09:20:24 host kernel: RBP: 0000000000000064 R08: 0000000000000001
R09: 0000000000000001
> Apr  9 09:20:24 host kernel: R10: 00007f1b0e539a30 R11: 00007f1b0e25bfb0
R12: 0000000000000000
> Apr  9 09:20:24 host kernel: R13: 00007f1b0fcd2aec R14: 0000000000000111
R15: 0000000000000071
> Apr  9 09:20:24 host kernel: FS:  00007f1b10528710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 09:20:24 host kernel: CR2: 00007fa913f53000 CR3: 000000006ad44000
CR4: 0000000000002660
> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:20:24 host kernel: Call Trace:
> Apr  9 09:20:24 host kernel: BUG: soft lockup - CPU#0 stuck for 492s!
[swapper:0]
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: CPU 0:
> Apr  9 09:20:24 host kernel: Modules linked in:
> Apr  9 09:20:24 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:20:24 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:20:24 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 09:20:24 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 09:20:24 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 09:20:24 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 09:20:24 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 09:20:24 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 09:20:24 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:20:24 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 09:20:24 host kernel: CR2: 00007fff892a1e30 CR3: 000000006acb1000
CR4: 0000000000002660
> Apr  9 09:20:24 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:20:24 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:20:24 host kernel: Call Trace:
> Apr  9 09:20:24 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 09:20:24 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 09:20:24 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 09:20:24 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 09:24:36 host kernel: BUG: soft lockup - CPU#1 stuck for 234s!
[gzip:4496]
> Apr  9 09:24:36 host kernel: Modules linked in:
> Apr  9 09:24:36 host kernel: CPU 1:
> Apr  9 09:24:36 host kernel: Modules linked in:
> Apr  9 09:24:36 host kernel: Pid: 4496, comm: gzip Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:24:36 host kernel: RIP: e033:[<0000000000409804>] 
[<0000000000409804>] 0x409804
> Apr  9 09:24:36 host kernel: RSP: e02b:00007fff87e009f8  EFLAGS: 00000202
> Apr  9 09:24:36 host kernel: RAX: 00000000f6019e6d RBX: 0000000000008000
RCX: 00007f529598c610
> Apr  9 09:24:36 host kernel: RDX: 0000000000f6019e RSI: 00000000000045a6
RDI: 00000000006534fb
> Apr  9 09:24:36 host kernel: RBP: 0000000000005800 R08: 0000000000004000
R09: 7464697720223333
> Apr  9 09:24:36 host kernel: R10: 6c61202236223d68 R11: 0000000000000246
R12: 00000000006522a0
> Apr  9 09:24:36 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 09:24:36 host kernel: FS:  00007f5295e2a6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 09:24:36 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 09:24:36 host kernel: CR2: 0000000000d550c8 CR3: 0000000026474000
CR4: 0000000000002660
> Apr  9 09:24:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:24:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:24:37 host kernel: Call Trace:
> Apr  9 09:24:37 host kernel: BUG: soft lockup - CPU#0 stuck for 234s!
[swapper:0]
> Apr  9 09:24:37 host kernel: Modules linked in:
> Apr  9 09:24:37 host kernel: CPU 0:
> Apr  9 09:24:37 host kernel: Modules linked in:
> Apr  9 09:24:37 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 09:24:37 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 09:24:38 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 09:24:38 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 09:24:38 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 09:24:38 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 09:24:38 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 09:24:38 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 09:24:38 host kernel: FS:  00007f2f6a31a710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 09:24:38 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 09:24:38 host kernel: CR2: 00007f2f6a327000 CR3: 000000003f62e000
CR4: 0000000000002660
> Apr  9 09:24:38 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 09:24:38 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 09:24:38 host kernel: Call Trace:
> Apr  9 09:24:38 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 09:24:38 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 09:24:38 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 09:24:38 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#1 stuck for 2305s!
[swapper:0]
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: CPU 1:
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:05:51 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 10:05:51 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 10:05:51 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 10:05:51 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 10:05:51 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:05:51 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 10:05:51 host kernel: CR2: 0000000004786158 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:05:51 host kernel: Call Trace:
> Apr  9 10:05:51 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 10:05:51 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 10:05:51 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 10:05:51 host kernel: BUG: soft lockup - CPU#0 stuck for 2305s!
[httpd:9922]
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: CPU 0:
> Apr  9 10:05:51 host kernel: Modules linked in:
> Apr  9 10:05:51 host kernel: Pid: 9922, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:05:51 host kernel: RIP: e033:[<00007ff39a44a36d>] 
[<00007ff39a44a36d>] 0x7ff39a44a36d
> Apr  9 10:05:51 host kernel: RSP: e02b:00007fff4b9ad118  EFLAGS: 00000a86
> Apr  9 10:05:51 host kernel: RAX: c0c176e403831574 RBX: 0000000003643b60
RCX: c0c176e403831574
> Apr  9 10:05:51 host kernel: RDX: 0000000000000000 RSI: 0000000000000004
RDI: 000000000420e43b
> Apr  9 10:05:51 host kernel: RBP: 0000000003643b60 R08: 0000000000000004
R09: 00007fff4b9ad270
> Apr  9 10:05:51 host kernel: R10: 00007ff39aaa9fa0 R11: 00007fff4b9ad188
R12: 000000000320e6d0
> Apr  9 10:05:51 host kernel: R13: 0000000000000000 R14: 00007fff4b9ad368
R15: 0000000000000000
> Apr  9 10:05:51 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:05:51 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 10:05:51 host kernel: CR2: 00000000047a4018 CR3: 000000007972c000
CR4: 0000000000002660
> Apr  9 10:05:51 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:05:51 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:05:51 host kernel: Call Trace:
> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#1 stuck for 2471s!
[swapper:0]
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: CPU 1:
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:50:04 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 10:50:04 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 10:50:04 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 10:50:04 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 10:50:04 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 10:50:04 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 10:50:04 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 10:50:04 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:50:04 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 10:50:04 host kernel: CR2: 00007ff398a0ce08 CR3: 000000004ddb4000
CR4: 0000000000002660
> Apr  9 10:50:04 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:50:04 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:50:04 host kernel: Call Trace:
> Apr  9 10:50:04 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 10:50:04 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 10:50:04 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 10:50:04 host kernel: BUG: soft lockup - CPU#0 stuck for 2471s!
[httpd:16797]
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: CPU 0:
> Apr  9 10:50:04 host kernel: Modules linked in:
> Apr  9 10:50:04 host kernel: Pid: 16797, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:50:04 host kernel: RIP: e033:[<00007ff39a49ae55>] 
[<00007ff39a49ae55>] 0x7ff39a49ae55
> Apr  9 10:50:04 host kernel: RSP: e02b:00007fff4b9b10b8  EFLAGS: 00000246
> Apr  9 10:50:04 host kernel: RAX: 000000000368dd40 RBX: 00000000036868d0
RCX: 222879726575713e
> Apr  9 10:50:05 host kernel: RDX: 000000000000000c RSI: 0000000003686a6c
RDI: 00007fff4b9b1d00
> Apr  9 10:50:05 host kernel: RBP: 00007fff4b9b1da0 R08: 0000000000000001
R09: 0101010101010101
> Apr  9 10:50:05 host kernel: R10: 0000000000000001 R11: 00007ff39c4ac0be
R12: 00007fff4b9b1d00
> Apr  9 10:50:05 host kernel: R13: 00007ff39aaa9fa0 R14: 00007ff39aaa9fa0
R15: 0000000003695df0
> Apr  9 10:50:05 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:50:05 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 10:50:05 host kernel: CR2: 00000000032068c0 CR3: 000000004ddb4000
CR4: 0000000000002660
> Apr  9 10:50:05 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:50:05 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:50:05 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#1 stuck for 389s!
[exim:17977]
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: CPU 1:
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: Pid: 17977, comm: exim Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 10:57:01 host kernel: RSP: e02b:ffff880056611ea0  EFLAGS: 00000246
> Apr  9 10:57:01 host kernel: RAX: 0000000000030004 RBX: 0000000000000000
RCX: ffffffff8100922a
> Apr  9 10:57:01 host kernel: RDX: 00007fd75a6d614e RSI: 0000000000000000
RDI: 0000000000000000
> Apr  9 10:57:01 host kernel: RBP: 00007fd75a6d614e R08: 00007fd75c048710
R09: 0000000000000003
> Apr  9 10:57:01 host kernel: R10: 00007fd75a909a30 R11: 0000000000000246
R12: 0000000000000004
> Apr  9 10:57:01 host kernel: R13: ffff880056611f58 R14: ffff88005aa996d0
R15: ffff88005a51b3c0
> Apr  9 10:57:01 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 10:57:01 host kernel: CR2: 00007fd75a6d614e CR3: 000000007eb9a000
CR4: 0000000000002660
> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:57:01 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 10:57:01 host kernel:  [<ffffffff814da477>] ?
do_page_fault+0xfb/0x28d
> Apr  9 10:57:01 host kernel:  [<ffffffff814d85b5>] ?
page_fault+0x25/0x30
> Apr  9 10:57:01 host kernel: BUG: soft lockup - CPU#0 stuck for 389s!
[netstat:17966]
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: CPU 0:
> Apr  9 10:57:01 host kernel: Modules linked in:
> Apr  9 10:57:01 host kernel: Pid: 17966, comm: netstat Not tainted
2.6.32.9xls-domU #2
> Apr  9 10:57:01 host kernel: RIP: e030:[<ffffffff814d8226>] 
[<ffffffff814d8226>] _spin_lock+0xa/0x15
> Apr  9 10:57:01 host kernel: RSP: e02b:ffff88006ad4dce0  EFLAGS: 00000202
> Apr  9 10:57:01 host kernel: RAX: 0000000000001c1c RBX: ffff88007ff77180
RCX: 0000000000000012
> Apr  9 10:57:01 host kernel: RDX: 018721dfffc35c9f RSI: ffff88006ad4ddc8
RDI: ffff88007ff77188
> Apr  9 10:57:01 host kernel: RBP: ffff88007ff77198 R08: 0000000000000000
R09: 0000000000300000
> Apr  9 10:57:01 host kernel: R10: 00000000fffffff2 R11: ffffffff8126158e
R12: ffff88000ff859c0
> Apr  9 10:57:01 host kernel: R13: ffff88006ad4ddc8 R14: 0000000000034b38
R15: 0000000000000002
> Apr  9 10:57:01 host kernel: FS:  00007fb435545710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 10:57:01 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 10:57:01 host kernel: CR2: 00007f0e429d6000 CR3: 000000007ee3d000
CR4: 0000000000002660
> Apr  9 10:57:01 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 10:57:01 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 10:57:01 host kernel: Call Trace:
> Apr  9 10:57:01 host kernel:  [<ffffffff810cb01b>] ?
__d_lookup+0x7a/0x105
> Apr  9 10:57:01 host kernel:  [<ffffffff810c251e>] ?
do_lookup+0x2a/0x1b3
> Apr  9 10:57:01 host kernel:  [<ffffffff810c4498>] ?
__link_path_walk+0x348/0xdcd
> Apr  9 10:57:01 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 10:57:01 host kernel:  [<ffffffff810c514b>] ?
path_walk+0x66/0xca
> Apr  9 10:57:01 host kernel:  [<ffffffff810c51cf>] ?
do_path_lookup+0x20/0x77
> Apr  9 10:57:01 host kernel:  [<ffffffff810c5ae9>] ?
user_path_at+0x48/0x79
> Apr  9 10:57:01 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ?
mntput_no_expire+0x23/0xed
> Apr  9 10:57:01 host kernel:  [<ffffffff8129b703>] ?
_atomic_dec_and_lock+0x33/0x50
> Apr  9 10:57:01 host kernel:  [<ffffffff810cfd6b>] ?
mntput_no_expire+0x23/0xed
> Apr  9 10:57:01 host kernel:  [<ffffffff810bdf4d>] ?
sys_readlinkat+0x25/0x8d
> Apr  9 10:57:01 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
> Apr  9 11:00:36 host kernel: BUG: soft lockup - CPU#1 stuck for 200s!
[swapper:0]
> Apr  9 11:00:36 host kernel: Modules linked in:
> Apr  9 11:00:36 host kernel: CPU 1:
> Apr  9 11:00:36 host kernel: Modules linked in:
> Apr  9 11:00:36 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 11:00:36 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:00:36 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 11:00:36 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 11:00:36 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 11:00:36 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 11:00:36 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 11:00:36 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 11:00:36 host kernel: FS:  00007f719f5fa6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 11:00:36 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 11:00:36 host kernel: CR2: 00007ff39c43cfa8 CR3: 000000007d0db000
CR4: 0000000000002660
> Apr  9 11:00:36 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 11:00:36 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 11:00:37 host kernel: Call Trace:
> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 11:00:37 host kernel: BUG: soft lockup - CPU#0 stuck for 200s!
[swapper:0]
> Apr  9 11:00:37 host kernel: Modules linked in:
> Apr  9 11:00:37 host kernel: CPU 0:
> Apr  9 11:00:37 host kernel: Modules linked in:
> Apr  9 11:00:37 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 11:00:37 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:00:37 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 11:00:37 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 11:00:37 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 11:00:37 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 11:00:37 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 11:00:37 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 11:00:37 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 11:00:37 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 11:00:37 host kernel: CR2: 00000000035750c8 CR3: 000000007ea99000
CR4: 0000000000002660
> Apr  9 11:00:37 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 11:00:37 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 11:00:37 host kernel: Call Trace:
> Apr  9 11:00:37 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 11:00:37 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 11:00:37 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 11:00:37 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#0 stuck for 460s!
[httpd:19360]
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: CPU 0:
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: Pid: 19360, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff813b995e>] 
[<ffffffff813b995e>] sock_alloc_send_skb+0x3/0xc
> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88005675dc80  EFLAGS: 00000246
> Apr  9 11:08:50 host kernel: RAX: 0000000000003e00 RBX: ffff88005675dcb8
RCX: ffff88005675dce4
> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000057
RDI: ffff88002678c900
> Apr  9 11:08:50 host kernel: RBP: ffff88007f740b80 R08: ffff88005675dce4
R09: 75646f72705f6d76
> Apr  9 11:08:50 host kernel: R10: 5245485720607463 R11: ffffffff81261683
R12: 0000000000000057
> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: ffff88006add3c80
R15: ffff88005675dd60
> Apr  9 11:08:50 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 11:08:50 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 11:08:50 host kernel: CR2: 00000000045cf000 CR3: 000000007f97d000
CR4: 0000000000002660
> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 11:08:50 host kernel: Call Trace:
> Apr  9 11:08:50 host kernel:  [<ffffffff814573c8>] ?
unix_stream_sendmsg+0x18a/0x31f
> Apr  9 11:08:50 host kernel:  [<ffffffff813b61ec>] ?
sock_aio_write+0xf6/0x10a
> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e39f>] ?
xen_restore_fl_direct_end+0x0/0x1
> Apr  9 11:08:50 host kernel:  [<ffffffff810bab7e>] ?
do_sync_write+0xce/0x113
> Apr  9 11:08:50 host kernel:  [<ffffffff810c5af3>] ?
user_path_at+0x52/0x79
> Apr  9 11:08:50 host kernel:  [<ffffffff810be0b0>] ?
cp_new_stat+0xe9/0xfc
> Apr  9 11:08:50 host kernel:  [<ffffffff810561b8>] ?
autoremove_wake_function+0x0/0x2e
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 11:08:50 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 11:08:50 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 11:08:50 host kernel:  [<ffffffff810bb4c5>] ?
vfs_write+0xc0/0x149
> Apr  9 11:08:50 host kernel:  [<ffffffff810bb60a>] ?
sys_write+0x45/0x6e
> Apr  9 11:08:50 host kernel:  [<ffffffff81010ac2>] ?
system_call_fastpath+0x16/0x1b
> Apr  9 11:08:50 host kernel: BUG: soft lockup - CPU#1 stuck for 460s!
[swapper:0]
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: CPU 1:
> Apr  9 11:08:50 host kernel: Modules linked in:
> Apr  9 11:08:50 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 11:08:50 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 11:08:50 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 11:08:50 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 11:08:50 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 11:08:50 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 11:08:50 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 11:08:50 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 11:08:50 host kernel: FS:  00007f0e429cb6e0(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 11:08:50 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 11:08:50 host kernel: CR2: 0000000004598000 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 11:08:50 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 11:08:50 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 11:08:50 host kernel: Call Trace:
> Apr  9 11:08:50 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 11:08:50 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 11:08:50 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#1 stuck for 2897s!
[httpd:26903]
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: CPU 1:
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: Pid: 26903, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:00:40 host kernel: RIP: e033:[<00007ff39a427cd2>] 
[<00007ff39a427cd2>] 0x7ff39a427cd2
> Apr  9 12:00:40 host kernel: RSP: e02b:00007fff4b9afa20  EFLAGS: 00000202
> Apr  9 12:00:40 host kernel: RAX: 0000000000000020 RBX: 00007fff4b9aff30
RCX: 0000000000000013
> Apr  9 12:00:40 host kernel: RDX: 00007ff39a77be20 RSI: 00007fff4b9aff50
RDI: 0000000003bd6418
> Apr  9 12:00:40 host kernel: RBP: 0000000000000000 R08: 00007ff39a782e60
R09: 00007ff39a783d80
> Apr  9 12:00:40 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2
R12: 00007ff39aaaa3e0
> Apr  9 12:00:40 host kernel: R13: 00007fff4b9aff50 R14: 00007fff4b9aff50
R15: 000000000000015e
> Apr  9 12:00:40 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 12:00:40 host kernel: CR2: 0000000003bdc000 CR3: 0000000056566000
CR4: 0000000000002660
> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:00:40 host kernel: Call Trace:
> Apr  9 12:00:40 host kernel: BUG: soft lockup - CPU#0 stuck for 2897s!
[swapper:0]
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: CPU 0:
> Apr  9 12:00:40 host kernel: Modules linked in:
> Apr  9 12:00:40 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:00:40 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:00:40 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 12:00:40 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 12:00:40 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 12:00:40 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 12:00:40 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 12:00:40 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 12:00:40 host kernel: FS:  00007f74fde746e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:00:40 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 12:00:40 host kernel: CR2: 00007ff39aec11a7 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 12:00:40 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:00:40 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:00:40 host kernel: Call Trace:
> Apr  9 12:00:40 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 12:00:40 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 12:00:40 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 12:00:40 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#1 stuck for 1065s!
[swapper:0]
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: CPU 1:
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:19:45 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 12:19:45 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:19:45 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 12:19:45 host kernel: CR2: 00007ff39c5002b0 CR3: 0000000079588000
CR4: 0000000000002660
> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:19:45 host kernel: Call Trace:
> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 12:19:45 host kernel: BUG: soft lockup - CPU#0 stuck for 1065s!
[swapper:0]
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: CPU 0:
> Apr  9 12:19:45 host kernel: Modules linked in:
> Apr  9 12:19:45 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:19:45 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:19:45 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 12:19:45 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 12:19:45 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 12:19:45 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 12:19:45 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 12:19:45 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 12:19:45 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:19:45 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 12:19:45 host kernel: CR2: 00007ff39e404000 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 12:19:45 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:19:45 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:19:45 host kernel: Call Trace:
> Apr  9 12:19:45 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 12:19:45 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 12:19:45 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 12:19:45 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#1 stuck for 989s!
[swapper:0]
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: CPU 1:
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:37:26 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 12:37:26 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 12:37:26 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 12:37:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 12:37:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 12:37:26 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 12:37:26 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 12:37:26 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 12:37:26 host kernel: CR2: 00000000039f7330 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:37:26 host kernel: Call Trace:
> Apr  9 12:37:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 12:37:26 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 12:37:26 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 12:37:26 host kernel: BUG: soft lockup - CPU#0 stuck for 989s!
[httpd:31684]
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: CPU 0:
> Apr  9 12:37:26 host kernel: Modules linked in:
> Apr  9 12:37:26 host kernel: Pid: 31684, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 12:37:26 host kernel: RIP: e033:[<00007ff39a41f7d3>] 
[<00007ff39a41f7d3>] 0x7ff39a41f7d3
> Apr  9 12:37:26 host kernel: RSP: e02b:00007fff4b9ad330  EFLAGS: 00000246
> Apr  9 12:37:26 host kernel: RAX: 00007ff39a41f7ac RBX: 00007ff39aaaa3e0
RCX: 0000000000000001
> Apr  9 12:37:26 host kernel: RDX: 00007ff39a781bdc RSI: 000000000000003e
RDI: 0000000000000000
> Apr  9 12:37:26 host kernel: RBP: 00007ff39aaa9e60 R08: 00007ff39a782e60
R09: 0000000000000003
> Apr  9 12:37:26 host kernel: R10: 0000000000000000 R11: 00007ff39c4ac0a2
R12: 0000000003a2b297
> Apr  9 12:37:26 host kernel: R13: 00007ff39aaaa3e0 R14: 00007ff39a784da0
R15: 00007ff39a783e80
> Apr  9 12:37:26 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 12:37:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 12:37:26 host kernel: CR2: 0000000003aa1000 CR3: 000000004debb000
CR4: 0000000000002660
> Apr  9 12:37:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 12:37:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 12:37:26 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#1 stuck for 3768s!
[swapper:0]
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: CPU 1:
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:44:52 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 13:44:52 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 13:44:52 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 13:44:52 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 13:44:52 host kernel: CR2: 00007fff892a1e30 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 13:44:52 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 13:44:52 host kernel: BUG: soft lockup - CPU#0 stuck for 3768s!
[swapper:0]
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: CPU 0:
> Apr  9 13:44:52 host kernel: Modules linked in:
> Apr  9 13:44:52 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 13:44:52 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:44:52 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 13:44:52 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 13:44:52 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 13:44:52 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 13:44:52 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 13:44:52 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 13:44:52 host kernel: FS:  00007f0e429cb6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 13:44:52 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 13:44:52 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d0de000
CR4: 0000000000002660
> Apr  9 13:44:52 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 13:44:52 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 13:44:52 host kernel: Call Trace:
> Apr  9 13:44:52 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 13:44:52 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 13:44:52 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 13:44:52 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#1 stuck for 607s!
[swapper:0]
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: CPU 1:
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:55:43 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 13:55:43 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 13:55:43 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 13:55:43 host kernel: CR2: 0000000003c7b27f CR3: 00000000266e8000
CR4: 0000000000002660
> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 13:55:43 host kernel: Call Trace:
> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 13:55:43 host kernel: BUG: soft lockup - CPU#0 stuck for 607s!
[swapper:0]
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: CPU 0:
> Apr  9 13:55:43 host kernel: Modules linked in:
> Apr  9 13:55:43 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 13:55:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 13:55:43 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 13:55:43 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 13:55:43 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 13:55:43 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 13:55:43 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 13:55:43 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 13:55:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 13:55:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 13:55:43 host kernel: CR2: 00007f2f28bf62e8 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 13:55:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 13:55:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 13:55:43 host kernel: Call Trace:
> Apr  9 13:55:43 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 13:55:43 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 13:55:43 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 13:55:43 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1153s!
[swapper:0]
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: CPU 1:
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 14:16:21 host kernel: RSP: e02b:ffff88007f8a1dd8  EFLAGS: 00000246
> Apr  9 14:16:21 host kernel: RAX: 0000000000030004 RBX: ffff88007db55810
RCX: ffffffff8100922a
> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000000
> Apr  9 14:16:21 host kernel: RBP: 0000000000000001 R08: ffff88007f8a0000
R09: ffffffff8100e3b2
> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 14:16:21 host kernel: R13: ffff88007e0040c0 R14: 0000000000000000
R15: 0000000000001000
> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 14:16:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 14:16:21 host kernel: Call Trace:
> Apr  9 14:16:21 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 14:16:21 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 14:16:21 host kernel:  [<ffffffff8103b070>] ?
finish_task_switch+0x27/0x6e
> Apr  9 14:16:21 host kernel:  [<ffffffff814d6a77>] ?
thread_return+0x3e/0x109
> Apr  9 14:16:21 host kernel:  [<ffffffff814d87ba>] ?
error_exit+0x2a/0x60
> Apr  9 14:16:21 host kernel:  [<ffffffff8101149d>] ?
retint_restore_args+0x5/0x6
> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe5f>] ?
cpu_idle+0x80/0x82
> Apr  9 14:16:21 host kernel: BUG: soft lockup - CPU#0 stuck for 1153s!
[swapper:0]
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: CPU 0:
> Apr  9 14:16:21 host kernel: Modules linked in:
> Apr  9 14:16:21 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 14:16:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 14:16:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 14:16:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 14:16:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 14:16:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 14:16:21 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 14:16:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 14:16:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 14:16:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 14:16:21 host kernel: CR2: 00007ff39e4a7000 CR3: 000000007e2a6000
CR4: 0000000000002660
> Apr  9 14:16:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 14:16:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 14:16:21 host kernel: Call Trace:
> Apr  9 14:16:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 14:16:21 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 14:16:21 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 14:16:21 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#1 stuck for 4104s!
[swapper:0]
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: CPU 1:
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 15:40:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 15:40:21 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 15:40:21 host kernel: FS:  00007fd75c048710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 15:40:21 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 15:40:21 host kernel: CR2: 000000000040d0e2 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 15:40:21 host kernel: Call Trace:
> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 15:40:21 host kernel: BUG: soft lockup - CPU#0 stuck for 4104s!
[swapper:0]
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: CPU 0:
> Apr  9 15:40:21 host kernel: Modules linked in:
> Apr  9 15:40:21 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 15:40:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 15:40:21 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 15:40:21 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 15:40:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 15:40:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 15:40:21 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 15:40:21 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 15:40:21 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 15:40:21 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 15:40:21 host kernel: CR2: 00000000004c04a8 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 15:40:21 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 15:40:21 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 15:40:21 host kernel: Call Trace:
> Apr  9 15:40:21 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 15:40:21 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 15:40:21 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 15:40:21 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 16:06:21 host kernel: BUG: soft lockup - CPU#1 stuck for 1452s!
[swapper:0]
> Apr  9 16:06:21 host kernel: Modules linked in:
> Apr  9 16:06:21 host kernel: CPU 1:
> Apr  9 16:06:21 host kernel: Modules linked in:
> Apr  9 16:06:21 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:06:21 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:06:21 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 16:06:21 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 16:06:21 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:06:21 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:06:21 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 16:06:21 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 16:06:21 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:06:25 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 16:06:25 host kernel: CR2: 00007fbd3c012909 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:06:26 host kernel: Call Trace:
> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:06:26 host kernel: BUG: soft lockup - CPU#0 stuck for 1452s!
[swapper:0]
> Apr  9 16:06:26 host kernel: Modules linked in:
> Apr  9 16:06:26 host kernel: CPU 0:
> Apr  9 16:06:26 host kernel: Modules linked in:
> Apr  9 16:06:26 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:06:26 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:06:26 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 16:06:26 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 16:06:26 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:06:26 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:06:26 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 16:06:26 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 16:06:26 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:06:26 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 16:06:26 host kernel: CR2: 00007f09a4413000 CR3: 000000007ea99000
CR4: 0000000000002660
> Apr  9 16:06:26 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:06:26 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:06:26 host kernel: Call Trace:
> Apr  9 16:06:26 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:06:26 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:06:26 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:06:26 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#1 stuck for 637s!
[httpd:32342]
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: CPU 1:
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: Pid: 32342, comm: httpd Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff8100922a>] 
[<ffffffff8100922a>] hypercall_page+0x22a/0x1001
> Apr  9 16:17:43 host kernel: RSP: e02b:ffff88007955fea0  EFLAGS: 00000246
> Apr  9 16:17:43 host kernel: RAX: 0000000000030004 RBX: 0000000000000000
RCX: ffffffff8100922a
> Apr  9 16:17:43 host kernel: RDX: 000000000002ba30 RSI: 0000000000000000
RDI: 0000000000000000
> Apr  9 16:17:43 host kernel: RBP: 000000000394ca98 R08: 0000000000033240
R09: 000000310f6de059
> Apr  9 16:17:43 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000006
> Apr  9 16:17:43 host kernel: R13: ffff88007955ff58 R14: ffff88007f9a8ab0
R15: ffff88007d093b40
> Apr  9 16:17:43 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:17:43 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 16:17:43 host kernel: CR2: 000000000394ca98 CR3: 000000006adde000
CR4: 0000000000002660
> Apr  9 16:17:43 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:17:43 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:17:43 host kernel: Call Trace:
> Apr  9 16:17:43 host kernel:  [<ffffffff8100dd39>] ?
xen_force_evtchn_callback+0x9/0xa
> Apr  9 16:17:43 host kernel:  [<ffffffff8100e3b2>] ?
check_events+0x12/0x20
> Apr  9 16:17:43 host kernel:  [<ffffffff8100e359>] ?
xen_irq_enable_direct_end+0x0/0x7
> Apr  9 16:17:43 host kernel:  [<ffffffff814da477>] ?
do_page_fault+0xfb/0x28d
> Apr  9 16:17:43 host kernel:  [<ffffffff814d85b5>] ?
page_fault+0x25/0x30
> Apr  9 16:17:43 host kernel: BUG: soft lockup - CPU#0 stuck for 637s!
[swapper:0]
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: CPU 0:
> Apr  9 16:17:43 host kernel: Modules linked in:
> Apr  9 16:17:43 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:17:43 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:17:44 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 16:17:44 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 16:17:44 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:17:44 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:17:44 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 16:17:44 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 16:17:44 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:17:44 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 16:17:44 host kernel: CR2: 00000000004505b0 CR3: 000000007ea99000
CR4: 0000000000002660
> Apr  9 16:17:44 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:17:44 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:17:44 host kernel: Call Trace:
> Apr  9 16:17:44 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:17:44 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:17:44 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:17:44 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#1 stuck for 147s!
[swapper:0]
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: CPU 1:
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:20:22 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 16:20:22 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 16:20:22 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:20:22 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 16:20:22 host kernel: CR2: 00007ff39aec11a7 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:20:22 host kernel: Call Trace:
> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:20:22 host kernel: BUG: soft lockup - CPU#0 stuck for 147s!
[swapper:0]
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: CPU 0:
> Apr  9 16:20:22 host kernel: Modules linked in:
> Apr  9 16:20:22 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:20:22 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:20:22 host kernel: RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
> Apr  9 16:20:22 host kernel: RAX: 0000000000000000 RBX: ffffffff81690000
RCX: ffffffff810093aa
> Apr  9 16:20:22 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:20:22 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:20:22 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: ffffffffffffffff
> Apr  9 16:20:22 host kernel: R13: ffffffff818ebf20 R14: ffffffff818eec70
R15: 0000000000000000
> Apr  9 16:20:22 host kernel: FS:  00007f07704fe6e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:20:22 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 16:20:22 host kernel: CR2: 00000000006ddc20 CR3: 0000000071c86000
CR4: 0000000000002660
> Apr  9 16:20:22 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:20:22 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:20:22 host kernel: Call Trace:
> Apr  9 16:20:22 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:20:22 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:20:22 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:20:22 host kernel:  [<ffffffff818b6c42>] ?
start_kernel+0x353/0x35f
> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#1 stuck for 119s!
[swapper:0]
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: CPU 1:
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: Pid: 0, comm: swapper Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:22:29 host kernel: RIP: e030:[<ffffffff810093aa>] 
[<ffffffff810093aa>] hypercall_page+0x3aa/0x1001
> Apr  9 16:22:29 host kernel: RSP: e02b:ffff88007f8a1f08  EFLAGS: 00000246
> Apr  9 16:22:29 host kernel: RAX: 0000000000000000 RBX: ffff88007f8a0000
RCX: ffffffff810093aa
> Apr  9 16:22:29 host kernel: RDX: 0000000000000000 RSI: 0000000000000000
RDI: 0000000000000001
> Apr  9 16:22:29 host kernel: RBP: ffffffff81896d30 R08: 0000000000000000
R09: ffffffff8100e3b2
> Apr  9 16:22:29 host kernel: R10: 0000000000000001 R11: 0000000000000246
R12: 0000000000000000
> Apr  9 16:22:29 host kernel: R13: 0000000000000000 R14: 0000000000000000
R15: 0000000000000000
> Apr  9 16:22:29 host kernel: FS:  00007ff39e495710(0000)
GS:ffff8800022c7000(0000) knlGS:0000000000000000
> Apr  9 16:22:29 host kernel: CS:  e033 DS: 002b ES: 002b CR0:
000000008005003b
> Apr  9 16:22:29 host kernel: CR2: 00007ff39c506460 CR3: 0000000026469000
CR4: 0000000000002660
> Apr  9 16:22:29 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:22:29 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:22:29 host kernel: Call Trace:
> Apr  9 16:22:29 host kernel:  [<ffffffff8100ddb7>] ?
xen_safe_halt+0xc/0x15
> Apr  9 16:22:29 host kernel:  [<ffffffff8100bdcf>] ?
xen_idle+0x37/0x40
> Apr  9 16:22:29 host kernel:  [<ffffffff8100fe2e>] ?
cpu_idle+0x4f/0x82
> Apr  9 16:22:29 host kernel: BUG: soft lockup - CPU#0 stuck for 119s!
[syslogd:1848]
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: CPU 0:
> Apr  9 16:22:29 host kernel: Modules linked in:
> Apr  9 16:22:29 host kernel: Pid: 1848, comm: syslogd Not tainted
2.6.32.9xls-domU #2
> Apr  9 16:22:29 host kernel: RIP: 0033:[<00007f74fd9e4ad5>] 
[<00007f74fd9e4ad5>] 0x7f74fd9e4ad5
> Apr  9 16:22:29 host kernel: RSP: 002b:00007fff22fa00d0  EFLAGS: 00000246
> Apr  9 16:22:29 host kernel: RAX: 000000000000002e RBX: 0000000000000000
RCX: ffffffff810092eb
> Apr  9 16:22:29 host kernel: RDX: 00000000000003fe RSI: 00007fff22fa01b0
RDI: 0000000000000000
> Apr  9 16:22:30 host kernel: RBP: 00000000000003fe R08: 0000000000000000
R09: 0000000000000000
> Apr  9 16:22:30 host kernel: R10: 0000000000000000 R11: 0000000000000246
R12: 00007fff22fa01b0
> Apr  9 16:22:30 host kernel: R13: 0000000000000000 R14: 00007fff22fa0120
R15: 00007f74fde74690
> Apr  9 16:22:30 host kernel: FS:  00007f74fde746e0(0000)
GS:ffff8800022ac000(0000) knlGS:0000000000000000
> Apr  9 16:22:30 host kernel: CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
> Apr  9 16:22:30 host kernel: CR2: 00007f0e429d6000 CR3: 000000007d03a000
CR4: 0000000000002660
> Apr  9 16:22:30 host kernel: DR0: 0000000000000000 DR1: 0000000000000000
DR2: 0000000000000000
> Apr  9 16:22:30 host kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7: 0000000000000400
> Apr  9 16:22:30 host kernel: Call Trace:
> 
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Yann Cezard

2010-Apr-15 07:33 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Pim van Riezen a écrit :> Good day,
>
> We''re trying to get 2.6.31 and 2.6.32 rolled out on our clusters
to offer newer features like FUSE fo our customers, but we''re ran into
a couple of showstopper issues when deploying these kernels on busier guests,
showing a lot of errors like this:
>
>   BUG: soft lockup - CPU#0 stuck for 561s! [swapper:0]
>   Modules linked in:
>   CPU 0:
>   Modules linked in:
>   Pid: 0, comm: swapper Not tainted 2.6.32.9xls-domU #2 
>   RIP: e030:[<ffffffff810093aa>]  [<ffffffff810093aa>]
hypercall_page+0x3aa/0x1001
>   RSP: e02b:ffffffff81691f70  EFLAGS: 00000246
>   RAX: 0000000000000000 RBX: ffffffff81690000 RCX: ffffffff810093aa
>   RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
>   RBP: ffffffff81896d30 R08: 0000000000000000 R09: ffffffff8100e3b2
>   R10: 0000000000000001 R11: 0000000000000246 R12: ffffffffffffffff
>   R13: ffffffff818ebf20 R14: ffffffff818eec70 R15: 0000000000000000
>   FS:  00007f8ac7a9c6e0(0000) GS:ffff8800022ac000(0000)
knlGS:0000000000000000
>   CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>   CR2: 00007f291b4c9000 CR3: 000000007d8c1000 CR4: 0000000000002660
>   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>   DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>   Call Trace:
>    [<ffffffff8100ddb7>] ? xen_safe_halt+0xc/0x15
>    [<ffffffff8100bdcf>] ? xen_idle+0x37/0x40
>    [<ffffffff8100fe2e>] ? cpu_idle+0x4f/0x82
>    [<ffffffff818b6c42>] ? start_kernel+0x353/0x35f
>
> in our hope to get rid of this issue we upgraded from Xen 3.3 to Xen
3.4.1.7 out of the gitco repos. The issue persisted. Is there a magic version of
Xen, preferably one that can be found in an rpm repository for CentOS 5, that
*does* properly support pvops kernels without these issues?
>
> I''m also seing this one:
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=543 where the bug is
still open but no activity since 2008. I don''t know if that bugzilla is
still being actively maintained?
>
> Cheers,
> Pim
>
>
>   Hi Pim,

I am having similar issues, with differents versions of Xen hypervisor
(3.2, 3.4) and
with domU kernels >= 2.6.26 (until 2.6.32-4 from Debian), always on the
same 2
or 3 VMs that are frequently under heavy load.

After searching a lot, I thought that my CPU softlock problems (which
sometimes
make my VMs freezing) was perhaps related to the xen clocksource, so I
decided
to give a try to this :
   
http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27

Using jiffies + independant wallclock + ntp in domU seems to have stop
the CPU
softlock error messages in kernel messages (at least I didn''t have any
since I use
it, but it''s only for 2 days...). Now I am crossing my fingers... :-)

I also read in xen-devel that you are using FC LUNs for storage, I also
use that,
perhaps you will want to have a look at the "Interrupt handling in
Xen"
message
that was post on this list yesterday, by defaults my domain-0 was doing
all its
interrupts (network and HBA) on the same CPU, which is probably some kind
of bottleneck under heavy load.

Cheers,

-- 
Yann Cézard - Administrateur Systèmes Serveurs
Centre de Ressources Informatiques    -    http://cri.univ-pau.fr
Université de Pau et des Pays de l''Adour - http://www.univ-pau.fr
Bat IFR, rue Jules Ferry, 64000 PAU - Tél.:  +33 (0)5 59 40 77 94


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pim van Riezen

2010-Apr-15 08:21 UTC

head link

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Yann,

On Apr 15, 2010, at 9:33 , Yann Cezard wrote:
> After searching a lot, I thought that my CPU softlock problems (which
> sometimes
> make my VMs freezing) was perhaps related to the xen clocksource, so I
> decided
> to give a try to this :
> 
> http://wiki.debian.org/Xen#A.27clocksource.2BAC8-0.3ATimewentbackwards.27
Giving that a shot now. Sounds plausibly related, also with regards to my seeing
timing-related wackiness inside the domUs where this happens.
> I also read in xen-devel that you are using FC LUNs for storage, I also
> use that,
> perhaps you will want to have a look at the "Interrupt handling in
Xen"
> message
> that was post on this list yesterday, by defaults my domain-0 was doing
> all its
> interrupts (network and HBA) on the same CPU, which is probably some kind
> of bottleneck under heavy load.
CentOS5 installs irqbalance by default.

Thanks for your suggestions.

Kind regards,
Pim van Riezen


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Apr 2010 - Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

[Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32

Re: [Xen-users] Xen3.3 / Xen3.4 CPU soft lockups under pvops 2.6.31/2.6.32