Hi,
I've created a new debian stable domU on an existing debian stable dom0
with xen 4.1 installed from packages.
I installed some applications/etc into the domU, but haven't had time to
actually do anything, so it should be very much idle.
In fact, the domU has been up almost 12 days:
11:51:52 up 11 days, 22:26, 1 user, load average: 0.00, 0.01, 0.05
and xm list shows less than 1 hour CPU time:
pabx 6 2048 2 -b---- 3325.7
I noticed when I looked at it today, there are a number of kernel errors
(BUG).
The last line of the normal bootup (from dmesg output) through to the
end of the first BUG are here:
[ 18.216073] eth0: no IPv6 routers present
[12957.032892] hrtimer: interrupt took 180298877 ns
[571901.759780] sched: RT throttling activated
[943100.068753] BUG: soft lockup - CPU#1 stuck for 23s! [swapper/1:0]
[943100.068753] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache
xen_blkfront xen_netfront
[943100.068753] CPU 1
[943100.068753] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache
xen_blkfront xen_netfront
[943100.068753]
[943100.068753] Pid: 0, comm: swapper/1 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u1
[943100.068753] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>]
hypercall_page+0x22a/0x1000
[943100.068753] RSP: e02b:ffff88007fd03e90 EFLAGS: 00000246
[943100.068753] RAX: 0000000000040001 RBX: ffffffff816040c0 RCX:
ffffffff8100122a
[943100.068753] RDX: ffff88007fd03e30 RSI: 0000000000000000 RDI:
0000000000000000
[943100.068753] RBP: ffff88007d371fd8 R08: 0000000000000005 R09:
0000000000000004
[943100.068753] R10: 0000000000000020 R11: 0000000000000246 R12:
0000000000000100
[943100.068753] R13: 0000000000000003 R14: 0000000000000008 R15:
ffff88007d371fd8
[943100.068753] FS: 00007fac5856c7c0(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
[943100.068753] CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
[943100.068753] CR2: 00007fac568e8850 CR3: 0000000001605000 CR4:
0000000000000660
[943100.068753] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[943100.068753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[943100.068753] Process swapper/1 (pid: 0, threadinfo ffff88007d370000,
task ffff88007d360780)
[943100.068753] Stack:
[943100.068753] ffff88007fd0e980 0000000000000001 ffffffff81006790
ffffffff81006d22
[943100.068753] ffff88007fd0e980 0000000000000020 0000000000000004
0000000000000005
[943100.068753] 0000000000000200 0000000000000001 ffff88007fd03e30
0000000000000001
[943100.068753] Call Trace:
[943100.068753] <IRQ>
[943100.068753] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753] [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753] [<ffffffff81006d0f>] ?
xen_restore_fl_direct_reloc+0x4/0x4
[943100.068753] [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[943100.068753] [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[943100.068753] [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[943100.068753] [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[943100.068753] [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[943100.068753] [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[943100.068753] [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[943100.068753] [<ffffffff8135783e>] ?
xen_do_hypervisor_callback+0x1e/0x30
[943100.068753] <EOI>
[943100.068753] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753] [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753] [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[943100.068753] [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[943100.068753] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00 00
0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[943100.068753] Call Trace:
[943100.068753] <IRQ> [<ffffffff81006790>] ?
xen_force_evtchn_callback+0x9/0xa
[943100.068753] [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753] [<ffffffff81006d0f>] ?
xen_restore_fl_direct_reloc+0x4/0x4
[943100.068753] [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[943100.068753] [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[943100.068753] [<ffffffff8121c9dd>] ? __xen_evtchn_do_upcall+0x24a/0x287
[943100.068753] [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[943100.068753] [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[943100.068753] [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[943100.068753] [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[943100.068753] [<ffffffff8135783e>] ?
xen_do_hypervisor_callback+0x1e/0x30
[943100.068753] <EOI> [<ffffffff8100122a>] ?
hypercall_page+0x22a/0x1000
[943100.068753] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[943100.068753] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[943100.068753] [<ffffffff81006d22>] ? check_events+0x12/0x20
[943100.068753] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[943100.068753] [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[943100.068753] [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[943100.068753] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
The second occurrence was similar:
[1024036.074678] BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0]
[1024036.074678] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache
xen_blkfront xen_netfront
[1024036.074678] CPU 1
[1024036.074678] Modules linked in: nfsd nfs nfs_acl auth_rpcgss fscache
lockd sunrpc loop evdev mperf snd_pcm processor thermal_sys
snd_page_alloc snd_timer snd soundcore pcspkr ext4 crc16 jbd2 mbcache
xen_blkfront xen_netfront
[1024036.074678]
[1024036.074678] Pid: 0, comm: swapper/1 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u1
[1024036.074678] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>]
hypercall_page+0x22a/0x1000
[1024036.074678] RSP: e02b:ffff88007fd03e90 EFLAGS: 00000246
[1024036.074678] RAX: 0000000000040001 RBX: ffffffff816040c0 RCX:
ffffffff8100122a
[1024036.074678] RDX: ffff88007fd03e30 RSI: 0000000000000000 RDI:
0000000000000000
[1024036.074678] RBP: ffff88007d371fd8 R08: 0000000000000020 R09:
0000000000000020
[1024036.074678] R10: 0000000000000020 R11: 0000000000000246 R12:
0000000000000100
[1024036.074678] R13: 0000000000000001 R14: 0000000000000008 R15:
ffff88007d371fd8
[1024036.074678] FS: 00007fac5856c7c0(0000) GS:ffff88007fd00000(0000)
knlGS:0000000000000000
[1024036.074678] CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
[1024036.074678] CR2: 00007fac568e8850 CR3: 0000000001605000 CR4:
0000000000000660
[1024036.074678] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[1024036.074678] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[1024036.074678] Process swapper/1 (pid: 0, threadinfo ffff88007d370000,
task ffff88007d360780)
[1024036.074678] Stack:
[1024036.074678] ffff88007fd0e980 0000000000000001 ffffffff81006790
ffffffff81006d22
[1024036.074678] ffff88007fd0e980 0000000000000020 0000000000000020
0000000000000020
[1024036.074678] 0000000000000200 0000000000000001 ffff88007fd03e30
0000000000000001
[1024036.074678] Call Trace:
[1024036.074678] <IRQ>
[1024036.074678] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678] [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678] [<ffffffff81006d0f>] ?
xen_restore_fl_direct_reloc+0x4/0x4
[1024036.074678] [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[1024036.074678] [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[1024036.074678] [<ffffffff8121c9dd>] ?
__xen_evtchn_do_upcall+0x24a/0x287
[1024036.074678] [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[1024036.074678] [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[1024036.074678] [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[1024036.074678] [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[1024036.074678] [<ffffffff8135783e>] ?
xen_do_hypervisor_callback+0x1e/0x30
[1024036.074678] <EOI>
[1024036.074678] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678] [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678] [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[1024036.074678] [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[1024036.074678] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00 00
00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
[1024036.074678] Call Trace:
[1024036.074678] <IRQ> [<ffffffff81006790>] ?
xen_force_evtchn_callback+0x9/0xa
[1024036.074678] [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678] [<ffffffff81006d0f>] ?
xen_restore_fl_direct_reloc+0x4/0x4
[1024036.074678] [<ffffffff8106218f>] ? arch_local_irq_restore+0x7/0x8
[1024036.074678] [<ffffffff8104c36e>] ? __do_softirq+0xb9/0x177
[1024036.074678] [<ffffffff8121c9dd>] ?
__xen_evtchn_do_upcall+0x24a/0x287
[1024036.074678] [<ffffffff813577ec>] ? call_softirq+0x1c/0x30
[1024036.074678] [<ffffffff8100fa21>] ? do_softirq+0x3c/0x7b
[1024036.074678] [<ffffffff8104c5d6>] ? irq_exit+0x3c/0x99
[1024036.074678] [<ffffffff8121dd9d>] ? xen_evtchn_do_upcall+0x27/0x32
[1024036.074678] [<ffffffff8135783e>] ?
xen_do_hypervisor_callback+0x1e/0x30
[1024036.074678] <EOI> [<ffffffff8100122a>] ?
hypercall_page+0x22a/0x1000
[1024036.074678] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[1024036.074678] [<ffffffff81006790>] ? xen_force_evtchn_callback+0x9/0xa
[1024036.074678] [<ffffffff81006d22>] ? check_events+0x12/0x20
[1024036.074678] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
[1024036.074678] [<ffffffff8106c19b>] ? arch_local_irq_enable+0x7/0x8
[1024036.074678] [<ffffffff8100d285>] ? cpu_idle+0xe8/0xf2
[1024036.074678] [<ffffffff81006cc9>] ?
xen_irq_enable_direct_reloc+0x4/0x4
Note there are no NFS mounts or similar.
Memory stats now:
free
total used free shared buffers cached
Mem: 2051044 262388 1788656 0 114992 79496
-/+ buffers/cache: 67900 1983144
Swap: 2171900 0 2171900
Disk info:
fdisk -l
Disk /dev/xvda: 53.7 GB, 53685415936 bytes
255 heads, 63 sectors/track, 6526 cylinders, total 104854328 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ee580
Device Boot Start End Blocks Id System
/dev/xvda1 2048 100507647 50252800 83 Linux
/dev/xvda2 100509694 104853503 2171905 5 Extended
/dev/xvda5 100509696 104853503 2171904 82 Linux swap /
Solaris
The disk is provided by dom0, which is a multipath iSCSI device.
Debian packages on dom0:
dpkg -l|grep xen
ii libxen-4.1 4.1.4-3+deb7u1 amd64 Public libs for Xen
ii libxenstore3.0 4.1.4-3+deb7u1 amd64 Xenstore
communications library for Xen
ii xen-hypervisor-4.1-amd64 4.1.4-3+deb7u1 amd64
Xen Hypervisor on AMD64
ii xen-linux-system-3.2.0-4-amd64 3.2.60-1+deb7u3
amd64 Xen system with Linux 3.2 on 64-bit PCs (meta-package)
ii xen-linux-system-amd64 3.2+46 amd64 Xen
system with Linux for 64-bit PCs (meta-package)
ii xen-system-amd64 4.1.4-3+deb7u1 amd64 Xen
System on AMD64 (meta-package)
ii xen-utils-4.1 4.1.4-3+deb7u1 amd64 XEN
administrative tools
ii xen-utils-common 4.1.4-3+deb7u1 all Xen
administrative tools - common files
ii xenstore-utils 4.1.4-3+deb7u1 amd64 Xenstore
utilities for Xen
ii linux-image-3.2.0-4-amd64 3.2.60-1+deb7u3 amd64
Linux 3.2 for 64-bit PCs
Packages on domU
ii linux-image-3.2.0-4-amd64 3.2.60-1+deb7u1
amd64 Linux 3.2 for 64-bit PCs
Can anyone provide any suggestions or information on how I might resolve
this before I need to actually use it?
Thanks,
Adam
--
Adam Goryachev Website Managers www.websitemanagers.com.au