thr3ads.net - Xen users - [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions [Apr 2010]

If this information is useful, please help other people find it:
Share via:

Ben Holt

2010-Apr-11 20:12 UTC

[Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Hello,

I have experienced a dom0 crash where the system became unreachable via 
the network and the console was unresponsive.  I would appreciate help 
interpretting the logs and any configuration change suggestions.

It is a stock Debian Lenny dom0 running xen 3.2.1 with kernel 
2.6.26-2-xen-amd64 and an AMD Athlon IIx4 with 4 GB of RAM.  It is 
running 4 VMs.  One VM has two PCI NICs being passed to it, while 
another has three PCI MPEG encoders passed to it (physically they are 
only two cards, but one, a hauppage pvr-500, has two encoders on it 
connected via a PCI bridge).  Three of the VMs were migrated from 
another machine and are running Debian kernel 2.6.26-2-xen-686 (32 bit), 
the fourth is running the same kernel as the dom0 (64 bit).

As you''ll see from the logs the system has three hard drives, a PATA 
(hdc) that holds the dom0 and VMs OS file systems, and 2 SATA (sda, sdb) 
that are configured in a software RAID 1 and hold the VMs data 
filesystems.  If I am reading things right all three drives appear to 
experience interrupt / DMA issues during the crash.  The peth0 NIC goes 
down and up during the crash.  There is also a mention of IRQ 22, which 
is used by the SATA controller.

Any pointers or suggests are greatly appreciated.

- Ben


 From /var/log/syslog:

Apr 11 04:00:01 xenserver /USR/SBIN/CRON[20831]: (root) CMD (test -x 
/usr/sbin/cron-apt && /usr/sbin/cron-apt)
Apr 11 04:00:01 xenserver /USR/SBIN/CRON[20832]: (root) CMD (if [ -x 
/usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then 
/usr/bin/vnstat -u; fi)
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/console/1/0
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/block: remove 
XENBUS_PATH=backend/vbd/1/2049
Apr 11 04:03:02 xenserver kernel: [275947.205259] eth0: port 2(vif1.0) 
entering disabled state
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/block: remove 
XENBUS_PATH=backend/vbd/1/2051
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/vif-bridge: 
bridge=eth0 offline XENBUS_PATH=backend/vif/1/0
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/block: remove 
XENBUS_PATH=backend/vbd/1/2050
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vbd/1/2049
Apr 11 04:03:02 xenserver kernel: [275947.239755] eth0: port 2(vif1.0) 
entering disabled state
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vbd/1/2051
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vbd/1/2050
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/vif-bridge: brctl 
delif eth0 vif1.0 failed
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/vif-bridge: ifconfig 
vif1.0 down failed
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/vif-bridge: 
Successful vif-bridge offline for vif1.0, bridge eth0.
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vif/1/0
Apr 11 04:03:02 xenserver logger: /etc/xen/scripts/vif-bridge: 
bridge=eth0 offline XENBUS_PATH=backend/vif/1/1
Apr 11 04:03:02 xenserver kernel: [275947.564179] xenbrDMZ: port 
1(vif1.1) entering disabled state
Apr 11 04:03:03 xenserver kernel: [275947.607309] xenbrDMZ: port 
1(vif1.1) entering disabled state
Apr 11 04:03:03 xenserver logger: /etc/xen/scripts/vif-bridge: brctl 
delif xenbrDMZ vif1.1 failed
Apr 11 04:03:03 xenserver logger: /etc/xen/scripts/vif-bridge: ifconfig 
vif1.1 down failed
Apr 11 04:03:03 xenserver logger: /etc/xen/scripts/vif-bridge: 
Successful vif-bridge offline for vif1.1, bridge xenbrDMZ.
Apr 11 04:03:03 xenserver kernel: [275948.415105] ACPI: PCI interrupt 
for device 0000:02:00.0 disabled
Apr 11 04:03:03 xenserver kernel: [275948.415130] ACPI: PCI interrupt 
for device 0000:04:07.0 disabled
Apr 11 04:03:03 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/vif/1/1
Apr 11 04:03:03 xenserver logger: /etc/xen/scripts/xen-hotplug-cleanup: 
XENBUS_PATH=backend/pci/1/0
Apr 11 04:05:01 xenserver /USR/SBIN/CRON[21159]: (root) CMD (if [ -x 
/usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then 
/usr/bin/vnstat -u; fi)
Apr 11 04:10:01 xenserver /USR/SBIN/CRON[21165]: (root) CMD (if [ -x 
/usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then 
/usr/bin/vnstat -u; fi)
Apr 11 04:15:01 xenserver /USR/SBIN/CRON[21171]: (root) CMD (if [ -x 
/usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then 
/usr/bin/vnstat -u; fi)
Apr 11 04:17:01 xenserver /USR/SBIN/CRON[21177]: (root) CMD (   cd / && 
run-parts --report /etc/cron.hourly)
Apr 11 04:20:01 xenserver /USR/SBIN/CRON[21181]: (root) CMD (if [ -x 
/usr/bin/vnstat ] && [ `ls /var/lib/vnstat/ | wc -l` -ge 1 ]; then 
/usr/bin/vnstat -u; fi)
Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody cared 
(try booting with the "irqpoll" option)
Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm: swapper 
Not tainted 2.6.26-2-xen-amd64 #1
Apr 11 04:22:29 xenserver kernel: [277113.497152]
Apr 11 04:22:29 xenserver kernel: [277113.497153] Call Trace:
Apr 11 04:22:29 xenserver kernel: [277113.497165]  <IRQ>  
[<ffffffff8037ce18>] irq_ignore_unhandled+0x1c/0x32
Apr 11 04:22:29 xenserver kernel: [277113.497188]  [<ffffffff8025f88f>] 
__report_bad_irq+0x30/0x72
Apr 11 04:22:29 xenserver kernel: [277113.497197]  [<ffffffff8025fb58>] 
note_interrupt+0x287/0x2c7
Apr 11 04:22:29 xenserver kernel: [277113.497207]  [<ffffffff80260440>] 
handle_level_irq+0xc3/0x118
Apr 11 04:22:29 xenserver kernel: [277113.497218]  [<ffffffff8020e146>] 
do_IRQ+0x4e/0x9a
Apr 11 04:22:29 xenserver kernel: [277113.497227]  [<ffffffff8037db2c>] 
evtchn_do_upcall+0x13c/0x1fc
Apr 11 04:22:29 xenserver kernel: [277113.497237]  [<ffffffff8020bbde>] 
do_hypervisor_callback+0x1e/0x30
Apr 11 04:22:29 xenserver kernel: [277113.497245]  <EOI>  
[<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
Apr 11 04:22:29 xenserver kernel: [277113.497263]  [<ffffffff8020a0ce>] 
xen_idle+0x2e/0x66
Apr 11 04:22:29 xenserver kernel: [277113.497271]  [<ffffffff80209d49>] 
cpu_idle+0x97/0xb9
Apr 11 04:22:29 xenserver kernel: [277113.497283]
Apr 11 04:22:29 xenserver kernel: [277113.497288] handlers:
Apr 11 04:22:29 xenserver kernel: [277113.497295] [<ffffffffa0099969>] 
(ahci_interrupt+0x0/0x46b [ahci])
Apr 11 04:22:29 xenserver kernel: [277113.497312] Disabling IRQ #22
Apr 11 04:22:54 xenserver kernel: [277138.645875] hdc: dma_timer_expiry: 
dma status == 0x64
Apr 11 04:23:00 xenserver kernel: [277145.647409] ata2.00: exception 
Emask 0x40 SAct 0x0 SErr 0x800 action 0x6 frozen
Apr 11 04:23:00 xenserver kernel: [277145.647429] ata2: SError: { HostInt }
Apr 11 04:23:00 xenserver kernel: [277145.647438] ata2.00: cmd 
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Apr 11 04:23:00 xenserver kernel: [277145.647439]          res 
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: { DRDY }
Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting link
Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception 
Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd 
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Apr 11 04:23:00 xenserver kernel: [277145.647497]          res 
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: { DRDY }
Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting link
Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset failed 
(device not ready)
Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to HW 
bug, retry pmp=0
Apr 11 04:23:01 xenserver kernel: [277146.131400] ata1: softreset failed 
(device not ready)
Apr 11 04:23:01 xenserver kernel: [277146.131409] ata1: failed due to HW 
bug, retry pmp=0
Apr 11 04:23:01 xenserver kernel: [277146.295250] ata1: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:23:01 xenserver kernel: [277146.295287] ata2: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:23:03 xenserver kernel: [277148.647303] hdc: DMA interrupt 
recovery
Apr 11 04:23:03 xenserver kernel: [277148.647316] hdc: lost interrupt
Apr 11 04:23:24 xenserver kernel: [277168.647352] hdc: dma_timer_expiry: 
dma status == 0x64
Apr 11 04:23:31 xenserver kernel: [277176.295271] ata2.00: qc timeout 
(cmd 0xec)
Apr 11 04:23:31 xenserver kernel: [277176.295292] ata2.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:23:31 xenserver kernel: [277176.295301] ata2.00: revalidation 
failed (errno=-5)
Apr 11 04:23:31 xenserver kernel: [277176.295310] ata2: failed to 
recover some devices, retrying in 5 secs
Apr 11 04:23:31 xenserver kernel: [277176.295322] ata1.00: qc timeout 
(cmd 0xec)
Apr 11 04:23:31 xenserver kernel: [277176.295336] ata1.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:23:31 xenserver kernel: [277176.295344] ata1.00: revalidation 
failed (errno=-5)
Apr 11 04:23:31 xenserver kernel: [277176.295352] ata1: failed to 
recover some devices, retrying in 5 secs
Apr 11 04:23:33 xenserver kernel: [277178.647242] hdc: DMA interrupt 
recovery
Apr 11 04:23:33 xenserver kernel: [277178.647255] hdc: lost interrupt
Apr 11 04:23:36 xenserver kernel: [277181.299161] ata1: hard resetting link
Apr 11 04:23:36 xenserver kernel: [277181.299185] ata2: hard resetting link
Apr 11 04:23:37 xenserver kernel: [277181.783130] ata2: softreset failed 
(device not ready)
Apr 11 04:23:37 xenserver kernel: [277181.783147] ata2: failed due to HW 
bug, retry pmp=0
Apr 11 04:23:37 xenserver kernel: [277181.783170] ata1: softreset failed 
(device not ready)
Apr 11 04:23:37 xenserver kernel: [277181.783178] ata1: failed due to HW 
bug, retry pmp=0
Apr 11 04:23:37 xenserver kernel: [277181.947140] ata1: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:23:37 xenserver kernel: [277181.947177] ata2: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:23:54 xenserver kernel: [277198.648158] hdc: dma_timer_expiry: 
dma status == 0x64
Apr 11 04:24:03 xenserver kernel: [277208.648119] hdc: DMA interrupt 
recovery
Apr 11 04:24:03 xenserver kernel: [277208.648131] hdc: lost interrupt
Apr 11 04:24:05 xenserver kernel: [277210.437511] NETDEV WATCHDOG: 
peth0: transmit timed out
Apr 11 04:24:05 xenserver kernel: [277210.437533] ------------[ cut here 
]------------
Apr 11 04:24:05 xenserver kernel: [277210.437541] WARNING: at 
net/sched/sch_generic.c:222 dev_watchdog+0xa6/0x119()
Apr 11 04:24:05 xenserver kernel: [277210.437553] Modules linked in: 
xt_tcpudp xfs xt_physdev iptable_filter ip_tables x_tables bridge 
netloop ipv6 raid1 md_mod loop snd_hda_intel snd_pcm snd_timer snd 
psmouse soundcore pcspkr serio_raw snd_page_alloc i2c_piix4 i2c_core 
button wmi shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log 
dm_snapshot dm_mod ide_cd_mod cdrom ide_disk ide_pci_generic sd_mod 
atiixp ide_core ata_generic ahci ehci_hcd libata scsi_mod dock ohci_hcd 
r8169 thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
Apr 11 04:24:05 xenserver kernel: [277210.437736] Pid: 0, comm: swapper 
Not tainted 2.6.26-2-xen-amd64 #1
Apr 11 04:24:05 xenserver kernel: [277210.437744]
Apr 11 04:24:05 xenserver kernel: [277210.437745] Call Trace:
Apr 11 04:24:05 xenserver kernel: [277210.437757]  <IRQ>  
[<ffffffff8022c9a2>] warn_on_slowpath+0x51/0x78
Apr 11 04:24:05 xenserver kernel: [277210.437774]  [<ffffffff8022d4f4>] 
printk+0x4e/0x56
Apr 11 04:24:05 xenserver kernel: [277210.437784]  [<ffffffff80235a70>] 
lock_timer_base+0x26/0x4b
Apr 11 04:24:05 xenserver kernel: [277210.437794]  [<ffffffff80235c16>] 
__mod_timer+0xd4/0xe3
Apr 11 04:24:05 xenserver kernel: [277210.437804]  [<ffffffff8023ca2f>] 
queue_delayed_work_on+0xb8/0xc8
Apr 11 04:24:05 xenserver kernel: [277210.437813]  [<ffffffff803d8161>] 
dev_watchdog+0x0/0x119
Apr 11 04:24:05 xenserver kernel: [277210.437822]  [<ffffffff803d8207>] 
dev_watchdog+0xa6/0x119
Apr 11 04:24:05 xenserver kernel: [277210.437831]  [<ffffffff802354df>] 
run_timer_softirq+0x190/0x237
Apr 11 04:24:05 xenserver kernel: [277210.437842]  [<ffffffff80231ad0>] 
__do_softirq+0x77/0x103
Apr 11 04:24:05 xenserver kernel: [277210.437852]  [<ffffffff8020c13c>] 
call_softirq+0x1c/0x28
Apr 11 04:24:05 xenserver kernel: [277210.437861]  [<ffffffff8020e092>] 
do_softirq+0x55/0xbb
Apr 11 04:24:05 xenserver kernel: [277210.437870]  [<ffffffff8020e175>] 
do_IRQ+0x7d/0x9a
Apr 11 04:24:05 xenserver kernel: [277210.437879]  [<ffffffff8037db2c>] 
evtchn_do_upcall+0x13c/0x1fc
Apr 11 04:24:05 xenserver kernel: [277210.437890]  [<ffffffff8020bbde>] 
do_hypervisor_callback+0x1e/0x30
Apr 11 04:24:05 xenserver kernel: [277210.437898]  <EOI>  
[<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
Apr 11 04:24:05 xenserver kernel: [277210.437916]  [<ffffffff8020a0ce>] 
xen_idle+0x2e/0x66
Apr 11 04:24:05 xenserver kernel: [277210.437924]  [<ffffffff80209d49>] 
cpu_idle+0x97/0xb9
Apr 11 04:24:05 xenserver kernel: [277210.437936]
Apr 11 04:24:05 xenserver kernel: [277210.437942] ---[ end trace 
470e1e7e4a9eb321 ]---
Apr 11 04:24:06 xenserver kernel: [277210.456294] r8169: peth0: link up
Apr 11 04:24:07 xenserver kernel: [277211.948161] ata2.00: qc timeout 
(cmd 0xec)
Apr 11 04:24:07 xenserver kernel: [277211.948182] ata2.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:24:07 xenserver kernel: [277211.948191] ata2.00: revalidation 
failed (errno=-5)
Apr 11 04:24:07 xenserver kernel: [277211.948199] ata2: failed to 
recover some devices, retrying in 5 secs
Apr 11 04:24:07 xenserver kernel: [277211.948211] ata1.00: qc timeout 
(cmd 0xec)
Apr 11 04:24:07 xenserver kernel: [277211.948225] ata1.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:24:07 xenserver kernel: [277211.948233] ata1.00: revalidation 
failed (errno=-5)
Apr 11 04:24:07 xenserver kernel: [277211.948240] ata1: failed to 
recover some devices, retrying in 5 secs
Apr 11 04:24:12 xenserver kernel: [277216.952165] ata1: hard resetting link
Apr 11 04:24:12 xenserver kernel: [277216.952189] ata2: hard resetting link
Apr 11 04:24:12 xenserver kernel: [277217.436134] ata1: softreset failed 
(device not ready)
Apr 11 04:24:12 xenserver kernel: [277217.436150] ata1: failed due to HW 
bug, retry pmp=0
Apr 11 04:24:12 xenserver kernel: [277217.436173] ata2: softreset failed 
(device not ready)
Apr 11 04:24:12 xenserver kernel: [277217.436181] ata2: failed due to HW 
bug, retry pmp=0
Apr 11 04:24:12 xenserver kernel: [277217.600168] ata2: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:24:12 xenserver kernel: [277217.600210] ata1: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:24:17 xenserver kernel: [277222.438507] NETDEV WATCHDOG: 
peth0: transmit timed out
Apr 11 04:24:18 xenserver kernel: [277222.456219] r8169: peth0: link up
Apr 11 04:24:24 xenserver kernel: [277228.648149] hdc: dma_timer_expiry: 
dma status == 0x64
Apr 11 04:24:29 xenserver kernel: [277234.439860] NETDEV WATCHDOG: 
peth0: transmit timed out
Apr 11 04:24:30 xenserver kernel: [277234.456241] r8169: peth0: link up
Apr 11 04:24:33 xenserver kernel: [277238.648044] hdc: DMA interrupt 
recovery
Apr 11 04:24:33 xenserver kernel: [277238.648058] hdc: lost interrupt
Apr 11 04:24:41 xenserver kernel: [277246.439954] NETDEV WATCHDOG: 
peth0: transmit timed out
Apr 11 04:24:42 xenserver kernel: [277246.456300] r8169: peth0: link up
Apr 11 04:24:42 xenserver kernel: [277246.633366] INFO: task 
md0_raid1:2194 blocked for more than 120 seconds.
Apr 11 04:24:42 xenserver kernel: [277246.633382] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 11 04:24:42 xenserver kernel: [277246.633394] md0_raid1     D 
ffffffff80449f00     0  2194      2
Apr 11 04:24:42 xenserver kernel: [277246.633408]  ffff8800e0169cc0 
0000000000000246 0000000000000000 ffff8800e2a2fac0
Apr 11 04:24:42 xenserver kernel: [277246.633429]  ffff8800e2ad2480 
ffffffff804fe460 ffff8800e2ad2700 00000000e02c2528
Apr 11 04:24:42 xenserver kernel: [277246.633450]  00000000ffffffff 
ffff880093da90c0 000000000000ffff 000000007470593f
Apr 11 04:24:42 xenserver kernel: [277246.633465] Call Trace:
Apr 11 04:24:42 xenserver kernel: [277246.634648]  [<ffffffff802ff7a6>] 
generic_make_request+0x2fe/0x339
Apr 11 04:24:42 xenserver kernel: [277246.634664]  [<ffffffffa0232413>] 
:md_mod:md_super_wait+0xee/0x107
Apr 11 04:24:42 xenserver kernel: [277246.634674]  [<ffffffff8023f4e5>] 
autoremove_wake_function+0x0/0x2e
Apr 11 04:24:42 xenserver kernel: [277246.634688]  [<ffffffffa02328a3>] 
:md_mod:md_update_sb+0x2a0/0x38f
Apr 11 04:24:42 xenserver kernel: [277246.634702]  [<ffffffffa0237016>] 
:md_mod:md_check_recovery+0x1c9/0x504
Apr 11 04:24:42 xenserver kernel: [277246.634714]  [<ffffffffa0246239>] 
:raid1:raid1d+0x26/0xd73
Apr 11 04:24:42 xenserver kernel: [277246.634724]  [<ffffffff804369b7>] 
error_exit+0x0/0x69
Apr 11 04:24:42 xenserver kernel: [277246.634734]  [<ffffffff804369b7>] 
error_exit+0x0/0x69
Apr 11 04:24:42 xenserver kernel: [277246.634744]  [<ffffffff80434efe>] 
schedule_timeout+0x1e/0xad
Apr 11 04:24:42 xenserver kernel: [277246.634754]  [<ffffffff8030dc91>] 
__next_cpu+0x19/0x26
Apr 11 04:24:42 xenserver kernel: [277246.634766]  [<ffffffffa02357d2>] 
:md_mod:md_thread+0xd7/0xed
Apr 11 04:24:42 xenserver kernel: [277246.634775]  [<ffffffff8023f4e5>] 
autoremove_wake_function+0x0/0x2e
Apr 11 04:24:42 xenserver kernel: [277246.634788]  [<ffffffffa02356fb>] 
:md_mod:md_thread+0x0/0xed
Apr 11 04:24:42 xenserver kernel: [277246.634796]  [<ffffffff8023f3b7>] 
kthread+0x47/0x74
Apr 11 04:24:42 xenserver kernel: [277246.634805]  [<ffffffff8022817c>] 
schedule_tail+0x27/0x5c
Apr 11 04:24:42 xenserver kernel: [277246.634814]  [<ffffffff8020be28>] 
child_rip+0xa/0x12
Apr 11 04:24:42 xenserver kernel: [277246.634825]  [<ffffffff8023f370>] 
kthread+0x0/0x74
Apr 11 04:24:42 xenserver kernel: [277246.634833]  [<ffffffff8020be1e>] 
child_rip+0x0/0x12
Apr 11 04:24:42 xenserver kernel: [277246.634841]
Apr 11 04:24:42 xenserver kernel: [277246.634856] INFO: task 
blkback.2.sda6:4518 blocked for more than 120 seconds.
Apr 11 04:24:42 xenserver kernel: [277246.634867] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 11 04:24:42 xenserver kernel: [277246.634878] blkback.2.sda D 
00000000000000ff     0  4518      2
Apr 11 04:24:42 xenserver kernel: [277246.634891]  ffff880093e999c0 
0000000000000246 0000000000000001 ffff8800e0169ee0
Apr 11 04:24:42 xenserver kernel: [277246.634911]  ffff8800e2057600 
ffff8800e2ad2480 ffff8800e2057880 0000000000000000
Apr 11 04:24:42 xenserver kernel: [277246.634932]  ffff880093e99980 
ffffffff8023f4ee ffff8800e23d3ec0 ffffffff802213e5
Apr 11 04:24:42 xenserver kernel: [277246.634947] Call Trace:
Apr 11 04:24:42 xenserver kernel: [277246.634961]  [<ffffffff8023f4ee>] 
autoremove_wake_function+0x9/0x2e
Apr 11 04:24:42 xenserver kernel: [277246.634970]  [<ffffffff802213e5>] 
__wake_up_common+0x41/0x74
Apr 11 04:24:42 xenserver kernel: [277246.634981]  [<ffffffff802d54cc>] 
sysfs_find_dirent+0x1b/0x2f
Apr 11 04:24:42 xenserver kernel: [277246.634994]  [<ffffffffa02364f3>] 
:md_mod:md_write_start+0x160/0x178
Apr 11 04:24:42 xenserver kernel: [277246.635003]  [<ffffffff8023f4e5>] 
autoremove_wake_function+0x0/0x2e
Apr 11 04:24:42 xenserver kernel: [277246.635016]  [<ffffffffa00f43f8>] 
:dm_mod:__split_bio+0x363/0x374
Apr 11 04:24:42 xenserver kernel: [277246.635026]  [<ffffffffa0246faf>] 
:raid1:make_request+0x29/0x5b6
Apr 11 04:24:42 xenserver kernel: [277246.635036]  [<ffffffff803117f7>] 
__up_read+0x13/0x8a
Apr 11 04:24:42 xenserver kernel: [277246.635046]  [<ffffffff802ff7a6>] 
generic_make_request+0x2fe/0x339
Apr 11 04:24:42 xenserver kernel: [277246.635056]  [<ffffffff80266983>] 
mempool_alloc+0x24/0xda
Apr 11 04:24:42 xenserver kernel: [277246.635067]  [<ffffffff80300a80>] 
submit_bio+0xdb/0xe2
Apr 11 04:24:42 xenserver kernel: [277246.635077]  [<ffffffff80386e68>] 
dispatch_rw_block_io+0x5b9/0x65d
Apr 11 04:24:42 xenserver kernel: [277246.635102]  [<ffffffff802ff83b>] 
blk_unplug+0x5a/0x60
Apr 11 04:24:42 xenserver kernel: [277246.635115]  [<ffffffffa00f5b19>] 
:dm_mod:dm_table_unplug_all+0x2a/0x3d
Apr 11 04:24:42 xenserver kernel: [277246.635125]  [<ffffffff8038735b>] 
blkif_schedule+0x3c7/0x4dc
Apr 11 04:24:42 xenserver kernel: [277246.635137]  [<ffffffff80386f94>] 
blkif_schedule+0x0/0x4dc
Apr 11 04:24:42 xenserver kernel: [277246.635146]  [<ffffffff8023f3b7>] 
kthread+0x47/0x74
Apr 11 04:24:42 xenserver kernel: [277246.635155]  [<ffffffff8022817c>] 
schedule_tail+0x27/0x5c
Apr 11 04:24:42 xenserver kernel: [277246.635163]  [<ffffffff8020be28>] 
child_rip+0xa/0x12
Apr 11 04:24:42 xenserver kernel: [277246.635174]  [<ffffffff8023f370>] 
kthread+0x0/0x74
Apr 11 04:24:42 xenserver kernel: [277246.635183]  [<ffffffff8020be1e>] 
child_rip+0x0/0x12
Apr 11 04:24:42 xenserver kernel: [277246.635191]
Apr 11 04:24:43 xenserver kernel: [277247.601345] ata1.00: qc timeout 
(cmd 0xec)
Apr 11 04:24:43 xenserver kernel: [277247.601366] ata1.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:24:43 xenserver kernel: [277247.601375] ata1.00: revalidation 
failed (errno=-5)
Apr 11 04:24:43 xenserver kernel: [277247.601383] ata1.00: disabled
Apr 11 04:24:43 xenserver kernel: [277247.601396] ata2.00: qc timeout 
(cmd 0xec)
Apr 11 04:24:43 xenserver kernel: [277247.601410] ata2.00: failed to 
IDENTIFY (I/O error, err_mask=0x4)
Apr 11 04:24:43 xenserver kernel: [277247.601418] ata2.00: revalidation 
failed (errno=-5)
Apr 11 04:24:43 xenserver kernel: [277247.601425] ata2.00: disabled
Apr 11 04:24:43 xenserver kernel: [277248.105346] ata2: hard resetting link
Apr 11 04:24:43 xenserver kernel: [277248.105372] ata1: hard resetting link
Apr 11 04:24:43 xenserver kernel: [277248.589302] ata1: softreset failed 
(device not ready)
Apr 11 04:24:43 xenserver kernel: [277248.589319] ata1: failed due to HW 
bug, retry pmp=0
Apr 11 04:24:43 xenserver kernel: [277248.589341] ata2: softreset failed 
(device not ready)
Apr 11 04:24:43 xenserver kernel: [277248.589350] ata2: failed due to HW 
bug, retry pmp=0
Apr 11 04:24:44 xenserver kernel: [277248.753334] ata2: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:24:44 xenserver kernel: [277248.753366] ata2: EH complete
Apr 11 04:24:44 xenserver kernel: [277248.753384] sd 1:0:0:0: [sdb] 
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
Apr 11 04:24:44 xenserver kernel: [277248.753399] end_request: I/O 
error, dev sdb, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753410] end_request: I/O 
error, dev sdb, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753421] md: super_written gets 
error=-5, uptodate=0
Apr 11 04:24:44 xenserver kernel: [277248.753431] raid1: Disk failure on 
sdb1, disabling device.
Apr 11 04:24:44 xenserver kernel: [277248.753432] raid1: Operation 
continuing on 1 devices.
Apr 11 04:24:44 xenserver kernel: [277248.753466] ata1: SATA link up 3.0 
Gbps (SStatus 123 SControl 300)
Apr 11 04:24:44 xenserver kernel: [277248.753479] ata1: EH complete
Apr 11 04:24:44 xenserver kernel: [277248.753490] sd 0:0:0:0: [sda] 
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
Apr 11 04:24:44 xenserver kernel: [277248.753505] end_request: I/O 
error, dev sda, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753515] end_request: I/O 
error, dev sda, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753524] md: super_written gets 
error=-5, uptodate=0
Apr 11 04:24:44 xenserver kernel: [277248.753551] sd 0:0:0:0: [sda] 
Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
Apr 11 04:24:44 xenserver kernel: [277248.753566] end_request: I/O 
error, dev sda, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753576] end_request: I/O 
error, dev sda, sector 1953519935
Apr 11 04:24:44 xenserver kernel: [277248.753585] md: super_written gets 
error=-5, uptodate=0
Apr 11 04:24:44 xenserver kernel: [277248.753596] RAID1 conf printout:
Apr 11 04:24:44 xenserver kernel: [277248.753603]  --- wd:1 rd:2
Apr 11 04:24:44 xenserver kernel: [277248.753610]  disk 0, wo:0, o:1, 
dev:sda1
Apr 11 04:24:44 xenserver kernel: [277248.753618]  disk 1, wo:1, o:0, 
dev:sdb1
Apr 11 04:24:44 xenserver kernel: [277248.765416] RAID1 conf printout:
Apr 11 04:24:44 xenserver kernel: [277248.765431]  --- wd:1 rd:2


 From lspci -v:

00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
        Subsystem: Advanced Micro Devices [AMD] RS780 Host Bridge
        Flags: bus master, 66MHz, medium devsel, latency 0
        Capabilities: [c4] HyperTransport: Slave or Primary Interface
        Capabilities: [54] HyperTransport: UnitID Clumping
        Capabilities: [40] HyperTransport: Retry Mode
        Capabilities: [9c] HyperTransport: #1a
        Capabilities: [f8] HyperTransport: #1c

00:01.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(int gfx) (prog-if 00 [Normal decode])
        Flags: bus master, 66MHz, medium devsel, latency 64
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
        I/O behind bridge: 0000b000-0000bfff
        Memory behind bridge: fe700000-fe8fffff
        Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
        Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] RS780 
PCI to PCI bridge (int gfx)
        Kernel modules: shpchp

00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(PCIE port 0) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
        I/O behind bridge: 0000c000-0000cfff
        Memory behind bridge: fe900000-fe9fffff
        Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Root Port (Slot+), MSI 00
        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit- 
Queue=0/0 Enable-
        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] 
Device 9600
        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [100] Vendor Specific Information <?>
        Capabilities: [110] Virtual Channel <?>
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

00:07.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge 
(PCIE port 3) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
        I/O behind bridge: 0000d000-0000dfff
        Memory behind bridge: fea00000-feafffff
        Prefetchable memory behind bridge: 00000000dff00000-00000000dfffffff
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Root Port (Slot+), MSI 00
        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit- 
Queue=0/0 Enable-
        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] 
Device 9600
        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
        Capabilities: [100] Vendor Specific Information <?>
        Capabilities: [110] Virtual Channel <?>
        Kernel driver in use: pcieport-driver
        Kernel modules: shpchp

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA 
Controller [IDE mode] (prog-if 01 [AHCI 1.0])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
        I/O ports at a000 [size=8]
        I/O ports at 9000 [size=4]
        I/O ports at 8000 [size=8]
        I/O ports at 7000 [size=4]
        I/O ports at 6000 [size=16]
        Memory at fe6ff800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [60] Power Management version 2
        Capabilities: [70] SATA HBA <?>
        Kernel driver in use: ahci
        Kernel modules: ahci

00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 
Controller (prog-if 10 [OHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
        Memory at fe6fe000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ohci_hcd
        Kernel modules: ohci-hcd

00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller 
(prog-if 10 [OHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
        Memory at fe6fd000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ohci_hcd
        Kernel modules: ohci-hcd

00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI 
Controller (prog-if 20 [EHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
        Memory at fe6ff000 (32-bit, non-prefetchable) [size=256]
        Capabilities: [c0] Power Management version 2
        Capabilities: [e4] Debug port: BAR=1 offset=00e0
        Kernel driver in use: ehci_hcd
        Kernel modules: ehci-hcd

00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 
Controller (prog-if 10 [OHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
        Memory at fe6fc000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ohci_hcd
        Kernel modules: ohci-hcd

00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller 
(prog-if 10 [OHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
        Memory at fe6fb000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ohci_hcd
        Kernel modules: ohci-hcd

00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI 
Controller (prog-if 20 [EHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
        Memory at fe6fa800 (32-bit, non-prefetchable) [size=256]
        Capabilities: [c0] Power Management version 2
        Capabilities: [e4] Debug port: BAR=1 offset=00e0
        Kernel driver in use: ehci_hcd
        Kernel modules: ehci-hcd

00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: 66MHz, medium devsel
        Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c-piix4

00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller 
(prog-if 8a [Master SecP PriP])
        Subsystem: ATI Technologies Inc SB700/SB800 IDE Controller
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 16
        I/O ports at 01f0 [size=8]
        I/O ports at 03f4 [size=1]
        I/O ports at 0170 [size=8]
        I/O ports at 0374 [size=1]
        I/O ports at ff00 [size=16]
        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit- 
Queue=0/1 Enable-
        Kernel driver in use: ATIIXP_IDE
        Kernel modules: atiixp, ide-pci-generic, ata_generic

00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
        Subsystem: Biostar Microtech Int''l Corp Device 8218
        Flags: bus master, slow devsel, latency 64, IRQ 16
        Memory at fe6f4000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 2
        Kernel driver in use: HDA Intel
        Kernel modules: snd-hda-intel

00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 0

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge 
(prog-if 01 [Subtractive decode])
        Flags: bus master, 66MHz, medium devsel, latency 64
        Bus: primary=00, secondary=04, subordinate=05, sec-latency=64
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: feb00000-febfffff
        Prefetchable memory behind bridge: f0000000-fbffffff

00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 
Controller (prog-if 10 [OHCI])
        Subsystem: Biostar Microtech Int''l Corp Device 3700
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
        Memory at fe6f9000 (32-bit, non-prefetchable) [size=4K]
        Kernel driver in use: ohci_hcd
        Kernel modules: ohci-hcd

00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, 
Athlon64, Sempron] HyperTransport Configuration
        Flags: fast devsel
        Capabilities: [80] HyperTransport: Host or Secondary Interface

00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, 
Athlon64, Sempron] Address Map
        Flags: fast devsel

00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, 
Athlon64, Sempron] DRAM Controller
        Flags: fast devsel

00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, 
Athlon64, Sempron] Miscellaneous Control
        Flags: fast devsel
        Capabilities: [f0] Secure device <?>

00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron, 
Athlon64, Sempron] Link Control
        Flags: fast devsel

01:05.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3300 
Graphics (prog-if 00 [VGA controller])
        Subsystem: Biostar Microtech Int''l Corp Device 0217
        Flags: bus master, fast devsel, latency 0, IRQ 7
        Memory at c0000000 (32-bit, prefetchable) [size=256M]
        I/O ports at b000 [size=256]
        Memory at fe8f0000 (32-bit, non-prefetchable) [size=64K]
        Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [50] Power Management version 3
        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/0 Enable-

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI 
Express Gigabit Ethernet controller
        Flags: bus master, fast devsel, latency 0, IRQ 16
        I/O ports at c800 [size=256]
        Memory at fe9ff000 (64-bit, non-prefetchable) [size=4K]
        [virtual] Expansion ROM at d0000000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 2
        Capabilities: [48] Vital Product Data <?>
        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/1 Enable-
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [84] Vendor Specific Information <?>
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [12c] Virtual Channel <?>
        Capabilities: [148] Device Serial Number 00-e0-4c-68-00-00-00-fd
        Capabilities: [154] Power Budgeting <?>
        Kernel driver in use: pciback
        Kernel modules: r8169

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. 
RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
        Subsystem: Biostar Microtech Int''l Corp Device 2309
        Flags: bus master, fast devsel, latency 0, IRQ 19
        I/O ports at d800 [size=256]
        Memory at dffff000 (64-bit, prefetchable) [size=4K]
        Memory at dfff8000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at feae0000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ 
Queue=0/0 Enable-
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4
        Capabilities: [cc] Vital Product Data <?>
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Virtual Channel <?>
        Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-02-61
        Kernel driver in use: r8169
        Kernel modules: r8169

04:05.0 PCI bridge: Pericom Semiconductor Device 8140 (prog-if 00 
[Normal decode])
        Flags: bus master, 66MHz, medium devsel, latency 64
        Bus: primary=04, secondary=05, subordinate=05, sec-latency=64
        Prefetchable memory behind bridge: 00000000f4000000-00000000fbffffff
        Capabilities: [80] Power Management version 2
        Capabilities: [90] CompactPCI hot-swap <?>
        Kernel modules: shpchp

04:06.0 Multimedia video controller: Internext Compression Inc iTVC16 
(CX23416) MPEG-2 Encoder (rev 01)
        Subsystem: Hauppauge computer works Inc. WinTV PVR 250
        Flags: medium devsel, IRQ 21
        Memory at f0000000 (32-bit, prefetchable) [disabled] [size=64M]
        Capabilities: [44] Power Management version 2
        Kernel driver in use: pciback
        Kernel modules: ivtv

04:07.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet 
Adapter (rev 11) (rev 11)
        Subsystem: D-Link System Inc DGE-530T Gigabit Ethernet Adapter 
(rev 11)
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
        Memory at febfc000 (32-bit, non-prefetchable) [size=16K]
        I/O ports at e800 [size=256]
        Expansion ROM at feb00000 [disabled] [size=128K]
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data <?>
        Kernel driver in use: pciback
        Kernel modules: skge

05:08.0 Multimedia video controller: Internext Compression Inc iTVC16 
(CX23416) MPEG-2 Encoder (rev 01)
        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (1st unit)
        Flags: medium devsel, IRQ 20
        Memory at f8000000 (32-bit, prefetchable) [disabled] [size=64M]
        Capabilities: [44] Power Management version 2
        Kernel driver in use: pciback
        Kernel modules: ivtv

05:09.0 Multimedia video controller: Internext Compression Inc iTVC16 
(CX23416) MPEG-2 Encoder (rev 01)
        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (2nd unit)
        Flags: medium devsel, IRQ 21
        Memory at f4000000 (32-bit, prefetchable) [disabled] [size=64M]
        Capabilities: [44] Power Management version 2
        Kernel driver in use: pciback
        Kernel modules: ivtv


 From /proc/interrupts:

           CPU0       CPU1       CPU2       CPU3      
  1:         58          0          0          0  Phys-irq-level     i8042
  8:          0          0          0          0  Phys-irq-level     rtc0
  9:          0          0          0          0  Phys-irq-level     acpi
 12:          4          0          0          0  Phys-irq-level     i8042
 14:          0          0          0          0  Phys-irq-level     ide0
 15:      59700          0          0          0  Phys-irq-level     ide1
 16:     339121          0          0          0  Phys-irq-level     
ohci_hcd:usb1, ohci_hcd:usb2, HDA Intel
 17:          0          0          0          0  Phys-irq-level     
ehci_hcd:usb5
 18:          3          0          0          0  Phys-irq-level     
ohci_hcd:usb3, ohci_hcd:usb4, ohci_hcd:usb7
 19:     174793          0          0          0  Phys-irq-level     
ehci_hcd:usb6, peth0
 22:     645023          0          0          0  Phys-irq-level     ahci
256:     450281          0          0          0  Dynamic-irq-level     
timer0
257:       4704          0          0          0  Dynamic-irq-level     
resched0
258:         43          0          0          0  Dynamic-irq-level     
callfunc0
259:        731          0          0          0  Dynamic-irq-level     
spinlock0
260:          0       2805          0          0  Dynamic-irq-level     
resched1
261:          0        189          0          0  Dynamic-irq-level     
callfunc1
262:          0        543          0          0  Dynamic-irq-level     
spinlock1
263:          0      58565          0          0  Dynamic-irq-level     
timer1
264:          0          0       4449          0  Dynamic-irq-level     
resched2
265:          0          0        177          0  Dynamic-irq-level     
callfunc2
266:          0          0        570          0  Dynamic-irq-level     
spinlock2
267:          0          0      43025          0  Dynamic-irq-level     
timer2
268:          0          0          0       3650  Dynamic-irq-level     
resched3
269:          0          0          0        176  Dynamic-irq-level     
callfunc3
270:          0          0          0        652  Dynamic-irq-level     
spinlock3
271:          0          0          0      24529  Dynamic-irq-level     
timer3
272:       5815          0          0          0  Dynamic-irq-level     
xenbus
273:          0          0          0          0  Dynamic-irq-level     
console
274:          0          0          0          0  Dynamic-irq-level     
suspend
275:        406          0          0          0  Dynamic-irq-level     
pciback
276:         36          0          0          0  Dynamic-irq-level     
blkif-backend
277:        162          0          0          0  Dynamic-irq-level     
blkif-backend
278:       1939          0          0          0  Dynamic-irq-level     
blkif-backend
279:     178850          0          0          0  Dynamic-irq-level     
vif1.0
280:      28670          0          0          0  Dynamic-irq-level     
vif1.1
281:       4372          0          0          0  Dynamic-irq-level     
blkif-backend
282:       1227          0          0          0  Dynamic-irq-level     
blkif-backend
283:       8079          0          0          0  Dynamic-irq-level     
blkif-backend
284:        177          0          0          0  Dynamic-irq-level     
blkif-backend
285:        106          0          0          0  Dynamic-irq-level     
blkif-backend
286:      43937          0          0          0  Dynamic-irq-level     
blkif-backend
287:      28211          0          0          0  Dynamic-irq-level     
vif2.0
288:       2950          0          0          0  Dynamic-irq-level     
blkif-backend
289:       5669          0          0          0  Dynamic-irq-level     
blkif-backend
290:         83          0          0          0  Dynamic-irq-level     
blkif-backend
291:        148          0          0          0  Dynamic-irq-level     
blkif-backend
292:         40          0          0          0  Dynamic-irq-level     
blkif-backend
293:      98438          0          0          0  Dynamic-irq-level     
blkif-backend
294:      64518          0          0          0  Dynamic-irq-level     
vif3.0
NMI:          0          0          0          0   Non-maskable interrupts
RES:       4704       2805       4449       3650   Rescheduling interrupts
CAL:         43        189        177        176   function call interrupts


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-12 06:31 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

On Sun, Apr 11, 2010 at 01:12:09PM -0700, Ben Holt
wrote:> Hello,
>
> I have experienced a dom0 crash where the system became unreachable via  
> the network and the console was unresponsive.  I would appreciate help  
> interpretting the logs and any configuration change suggestions.
>
> It is a stock Debian Lenny dom0 running xen 3.2.1 with kernel  
> 2.6.26-2-xen-amd64 and an AMD Athlon IIx4 with 4 GB of RAM.  It is  
> running 4 VMs.  One VM has two PCI NICs being passed to it, while  
> another has three PCI MPEG encoders passed to it (physically they are  
> only two cards, but one, a hauppage pvr-500, has two encoders on it  
> connected via a PCI bridge).  Three of the VMs were migrated from  
> another machine and are running Debian kernel 2.6.26-2-xen-686 (32 bit),  
> the fourth is running the same kernel as the dom0 (64 bit).
>
> As you''ll see from the logs the system has three hard drives, a
PATA
> (hdc) that holds the dom0 and VMs OS file systems, and 2 SATA (sda, sdb)  
> that are configured in a software RAID 1 and hold the VMs data  
> filesystems.  If I am reading things right all three drives appear to  
> experience interrupt / DMA issues during the crash.  The peth0 NIC goes  
> down and up during the crash.  There is also a mention of IRQ 22, which  
> is used by the SATA controller.
>
> Any pointers or suggests are greatly appreciated.
>
>
>
> From /var/log/syslog:
>
> Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody cared  
> (try booting with the "irqpoll" option)
> Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm: swapper  
> Not tainted 2.6.26-2-xen-amd64 #1
>
So I assume this is the host/dom0 kernel.

What device is using IRQ 22? 
Try checking from /proc/interrupts or from lspci.

Is it a shared IRQ (multiple devices using the same IRQ)? 

-- Pasi

> Apr 11 04:22:29 xenserver kernel: [277113.497152]
> Apr 11 04:22:29 xenserver kernel: [277113.497153] Call Trace:
> Apr 11 04:22:29 xenserver kernel: [277113.497165]  <IRQ>   
> [<ffffffff8037ce18>] irq_ignore_unhandled+0x1c/0x32
> Apr 11 04:22:29 xenserver kernel: [277113.497188] 
[<ffffffff8025f88f>]
> __report_bad_irq+0x30/0x72
> Apr 11 04:22:29 xenserver kernel: [277113.497197] 
[<ffffffff8025fb58>]
> note_interrupt+0x287/0x2c7
> Apr 11 04:22:29 xenserver kernel: [277113.497207] 
[<ffffffff80260440>]
> handle_level_irq+0xc3/0x118
> Apr 11 04:22:29 xenserver kernel: [277113.497218] 
[<ffffffff8020e146>]
> do_IRQ+0x4e/0x9a
> Apr 11 04:22:29 xenserver kernel: [277113.497227] 
[<ffffffff8037db2c>]
> evtchn_do_upcall+0x13c/0x1fc
> Apr 11 04:22:29 xenserver kernel: [277113.497237] 
[<ffffffff8020bbde>]
> do_hypervisor_callback+0x1e/0x30
> Apr 11 04:22:29 xenserver kernel: [277113.497245]  <EOI>   
> [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
> Apr 11 04:22:29 xenserver kernel: [277113.497263] 
[<ffffffff8020a0ce>]
> xen_idle+0x2e/0x66
> Apr 11 04:22:29 xenserver kernel: [277113.497271] 
[<ffffffff80209d49>]
> cpu_idle+0x97/0xb9
> Apr 11 04:22:29 xenserver kernel: [277113.497283]
> Apr 11 04:22:29 xenserver kernel: [277113.497288] handlers:
> Apr 11 04:22:29 xenserver kernel: [277113.497295]
[<ffffffffa0099969>]
> (ahci_interrupt+0x0/0x46b [ahci])
> Apr 11 04:22:29 xenserver kernel: [277113.497312] Disabling IRQ #22
> Apr 11 04:22:54 xenserver kernel: [277138.645875] hdc: dma_timer_expiry:  
> dma status == 0x64
> Apr 11 04:23:00 xenserver kernel: [277145.647409] ata2.00: exception  
> Emask 0x40 SAct 0x0 SErr 0x800 action 0x6 frozen
> Apr 11 04:23:00 xenserver kernel: [277145.647429] ata2: SError: { HostInt }
> Apr 11 04:23:00 xenserver kernel: [277145.647438] ata2.00: cmd  
> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> Apr 11 04:23:00 xenserver kernel: [277145.647439]          res  
> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
> Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: { DRDY }
> Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting link
> Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception  
> Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
> Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd  
> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> Apr 11 04:23:00 xenserver kernel: [277145.647497]          res  
> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: { DRDY }
> Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting link
> Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset failed  
> (device not ready)
> Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:23:01 xenserver kernel: [277146.131400] ata1: softreset failed  
> (device not ready)
> Apr 11 04:23:01 xenserver kernel: [277146.131409] ata1: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:23:01 xenserver kernel: [277146.295250] ata1: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:23:01 xenserver kernel: [277146.295287] ata2: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:23:03 xenserver kernel: [277148.647303] hdc: DMA interrupt  
> recovery
> Apr 11 04:23:03 xenserver kernel: [277148.647316] hdc: lost interrupt
> Apr 11 04:23:24 xenserver kernel: [277168.647352] hdc: dma_timer_expiry:  
> dma status == 0x64
> Apr 11 04:23:31 xenserver kernel: [277176.295271] ata2.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:23:31 xenserver kernel: [277176.295292] ata2.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:23:31 xenserver kernel: [277176.295301] ata2.00: revalidation  
> failed (errno=-5)
> Apr 11 04:23:31 xenserver kernel: [277176.295310] ata2: failed to  
> recover some devices, retrying in 5 secs
> Apr 11 04:23:31 xenserver kernel: [277176.295322] ata1.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:23:31 xenserver kernel: [277176.295336] ata1.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:23:31 xenserver kernel: [277176.295344] ata1.00: revalidation  
> failed (errno=-5)
> Apr 11 04:23:31 xenserver kernel: [277176.295352] ata1: failed to  
> recover some devices, retrying in 5 secs
> Apr 11 04:23:33 xenserver kernel: [277178.647242] hdc: DMA interrupt  
> recovery
> Apr 11 04:23:33 xenserver kernel: [277178.647255] hdc: lost interrupt
> Apr 11 04:23:36 xenserver kernel: [277181.299161] ata1: hard resetting link
> Apr 11 04:23:36 xenserver kernel: [277181.299185] ata2: hard resetting link
> Apr 11 04:23:37 xenserver kernel: [277181.783130] ata2: softreset failed  
> (device not ready)
> Apr 11 04:23:37 xenserver kernel: [277181.783147] ata2: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:23:37 xenserver kernel: [277181.783170] ata1: softreset failed  
> (device not ready)
> Apr 11 04:23:37 xenserver kernel: [277181.783178] ata1: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:23:37 xenserver kernel: [277181.947140] ata1: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:23:37 xenserver kernel: [277181.947177] ata2: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:23:54 xenserver kernel: [277198.648158] hdc: dma_timer_expiry:  
> dma status == 0x64
> Apr 11 04:24:03 xenserver kernel: [277208.648119] hdc: DMA interrupt  
> recovery
> Apr 11 04:24:03 xenserver kernel: [277208.648131] hdc: lost interrupt
> Apr 11 04:24:05 xenserver kernel: [277210.437511] NETDEV WATCHDOG:  
> peth0: transmit timed out
> Apr 11 04:24:05 xenserver kernel: [277210.437533] ------------[ cut here  
> ]------------
> Apr 11 04:24:05 xenserver kernel: [277210.437541] WARNING: at  
> net/sched/sch_generic.c:222 dev_watchdog+0xa6/0x119()
> Apr 11 04:24:05 xenserver kernel: [277210.437553] Modules linked in:  
> xt_tcpudp xfs xt_physdev iptable_filter ip_tables x_tables bridge  
> netloop ipv6 raid1 md_mod loop snd_hda_intel snd_pcm snd_timer snd  
> psmouse soundcore pcspkr serio_raw snd_page_alloc i2c_piix4 i2c_core  
> button wmi shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log  
> dm_snapshot dm_mod ide_cd_mod cdrom ide_disk ide_pci_generic sd_mod  
> atiixp ide_core ata_generic ahci ehci_hcd libata scsi_mod dock ohci_hcd  
> r8169 thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
> Apr 11 04:24:05 xenserver kernel: [277210.437736] Pid: 0, comm: swapper  
> Not tainted 2.6.26-2-xen-amd64 #1
> Apr 11 04:24:05 xenserver kernel: [277210.437744]
> Apr 11 04:24:05 xenserver kernel: [277210.437745] Call Trace:
> Apr 11 04:24:05 xenserver kernel: [277210.437757]  <IRQ>   
> [<ffffffff8022c9a2>] warn_on_slowpath+0x51/0x78
> Apr 11 04:24:05 xenserver kernel: [277210.437774] 
[<ffffffff8022d4f4>]
> printk+0x4e/0x56
> Apr 11 04:24:05 xenserver kernel: [277210.437784] 
[<ffffffff80235a70>]
> lock_timer_base+0x26/0x4b
> Apr 11 04:24:05 xenserver kernel: [277210.437794] 
[<ffffffff80235c16>]
> __mod_timer+0xd4/0xe3
> Apr 11 04:24:05 xenserver kernel: [277210.437804] 
[<ffffffff8023ca2f>]
> queue_delayed_work_on+0xb8/0xc8
> Apr 11 04:24:05 xenserver kernel: [277210.437813] 
[<ffffffff803d8161>]
> dev_watchdog+0x0/0x119
> Apr 11 04:24:05 xenserver kernel: [277210.437822] 
[<ffffffff803d8207>]
> dev_watchdog+0xa6/0x119
> Apr 11 04:24:05 xenserver kernel: [277210.437831] 
[<ffffffff802354df>]
> run_timer_softirq+0x190/0x237
> Apr 11 04:24:05 xenserver kernel: [277210.437842] 
[<ffffffff80231ad0>]
> __do_softirq+0x77/0x103
> Apr 11 04:24:05 xenserver kernel: [277210.437852] 
[<ffffffff8020c13c>]
> call_softirq+0x1c/0x28
> Apr 11 04:24:05 xenserver kernel: [277210.437861] 
[<ffffffff8020e092>]
> do_softirq+0x55/0xbb
> Apr 11 04:24:05 xenserver kernel: [277210.437870] 
[<ffffffff8020e175>]
> do_IRQ+0x7d/0x9a
> Apr 11 04:24:05 xenserver kernel: [277210.437879] 
[<ffffffff8037db2c>]
> evtchn_do_upcall+0x13c/0x1fc
> Apr 11 04:24:05 xenserver kernel: [277210.437890] 
[<ffffffff8020bbde>]
> do_hypervisor_callback+0x1e/0x30
> Apr 11 04:24:05 xenserver kernel: [277210.437898]  <EOI>   
> [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
> Apr 11 04:24:05 xenserver kernel: [277210.437916] 
[<ffffffff8020a0ce>]
> xen_idle+0x2e/0x66
> Apr 11 04:24:05 xenserver kernel: [277210.437924] 
[<ffffffff80209d49>]
> cpu_idle+0x97/0xb9
> Apr 11 04:24:05 xenserver kernel: [277210.437936]
> Apr 11 04:24:05 xenserver kernel: [277210.437942] ---[ end trace  
> 470e1e7e4a9eb321 ]---
> Apr 11 04:24:06 xenserver kernel: [277210.456294] r8169: peth0: link up
> Apr 11 04:24:07 xenserver kernel: [277211.948161] ata2.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:24:07 xenserver kernel: [277211.948182] ata2.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:24:07 xenserver kernel: [277211.948191] ata2.00: revalidation  
> failed (errno=-5)
> Apr 11 04:24:07 xenserver kernel: [277211.948199] ata2: failed to  
> recover some devices, retrying in 5 secs
> Apr 11 04:24:07 xenserver kernel: [277211.948211] ata1.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:24:07 xenserver kernel: [277211.948225] ata1.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:24:07 xenserver kernel: [277211.948233] ata1.00: revalidation  
> failed (errno=-5)
> Apr 11 04:24:07 xenserver kernel: [277211.948240] ata1: failed to  
> recover some devices, retrying in 5 secs
> Apr 11 04:24:12 xenserver kernel: [277216.952165] ata1: hard resetting link
> Apr 11 04:24:12 xenserver kernel: [277216.952189] ata2: hard resetting link
> Apr 11 04:24:12 xenserver kernel: [277217.436134] ata1: softreset failed  
> (device not ready)
> Apr 11 04:24:12 xenserver kernel: [277217.436150] ata1: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:24:12 xenserver kernel: [277217.436173] ata2: softreset failed  
> (device not ready)
> Apr 11 04:24:12 xenserver kernel: [277217.436181] ata2: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:24:12 xenserver kernel: [277217.600168] ata2: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:24:12 xenserver kernel: [277217.600210] ata1: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:24:17 xenserver kernel: [277222.438507] NETDEV WATCHDOG:  
> peth0: transmit timed out
> Apr 11 04:24:18 xenserver kernel: [277222.456219] r8169: peth0: link up
> Apr 11 04:24:24 xenserver kernel: [277228.648149] hdc: dma_timer_expiry:  
> dma status == 0x64
> Apr 11 04:24:29 xenserver kernel: [277234.439860] NETDEV WATCHDOG:  
> peth0: transmit timed out
> Apr 11 04:24:30 xenserver kernel: [277234.456241] r8169: peth0: link up
> Apr 11 04:24:33 xenserver kernel: [277238.648044] hdc: DMA interrupt  
> recovery
> Apr 11 04:24:33 xenserver kernel: [277238.648058] hdc: lost interrupt
> Apr 11 04:24:41 xenserver kernel: [277246.439954] NETDEV WATCHDOG:  
> peth0: transmit timed out
> Apr 11 04:24:42 xenserver kernel: [277246.456300] r8169: peth0: link up
> Apr 11 04:24:42 xenserver kernel: [277246.633366] INFO: task  
> md0_raid1:2194 blocked for more than 120 seconds.
> Apr 11 04:24:42 xenserver kernel: [277246.633382] "echo 0 >  
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 11 04:24:42 xenserver kernel: [277246.633394] md0_raid1     D  
> ffffffff80449f00     0  2194      2
> Apr 11 04:24:42 xenserver kernel: [277246.633408]  ffff8800e0169cc0  
> 0000000000000246 0000000000000000 ffff8800e2a2fac0
> Apr 11 04:24:42 xenserver kernel: [277246.633429]  ffff8800e2ad2480  
> ffffffff804fe460 ffff8800e2ad2700 00000000e02c2528
> Apr 11 04:24:42 xenserver kernel: [277246.633450]  00000000ffffffff  
> ffff880093da90c0 000000000000ffff 000000007470593f
> Apr 11 04:24:42 xenserver kernel: [277246.633465] Call Trace:
> Apr 11 04:24:42 xenserver kernel: [277246.634648] 
[<ffffffff802ff7a6>]
> generic_make_request+0x2fe/0x339
> Apr 11 04:24:42 xenserver kernel: [277246.634664] 
[<ffffffffa0232413>]
> :md_mod:md_super_wait+0xee/0x107
> Apr 11 04:24:42 xenserver kernel: [277246.634674] 
[<ffffffff8023f4e5>]
> autoremove_wake_function+0x0/0x2e
> Apr 11 04:24:42 xenserver kernel: [277246.634688] 
[<ffffffffa02328a3>]
> :md_mod:md_update_sb+0x2a0/0x38f
> Apr 11 04:24:42 xenserver kernel: [277246.634702] 
[<ffffffffa0237016>]
> :md_mod:md_check_recovery+0x1c9/0x504
> Apr 11 04:24:42 xenserver kernel: [277246.634714] 
[<ffffffffa0246239>]
> :raid1:raid1d+0x26/0xd73
> Apr 11 04:24:42 xenserver kernel: [277246.634724] 
[<ffffffff804369b7>]
> error_exit+0x0/0x69
> Apr 11 04:24:42 xenserver kernel: [277246.634734] 
[<ffffffff804369b7>]
> error_exit+0x0/0x69
> Apr 11 04:24:42 xenserver kernel: [277246.634744] 
[<ffffffff80434efe>]
> schedule_timeout+0x1e/0xad
> Apr 11 04:24:42 xenserver kernel: [277246.634754] 
[<ffffffff8030dc91>]
> __next_cpu+0x19/0x26
> Apr 11 04:24:42 xenserver kernel: [277246.634766] 
[<ffffffffa02357d2>]
> :md_mod:md_thread+0xd7/0xed
> Apr 11 04:24:42 xenserver kernel: [277246.634775] 
[<ffffffff8023f4e5>]
> autoremove_wake_function+0x0/0x2e
> Apr 11 04:24:42 xenserver kernel: [277246.634788] 
[<ffffffffa02356fb>]
> :md_mod:md_thread+0x0/0xed
> Apr 11 04:24:42 xenserver kernel: [277246.634796] 
[<ffffffff8023f3b7>]
> kthread+0x47/0x74
> Apr 11 04:24:42 xenserver kernel: [277246.634805] 
[<ffffffff8022817c>]
> schedule_tail+0x27/0x5c
> Apr 11 04:24:42 xenserver kernel: [277246.634814] 
[<ffffffff8020be28>]
> child_rip+0xa/0x12
> Apr 11 04:24:42 xenserver kernel: [277246.634825] 
[<ffffffff8023f370>]
> kthread+0x0/0x74
> Apr 11 04:24:42 xenserver kernel: [277246.634833] 
[<ffffffff8020be1e>]
> child_rip+0x0/0x12
> Apr 11 04:24:42 xenserver kernel: [277246.634841]
> Apr 11 04:24:42 xenserver kernel: [277246.634856] INFO: task  
> blkback.2.sda6:4518 blocked for more than 120 seconds.
> Apr 11 04:24:42 xenserver kernel: [277246.634867] "echo 0 >  
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 11 04:24:42 xenserver kernel: [277246.634878] blkback.2.sda D  
> 00000000000000ff     0  4518      2
> Apr 11 04:24:42 xenserver kernel: [277246.634891]  ffff880093e999c0  
> 0000000000000246 0000000000000001 ffff8800e0169ee0
> Apr 11 04:24:42 xenserver kernel: [277246.634911]  ffff8800e2057600  
> ffff8800e2ad2480 ffff8800e2057880 0000000000000000
> Apr 11 04:24:42 xenserver kernel: [277246.634932]  ffff880093e99980  
> ffffffff8023f4ee ffff8800e23d3ec0 ffffffff802213e5
> Apr 11 04:24:42 xenserver kernel: [277246.634947] Call Trace:
> Apr 11 04:24:42 xenserver kernel: [277246.634961] 
[<ffffffff8023f4ee>]
> autoremove_wake_function+0x9/0x2e
> Apr 11 04:24:42 xenserver kernel: [277246.634970] 
[<ffffffff802213e5>]
> __wake_up_common+0x41/0x74
> Apr 11 04:24:42 xenserver kernel: [277246.634981] 
[<ffffffff802d54cc>]
> sysfs_find_dirent+0x1b/0x2f
> Apr 11 04:24:42 xenserver kernel: [277246.634994] 
[<ffffffffa02364f3>]
> :md_mod:md_write_start+0x160/0x178
> Apr 11 04:24:42 xenserver kernel: [277246.635003] 
[<ffffffff8023f4e5>]
> autoremove_wake_function+0x0/0x2e
> Apr 11 04:24:42 xenserver kernel: [277246.635016] 
[<ffffffffa00f43f8>]
> :dm_mod:__split_bio+0x363/0x374
> Apr 11 04:24:42 xenserver kernel: [277246.635026] 
[<ffffffffa0246faf>]
> :raid1:make_request+0x29/0x5b6
> Apr 11 04:24:42 xenserver kernel: [277246.635036] 
[<ffffffff803117f7>]
> __up_read+0x13/0x8a
> Apr 11 04:24:42 xenserver kernel: [277246.635046] 
[<ffffffff802ff7a6>]
> generic_make_request+0x2fe/0x339
> Apr 11 04:24:42 xenserver kernel: [277246.635056] 
[<ffffffff80266983>]
> mempool_alloc+0x24/0xda
> Apr 11 04:24:42 xenserver kernel: [277246.635067] 
[<ffffffff80300a80>]
> submit_bio+0xdb/0xe2
> Apr 11 04:24:42 xenserver kernel: [277246.635077] 
[<ffffffff80386e68>]
> dispatch_rw_block_io+0x5b9/0x65d
> Apr 11 04:24:42 xenserver kernel: [277246.635102] 
[<ffffffff802ff83b>]
> blk_unplug+0x5a/0x60
> Apr 11 04:24:42 xenserver kernel: [277246.635115] 
[<ffffffffa00f5b19>]
> :dm_mod:dm_table_unplug_all+0x2a/0x3d
> Apr 11 04:24:42 xenserver kernel: [277246.635125] 
[<ffffffff8038735b>]
> blkif_schedule+0x3c7/0x4dc
> Apr 11 04:24:42 xenserver kernel: [277246.635137] 
[<ffffffff80386f94>]
> blkif_schedule+0x0/0x4dc
> Apr 11 04:24:42 xenserver kernel: [277246.635146] 
[<ffffffff8023f3b7>]
> kthread+0x47/0x74
> Apr 11 04:24:42 xenserver kernel: [277246.635155] 
[<ffffffff8022817c>]
> schedule_tail+0x27/0x5c
> Apr 11 04:24:42 xenserver kernel: [277246.635163] 
[<ffffffff8020be28>]
> child_rip+0xa/0x12
> Apr 11 04:24:42 xenserver kernel: [277246.635174] 
[<ffffffff8023f370>]
> kthread+0x0/0x74
> Apr 11 04:24:42 xenserver kernel: [277246.635183] 
[<ffffffff8020be1e>]
> child_rip+0x0/0x12
> Apr 11 04:24:42 xenserver kernel: [277246.635191]
> Apr 11 04:24:43 xenserver kernel: [277247.601345] ata1.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:24:43 xenserver kernel: [277247.601366] ata1.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:24:43 xenserver kernel: [277247.601375] ata1.00: revalidation  
> failed (errno=-5)
> Apr 11 04:24:43 xenserver kernel: [277247.601383] ata1.00: disabled
> Apr 11 04:24:43 xenserver kernel: [277247.601396] ata2.00: qc timeout  
> (cmd 0xec)
> Apr 11 04:24:43 xenserver kernel: [277247.601410] ata2.00: failed to  
> IDENTIFY (I/O error, err_mask=0x4)
> Apr 11 04:24:43 xenserver kernel: [277247.601418] ata2.00: revalidation  
> failed (errno=-5)
> Apr 11 04:24:43 xenserver kernel: [277247.601425] ata2.00: disabled
> Apr 11 04:24:43 xenserver kernel: [277248.105346] ata2: hard resetting link
> Apr 11 04:24:43 xenserver kernel: [277248.105372] ata1: hard resetting link
> Apr 11 04:24:43 xenserver kernel: [277248.589302] ata1: softreset failed  
> (device not ready)
> Apr 11 04:24:43 xenserver kernel: [277248.589319] ata1: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:24:43 xenserver kernel: [277248.589341] ata2: softreset failed  
> (device not ready)
> Apr 11 04:24:43 xenserver kernel: [277248.589350] ata2: failed due to HW  
> bug, retry pmp=0
> Apr 11 04:24:44 xenserver kernel: [277248.753334] ata2: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:24:44 xenserver kernel: [277248.753366] ata2: EH complete
> Apr 11 04:24:44 xenserver kernel: [277248.753384] sd 1:0:0:0: [sdb]  
> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
> Apr 11 04:24:44 xenserver kernel: [277248.753399] end_request: I/O  
> error, dev sdb, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753410] end_request: I/O  
> error, dev sdb, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753421] md: super_written gets  
> error=-5, uptodate=0
> Apr 11 04:24:44 xenserver kernel: [277248.753431] raid1: Disk failure on  
> sdb1, disabling device.
> Apr 11 04:24:44 xenserver kernel: [277248.753432] raid1: Operation  
> continuing on 1 devices.
> Apr 11 04:24:44 xenserver kernel: [277248.753466] ata1: SATA link up 3.0  
> Gbps (SStatus 123 SControl 300)
> Apr 11 04:24:44 xenserver kernel: [277248.753479] ata1: EH complete
> Apr 11 04:24:44 xenserver kernel: [277248.753490] sd 0:0:0:0: [sda]  
> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
> Apr 11 04:24:44 xenserver kernel: [277248.753505] end_request: I/O  
> error, dev sda, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753515] end_request: I/O  
> error, dev sda, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753524] md: super_written gets  
> error=-5, uptodate=0
> Apr 11 04:24:44 xenserver kernel: [277248.753551] sd 0:0:0:0: [sda]  
> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
> Apr 11 04:24:44 xenserver kernel: [277248.753566] end_request: I/O  
> error, dev sda, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753576] end_request: I/O  
> error, dev sda, sector 1953519935
> Apr 11 04:24:44 xenserver kernel: [277248.753585] md: super_written gets  
> error=-5, uptodate=0
> Apr 11 04:24:44 xenserver kernel: [277248.753596] RAID1 conf printout:
> Apr 11 04:24:44 xenserver kernel: [277248.753603]  --- wd:1 rd:2
> Apr 11 04:24:44 xenserver kernel: [277248.753610]  disk 0, wo:0, o:1,  
> dev:sda1
> Apr 11 04:24:44 xenserver kernel: [277248.753618]  disk 1, wo:1, o:0,  
> dev:sdb1
> Apr 11 04:24:44 xenserver kernel: [277248.765416] RAID1 conf printout:
> Apr 11 04:24:44 xenserver kernel: [277248.765431]  --- wd:1 rd:2
>
>
> From lspci -v:
>
> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
>        Subsystem: Advanced Micro Devices [AMD] RS780 Host Bridge
>        Flags: bus master, 66MHz, medium devsel, latency 0
>        Capabilities: [c4] HyperTransport: Slave or Primary Interface
>        Capabilities: [54] HyperTransport: UnitID Clumping
>        Capabilities: [40] HyperTransport: Retry Mode
>        Capabilities: [9c] HyperTransport: #1a
>        Capabilities: [f8] HyperTransport: #1c
>
> 00:01.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge  
> (int gfx) (prog-if 00 [Normal decode])
>        Flags: bus master, 66MHz, medium devsel, latency 64
>        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>        I/O behind bridge: 0000b000-0000bfff
>        Memory behind bridge: fe700000-fe8fffff
>        Prefetchable memory behind bridge: 00000000c0000000-00000000cfffffff
>        Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+
>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] RS780  
> PCI to PCI bridge (int gfx)
>        Kernel modules: shpchp
>
> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge  
> (PCIE port 0) (prog-if 00 [Normal decode])
>        Flags: bus master, fast devsel, latency 0
>        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>        I/O behind bridge: 0000c000-0000cfff
>        Memory behind bridge: fe900000-fe9fffff
>        Prefetchable memory behind bridge: 00000000d0000000-00000000d00fffff
>        Capabilities: [50] Power Management version 3
>        Capabilities: [58] Express Root Port (Slot+), MSI 00
>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-  
> Queue=0/0 Enable-
>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device 
> 9600
>        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>        Capabilities: [100] Vendor Specific Information <?>
>        Capabilities: [110] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:07.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge  
> (PCIE port 3) (prog-if 00 [Normal decode])
>        Flags: bus master, fast devsel, latency 0
>        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>        I/O behind bridge: 0000d000-0000dfff
>        Memory behind bridge: fea00000-feafffff
>        Prefetchable memory behind bridge: 00000000dff00000-00000000dfffffff
>        Capabilities: [50] Power Management version 3
>        Capabilities: [58] Express Root Port (Slot+), MSI 00
>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-  
> Queue=0/0 Enable-
>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device 
> 9600
>        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>        Capabilities: [100] Vendor Specific Information <?>
>        Capabilities: [110] Virtual Channel <?>
>        Kernel driver in use: pcieport-driver
>        Kernel modules: shpchp
>
> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA  
> Controller [IDE mode] (prog-if 01 [AHCI 1.0])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>        I/O ports at a000 [size=8]
>        I/O ports at 9000 [size=4]
>        I/O ports at 8000 [size=8]
>        I/O ports at 7000 [size=4]
>        I/O ports at 6000 [size=16]
>        Memory at fe6ff800 (32-bit, non-prefetchable) [size=1K]
>        Capabilities: [60] Power Management version 2
>        Capabilities: [70] SATA HBA <?>
>        Kernel driver in use: ahci
>        Kernel modules: ahci
>
> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0  
> Controller (prog-if 10 [OHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>        Memory at fe6fe000 (32-bit, non-prefetchable) [size=4K]
>        Kernel driver in use: ohci_hcd
>        Kernel modules: ohci-hcd
>
> 00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller  
> (prog-if 10 [OHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>        Memory at fe6fd000 (32-bit, non-prefetchable) [size=4K]
>        Kernel driver in use: ohci_hcd
>        Kernel modules: ohci-hcd
>
> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI  
> Controller (prog-if 20 [EHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
>        Memory at fe6ff000 (32-bit, non-prefetchable) [size=256]
>        Capabilities: [c0] Power Management version 2
>        Capabilities: [e4] Debug port: BAR=1 offset=00e0
>        Kernel driver in use: ehci_hcd
>        Kernel modules: ehci-hcd
>
> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0  
> Controller (prog-if 10 [OHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>        Memory at fe6fc000 (32-bit, non-prefetchable) [size=4K]
>        Kernel driver in use: ohci_hcd
>        Kernel modules: ohci-hcd
>
> 00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller  
> (prog-if 10 [OHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>        Memory at fe6fb000 (32-bit, non-prefetchable) [size=4K]
>        Kernel driver in use: ohci_hcd
>        Kernel modules: ohci-hcd
>
> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI  
> Controller (prog-if 20 [EHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
>        Memory at fe6fa800 (32-bit, non-prefetchable) [size=256]
>        Capabilities: [c0] Power Management version 2
>        Capabilities: [e4] Debug port: BAR=1 offset=00e0
>        Kernel driver in use: ehci_hcd
>        Kernel modules: ehci-hcd
>
> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: 66MHz, medium devsel
>        Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+
>        Kernel driver in use: piix4_smbus
>        Kernel modules: i2c-piix4
>
> 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller  
> (prog-if 8a [Master SecP PriP])
>        Subsystem: ATI Technologies Inc SB700/SB800 IDE Controller
>        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 16
>        I/O ports at 01f0 [size=8]
>        I/O ports at 03f4 [size=1]
>        I/O ports at 0170 [size=8]
>        I/O ports at 0374 [size=1]
>        I/O ports at ff00 [size=16]
>        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit-  
> Queue=0/1 Enable-
>        Kernel driver in use: ATIIXP_IDE
>        Kernel modules: atiixp, ide-pci-generic, ata_generic
>
> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
>        Subsystem: Biostar Microtech Int''l Corp Device 8218
>        Flags: bus master, slow devsel, latency 64, IRQ 16
>        Memory at fe6f4000 (64-bit, non-prefetchable) [size=16K]
>        Capabilities: [50] Power Management version 2
>        Kernel driver in use: HDA Intel
>        Kernel modules: snd-hda-intel
>
> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 0
>
> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge  
> (prog-if 01 [Subtractive decode])
>        Flags: bus master, 66MHz, medium devsel, latency 64
>        Bus: primary=00, secondary=04, subordinate=05, sec-latency=64
>        I/O behind bridge: 0000e000-0000efff
>        Memory behind bridge: feb00000-febfffff
>        Prefetchable memory behind bridge: f0000000-fbffffff
>
> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2  
> Controller (prog-if 10 [OHCI])
>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>        Memory at fe6f9000 (32-bit, non-prefetchable) [size=4K]
>        Kernel driver in use: ohci_hcd
>        Kernel modules: ohci-hcd
>
> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,  
> Athlon64, Sempron] HyperTransport Configuration
>        Flags: fast devsel
>        Capabilities: [80] HyperTransport: Host or Secondary Interface
>
> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,  
> Athlon64, Sempron] Address Map
>        Flags: fast devsel
>
> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,  
> Athlon64, Sempron] DRAM Controller
>        Flags: fast devsel
>
> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,  
> Athlon64, Sempron] Miscellaneous Control
>        Flags: fast devsel
>        Capabilities: [f0] Secure device <?>
>
> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,  
> Athlon64, Sempron] Link Control
>        Flags: fast devsel
>
> 01:05.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3300  
> Graphics (prog-if 00 [VGA controller])
>        Subsystem: Biostar Microtech Int''l Corp Device 0217
>        Flags: bus master, fast devsel, latency 0, IRQ 7
>        Memory at c0000000 (32-bit, prefetchable) [size=256M]
>        I/O ports at b000 [size=256]
>        Memory at fe8f0000 (32-bit, non-prefetchable) [size=64K]
>        Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
>        Capabilities: [50] Power Management version 3
>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+  
> Queue=0/0 Enable-
>
> 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.  
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
>        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI  
> Express Gigabit Ethernet controller
>        Flags: bus master, fast devsel, latency 0, IRQ 16
>        I/O ports at c800 [size=256]
>        Memory at fe9ff000 (64-bit, non-prefetchable) [size=4K]
>        [virtual] Expansion ROM at d0000000 [disabled] [size=128K]
>        Capabilities: [40] Power Management version 2
>        Capabilities: [48] Vital Product Data <?>
>        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+  
> Queue=0/1 Enable-
>        Capabilities: [60] Express Endpoint, MSI 00
>        Capabilities: [84] Vendor Specific Information <?>
>        Capabilities: [100] Advanced Error Reporting <?>
>        Capabilities: [12c] Virtual Channel <?>
>        Capabilities: [148] Device Serial Number 00-e0-4c-68-00-00-00-fd
>        Capabilities: [154] Power Budgeting <?>
>        Kernel driver in use: pciback
>        Kernel modules: r8169
>
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.  
> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
>        Subsystem: Biostar Microtech Int''l Corp Device 2309
>        Flags: bus master, fast devsel, latency 0, IRQ 19
>        I/O ports at d800 [size=256]
>        Memory at dffff000 (64-bit, prefetchable) [size=4K]
>        Memory at dfff8000 (64-bit, prefetchable) [size=16K]
>        Expansion ROM at feae0000 [disabled] [size=128K]
>        Capabilities: [40] Power Management version 3
>        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+  
> Queue=0/0 Enable-
>        Capabilities: [70] Express Endpoint, MSI 01
>        Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4
>        Capabilities: [cc] Vital Product Data <?>
>        Capabilities: [100] Advanced Error Reporting <?>
>        Capabilities: [140] Virtual Channel <?>
>        Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-02-61
>        Kernel driver in use: r8169
>        Kernel modules: r8169
>
> 04:05.0 PCI bridge: Pericom Semiconductor Device 8140 (prog-if 00  
> [Normal decode])
>        Flags: bus master, 66MHz, medium devsel, latency 64
>        Bus: primary=04, secondary=05, subordinate=05, sec-latency=64
>        Prefetchable memory behind bridge: 00000000f4000000-00000000fbffffff
>        Capabilities: [80] Power Management version 2
>        Capabilities: [90] CompactPCI hot-swap <?>
>        Kernel modules: shpchp
>
> 04:06.0 Multimedia video controller: Internext Compression Inc iTVC16  
> (CX23416) MPEG-2 Encoder (rev 01)
>        Subsystem: Hauppauge computer works Inc. WinTV PVR 250
>        Flags: medium devsel, IRQ 21
>        Memory at f0000000 (32-bit, prefetchable) [disabled] [size=64M]
>        Capabilities: [44] Power Management version 2
>        Kernel driver in use: pciback
>        Kernel modules: ivtv
>
> 04:07.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet  
> Adapter (rev 11) (rev 11)
>        Subsystem: D-Link System Inc DGE-530T Gigabit Ethernet Adapter  
> (rev 11)
>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>        Memory at febfc000 (32-bit, non-prefetchable) [size=16K]
>        I/O ports at e800 [size=256]
>        Expansion ROM at feb00000 [disabled] [size=128K]
>        Capabilities: [48] Power Management version 2
>        Capabilities: [50] Vital Product Data <?>
>        Kernel driver in use: pciback
>        Kernel modules: skge
>
> 05:08.0 Multimedia video controller: Internext Compression Inc iTVC16  
> (CX23416) MPEG-2 Encoder (rev 01)
>        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (1st unit)
>        Flags: medium devsel, IRQ 20
>        Memory at f8000000 (32-bit, prefetchable) [disabled] [size=64M]
>        Capabilities: [44] Power Management version 2
>        Kernel driver in use: pciback
>        Kernel modules: ivtv
>
> 05:09.0 Multimedia video controller: Internext Compression Inc iTVC16  
> (CX23416) MPEG-2 Encoder (rev 01)
>        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (2nd unit)
>        Flags: medium devsel, IRQ 21
>        Memory at f4000000 (32-bit, prefetchable) [disabled] [size=64M]
>        Capabilities: [44] Power Management version 2
>        Kernel driver in use: pciback
>        Kernel modules: ivtv
>
>
> From /proc/interrupts:
>
>           CPU0       CPU1       CPU2       CPU3       1:         58       
>    0          0          0  Phys-irq-level     i8042
>  8:          0          0          0          0  Phys-irq-level     rtc0
>  9:          0          0          0          0  Phys-irq-level     acpi
> 12:          4          0          0          0  Phys-irq-level     i8042
> 14:          0          0          0          0  Phys-irq-level     ide0
> 15:      59700          0          0          0  Phys-irq-level     ide1
> 16:     339121          0          0          0  Phys-irq-level      
> ohci_hcd:usb1, ohci_hcd:usb2, HDA Intel
> 17:          0          0          0          0  Phys-irq-level      
> ehci_hcd:usb5
> 18:          3          0          0          0  Phys-irq-level      
> ohci_hcd:usb3, ohci_hcd:usb4, ohci_hcd:usb7
> 19:     174793          0          0          0  Phys-irq-level      
> ehci_hcd:usb6, peth0
> 22:     645023          0          0          0  Phys-irq-level     ahci
> 256:     450281          0          0          0  Dynamic-irq-level      
> timer0
> 257:       4704          0          0          0  Dynamic-irq-level      
> resched0
> 258:         43          0          0          0  Dynamic-irq-level      
> callfunc0
> 259:        731          0          0          0  Dynamic-irq-level      
> spinlock0
> 260:          0       2805          0          0  Dynamic-irq-level      
> resched1
> 261:          0        189          0          0  Dynamic-irq-level      
> callfunc1
> 262:          0        543          0          0  Dynamic-irq-level      
> spinlock1
> 263:          0      58565          0          0  Dynamic-irq-level      
> timer1
> 264:          0          0       4449          0  Dynamic-irq-level      
> resched2
> 265:          0          0        177          0  Dynamic-irq-level      
> callfunc2
> 266:          0          0        570          0  Dynamic-irq-level      
> spinlock2
> 267:          0          0      43025          0  Dynamic-irq-level      
> timer2
> 268:          0          0          0       3650  Dynamic-irq-level      
> resched3
> 269:          0          0          0        176  Dynamic-irq-level      
> callfunc3
> 270:          0          0          0        652  Dynamic-irq-level      
> spinlock3
> 271:          0          0          0      24529  Dynamic-irq-level      
> timer3
> 272:       5815          0          0          0  Dynamic-irq-level      
> xenbus
> 273:          0          0          0          0  Dynamic-irq-level      
> console
> 274:          0          0          0          0  Dynamic-irq-level      
> suspend
> 275:        406          0          0          0  Dynamic-irq-level      
> pciback
> 276:         36          0          0          0  Dynamic-irq-level      
> blkif-backend
> 277:        162          0          0          0  Dynamic-irq-level      
> blkif-backend
> 278:       1939          0          0          0  Dynamic-irq-level      
> blkif-backend
> 279:     178850          0          0          0  Dynamic-irq-level      
> vif1.0
> 280:      28670          0          0          0  Dynamic-irq-level      
> vif1.1
> 281:       4372          0          0          0  Dynamic-irq-level      
> blkif-backend
> 282:       1227          0          0          0  Dynamic-irq-level      
> blkif-backend
> 283:       8079          0          0          0  Dynamic-irq-level      
> blkif-backend
> 284:        177          0          0          0  Dynamic-irq-level      
> blkif-backend
> 285:        106          0          0          0  Dynamic-irq-level      
> blkif-backend
> 286:      43937          0          0          0  Dynamic-irq-level      
> blkif-backend
> 287:      28211          0          0          0  Dynamic-irq-level      
> vif2.0
> 288:       2950          0          0          0  Dynamic-irq-level      
> blkif-backend
> 289:       5669          0          0          0  Dynamic-irq-level      
> blkif-backend
> 290:         83          0          0          0  Dynamic-irq-level      
> blkif-backend
> 291:        148          0          0          0  Dynamic-irq-level      
> blkif-backend
> 292:         40          0          0          0  Dynamic-irq-level      
> blkif-backend
> 293:      98438          0          0          0  Dynamic-irq-level      
> blkif-backend
> 294:      64518          0          0          0  Dynamic-irq-level      
> vif3.0
> NMI:          0          0          0          0   Non-maskable interrupts
> RES:       4704       2805       4449       3650   Rescheduling interrupts
> CAL:         43        189        177        176   function call interrupts
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ben Holt

2010-Apr-12 21:47 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Pasi Kärkkäinen wrote:> On Sun, Apr 11, 2010 at 01:12:09PM -0700, Ben Holt wrote:
>   
>
>> From /var/log/syslog:
>>
>> Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody cared
>> (try booting with the "irqpoll" option)
>> Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm: swapper
>> Not tainted 2.6.26-2-xen-amd64 #1
>>
>>     
>
> So I assume this is the host/dom0 kernel.
>   
Yes.> What device is using IRQ 22? 
> Try checking from /proc/interrupts or from lspci.
>
> Is it a shared IRQ (multiple devices using the same IRQ)? 
>   
IRQ 22 is being shared between the SATA controller and a NIC.

Both /proc/interrupts and lspci are posted below, I should have 
mentioned as much in my first message.

Thanks for your response.

- Ben

> -- Pasi
>
>
>   
>> Apr 11 04:22:29 xenserver kernel: [277113.497152]
>> Apr 11 04:22:29 xenserver kernel: [277113.497153] Call Trace:
>> Apr 11 04:22:29 xenserver kernel: [277113.497165]  <IRQ>   
>> [<ffffffff8037ce18>] irq_ignore_unhandled+0x1c/0x32
>> Apr 11 04:22:29 xenserver kernel: [277113.497188] 
[<ffffffff8025f88f>]
>> __report_bad_irq+0x30/0x72
>> Apr 11 04:22:29 xenserver kernel: [277113.497197] 
[<ffffffff8025fb58>]
>> note_interrupt+0x287/0x2c7
>> Apr 11 04:22:29 xenserver kernel: [277113.497207] 
[<ffffffff80260440>]
>> handle_level_irq+0xc3/0x118
>> Apr 11 04:22:29 xenserver kernel: [277113.497218] 
[<ffffffff8020e146>]
>> do_IRQ+0x4e/0x9a
>> Apr 11 04:22:29 xenserver kernel: [277113.497227] 
[<ffffffff8037db2c>]
>> evtchn_do_upcall+0x13c/0x1fc
>> Apr 11 04:22:29 xenserver kernel: [277113.497237] 
[<ffffffff8020bbde>]
>> do_hypervisor_callback+0x1e/0x30
>> Apr 11 04:22:29 xenserver kernel: [277113.497245]  <EOI>   
>> [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
>> Apr 11 04:22:29 xenserver kernel: [277113.497263] 
[<ffffffff8020a0ce>]
>> xen_idle+0x2e/0x66
>> Apr 11 04:22:29 xenserver kernel: [277113.497271] 
[<ffffffff80209d49>]
>> cpu_idle+0x97/0xb9
>> Apr 11 04:22:29 xenserver kernel: [277113.497283]
>> Apr 11 04:22:29 xenserver kernel: [277113.497288] handlers:
>> Apr 11 04:22:29 xenserver kernel: [277113.497295]
[<ffffffffa0099969>]
>> (ahci_interrupt+0x0/0x46b [ahci])
>> Apr 11 04:22:29 xenserver kernel: [277113.497312] Disabling IRQ #22
>> Apr 11 04:22:54 xenserver kernel: [277138.645875] hdc:
dma_timer_expiry:
>> dma status == 0x64
>> Apr 11 04:23:00 xenserver kernel: [277145.647409] ata2.00: exception  
>> Emask 0x40 SAct 0x0 SErr 0x800 action 0x6 frozen
>> Apr 11 04:23:00 xenserver kernel: [277145.647429] ata2: SError: {
HostInt }
>> Apr 11 04:23:00 xenserver kernel: [277145.647438] ata2.00: cmd  
>> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>> Apr 11 04:23:00 xenserver kernel: [277145.647439]          res  
>> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
>> Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: {
DRDY }
>> Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting
link
>> Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception  
>> Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>> Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd  
>> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>> Apr 11 04:23:00 xenserver kernel: [277145.647497]          res  
>> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
>> Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: {
DRDY }
>> Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting
link
>> Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset
failed
>> (device not ready)
>> Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:23:01 xenserver kernel: [277146.131400] ata1: softreset
failed
>> (device not ready)
>> Apr 11 04:23:01 xenserver kernel: [277146.131409] ata1: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:23:01 xenserver kernel: [277146.295250] ata1: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:23:01 xenserver kernel: [277146.295287] ata2: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:23:03 xenserver kernel: [277148.647303] hdc: DMA interrupt  
>> recovery
>> Apr 11 04:23:03 xenserver kernel: [277148.647316] hdc: lost interrupt
>> Apr 11 04:23:24 xenserver kernel: [277168.647352] hdc:
dma_timer_expiry:
>> dma status == 0x64
>> Apr 11 04:23:31 xenserver kernel: [277176.295271] ata2.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:23:31 xenserver kernel: [277176.295292] ata2.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:23:31 xenserver kernel: [277176.295301] ata2.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:23:31 xenserver kernel: [277176.295310] ata2: failed to  
>> recover some devices, retrying in 5 secs
>> Apr 11 04:23:31 xenserver kernel: [277176.295322] ata1.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:23:31 xenserver kernel: [277176.295336] ata1.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:23:31 xenserver kernel: [277176.295344] ata1.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:23:31 xenserver kernel: [277176.295352] ata1: failed to  
>> recover some devices, retrying in 5 secs
>> Apr 11 04:23:33 xenserver kernel: [277178.647242] hdc: DMA interrupt  
>> recovery
>> Apr 11 04:23:33 xenserver kernel: [277178.647255] hdc: lost interrupt
>> Apr 11 04:23:36 xenserver kernel: [277181.299161] ata1: hard resetting
link
>> Apr 11 04:23:36 xenserver kernel: [277181.299185] ata2: hard resetting
link
>> Apr 11 04:23:37 xenserver kernel: [277181.783130] ata2: softreset
failed
>> (device not ready)
>> Apr 11 04:23:37 xenserver kernel: [277181.783147] ata2: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:23:37 xenserver kernel: [277181.783170] ata1: softreset
failed
>> (device not ready)
>> Apr 11 04:23:37 xenserver kernel: [277181.783178] ata1: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:23:37 xenserver kernel: [277181.947140] ata1: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:23:37 xenserver kernel: [277181.947177] ata2: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:23:54 xenserver kernel: [277198.648158] hdc:
dma_timer_expiry:
>> dma status == 0x64
>> Apr 11 04:24:03 xenserver kernel: [277208.648119] hdc: DMA interrupt  
>> recovery
>> Apr 11 04:24:03 xenserver kernel: [277208.648131] hdc: lost interrupt
>> Apr 11 04:24:05 xenserver kernel: [277210.437511] NETDEV WATCHDOG:  
>> peth0: transmit timed out
>> Apr 11 04:24:05 xenserver kernel: [277210.437533] ------------[ cut
here
>> ]------------
>> Apr 11 04:24:05 xenserver kernel: [277210.437541] WARNING: at  
>> net/sched/sch_generic.c:222 dev_watchdog+0xa6/0x119()
>> Apr 11 04:24:05 xenserver kernel: [277210.437553] Modules linked in:  
>> xt_tcpudp xfs xt_physdev iptable_filter ip_tables x_tables bridge  
>> netloop ipv6 raid1 md_mod loop snd_hda_intel snd_pcm snd_timer snd  
>> psmouse soundcore pcspkr serio_raw snd_page_alloc i2c_piix4 i2c_core  
>> button wmi shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log  
>> dm_snapshot dm_mod ide_cd_mod cdrom ide_disk ide_pci_generic sd_mod  
>> atiixp ide_core ata_generic ahci ehci_hcd libata scsi_mod dock ohci_hcd
>> r8169 thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
>> Apr 11 04:24:05 xenserver kernel: [277210.437736] Pid: 0, comm: swapper
>> Not tainted 2.6.26-2-xen-amd64 #1
>> Apr 11 04:24:05 xenserver kernel: [277210.437744]
>> Apr 11 04:24:05 xenserver kernel: [277210.437745] Call Trace:
>> Apr 11 04:24:05 xenserver kernel: [277210.437757]  <IRQ>   
>> [<ffffffff8022c9a2>] warn_on_slowpath+0x51/0x78
>> Apr 11 04:24:05 xenserver kernel: [277210.437774] 
[<ffffffff8022d4f4>]
>> printk+0x4e/0x56
>> Apr 11 04:24:05 xenserver kernel: [277210.437784] 
[<ffffffff80235a70>]
>> lock_timer_base+0x26/0x4b
>> Apr 11 04:24:05 xenserver kernel: [277210.437794] 
[<ffffffff80235c16>]
>> __mod_timer+0xd4/0xe3
>> Apr 11 04:24:05 xenserver kernel: [277210.437804] 
[<ffffffff8023ca2f>]
>> queue_delayed_work_on+0xb8/0xc8
>> Apr 11 04:24:05 xenserver kernel: [277210.437813] 
[<ffffffff803d8161>]
>> dev_watchdog+0x0/0x119
>> Apr 11 04:24:05 xenserver kernel: [277210.437822] 
[<ffffffff803d8207>]
>> dev_watchdog+0xa6/0x119
>> Apr 11 04:24:05 xenserver kernel: [277210.437831] 
[<ffffffff802354df>]
>> run_timer_softirq+0x190/0x237
>> Apr 11 04:24:05 xenserver kernel: [277210.437842] 
[<ffffffff80231ad0>]
>> __do_softirq+0x77/0x103
>> Apr 11 04:24:05 xenserver kernel: [277210.437852] 
[<ffffffff8020c13c>]
>> call_softirq+0x1c/0x28
>> Apr 11 04:24:05 xenserver kernel: [277210.437861] 
[<ffffffff8020e092>]
>> do_softirq+0x55/0xbb
>> Apr 11 04:24:05 xenserver kernel: [277210.437870] 
[<ffffffff8020e175>]
>> do_IRQ+0x7d/0x9a
>> Apr 11 04:24:05 xenserver kernel: [277210.437879] 
[<ffffffff8037db2c>]
>> evtchn_do_upcall+0x13c/0x1fc
>> Apr 11 04:24:05 xenserver kernel: [277210.437890] 
[<ffffffff8020bbde>]
>> do_hypervisor_callback+0x1e/0x30
>> Apr 11 04:24:05 xenserver kernel: [277210.437898]  <EOI>   
>> [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
>> Apr 11 04:24:05 xenserver kernel: [277210.437916] 
[<ffffffff8020a0ce>]
>> xen_idle+0x2e/0x66
>> Apr 11 04:24:05 xenserver kernel: [277210.437924] 
[<ffffffff80209d49>]
>> cpu_idle+0x97/0xb9
>> Apr 11 04:24:05 xenserver kernel: [277210.437936]
>> Apr 11 04:24:05 xenserver kernel: [277210.437942] ---[ end trace  
>> 470e1e7e4a9eb321 ]---
>> Apr 11 04:24:06 xenserver kernel: [277210.456294] r8169: peth0: link up
>> Apr 11 04:24:07 xenserver kernel: [277211.948161] ata2.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:24:07 xenserver kernel: [277211.948182] ata2.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:24:07 xenserver kernel: [277211.948191] ata2.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:24:07 xenserver kernel: [277211.948199] ata2: failed to  
>> recover some devices, retrying in 5 secs
>> Apr 11 04:24:07 xenserver kernel: [277211.948211] ata1.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:24:07 xenserver kernel: [277211.948225] ata1.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:24:07 xenserver kernel: [277211.948233] ata1.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:24:07 xenserver kernel: [277211.948240] ata1: failed to  
>> recover some devices, retrying in 5 secs
>> Apr 11 04:24:12 xenserver kernel: [277216.952165] ata1: hard resetting
link
>> Apr 11 04:24:12 xenserver kernel: [277216.952189] ata2: hard resetting
link
>> Apr 11 04:24:12 xenserver kernel: [277217.436134] ata1: softreset
failed
>> (device not ready)
>> Apr 11 04:24:12 xenserver kernel: [277217.436150] ata1: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:24:12 xenserver kernel: [277217.436173] ata2: softreset
failed
>> (device not ready)
>> Apr 11 04:24:12 xenserver kernel: [277217.436181] ata2: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:24:12 xenserver kernel: [277217.600168] ata2: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:24:12 xenserver kernel: [277217.600210] ata1: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:24:17 xenserver kernel: [277222.438507] NETDEV WATCHDOG:  
>> peth0: transmit timed out
>> Apr 11 04:24:18 xenserver kernel: [277222.456219] r8169: peth0: link up
>> Apr 11 04:24:24 xenserver kernel: [277228.648149] hdc:
dma_timer_expiry:
>> dma status == 0x64
>> Apr 11 04:24:29 xenserver kernel: [277234.439860] NETDEV WATCHDOG:  
>> peth0: transmit timed out
>> Apr 11 04:24:30 xenserver kernel: [277234.456241] r8169: peth0: link up
>> Apr 11 04:24:33 xenserver kernel: [277238.648044] hdc: DMA interrupt  
>> recovery
>> Apr 11 04:24:33 xenserver kernel: [277238.648058] hdc: lost interrupt
>> Apr 11 04:24:41 xenserver kernel: [277246.439954] NETDEV WATCHDOG:  
>> peth0: transmit timed out
>> Apr 11 04:24:42 xenserver kernel: [277246.456300] r8169: peth0: link up
>> Apr 11 04:24:42 xenserver kernel: [277246.633366] INFO: task  
>> md0_raid1:2194 blocked for more than 120 seconds.
>> Apr 11 04:24:42 xenserver kernel: [277246.633382] "echo 0 >  
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 11 04:24:42 xenserver kernel: [277246.633394] md0_raid1     D  
>> ffffffff80449f00     0  2194      2
>> Apr 11 04:24:42 xenserver kernel: [277246.633408]  ffff8800e0169cc0  
>> 0000000000000246 0000000000000000 ffff8800e2a2fac0
>> Apr 11 04:24:42 xenserver kernel: [277246.633429]  ffff8800e2ad2480  
>> ffffffff804fe460 ffff8800e2ad2700 00000000e02c2528
>> Apr 11 04:24:42 xenserver kernel: [277246.633450]  00000000ffffffff  
>> ffff880093da90c0 000000000000ffff 000000007470593f
>> Apr 11 04:24:42 xenserver kernel: [277246.633465] Call Trace:
>> Apr 11 04:24:42 xenserver kernel: [277246.634648] 
[<ffffffff802ff7a6>]
>> generic_make_request+0x2fe/0x339
>> Apr 11 04:24:42 xenserver kernel: [277246.634664] 
[<ffffffffa0232413>]
>> :md_mod:md_super_wait+0xee/0x107
>> Apr 11 04:24:42 xenserver kernel: [277246.634674] 
[<ffffffff8023f4e5>]
>> autoremove_wake_function+0x0/0x2e
>> Apr 11 04:24:42 xenserver kernel: [277246.634688] 
[<ffffffffa02328a3>]
>> :md_mod:md_update_sb+0x2a0/0x38f
>> Apr 11 04:24:42 xenserver kernel: [277246.634702] 
[<ffffffffa0237016>]
>> :md_mod:md_check_recovery+0x1c9/0x504
>> Apr 11 04:24:42 xenserver kernel: [277246.634714] 
[<ffffffffa0246239>]
>> :raid1:raid1d+0x26/0xd73
>> Apr 11 04:24:42 xenserver kernel: [277246.634724] 
[<ffffffff804369b7>]
>> error_exit+0x0/0x69
>> Apr 11 04:24:42 xenserver kernel: [277246.634734] 
[<ffffffff804369b7>]
>> error_exit+0x0/0x69
>> Apr 11 04:24:42 xenserver kernel: [277246.634744] 
[<ffffffff80434efe>]
>> schedule_timeout+0x1e/0xad
>> Apr 11 04:24:42 xenserver kernel: [277246.634754] 
[<ffffffff8030dc91>]
>> __next_cpu+0x19/0x26
>> Apr 11 04:24:42 xenserver kernel: [277246.634766] 
[<ffffffffa02357d2>]
>> :md_mod:md_thread+0xd7/0xed
>> Apr 11 04:24:42 xenserver kernel: [277246.634775] 
[<ffffffff8023f4e5>]
>> autoremove_wake_function+0x0/0x2e
>> Apr 11 04:24:42 xenserver kernel: [277246.634788] 
[<ffffffffa02356fb>]
>> :md_mod:md_thread+0x0/0xed
>> Apr 11 04:24:42 xenserver kernel: [277246.634796] 
[<ffffffff8023f3b7>]
>> kthread+0x47/0x74
>> Apr 11 04:24:42 xenserver kernel: [277246.634805] 
[<ffffffff8022817c>]
>> schedule_tail+0x27/0x5c
>> Apr 11 04:24:42 xenserver kernel: [277246.634814] 
[<ffffffff8020be28>]
>> child_rip+0xa/0x12
>> Apr 11 04:24:42 xenserver kernel: [277246.634825] 
[<ffffffff8023f370>]
>> kthread+0x0/0x74
>> Apr 11 04:24:42 xenserver kernel: [277246.634833] 
[<ffffffff8020be1e>]
>> child_rip+0x0/0x12
>> Apr 11 04:24:42 xenserver kernel: [277246.634841]
>> Apr 11 04:24:42 xenserver kernel: [277246.634856] INFO: task  
>> blkback.2.sda6:4518 blocked for more than 120 seconds.
>> Apr 11 04:24:42 xenserver kernel: [277246.634867] "echo 0 >  
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 11 04:24:42 xenserver kernel: [277246.634878] blkback.2.sda D  
>> 00000000000000ff     0  4518      2
>> Apr 11 04:24:42 xenserver kernel: [277246.634891]  ffff880093e999c0  
>> 0000000000000246 0000000000000001 ffff8800e0169ee0
>> Apr 11 04:24:42 xenserver kernel: [277246.634911]  ffff8800e2057600  
>> ffff8800e2ad2480 ffff8800e2057880 0000000000000000
>> Apr 11 04:24:42 xenserver kernel: [277246.634932]  ffff880093e99980  
>> ffffffff8023f4ee ffff8800e23d3ec0 ffffffff802213e5
>> Apr 11 04:24:42 xenserver kernel: [277246.634947] Call Trace:
>> Apr 11 04:24:42 xenserver kernel: [277246.634961] 
[<ffffffff8023f4ee>]
>> autoremove_wake_function+0x9/0x2e
>> Apr 11 04:24:42 xenserver kernel: [277246.634970] 
[<ffffffff802213e5>]
>> __wake_up_common+0x41/0x74
>> Apr 11 04:24:42 xenserver kernel: [277246.634981] 
[<ffffffff802d54cc>]
>> sysfs_find_dirent+0x1b/0x2f
>> Apr 11 04:24:42 xenserver kernel: [277246.634994] 
[<ffffffffa02364f3>]
>> :md_mod:md_write_start+0x160/0x178
>> Apr 11 04:24:42 xenserver kernel: [277246.635003] 
[<ffffffff8023f4e5>]
>> autoremove_wake_function+0x0/0x2e
>> Apr 11 04:24:42 xenserver kernel: [277246.635016] 
[<ffffffffa00f43f8>]
>> :dm_mod:__split_bio+0x363/0x374
>> Apr 11 04:24:42 xenserver kernel: [277246.635026] 
[<ffffffffa0246faf>]
>> :raid1:make_request+0x29/0x5b6
>> Apr 11 04:24:42 xenserver kernel: [277246.635036] 
[<ffffffff803117f7>]
>> __up_read+0x13/0x8a
>> Apr 11 04:24:42 xenserver kernel: [277246.635046] 
[<ffffffff802ff7a6>]
>> generic_make_request+0x2fe/0x339
>> Apr 11 04:24:42 xenserver kernel: [277246.635056] 
[<ffffffff80266983>]
>> mempool_alloc+0x24/0xda
>> Apr 11 04:24:42 xenserver kernel: [277246.635067] 
[<ffffffff80300a80>]
>> submit_bio+0xdb/0xe2
>> Apr 11 04:24:42 xenserver kernel: [277246.635077] 
[<ffffffff80386e68>]
>> dispatch_rw_block_io+0x5b9/0x65d
>> Apr 11 04:24:42 xenserver kernel: [277246.635102] 
[<ffffffff802ff83b>]
>> blk_unplug+0x5a/0x60
>> Apr 11 04:24:42 xenserver kernel: [277246.635115] 
[<ffffffffa00f5b19>]
>> :dm_mod:dm_table_unplug_all+0x2a/0x3d
>> Apr 11 04:24:42 xenserver kernel: [277246.635125] 
[<ffffffff8038735b>]
>> blkif_schedule+0x3c7/0x4dc
>> Apr 11 04:24:42 xenserver kernel: [277246.635137] 
[<ffffffff80386f94>]
>> blkif_schedule+0x0/0x4dc
>> Apr 11 04:24:42 xenserver kernel: [277246.635146] 
[<ffffffff8023f3b7>]
>> kthread+0x47/0x74
>> Apr 11 04:24:42 xenserver kernel: [277246.635155] 
[<ffffffff8022817c>]
>> schedule_tail+0x27/0x5c
>> Apr 11 04:24:42 xenserver kernel: [277246.635163] 
[<ffffffff8020be28>]
>> child_rip+0xa/0x12
>> Apr 11 04:24:42 xenserver kernel: [277246.635174] 
[<ffffffff8023f370>]
>> kthread+0x0/0x74
>> Apr 11 04:24:42 xenserver kernel: [277246.635183] 
[<ffffffff8020be1e>]
>> child_rip+0x0/0x12
>> Apr 11 04:24:42 xenserver kernel: [277246.635191]
>> Apr 11 04:24:43 xenserver kernel: [277247.601345] ata1.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:24:43 xenserver kernel: [277247.601366] ata1.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:24:43 xenserver kernel: [277247.601375] ata1.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:24:43 xenserver kernel: [277247.601383] ata1.00: disabled
>> Apr 11 04:24:43 xenserver kernel: [277247.601396] ata2.00: qc timeout  
>> (cmd 0xec)
>> Apr 11 04:24:43 xenserver kernel: [277247.601410] ata2.00: failed to  
>> IDENTIFY (I/O error, err_mask=0x4)
>> Apr 11 04:24:43 xenserver kernel: [277247.601418] ata2.00: revalidation
>> failed (errno=-5)
>> Apr 11 04:24:43 xenserver kernel: [277247.601425] ata2.00: disabled
>> Apr 11 04:24:43 xenserver kernel: [277248.105346] ata2: hard resetting
link
>> Apr 11 04:24:43 xenserver kernel: [277248.105372] ata1: hard resetting
link
>> Apr 11 04:24:43 xenserver kernel: [277248.589302] ata1: softreset
failed
>> (device not ready)
>> Apr 11 04:24:43 xenserver kernel: [277248.589319] ata1: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:24:43 xenserver kernel: [277248.589341] ata2: softreset
failed
>> (device not ready)
>> Apr 11 04:24:43 xenserver kernel: [277248.589350] ata2: failed due to
HW
>> bug, retry pmp=0
>> Apr 11 04:24:44 xenserver kernel: [277248.753334] ata2: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:24:44 xenserver kernel: [277248.753366] ata2: EH complete
>> Apr 11 04:24:44 xenserver kernel: [277248.753384] sd 1:0:0:0: [sdb]  
>> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>> Apr 11 04:24:44 xenserver kernel: [277248.753399] end_request: I/O  
>> error, dev sdb, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753410] end_request: I/O  
>> error, dev sdb, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753421] md: super_written
gets
>> error=-5, uptodate=0
>> Apr 11 04:24:44 xenserver kernel: [277248.753431] raid1: Disk failure
on
>> sdb1, disabling device.
>> Apr 11 04:24:44 xenserver kernel: [277248.753432] raid1: Operation  
>> continuing on 1 devices.
>> Apr 11 04:24:44 xenserver kernel: [277248.753466] ata1: SATA link up
3.0
>> Gbps (SStatus 123 SControl 300)
>> Apr 11 04:24:44 xenserver kernel: [277248.753479] ata1: EH complete
>> Apr 11 04:24:44 xenserver kernel: [277248.753490] sd 0:0:0:0: [sda]  
>> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>> Apr 11 04:24:44 xenserver kernel: [277248.753505] end_request: I/O  
>> error, dev sda, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753515] end_request: I/O  
>> error, dev sda, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753524] md: super_written
gets
>> error=-5, uptodate=0
>> Apr 11 04:24:44 xenserver kernel: [277248.753551] sd 0:0:0:0: [sda]  
>> Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>> Apr 11 04:24:44 xenserver kernel: [277248.753566] end_request: I/O  
>> error, dev sda, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753576] end_request: I/O  
>> error, dev sda, sector 1953519935
>> Apr 11 04:24:44 xenserver kernel: [277248.753585] md: super_written
gets
>> error=-5, uptodate=0
>> Apr 11 04:24:44 xenserver kernel: [277248.753596] RAID1 conf printout:
>> Apr 11 04:24:44 xenserver kernel: [277248.753603]  --- wd:1 rd:2
>> Apr 11 04:24:44 xenserver kernel: [277248.753610]  disk 0, wo:0, o:1,  
>> dev:sda1
>> Apr 11 04:24:44 xenserver kernel: [277248.753618]  disk 1, wo:1, o:0,  
>> dev:sdb1
>> Apr 11 04:24:44 xenserver kernel: [277248.765416] RAID1 conf printout:
>> Apr 11 04:24:44 xenserver kernel: [277248.765431]  --- wd:1 rd:2
>>
>>
>> From lspci -v:
>>
>> 00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
>>        Subsystem: Advanced Micro Devices [AMD] RS780 Host Bridge
>>        Flags: bus master, 66MHz, medium devsel, latency 0
>>        Capabilities: [c4] HyperTransport: Slave or Primary Interface
>>        Capabilities: [54] HyperTransport: UnitID Clumping
>>        Capabilities: [40] HyperTransport: Retry Mode
>>        Capabilities: [9c] HyperTransport: #1a
>>        Capabilities: [f8] HyperTransport: #1c
>>
>> 00:01.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
bridge
>> (int gfx) (prog-if 00 [Normal decode])
>>        Flags: bus master, 66MHz, medium devsel, latency 64
>>        Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>>        I/O behind bridge: 0000b000-0000bfff
>>        Memory behind bridge: fe700000-fe8fffff
>>        Prefetchable memory behind bridge:
00000000c0000000-00000000cfffffff
>>        Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+
>>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] RS780
>> PCI to PCI bridge (int gfx)
>>        Kernel modules: shpchp
>>
>> 00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
bridge
>> (PCIE port 0) (prog-if 00 [Normal decode])
>>        Flags: bus master, fast devsel, latency 0
>>        Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>>        I/O behind bridge: 0000c000-0000cfff
>>        Memory behind bridge: fe900000-fe9fffff
>>        Prefetchable memory behind bridge:
00000000d0000000-00000000d00fffff
>>        Capabilities: [50] Power Management version 3
>>        Capabilities: [58] Express Root Port (Slot+), MSI 00
>>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-  
>> Queue=0/0 Enable-
>>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD]
Device
>> 9600
>>        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>>        Capabilities: [100] Vendor Specific Information <?>
>>        Capabilities: [110] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:07.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI
bridge
>> (PCIE port 3) (prog-if 00 [Normal decode])
>>        Flags: bus master, fast devsel, latency 0
>>        Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>>        I/O behind bridge: 0000d000-0000dfff
>>        Memory behind bridge: fea00000-feafffff
>>        Prefetchable memory behind bridge:
00000000dff00000-00000000dfffffff
>>        Capabilities: [50] Power Management version 3
>>        Capabilities: [58] Express Root Port (Slot+), MSI 00
>>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-  
>> Queue=0/0 Enable-
>>        Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD]
Device
>> 9600
>>        Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>>        Capabilities: [100] Vendor Specific Information <?>
>>        Capabilities: [110] Virtual Channel <?>
>>        Kernel driver in use: pcieport-driver
>>        Kernel modules: shpchp
>>
>> 00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA  
>> Controller [IDE mode] (prog-if 01 [AHCI 1.0])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>>        I/O ports at a000 [size=8]
>>        I/O ports at 9000 [size=4]
>>        I/O ports at 8000 [size=8]
>>        I/O ports at 7000 [size=4]
>>        I/O ports at 6000 [size=16]
>>        Memory at fe6ff800 (32-bit, non-prefetchable) [size=1K]
>>        Capabilities: [60] Power Management version 2
>>        Capabilities: [70] SATA HBA <?>
>>        Kernel driver in use: ahci
>>        Kernel modules: ahci
>>
>> 00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0  
>> Controller (prog-if 10 [OHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>>        Memory at fe6fe000 (32-bit, non-prefetchable) [size=4K]
>>        Kernel driver in use: ohci_hcd
>>        Kernel modules: ohci-hcd
>>
>> 00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
>> (prog-if 10 [OHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>>        Memory at fe6fd000 (32-bit, non-prefetchable) [size=4K]
>>        Kernel driver in use: ohci_hcd
>>        Kernel modules: ohci-hcd
>>
>> 00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI  
>> Controller (prog-if 20 [EHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
>>        Memory at fe6ff000 (32-bit, non-prefetchable) [size=256]
>>        Capabilities: [c0] Power Management version 2
>>        Capabilities: [e4] Debug port: BAR=1 offset=00e0
>>        Kernel driver in use: ehci_hcd
>>        Kernel modules: ehci-hcd
>>
>> 00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0  
>> Controller (prog-if 10 [OHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>>        Memory at fe6fc000 (32-bit, non-prefetchable) [size=4K]
>>        Kernel driver in use: ohci_hcd
>>        Kernel modules: ohci-hcd
>>
>> 00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
>> (prog-if 10 [OHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>>        Memory at fe6fb000 (32-bit, non-prefetchable) [size=4K]
>>        Kernel driver in use: ohci_hcd
>>        Kernel modules: ohci-hcd
>>
>> 00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI  
>> Controller (prog-if 20 [EHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
>>        Memory at fe6fa800 (32-bit, non-prefetchable) [size=256]
>>        Capabilities: [c0] Power Management version 2
>>        Capabilities: [e4] Debug port: BAR=1 offset=00e0
>>        Kernel driver in use: ehci_hcd
>>        Kernel modules: ehci-hcd
>>
>> 00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: 66MHz, medium devsel
>>        Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+
>>        Kernel driver in use: piix4_smbus
>>        Kernel modules: i2c-piix4
>>
>> 00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller
>> (prog-if 8a [Master SecP PriP])
>>        Subsystem: ATI Technologies Inc SB700/SB800 IDE Controller
>>        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 16
>>        I/O ports at 01f0 [size=8]
>>        I/O ports at 03f4 [size=1]
>>        I/O ports at 0170 [size=8]
>>        I/O ports at 0374 [size=1]
>>        I/O ports at ff00 [size=16]
>>        Capabilities: [70] Message Signalled Interrupts: Mask- 64bit-  
>> Queue=0/1 Enable-
>>        Kernel driver in use: ATIIXP_IDE
>>        Kernel modules: atiixp, ide-pci-generic, ata_generic
>>
>> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
>>        Subsystem: Biostar Microtech Int''l Corp Device 8218
>>        Flags: bus master, slow devsel, latency 64, IRQ 16
>>        Memory at fe6f4000 (64-bit, non-prefetchable) [size=16K]
>>        Capabilities: [50] Power Management version 2
>>        Kernel driver in use: HDA Intel
>>        Kernel modules: snd-hda-intel
>>
>> 00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host
controller
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 0
>>
>> 00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge  
>> (prog-if 01 [Subtractive decode])
>>        Flags: bus master, 66MHz, medium devsel, latency 64
>>        Bus: primary=00, secondary=04, subordinate=05, sec-latency=64
>>        I/O behind bridge: 0000e000-0000efff
>>        Memory behind bridge: feb00000-febfffff
>>        Prefetchable memory behind bridge: f0000000-fbffffff
>>
>> 00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2  
>> Controller (prog-if 10 [OHCI])
>>        Subsystem: Biostar Microtech Int''l Corp Device 3700
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>>        Memory at fe6f9000 (32-bit, non-prefetchable) [size=4K]
>>        Kernel driver in use: ohci_hcd
>>        Kernel modules: ohci-hcd
>>
>> 00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] HyperTransport Configuration
>>        Flags: fast devsel
>>        Capabilities: [80] HyperTransport: Host or Secondary Interface
>>
>> 00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Address Map
>>        Flags: fast devsel
>>
>> 00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] DRAM Controller
>>        Flags: fast devsel
>>
>> 00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Miscellaneous Control
>>        Flags: fast devsel
>>        Capabilities: [f0] Secure device <?>
>>
>> 00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>> Athlon64, Sempron] Link Control
>>        Flags: fast devsel
>>
>> 01:05.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3300
>> Graphics (prog-if 00 [VGA controller])
>>        Subsystem: Biostar Microtech Int''l Corp Device 0217
>>        Flags: bus master, fast devsel, latency 0, IRQ 7
>>        Memory at c0000000 (32-bit, prefetchable) [size=256M]
>>        I/O ports at b000 [size=256]
>>        Memory at fe8f0000 (32-bit, non-prefetchable) [size=64K]
>>        Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
>>        Capabilities: [50] Power Management version 3
>>        Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+  
>> Queue=0/0 Enable-
>>
>> 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.  
>> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
>>        Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI  
>> Express Gigabit Ethernet controller
>>        Flags: bus master, fast devsel, latency 0, IRQ 16
>>        I/O ports at c800 [size=256]
>>        Memory at fe9ff000 (64-bit, non-prefetchable) [size=4K]
>>        [virtual] Expansion ROM at d0000000 [disabled] [size=128K]
>>        Capabilities: [40] Power Management version 2
>>        Capabilities: [48] Vital Product Data <?>
>>        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+  
>> Queue=0/1 Enable-
>>        Capabilities: [60] Express Endpoint, MSI 00
>>        Capabilities: [84] Vendor Specific Information <?>
>>        Capabilities: [100] Advanced Error Reporting <?>
>>        Capabilities: [12c] Virtual Channel <?>
>>        Capabilities: [148] Device Serial Number 00-e0-4c-68-00-00-00-fd
>>        Capabilities: [154] Power Budgeting <?>
>>        Kernel driver in use: pciback
>>        Kernel modules: r8169
>>
>> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.  
>> RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
>>        Subsystem: Biostar Microtech Int''l Corp Device 2309
>>        Flags: bus master, fast devsel, latency 0, IRQ 19
>>        I/O ports at d800 [size=256]
>>        Memory at dffff000 (64-bit, prefetchable) [size=4K]
>>        Memory at dfff8000 (64-bit, prefetchable) [size=16K]
>>        Expansion ROM at feae0000 [disabled] [size=128K]
>>        Capabilities: [40] Power Management version 3
>>        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+  
>> Queue=0/0 Enable-
>>        Capabilities: [70] Express Endpoint, MSI 01
>>        Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4
>>        Capabilities: [cc] Vital Product Data <?>
>>        Capabilities: [100] Advanced Error Reporting <?>
>>        Capabilities: [140] Virtual Channel <?>
>>        Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-02-61
>>        Kernel driver in use: r8169
>>        Kernel modules: r8169
>>
>> 04:05.0 PCI bridge: Pericom Semiconductor Device 8140 (prog-if 00  
>> [Normal decode])
>>        Flags: bus master, 66MHz, medium devsel, latency 64
>>        Bus: primary=04, secondary=05, subordinate=05, sec-latency=64
>>        Prefetchable memory behind bridge:
00000000f4000000-00000000fbffffff
>>        Capabilities: [80] Power Management version 2
>>        Capabilities: [90] CompactPCI hot-swap <?>
>>        Kernel modules: shpchp
>>
>> 04:06.0 Multimedia video controller: Internext Compression Inc iTVC16  
>> (CX23416) MPEG-2 Encoder (rev 01)
>>        Subsystem: Hauppauge computer works Inc. WinTV PVR 250
>>        Flags: medium devsel, IRQ 21
>>        Memory at f0000000 (32-bit, prefetchable) [disabled] [size=64M]
>>        Capabilities: [44] Power Management version 2
>>        Kernel driver in use: pciback
>>        Kernel modules: ivtv
>>
>> 04:07.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit
Ethernet
>> Adapter (rev 11) (rev 11)
>>        Subsystem: D-Link System Inc DGE-530T Gigabit Ethernet Adapter  
>> (rev 11)
>>        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>>        Memory at febfc000 (32-bit, non-prefetchable) [size=16K]
>>        I/O ports at e800 [size=256]
>>        Expansion ROM at feb00000 [disabled] [size=128K]
>>        Capabilities: [48] Power Management version 2
>>        Capabilities: [50] Vital Product Data <?>
>>        Kernel driver in use: pciback
>>        Kernel modules: skge
>>
>> 05:08.0 Multimedia video controller: Internext Compression Inc iTVC16  
>> (CX23416) MPEG-2 Encoder (rev 01)
>>        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (1st
unit)
>>        Flags: medium devsel, IRQ 20
>>        Memory at f8000000 (32-bit, prefetchable) [disabled] [size=64M]
>>        Capabilities: [44] Power Management version 2
>>        Kernel driver in use: pciback
>>        Kernel modules: ivtv
>>
>> 05:09.0 Multimedia video controller: Internext Compression Inc iTVC16  
>> (CX23416) MPEG-2 Encoder (rev 01)
>>        Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (2nd
unit)
>>        Flags: medium devsel, IRQ 21
>>        Memory at f4000000 (32-bit, prefetchable) [disabled] [size=64M]
>>        Capabilities: [44] Power Management version 2
>>        Kernel driver in use: pciback
>>        Kernel modules: ivtv
>>
>>
>> From /proc/interrupts:
>>
>>           CPU0       CPU1       CPU2       CPU3       1:         58
>>    0          0          0  Phys-irq-level     i8042
>>  8:          0          0          0          0  Phys-irq-level    
rtc0
>>  9:          0          0          0          0  Phys-irq-level    
acpi
>> 12:          4          0          0          0  Phys-irq-level    
i8042
>> 14:          0          0          0          0  Phys-irq-level    
ide0
>> 15:      59700          0          0          0  Phys-irq-level    
ide1
>> 16:     339121          0          0          0  Phys-irq-level      
>> ohci_hcd:usb1, ohci_hcd:usb2, HDA Intel
>> 17:          0          0          0          0  Phys-irq-level      
>> ehci_hcd:usb5
>> 18:          3          0          0          0  Phys-irq-level      
>> ohci_hcd:usb3, ohci_hcd:usb4, ohci_hcd:usb7
>> 19:     174793          0          0          0  Phys-irq-level      
>> ehci_hcd:usb6, peth0
>> 22:     645023          0          0          0  Phys-irq-level    
ahci
>> 256:     450281          0          0          0  Dynamic-irq-level
>> timer0
>> 257:       4704          0          0          0  Dynamic-irq-level
>> resched0
>> 258:         43          0          0          0  Dynamic-irq-level
>> callfunc0
>> 259:        731          0          0          0  Dynamic-irq-level
>> spinlock0
>> 260:          0       2805          0          0  Dynamic-irq-level
>> resched1
>> 261:          0        189          0          0  Dynamic-irq-level
>> callfunc1
>> 262:          0        543          0          0  Dynamic-irq-level
>> spinlock1
>> 263:          0      58565          0          0  Dynamic-irq-level
>> timer1
>> 264:          0          0       4449          0  Dynamic-irq-level
>> resched2
>> 265:          0          0        177          0  Dynamic-irq-level
>> callfunc2
>> 266:          0          0        570          0  Dynamic-irq-level
>> spinlock2
>> 267:          0          0      43025          0  Dynamic-irq-level
>> timer2
>> 268:          0          0          0       3650  Dynamic-irq-level
>> resched3
>> 269:          0          0          0        176  Dynamic-irq-level
>> callfunc3
>> 270:          0          0          0        652  Dynamic-irq-level
>> spinlock3
>> 271:          0          0          0      24529  Dynamic-irq-level
>> timer3
>> 272:       5815          0          0          0  Dynamic-irq-level
>> xenbus
>> 273:          0          0          0          0  Dynamic-irq-level
>> console
>> 274:          0          0          0          0  Dynamic-irq-level
>> suspend
>> 275:        406          0          0          0  Dynamic-irq-level
>> pciback
>> 276:         36          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 277:        162          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 278:       1939          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 279:     178850          0          0          0  Dynamic-irq-level
>> vif1.0
>> 280:      28670          0          0          0  Dynamic-irq-level
>> vif1.1
>> 281:       4372          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 282:       1227          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 283:       8079          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 284:        177          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 285:        106          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 286:      43937          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 287:      28211          0          0          0  Dynamic-irq-level
>> vif2.0
>> 288:       2950          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 289:       5669          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 290:         83          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 291:        148          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 292:         40          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 293:      98438          0          0          0  Dynamic-irq-level
>> blkif-backend
>> 294:      64518          0          0          0  Dynamic-irq-level
>> vif3.0
>> NMI:          0          0          0          0   Non-maskable
interrupts
>> RES:       4704       2805       4449       3650   Rescheduling
interrupts
>> CAL:         43        189        177        176   function call
interrupts
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>     
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>   


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Apr-13 05:55 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

On Mon, Apr 12, 2010 at 02:47:19PM -0700, Ben Holt
wrote:>    Pasi Kärkkäinen wrote:
> 
>  On Sun, Apr 11, 2010 at 01:12:09PM -0700, Ben Holt wrote:
> 
> 
>  From /var/log/syslog:
> 
>  Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody cared
>  (try booting with the "irqpoll" option)
>  Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm: swapper
>  Not tainted 2.6.26-2-xen-amd64 #1
> 
> 
> 
>  So I assume this is the host/dom0 kernel.
> 
> 
>    Yes.
> 
>  What device is using IRQ 22?
>  Try checking from /proc/interrupts or from lspci.
> 
>  Is it a shared IRQ (multiple devices using the same IRQ)?
> 
> 
>    IRQ 22 is being shared between the SATA controller and a NIC.
> 
>    Both /proc/interrupts and lspci are posted below, I should have
mentioned
>    as much in my first message.
> 
>    Thanks for your response.
> 
Is this onboard NIC? Or additional PCI card? Can you move it to a different PCI
slot?

-- Pasi
> 
> 
> 
> 
>  Apr 11 04:22:29 xenserver kernel: [277113.497152]
>  Apr 11 04:22:29 xenserver kernel: [277113.497153] Call Trace:
>  Apr 11 04:22:29 xenserver kernel: [277113.497165]  <IRQ>
>  [<ffffffff8037ce18>] irq_ignore_unhandled+0x1c/0x32
>  Apr 11 04:22:29 xenserver kernel: [277113.497188] 
[<ffffffff8025f88f>]
>  __report_bad_irq+0x30/0x72
>  Apr 11 04:22:29 xenserver kernel: [277113.497197] 
[<ffffffff8025fb58>]
>  note_interrupt+0x287/0x2c7
>  Apr 11 04:22:29 xenserver kernel: [277113.497207] 
[<ffffffff80260440>]
>  handle_level_irq+0xc3/0x118
>  Apr 11 04:22:29 xenserver kernel: [277113.497218] 
[<ffffffff8020e146>]
>  do_IRQ+0x4e/0x9a
>  Apr 11 04:22:29 xenserver kernel: [277113.497227] 
[<ffffffff8037db2c>]
>  evtchn_do_upcall+0x13c/0x1fc
>  Apr 11 04:22:29 xenserver kernel: [277113.497237] 
[<ffffffff8020bbde>]
>  do_hypervisor_callback+0x1e/0x30
>  Apr 11 04:22:29 xenserver kernel: [277113.497245]  <EOI>
>  [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
>  Apr 11 04:22:29 xenserver kernel: [277113.497263] 
[<ffffffff8020a0ce>]
>  xen_idle+0x2e/0x66
>  Apr 11 04:22:29 xenserver kernel: [277113.497271] 
[<ffffffff80209d49>]
>  cpu_idle+0x97/0xb9
>  Apr 11 04:22:29 xenserver kernel: [277113.497283]
>  Apr 11 04:22:29 xenserver kernel: [277113.497288] handlers:
>  Apr 11 04:22:29 xenserver kernel: [277113.497295]
[<ffffffffa0099969>]
>  (ahci_interrupt+0x0/0x46b [ahci])
>  Apr 11 04:22:29 xenserver kernel: [277113.497312] Disabling IRQ #22
>  Apr 11 04:22:54 xenserver kernel: [277138.645875] hdc: dma_timer_expiry:
>  dma status == 0x64
>  Apr 11 04:23:00 xenserver kernel: [277145.647409] ata2.00: exception
>  Emask 0x40 SAct 0x0 SErr 0x800 action 0x6 frozen
>  Apr 11 04:23:00 xenserver kernel: [277145.647429] ata2: SError: { HostInt
}
>  Apr 11 04:23:00 xenserver kernel: [277145.647438] ata2.00: cmd
>  ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>  Apr 11 04:23:00 xenserver kernel: [277145.647439]          res
>  40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x44 (timeout)
>  Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: { DRDY
}
>  Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting
link
>  Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception
>  Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>  Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd
>  ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>  Apr 11 04:23:00 xenserver kernel: [277145.647497]          res
>  40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
>  Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: { DRDY
}
>  Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting
link
>  Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset failed
>  (device not ready)
>  Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:23:01 xenserver kernel: [277146.131400] ata1: softreset failed
>  (device not ready)
>  Apr 11 04:23:01 xenserver kernel: [277146.131409] ata1: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:23:01 xenserver kernel: [277146.295250] ata1: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:23:01 xenserver kernel: [277146.295287] ata2: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:23:03 xenserver kernel: [277148.647303] hdc: DMA interrupt
>  recovery
>  Apr 11 04:23:03 xenserver kernel: [277148.647316] hdc: lost interrupt
>  Apr 11 04:23:24 xenserver kernel: [277168.647352] hdc: dma_timer_expiry:
>  dma status == 0x64
>  Apr 11 04:23:31 xenserver kernel: [277176.295271] ata2.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:23:31 xenserver kernel: [277176.295292] ata2.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:23:31 xenserver kernel: [277176.295301] ata2.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:23:31 xenserver kernel: [277176.295310] ata2: failed to
>  recover some devices, retrying in 5 secs
>  Apr 11 04:23:31 xenserver kernel: [277176.295322] ata1.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:23:31 xenserver kernel: [277176.295336] ata1.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:23:31 xenserver kernel: [277176.295344] ata1.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:23:31 xenserver kernel: [277176.295352] ata1: failed to
>  recover some devices, retrying in 5 secs
>  Apr 11 04:23:33 xenserver kernel: [277178.647242] hdc: DMA interrupt
>  recovery
>  Apr 11 04:23:33 xenserver kernel: [277178.647255] hdc: lost interrupt
>  Apr 11 04:23:36 xenserver kernel: [277181.299161] ata1: hard resetting
link
>  Apr 11 04:23:36 xenserver kernel: [277181.299185] ata2: hard resetting
link
>  Apr 11 04:23:37 xenserver kernel: [277181.783130] ata2: softreset failed
>  (device not ready)
>  Apr 11 04:23:37 xenserver kernel: [277181.783147] ata2: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:23:37 xenserver kernel: [277181.783170] ata1: softreset failed
>  (device not ready)
>  Apr 11 04:23:37 xenserver kernel: [277181.783178] ata1: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:23:37 xenserver kernel: [277181.947140] ata1: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:23:37 xenserver kernel: [277181.947177] ata2: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:23:54 xenserver kernel: [277198.648158] hdc: dma_timer_expiry:
>  dma status == 0x64
>  Apr 11 04:24:03 xenserver kernel: [277208.648119] hdc: DMA interrupt
>  recovery
>  Apr 11 04:24:03 xenserver kernel: [277208.648131] hdc: lost interrupt
>  Apr 11 04:24:05 xenserver kernel: [277210.437511] NETDEV WATCHDOG:
>  peth0: transmit timed out
>  Apr 11 04:24:05 xenserver kernel: [277210.437533] ------------[ cut here
>  ]------------
>  Apr 11 04:24:05 xenserver kernel: [277210.437541] WARNING: at
>  net/sched/sch_generic.c:222 dev_watchdog+0xa6/0x119()
>  Apr 11 04:24:05 xenserver kernel: [277210.437553] Modules linked in:
>  xt_tcpudp xfs xt_physdev iptable_filter ip_tables x_tables bridge
>  netloop ipv6 raid1 md_mod loop snd_hda_intel snd_pcm snd_timer snd
>  psmouse soundcore pcspkr serio_raw snd_page_alloc i2c_piix4 i2c_core
>  button wmi shpchp pci_hotplug evdev ext3 jbd mbcache dm_mirror dm_log
>  dm_snapshot dm_mod ide_cd_mod cdrom ide_disk ide_pci_generic sd_mod
>  atiixp ide_core ata_generic ahci ehci_hcd libata scsi_mod dock ohci_hcd
>  r8169 thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
>  Apr 11 04:24:05 xenserver kernel: [277210.437736] Pid: 0, comm: swapper
>  Not tainted 2.6.26-2-xen-amd64 #1
>  Apr 11 04:24:05 xenserver kernel: [277210.437744]
>  Apr 11 04:24:05 xenserver kernel: [277210.437745] Call Trace:
>  Apr 11 04:24:05 xenserver kernel: [277210.437757]  <IRQ>
>  [<ffffffff8022c9a2>] warn_on_slowpath+0x51/0x78
>  Apr 11 04:24:05 xenserver kernel: [277210.437774] 
[<ffffffff8022d4f4>]
>  printk+0x4e/0x56
>  Apr 11 04:24:05 xenserver kernel: [277210.437784] 
[<ffffffff80235a70>]
>  lock_timer_base+0x26/0x4b
>  Apr 11 04:24:05 xenserver kernel: [277210.437794] 
[<ffffffff80235c16>]
>  __mod_timer+0xd4/0xe3
>  Apr 11 04:24:05 xenserver kernel: [277210.437804] 
[<ffffffff8023ca2f>]
>  queue_delayed_work_on+0xb8/0xc8
>  Apr 11 04:24:05 xenserver kernel: [277210.437813] 
[<ffffffff803d8161>]
>  dev_watchdog+0x0/0x119
>  Apr 11 04:24:05 xenserver kernel: [277210.437822] 
[<ffffffff803d8207>]
>  dev_watchdog+0xa6/0x119
>  Apr 11 04:24:05 xenserver kernel: [277210.437831] 
[<ffffffff802354df>]
>  run_timer_softirq+0x190/0x237
>  Apr 11 04:24:05 xenserver kernel: [277210.437842] 
[<ffffffff80231ad0>]
>  __do_softirq+0x77/0x103
>  Apr 11 04:24:05 xenserver kernel: [277210.437852] 
[<ffffffff8020c13c>]
>  call_softirq+0x1c/0x28
>  Apr 11 04:24:05 xenserver kernel: [277210.437861] 
[<ffffffff8020e092>]
>  do_softirq+0x55/0xbb
>  Apr 11 04:24:05 xenserver kernel: [277210.437870] 
[<ffffffff8020e175>]
>  do_IRQ+0x7d/0x9a
>  Apr 11 04:24:05 xenserver kernel: [277210.437879] 
[<ffffffff8037db2c>]
>  evtchn_do_upcall+0x13c/0x1fc
>  Apr 11 04:24:05 xenserver kernel: [277210.437890] 
[<ffffffff8020bbde>]
>  do_hypervisor_callback+0x1e/0x30
>  Apr 11 04:24:05 xenserver kernel: [277210.437898]  <EOI>
>  [<ffffffff8020e79d>] xen_safe_halt+0x90/0xa6
>  Apr 11 04:24:05 xenserver kernel: [277210.437916] 
[<ffffffff8020a0ce>]
>  xen_idle+0x2e/0x66
>  Apr 11 04:24:05 xenserver kernel: [277210.437924] 
[<ffffffff80209d49>]
>  cpu_idle+0x97/0xb9
>  Apr 11 04:24:05 xenserver kernel: [277210.437936]
>  Apr 11 04:24:05 xenserver kernel: [277210.437942] ---[ end trace
>  470e1e7e4a9eb321 ]---
>  Apr 11 04:24:06 xenserver kernel: [277210.456294] r8169: peth0: link up
>  Apr 11 04:24:07 xenserver kernel: [277211.948161] ata2.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:24:07 xenserver kernel: [277211.948182] ata2.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:24:07 xenserver kernel: [277211.948191] ata2.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:24:07 xenserver kernel: [277211.948199] ata2: failed to
>  recover some devices, retrying in 5 secs
>  Apr 11 04:24:07 xenserver kernel: [277211.948211] ata1.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:24:07 xenserver kernel: [277211.948225] ata1.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:24:07 xenserver kernel: [277211.948233] ata1.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:24:07 xenserver kernel: [277211.948240] ata1: failed to
>  recover some devices, retrying in 5 secs
>  Apr 11 04:24:12 xenserver kernel: [277216.952165] ata1: hard resetting
link
>  Apr 11 04:24:12 xenserver kernel: [277216.952189] ata2: hard resetting
link
>  Apr 11 04:24:12 xenserver kernel: [277217.436134] ata1: softreset failed
>  (device not ready)
>  Apr 11 04:24:12 xenserver kernel: [277217.436150] ata1: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:24:12 xenserver kernel: [277217.436173] ata2: softreset failed
>  (device not ready)
>  Apr 11 04:24:12 xenserver kernel: [277217.436181] ata2: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:24:12 xenserver kernel: [277217.600168] ata2: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:24:12 xenserver kernel: [277217.600210] ata1: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:24:17 xenserver kernel: [277222.438507] NETDEV WATCHDOG:
>  peth0: transmit timed out
>  Apr 11 04:24:18 xenserver kernel: [277222.456219] r8169: peth0: link up
>  Apr 11 04:24:24 xenserver kernel: [277228.648149] hdc: dma_timer_expiry:
>  dma status == 0x64
>  Apr 11 04:24:29 xenserver kernel: [277234.439860] NETDEV WATCHDOG:
>  peth0: transmit timed out
>  Apr 11 04:24:30 xenserver kernel: [277234.456241] r8169: peth0: link up
>  Apr 11 04:24:33 xenserver kernel: [277238.648044] hdc: DMA interrupt
>  recovery
>  Apr 11 04:24:33 xenserver kernel: [277238.648058] hdc: lost interrupt
>  Apr 11 04:24:41 xenserver kernel: [277246.439954] NETDEV WATCHDOG:
>  peth0: transmit timed out
>  Apr 11 04:24:42 xenserver kernel: [277246.456300] r8169: peth0: link up
>  Apr 11 04:24:42 xenserver kernel: [277246.633366] INFO: task
>  md0_raid1:2194 blocked for more than 120 seconds.
>  Apr 11 04:24:42 xenserver kernel: [277246.633382] "echo 0 >
>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>  Apr 11 04:24:42 xenserver kernel: [277246.633394] md0_raid1     D
>  ffffffff80449f00     0  2194      2
>  Apr 11 04:24:42 xenserver kernel: [277246.633408]  ffff8800e0169cc0
>  0000000000000246 0000000000000000 ffff8800e2a2fac0
>  Apr 11 04:24:42 xenserver kernel: [277246.633429]  ffff8800e2ad2480
>  ffffffff804fe460 ffff8800e2ad2700 00000000e02c2528
>  Apr 11 04:24:42 xenserver kernel: [277246.633450]  00000000ffffffff
>  ffff880093da90c0 000000000000ffff 000000007470593f
>  Apr 11 04:24:42 xenserver kernel: [277246.633465] Call Trace:
>  Apr 11 04:24:42 xenserver kernel: [277246.634648] 
[<ffffffff802ff7a6>]
>  generic_make_request+0x2fe/0x339
>  Apr 11 04:24:42 xenserver kernel: [277246.634664] 
[<ffffffffa0232413>]
>  :md_mod:md_super_wait+0xee/0x107
>  Apr 11 04:24:42 xenserver kernel: [277246.634674] 
[<ffffffff8023f4e5>]
>  autoremove_wake_function+0x0/0x2e
>  Apr 11 04:24:42 xenserver kernel: [277246.634688] 
[<ffffffffa02328a3>]
>  :md_mod:md_update_sb+0x2a0/0x38f
>  Apr 11 04:24:42 xenserver kernel: [277246.634702] 
[<ffffffffa0237016>]
>  :md_mod:md_check_recovery+0x1c9/0x504
>  Apr 11 04:24:42 xenserver kernel: [277246.634714] 
[<ffffffffa0246239>]
>  :raid1:raid1d+0x26/0xd73
>  Apr 11 04:24:42 xenserver kernel: [277246.634724] 
[<ffffffff804369b7>]
>  error_exit+0x0/0x69
>  Apr 11 04:24:42 xenserver kernel: [277246.634734] 
[<ffffffff804369b7>]
>  error_exit+0x0/0x69
>  Apr 11 04:24:42 xenserver kernel: [277246.634744] 
[<ffffffff80434efe>]
>  schedule_timeout+0x1e/0xad
>  Apr 11 04:24:42 xenserver kernel: [277246.634754] 
[<ffffffff8030dc91>]
>  __next_cpu+0x19/0x26
>  Apr 11 04:24:42 xenserver kernel: [277246.634766] 
[<ffffffffa02357d2>]
>  :md_mod:md_thread+0xd7/0xed
>  Apr 11 04:24:42 xenserver kernel: [277246.634775] 
[<ffffffff8023f4e5>]
>  autoremove_wake_function+0x0/0x2e
>  Apr 11 04:24:42 xenserver kernel: [277246.634788] 
[<ffffffffa02356fb>]
>  :md_mod:md_thread+0x0/0xed
>  Apr 11 04:24:42 xenserver kernel: [277246.634796] 
[<ffffffff8023f3b7>]
>  kthread+0x47/0x74
>  Apr 11 04:24:42 xenserver kernel: [277246.634805] 
[<ffffffff8022817c>]
>  schedule_tail+0x27/0x5c
>  Apr 11 04:24:42 xenserver kernel: [277246.634814] 
[<ffffffff8020be28>]
>  child_rip+0xa/0x12
>  Apr 11 04:24:42 xenserver kernel: [277246.634825] 
[<ffffffff8023f370>]
>  kthread+0x0/0x74
>  Apr 11 04:24:42 xenserver kernel: [277246.634833] 
[<ffffffff8020be1e>]
>  child_rip+0x0/0x12
>  Apr 11 04:24:42 xenserver kernel: [277246.634841]
>  Apr 11 04:24:42 xenserver kernel: [277246.634856] INFO: task
>  blkback.2.sda6:4518 blocked for more than 120 seconds.
>  Apr 11 04:24:42 xenserver kernel: [277246.634867] "echo 0 >
>  /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>  Apr 11 04:24:42 xenserver kernel: [277246.634878] blkback.2.sda D
>  00000000000000ff     0  4518      2
>  Apr 11 04:24:42 xenserver kernel: [277246.634891]  ffff880093e999c0
>  0000000000000246 0000000000000001 ffff8800e0169ee0
>  Apr 11 04:24:42 xenserver kernel: [277246.634911]  ffff8800e2057600
>  ffff8800e2ad2480 ffff8800e2057880 0000000000000000
>  Apr 11 04:24:42 xenserver kernel: [277246.634932]  ffff880093e99980
>  ffffffff8023f4ee ffff8800e23d3ec0 ffffffff802213e5
>  Apr 11 04:24:42 xenserver kernel: [277246.634947] Call Trace:
>  Apr 11 04:24:42 xenserver kernel: [277246.634961] 
[<ffffffff8023f4ee>]
>  autoremove_wake_function+0x9/0x2e
>  Apr 11 04:24:42 xenserver kernel: [277246.634970] 
[<ffffffff802213e5>]
>  __wake_up_common+0x41/0x74
>  Apr 11 04:24:42 xenserver kernel: [277246.634981] 
[<ffffffff802d54cc>]
>  sysfs_find_dirent+0x1b/0x2f
>  Apr 11 04:24:42 xenserver kernel: [277246.634994] 
[<ffffffffa02364f3>]
>  :md_mod:md_write_start+0x160/0x178
>  Apr 11 04:24:42 xenserver kernel: [277246.635003] 
[<ffffffff8023f4e5>]
>  autoremove_wake_function+0x0/0x2e
>  Apr 11 04:24:42 xenserver kernel: [277246.635016] 
[<ffffffffa00f43f8>]
>  :dm_mod:__split_bio+0x363/0x374
>  Apr 11 04:24:42 xenserver kernel: [277246.635026] 
[<ffffffffa0246faf>]
>  :raid1:make_request+0x29/0x5b6
>  Apr 11 04:24:42 xenserver kernel: [277246.635036] 
[<ffffffff803117f7>]
>  __up_read+0x13/0x8a
>  Apr 11 04:24:42 xenserver kernel: [277246.635046] 
[<ffffffff802ff7a6>]
>  generic_make_request+0x2fe/0x339
>  Apr 11 04:24:42 xenserver kernel: [277246.635056] 
[<ffffffff80266983>]
>  mempool_alloc+0x24/0xda
>  Apr 11 04:24:42 xenserver kernel: [277246.635067] 
[<ffffffff80300a80>]
>  submit_bio+0xdb/0xe2
>  Apr 11 04:24:42 xenserver kernel: [277246.635077] 
[<ffffffff80386e68>]
>  dispatch_rw_block_io+0x5b9/0x65d
>  Apr 11 04:24:42 xenserver kernel: [277246.635102] 
[<ffffffff802ff83b>]
>  blk_unplug+0x5a/0x60
>  Apr 11 04:24:42 xenserver kernel: [277246.635115] 
[<ffffffffa00f5b19>]
>  :dm_mod:dm_table_unplug_all+0x2a/0x3d
>  Apr 11 04:24:42 xenserver kernel: [277246.635125] 
[<ffffffff8038735b>]
>  blkif_schedule+0x3c7/0x4dc
>  Apr 11 04:24:42 xenserver kernel: [277246.635137] 
[<ffffffff80386f94>]
>  blkif_schedule+0x0/0x4dc
>  Apr 11 04:24:42 xenserver kernel: [277246.635146] 
[<ffffffff8023f3b7>]
>  kthread+0x47/0x74
>  Apr 11 04:24:42 xenserver kernel: [277246.635155] 
[<ffffffff8022817c>]
>  schedule_tail+0x27/0x5c
>  Apr 11 04:24:42 xenserver kernel: [277246.635163] 
[<ffffffff8020be28>]
>  child_rip+0xa/0x12
>  Apr 11 04:24:42 xenserver kernel: [277246.635174] 
[<ffffffff8023f370>]
>  kthread+0x0/0x74
>  Apr 11 04:24:42 xenserver kernel: [277246.635183] 
[<ffffffff8020be1e>]
>  child_rip+0x0/0x12
>  Apr 11 04:24:42 xenserver kernel: [277246.635191]
>  Apr 11 04:24:43 xenserver kernel: [277247.601345] ata1.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:24:43 xenserver kernel: [277247.601366] ata1.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:24:43 xenserver kernel: [277247.601375] ata1.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:24:43 xenserver kernel: [277247.601383] ata1.00: disabled
>  Apr 11 04:24:43 xenserver kernel: [277247.601396] ata2.00: qc timeout
>  (cmd 0xec)
>  Apr 11 04:24:43 xenserver kernel: [277247.601410] ata2.00: failed to
>  IDENTIFY (I/O error, err_mask=0x4)
>  Apr 11 04:24:43 xenserver kernel: [277247.601418] ata2.00: revalidation
>  failed (errno=-5)
>  Apr 11 04:24:43 xenserver kernel: [277247.601425] ata2.00: disabled
>  Apr 11 04:24:43 xenserver kernel: [277248.105346] ata2: hard resetting
link
>  Apr 11 04:24:43 xenserver kernel: [277248.105372] ata1: hard resetting
link
>  Apr 11 04:24:43 xenserver kernel: [277248.589302] ata1: softreset failed
>  (device not ready)
>  Apr 11 04:24:43 xenserver kernel: [277248.589319] ata1: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:24:43 xenserver kernel: [277248.589341] ata2: softreset failed
>  (device not ready)
>  Apr 11 04:24:43 xenserver kernel: [277248.589350] ata2: failed due to HW
>  bug, retry pmp=0
>  Apr 11 04:24:44 xenserver kernel: [277248.753334] ata2: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:24:44 xenserver kernel: [277248.753366] ata2: EH complete
>  Apr 11 04:24:44 xenserver kernel: [277248.753384] sd 1:0:0:0: [sdb]
>  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>  Apr 11 04:24:44 xenserver kernel: [277248.753399] end_request: I/O
>  error, dev sdb, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753410] end_request: I/O
>  error, dev sdb, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753421] md: super_written gets
>  error=-5, uptodate=0
>  Apr 11 04:24:44 xenserver kernel: [277248.753431] raid1: Disk failure on
>  sdb1, disabling device.
>  Apr 11 04:24:44 xenserver kernel: [277248.753432] raid1: Operation
>  continuing on 1 devices.
>  Apr 11 04:24:44 xenserver kernel: [277248.753466] ata1: SATA link up 3.0
>  Gbps (SStatus 123 SControl 300)
>  Apr 11 04:24:44 xenserver kernel: [277248.753479] ata1: EH complete
>  Apr 11 04:24:44 xenserver kernel: [277248.753490] sd 0:0:0:0: [sda]
>  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>  Apr 11 04:24:44 xenserver kernel: [277248.753505] end_request: I/O
>  error, dev sda, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753515] end_request: I/O
>  error, dev sda, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753524] md: super_written gets
>  error=-5, uptodate=0
>  Apr 11 04:24:44 xenserver kernel: [277248.753551] sd 0:0:0:0: [sda]
>  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
>  Apr 11 04:24:44 xenserver kernel: [277248.753566] end_request: I/O
>  error, dev sda, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753576] end_request: I/O
>  error, dev sda, sector 1953519935
>  Apr 11 04:24:44 xenserver kernel: [277248.753585] md: super_written gets
>  error=-5, uptodate=0
>  Apr 11 04:24:44 xenserver kernel: [277248.753596] RAID1 conf printout:
>  Apr 11 04:24:44 xenserver kernel: [277248.753603]  --- wd:1 rd:2
>  Apr 11 04:24:44 xenserver kernel: [277248.753610]  disk 0, wo:0, o:1,
>  dev:sda1
>  Apr 11 04:24:44 xenserver kernel: [277248.753618]  disk 1, wo:1, o:0,
>  dev:sdb1
>  Apr 11 04:24:44 xenserver kernel: [277248.765416] RAID1 conf printout:
>  Apr 11 04:24:44 xenserver kernel: [277248.765431]  --- wd:1 rd:2
> 
> 
>  From lspci -v:
> 
>  00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
>         Subsystem: Advanced Micro Devices [AMD] RS780 Host Bridge
>         Flags: bus master, 66MHz, medium devsel, latency 0
>         Capabilities: [c4] HyperTransport: Slave or Primary Interface
>         Capabilities: [54] HyperTransport: UnitID Clumping
>         Capabilities: [40] HyperTransport: Retry Mode
>         Capabilities: [9c] HyperTransport: #1a
>         Capabilities: [f8] HyperTransport: #1c
> 
>  00:01.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
>  (int gfx) (prog-if 00 [Normal decode])
>         Flags: bus master, 66MHz, medium devsel, latency 64
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
>         I/O behind bridge: 0000b000-0000bfff
>         Memory behind bridge: fe700000-fe8fffff
>         Prefetchable memory behind bridge:
00000000c0000000-00000000cfffffff
>         Capabilities: [44] HyperTransport: MSI Mapping Enable+ Fixed+
>         Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] RS780
>  PCI to PCI bridge (int gfx)
>         Kernel modules: shpchp
> 
>  00:04.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
>  (PCIE port 0) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>         I/O behind bridge: 0000c000-0000cfff
>         Memory behind bridge: fe900000-fe9fffff
>         Prefetchable memory behind bridge:
00000000d0000000-00000000d00fffff
>         Capabilities: [50] Power Management version 3
>         Capabilities: [58] Express Root Port (Slot+), MSI 00
>         Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-
>  Queue=0/0 Enable-
>         Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device
>  9600
>         Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>         Capabilities: [100] Vendor Specific Information <?>
>         Capabilities: [110] Virtual Channel <?>
>         Kernel driver in use: pcieport-driver
>         Kernel modules: shpchp
> 
>  00:07.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge
>  (PCIE port 3) (prog-if 00 [Normal decode])
>         Flags: bus master, fast devsel, latency 0
>         Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>         I/O behind bridge: 0000d000-0000dfff
>         Memory behind bridge: fea00000-feafffff
>         Prefetchable memory behind bridge:
00000000dff00000-00000000dfffffff
>         Capabilities: [50] Power Management version 3
>         Capabilities: [58] Express Root Port (Slot+), MSI 00
>         Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit-
>  Queue=0/0 Enable-
>         Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device
>  9600
>         Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
>         Capabilities: [100] Vendor Specific Information <?>
>         Capabilities: [110] Virtual Channel <?>
>         Kernel driver in use: pcieport-driver
>         Kernel modules: shpchp
> 
>  00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA
>  Controller [IDE mode] (prog-if 01 [AHCI 1.0])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>         I/O ports at a000 [size=8]
>         I/O ports at 9000 [size=4]
>         I/O ports at 8000 [size=8]
>         I/O ports at 7000 [size=4]
>         I/O ports at 6000 [size=16]
>         Memory at fe6ff800 (32-bit, non-prefetchable) [size=1K]
>         Capabilities: [60] Power Management version 2
>         Capabilities: [70] SATA HBA <?>
>         Kernel driver in use: ahci
>         Kernel modules: ahci
> 
>  00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
>  Controller (prog-if 10 [OHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>         Memory at fe6fe000 (32-bit, non-prefetchable) [size=4K]
>         Kernel driver in use: ohci_hcd
>         Kernel modules: ohci-hcd
> 
>  00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
>  (prog-if 10 [OHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16
>         Memory at fe6fd000 (32-bit, non-prefetchable) [size=4K]
>         Kernel driver in use: ohci_hcd
>         Kernel modules: ohci-hcd
> 
>  00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI
>  Controller (prog-if 20 [EHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 17
>         Memory at fe6ff000 (32-bit, non-prefetchable) [size=256]
>         Capabilities: [c0] Power Management version 2
>         Capabilities: [e4] Debug port: BAR=1 offset=00e0
>         Kernel driver in use: ehci_hcd
>         Kernel modules: ehci-hcd
> 
>  00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0
>  Controller (prog-if 10 [OHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>         Memory at fe6fc000 (32-bit, non-prefetchable) [size=4K]
>         Kernel driver in use: ohci_hcd
>         Kernel modules: ohci-hcd
> 
>  00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller
>  (prog-if 10 [OHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>         Memory at fe6fb000 (32-bit, non-prefetchable) [size=4K]
>         Kernel driver in use: ohci_hcd
>         Kernel modules: ohci-hcd
> 
>  00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI
>  Controller (prog-if 20 [EHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
>         Memory at fe6fa800 (32-bit, non-prefetchable) [size=256]
>         Capabilities: [c0] Power Management version 2
>         Capabilities: [e4] Debug port: BAR=1 offset=00e0
>         Kernel driver in use: ehci_hcd
>         Kernel modules: ehci-hcd
> 
>  00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: 66MHz, medium devsel
>         Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+
>         Kernel driver in use: piix4_smbus
>         Kernel modules: i2c-piix4
> 
>  00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller
>  (prog-if 8a [Master SecP PriP])
>         Subsystem: ATI Technologies Inc SB700/SB800 IDE Controller
>         Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 16
>         I/O ports at 01f0 [size=8]
>         I/O ports at 03f4 [size=1]
>         I/O ports at 0170 [size=8]
>         I/O ports at 0374 [size=1]
>         I/O ports at ff00 [size=16]
>         Capabilities: [70] Message Signalled Interrupts: Mask- 64bit-
>  Queue=0/1 Enable-
>         Kernel driver in use: ATIIXP_IDE
>         Kernel modules: atiixp, ide-pci-generic, ata_generic
> 
>  00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
>         Subsystem: Biostar Microtech Int''l Corp Device 8218
>         Flags: bus master, slow devsel, latency 64, IRQ 16
>         Memory at fe6f4000 (64-bit, non-prefetchable) [size=16K]
>         Capabilities: [50] Power Management version 2
>         Kernel driver in use: HDA Intel
>         Kernel modules: snd-hda-intel
> 
>  00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 0
> 
>  00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge
>  (prog-if 01 [Subtractive decode])
>         Flags: bus master, 66MHz, medium devsel, latency 64
>         Bus: primary=00, secondary=04, subordinate=05, sec-latency=64
>         I/O behind bridge: 0000e000-0000efff
>         Memory behind bridge: feb00000-febfffff
>         Prefetchable memory behind bridge: f0000000-fbffffff
> 
>  00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2
>  Controller (prog-if 10 [OHCI])
>         Subsystem: Biostar Microtech Int''l Corp Device 3700
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
>         Memory at fe6f9000 (32-bit, non-prefetchable) [size=4K]
>         Kernel driver in use: ohci_hcd
>         Kernel modules: ohci-hcd
> 
>  00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>  Athlon64, Sempron] HyperTransport Configuration
>         Flags: fast devsel
>         Capabilities: [80] HyperTransport: Host or Secondary Interface
> 
>  00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>  Athlon64, Sempron] Address Map
>         Flags: fast devsel
> 
>  00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>  Athlon64, Sempron] DRAM Controller
>         Flags: fast devsel
> 
>  00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>  Athlon64, Sempron] Miscellaneous Control
>         Flags: fast devsel
>         Capabilities: [f0] Secure device <?>
> 
>  00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h [Opteron,
>  Athlon64, Sempron] Link Control
>         Flags: fast devsel
> 
>  01:05.0 VGA compatible controller: ATI Technologies Inc Radeon HD 3300
>  Graphics (prog-if 00 [VGA controller])
>         Subsystem: Biostar Microtech Int''l Corp Device 0217
>         Flags: bus master, fast devsel, latency 0, IRQ 7
>         Memory at c0000000 (32-bit, prefetchable) [size=256M]
>         I/O ports at b000 [size=256]
>         Memory at fe8f0000 (32-bit, non-prefetchable) [size=64K]
>         Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+
>  Queue=0/0 Enable-
> 
>  02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>  RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
>         Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI
>  Express Gigabit Ethernet controller
>         Flags: bus master, fast devsel, latency 0, IRQ 16
>         I/O ports at c800 [size=256]
>         Memory at fe9ff000 (64-bit, non-prefetchable) [size=4K]
>         [virtual] Expansion ROM at d0000000 [disabled] [size=128K]
>         Capabilities: [40] Power Management version 2
>         Capabilities: [48] Vital Product Data <?>
>         Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
>  Queue=0/1 Enable-
>         Capabilities: [60] Express Endpoint, MSI 00
>         Capabilities: [84] Vendor Specific Information <?>
>         Capabilities: [100] Advanced Error Reporting <?>
>         Capabilities: [12c] Virtual Channel <?>
>         Capabilities: [148] Device Serial Number 00-e0-4c-68-00-00-00-fd
>         Capabilities: [154] Power Budgeting <?>
>         Kernel driver in use: pciback
>         Kernel modules: r8169
> 
>  03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
>  RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
>         Subsystem: Biostar Microtech Int''l Corp Device 2309
>         Flags: bus master, fast devsel, latency 0, IRQ 19
>         I/O ports at d800 [size=256]
>         Memory at dffff000 (64-bit, prefetchable) [size=4K]
>         Memory at dfff8000 (64-bit, prefetchable) [size=16K]
>         Expansion ROM at feae0000 [disabled] [size=128K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
>  Queue=0/0 Enable-
>         Capabilities: [70] Express Endpoint, MSI 01
>         Capabilities: [ac] MSI-X: Enable- Mask- TabSize=4
>         Capabilities: [cc] Vital Product Data <?>
>         Capabilities: [100] Advanced Error Reporting <?>
>         Capabilities: [140] Virtual Channel <?>
>         Capabilities: [160] Device Serial Number 00-e0-4c-68-00-00-02-61
>         Kernel driver in use: r8169
>         Kernel modules: r8169
> 
>  04:05.0 PCI bridge: Pericom Semiconductor Device 8140 (prog-if 00
>  [Normal decode])
>         Flags: bus master, 66MHz, medium devsel, latency 64
>         Bus: primary=04, secondary=05, subordinate=05, sec-latency=64
>         Prefetchable memory behind bridge:
00000000f4000000-00000000fbffffff
>         Capabilities: [80] Power Management version 2
>         Capabilities: [90] CompactPCI hot-swap <?>
>         Kernel modules: shpchp
> 
>  04:06.0 Multimedia video controller: Internext Compression Inc iTVC16
>  (CX23416) MPEG-2 Encoder (rev 01)
>         Subsystem: Hauppauge computer works Inc. WinTV PVR 250
>         Flags: medium devsel, IRQ 21
>         Memory at f0000000 (32-bit, prefetchable) [disabled] [size=64M]
>         Capabilities: [44] Power Management version 2
>         Kernel driver in use: pciback
>         Kernel modules: ivtv
> 
>  04:07.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet
>  Adapter (rev 11) (rev 11)
>         Subsystem: D-Link System Inc DGE-530T Gigabit Ethernet Adapter
>  (rev 11)
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 22
>         Memory at febfc000 (32-bit, non-prefetchable) [size=16K]
>         I/O ports at e800 [size=256]
>         Expansion ROM at feb00000 [disabled] [size=128K]
>         Capabilities: [48] Power Management version 2
>         Capabilities: [50] Vital Product Data <?>
>         Kernel driver in use: pciback
>         Kernel modules: skge
> 
>  05:08.0 Multimedia video controller: Internext Compression Inc iTVC16
>  (CX23416) MPEG-2 Encoder (rev 01)
>         Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (1st unit)
>         Flags: medium devsel, IRQ 20
>         Memory at f8000000 (32-bit, prefetchable) [disabled] [size=64M]
>         Capabilities: [44] Power Management version 2
>         Kernel driver in use: pciback
>         Kernel modules: ivtv
> 
>  05:09.0 Multimedia video controller: Internext Compression Inc iTVC16
>  (CX23416) MPEG-2 Encoder (rev 01)
>         Subsystem: Hauppauge computer works Inc. WinTV PVR 500 (2nd unit)
>         Flags: medium devsel, IRQ 21
>         Memory at f4000000 (32-bit, prefetchable) [disabled] [size=64M]
>         Capabilities: [44] Power Management version 2
>         Kernel driver in use: pciback
>         Kernel modules: ivtv
> 
> 
>  From /proc/interrupts:
> 
>            CPU0       CPU1       CPU2       CPU3       1:         58
>     0          0          0  Phys-irq-level     i8042
>   8:          0          0          0          0  Phys-irq-level     rtc0
>   9:          0          0          0          0  Phys-irq-level     acpi
>  12:          4          0          0          0  Phys-irq-level     i8042
>  14:          0          0          0          0  Phys-irq-level     ide0
>  15:      59700          0          0          0  Phys-irq-level     ide1
>  16:     339121          0          0          0  Phys-irq-level
>  ohci_hcd:usb1, ohci_hcd:usb2, HDA Intel
>  17:          0          0          0          0  Phys-irq-level
>  ehci_hcd:usb5
>  18:          3          0          0          0  Phys-irq-level
>  ohci_hcd:usb3, ohci_hcd:usb4, ohci_hcd:usb7
>  19:     174793          0          0          0  Phys-irq-level
>  ehci_hcd:usb6, peth0
>  22:     645023          0          0          0  Phys-irq-level     ahci
>  256:     450281          0          0          0  Dynamic-irq-level
>  timer0
>  257:       4704          0          0          0  Dynamic-irq-level
>  resched0
>  258:         43          0          0          0  Dynamic-irq-level
>  callfunc0
>  259:        731          0          0          0  Dynamic-irq-level
>  spinlock0
>  260:          0       2805          0          0  Dynamic-irq-level
>  resched1
>  261:          0        189          0          0  Dynamic-irq-level
>  callfunc1
>  262:          0        543          0          0  Dynamic-irq-level
>  spinlock1
>  263:          0      58565          0          0  Dynamic-irq-level
>  timer1
>  264:          0          0       4449          0  Dynamic-irq-level
>  resched2
>  265:          0          0        177          0  Dynamic-irq-level
>  callfunc2
>  266:          0          0        570          0  Dynamic-irq-level
>  spinlock2
>  267:          0          0      43025          0  Dynamic-irq-level
>  timer2
>  268:          0          0          0       3650  Dynamic-irq-level
>  resched3
>  269:          0          0          0        176  Dynamic-irq-level
>  callfunc3
>  270:          0          0          0        652  Dynamic-irq-level
>  spinlock3
>  271:          0          0          0      24529  Dynamic-irq-level
>  timer3
>  272:       5815          0          0          0  Dynamic-irq-level
>  xenbus
>  273:          0          0          0          0  Dynamic-irq-level
>  console
>  274:          0          0          0          0  Dynamic-irq-level
>  suspend
>  275:        406          0          0          0  Dynamic-irq-level
>  pciback
>  276:         36          0          0          0  Dynamic-irq-level
>  blkif-backend
>  277:        162          0          0          0  Dynamic-irq-level
>  blkif-backend
>  278:       1939          0          0          0  Dynamic-irq-level
>  blkif-backend
>  279:     178850          0          0          0  Dynamic-irq-level
>  vif1.0
>  280:      28670          0          0          0  Dynamic-irq-level
>  vif1.1
>  281:       4372          0          0          0  Dynamic-irq-level
>  blkif-backend
>  282:       1227          0          0          0  Dynamic-irq-level
>  blkif-backend
>  283:       8079          0          0          0  Dynamic-irq-level
>  blkif-backend
>  284:        177          0          0          0  Dynamic-irq-level
>  blkif-backend
>  285:        106          0          0          0  Dynamic-irq-level
>  blkif-backend
>  286:      43937          0          0          0  Dynamic-irq-level
>  blkif-backend
>  287:      28211          0          0          0  Dynamic-irq-level
>  vif2.0
>  288:       2950          0          0          0  Dynamic-irq-level
>  blkif-backend
>  289:       5669          0          0          0  Dynamic-irq-level
>  blkif-backend
>  290:         83          0          0          0  Dynamic-irq-level
>  blkif-backend
>  291:        148          0          0          0  Dynamic-irq-level
>  blkif-backend
>  292:         40          0          0          0  Dynamic-irq-level
>  blkif-backend
>  293:      98438          0          0          0  Dynamic-irq-level
>  blkif-backend
>  294:      64518          0          0          0  Dynamic-irq-level
>  vif3.0
>  NMI:          0          0          0          0   Non-maskable interrupts
>  RES:       4704       2805       4449       3650   Rescheduling interrupts
>  CAL:         43        189        177        176   function call
interrupts
> 
> 
>  _______________________________________________
>  Xen-users mailing list
>  [1]Xen-users@lists.xensource.com
>  [2]http://lists.xensource.com/xen-users
> 
> 
>  _______________________________________________
>  Xen-users mailing list
>  [3]Xen-users@lists.xensource.com
>  [4]http://lists.xensource.com/xen-users
> 
> 
> References
> 
>    Visible links
>    1. mailto:Xen-users@lists.xensource.com
>    2. http://lists.xensource.com/xen-users
>    3. mailto:Xen-users@lists.xensource.com
>    4. http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Fajar A. Nugraha

2010-Apr-13 06:16 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

On Mon, Apr 12, 2010 at 3:12 AM, Ben Holt <ben@computerdolt.com>
wrote:> Hello,
>
> I have experienced a dom0 crash where the system became unreachable via the
> network and the console was unresponsive.  I would appreciate help
> interpretting the logs and any configuration change suggestions.
>
> It is a stock Debian Lenny dom0 running xen 3.2.1 with kernel
> 2.6.26-2-xen-amd64 and an AMD Athlon IIx4 with 4 GB of RAM.
that''s a somewhat old, known-to-have-bugs kernel. The bugs might not
be directly related to your problems though.

> Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: { DRDY }
> Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting link
> Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception Emask
> 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
> Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd
> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
> Apr 11 04:23:00 xenserver kernel: [277145.647497]          res
> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
> Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: { DRDY }
> Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting link
> Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset failed
> (device not ready)
> Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to HW
> bug, retry pmp=0
I''d start by making sure the HW itself is fine (perhaps by booting a
normal, non-xen, known-good kernel and doing something like "dd
if=/dev/sda of=/dev/null bs=1M").

Have you tried latest opensuse xen kernel?
http://code.google.com/p/gentoo-xen-kernel/downloads/list

-- 
Fajar

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ben Holt

2010-Apr-13 16:41 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Pasi Kärkkäinen said the following:> On Mon, Apr 12, 2010 at 02:47:19PM -0700, Ben Holt wrote:
>   
>>    Pasi Kärkkäinen wrote:
>>
>>  On Sun, Apr 11, 2010 at 01:12:09PM -0700, Ben Holt wrote:
>>
>>
>>  From /var/log/syslog:
>>
>>  Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody cared
>>  (try booting with the "irqpoll" option)
>>  Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm:
swapper
>>  Not tainted 2.6.26-2-xen-amd64 #1
>>
>>
>>
>>  So I assume this is the host/dom0 kernel.
>>
>>
>>    Yes.
>>
>>  What device is using IRQ 22?
>>  Try checking from /proc/interrupts or from lspci.
>>
>>  Is it a shared IRQ (multiple devices using the same IRQ)?
>>
>>
>>    IRQ 22 is being shared between the SATA controller and a NIC.
>>
>>    Both /proc/interrupts and lspci are posted below, I should have
mentioned
>>    as much in my first message.
>>
>>    Thanks for your response.
>>
>>     
>
> Is this onboard NIC? Or additional PCI card? Can you move it to a different
PCI slot?
>
> -- Pasi
>
>   It''s a PCI NIC.  Unfortunately the other available slots (there are
only
three on this MB) are being used by the MPEG encoders.  I have a free 
PCI express slot I could use for a PCIe NIC if necessary.  Whether 
moving to a PCIe NIC would actually free up an IRQ is hard to say, 
though for the price of a NIC it is probably worth trying.

- Ben


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ben Holt

2010-Apr-13 16:53 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Fajar A. Nugraha said the following:> On Mon, Apr 12, 2010 at 3:12 AM, Ben Holt <ben@computerdolt.com>
wrote:
>   
>> Hello,
>>
>> I have experienced a dom0 crash where the system became unreachable via
the
>> network and the console was unresponsive.  I would appreciate help
>> interpretting the logs and any configuration change suggestions.
>>
>> It is a stock Debian Lenny dom0 running xen 3.2.1 with kernel
>> 2.6.26-2-xen-amd64 and an AMD Athlon IIx4 with 4 GB of RAM.
>>     
>
> that''s a somewhat old, known-to-have-bugs kernel. The bugs might
not
> be directly related to your problems though.
>   
This is the stock debian stable (lenny) kernel.  Bug fixes are typically 
back-ported by debian maintainers, so I would be surprised if the kernel 
is the issue.

>
>   
>> Apr 11 04:23:00 xenserver kernel: [277145.647458] ata2.00: status: {
DRDY }
>> Apr 11 04:23:00 xenserver kernel: [277145.647467] ata2: hard resetting
link
>> Apr 11 04:23:00 xenserver kernel: [277145.647486] ata1.00: exception
Emask
>> 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
>> Apr 11 04:23:00 xenserver kernel: [277145.647496] ata1.00: cmd
>> ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
>> Apr 11 04:23:00 xenserver kernel: [277145.647497]          res
>> 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
>> Apr 11 04:23:00 xenserver kernel: [277145.647515] ata1.00: status: {
DRDY }
>> Apr 11 04:23:00 xenserver kernel: [277145.647523] ata1: hard resetting
link
>> Apr 11 04:23:01 xenserver kernel: [277146.131362] ata2: softreset
failed
>> (device not ready)
>> Apr 11 04:23:01 xenserver kernel: [277146.131378] ata2: failed due to
HW
>> bug, retry pmp=0
>>     
>
> I''d start by making sure the HW itself is fine (perhaps by booting
a
> normal, non-xen, known-good kernel and doing something like "dd
> if=/dev/sda of=/dev/null bs=1M").
>   
I''ll give this a shot and let you know what I learn.  The hardware 
itself is less than a week old and I haven''t done any serious testing.
> Have you tried latest opensuse xen kernel?
> http://code.google.com/p/gentoo-xen-kernel/downloads/list
>
>   Not yet, I''ll try loading the I/O as you''ve suggested above
with with my
existing non-xen debian stable kernel first and see how it goes.  If 
that causes problems I''ll try again with 2.6.32.11 and the patches from
your link.

Thanks for the responses!

- Ben


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Sergey Vlasov

2010-Apr-13 18:30 UTC

head link

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

On Tue, Apr 13, 2010 at 09:41:57AM -0700, Ben Holt
wrote:> Pasi Kärkkäinen said the following:
> > On Mon, Apr 12, 2010 at 02:47:19PM -0700, Ben Holt wrote:
> >   
> >>    Pasi Kärkkäinen wrote:
> >>
> >>  On Sun, Apr 11, 2010 at 01:12:09PM -0700, Ben Holt wrote:
> >>
> >>  From /var/log/syslog:
> >>
> >>  Apr 11 04:22:29 xenserver kernel: [277113.497127] irq 22: nobody
cared
> >>  (try booting with the "irqpoll" option)
> >>  Apr 11 04:22:29 xenserver kernel: [277113.497144] Pid: 0, comm:
swapper
> >>  Not tainted 2.6.26-2-xen-amd64 #1
> >>
> >>  So I assume this is the host/dom0 kernel.
> >>
> >>    Yes.
> >>
> >>  What device is using IRQ 22?
> >>  Try checking from /proc/interrupts or from lspci.
> >>
> >>  Is it a shared IRQ (multiple devices using the same IRQ)?
> >>
> >>    IRQ 22 is being shared between the SATA controller and a NIC.
> >>
> >>    Both /proc/interrupts and lspci are posted below, I should have
mentioned
> >>    as much in my first message.
> >>
> >>    Thanks for your response.
> >
> > Is this onboard NIC? Or additional PCI card? Can you move it to a
different PCI slot?
> >   
> It''s a PCI NIC.  Unfortunately the other available slots (there
are only
> three on this MB) are being used by the MPEG encoders.  I have a free 
> PCI express slot I could use for a PCIe NIC if necessary.  Whether 
> moving to a PCIe NIC would actually free up an IRQ is hard to say, 
> though for the price of a NIC it is probably worth trying.
And you did not explicitly mention that the NIC is passed to a domU -
maybe this is causing the problem.  Switching NIC roles so that all
devices using IRQ 22 will belong to dom0 may work better.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Apparently Analagous Threads

Search for more reasonably related threads

Xen users - Apr 2010 - dom0 crash, require assistance interpretting logs and config suggestions

[Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Re: [Xen-users] dom0 crash, require assistance interpretting logs and config suggestions

Apparently Analagous Threads