freebsd stable - Oct 2015 - mfi0: I/O error on MegaRAID SAS 9361-4i

System ist running ZFS data pool on 12 disk JBOD using
MegaRAID SAS 9361-4i controller.

After some time it starts printing the following errors:
Oct 16 22:26:46 hostname kernel: mfi0: I/O error, cmd=0xfffffe0001079c90,
status=0x3c, scsi_status=0
Oct 16 22:26:46 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:46 hostname kernel: mfisyspd6: hard error cmd=write
410262844-410263362
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe0001077188,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd5: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010780f0,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd6: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010778f8,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd7: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: 
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe000107a048,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd7: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe0001078db0,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd6: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010784a8,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd5: hard error cmd=write
410267090-410267593
Oct 16 22:26:49 hostname kernel: mfi0: 
Oct 16 22:26:49 hostname kernel: I/O error, cmd=0xfffffe0001078750, status=0x3c,
scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd10: hard error cmd=write
362223336-362223832
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe0001076660,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd11: hard error cmd=write
362223336-362223831
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010774b8,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd10: hard error cmd=write
362223336-362223832
Oct 16 22:26:49 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010788e8,
status=0x3c, scsi_status=0
Oct 16 22:26:49 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:49 hostname kernel: mfisyspd11: hard error cmd=write
362223336-362223831
Oct 16 22:26:59 hostname kernel: mfi0: I/O error, cmd=0xfffffe0001076f68,
status=0x3c, scsi_status=0
Oct 16 22:26:59 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:59 hostname kernel: mfisyspd3: hard error cmd=write
295491976-295492474
Oct 16 22:26:59 hostname kernel: mfi0: I/O error, cmd=0xfffffe00010797c8,
status=0x3c, scsi_status=0
Oct 16 22:26:59 hostname kernel: mfi0: sense error 0, sense_key 0, asc 0, ascq 0
Oct 16 22:26:59 hostname kernel: mfisyspd0: hard error cmd=write
295491977-295492475
[...] continues endless [...]

Interruptload in MFI is high and gstat shows disk load, but there is no
ZFS progress anymore.
I can only log into the machine because / is running on UFS.

After switching to mrsas things are different.
I got the follogin messages:
ses0: da0,pass1: Element descriptor: 'Slot00'
ses0: da0,pass1: SAS Device Slot Element: 1 Phys at Slot 0
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe080
ses0: da7,pass8: Element descriptor: 'Slot01'
ses0: da7,pass8: SAS Device Slot Element: 1 Phys at Slot 1
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe081
ses0: da1,pass2: Element descriptor: 'Slot02'
ses0: da1,pass2: SAS Device Slot Element: 1 Phys at Slot 2
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe082
ses0: da6,pass7: Element descriptor: 'Slot03'
ses0: da6,pass7: SAS Device Slot Element: 1 Phys at Slot 3
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe083
ses0: da3,pass4: Element descriptor: 'Slot04'
ses0: da3,pass4: SAS Device Slot Element: 1 Phys at Slot 4
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe084
ses0: da9,pass10: Element descriptor: 'Slot05'
ses0: da9,pass10: SAS Device Slot Element: 1 Phys at Slot 5
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe085
ses0: da2,pass3: Element descriptor: 'Slot06'
ses0: da2,pass3: SAS Device Slot Element: 1 Phys at Slot 6
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe086
ses0: da8,pass9: Element descriptor: 'Slot07'
ses0: da8,pass9: SAS Device Slot Element: 1 Phys at Slot 7
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe087
ses0: da5,pass6: Element descriptor: 'Slot08'
ses0: da5,pass6: SAS Device Slot Element: 1 Phys at Slot 8
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe088
ses0: da10,pass11: Element descriptor: 'Slot09'
ses0: da10,pass11: SAS Device Slot Element: 1 Phys at Slot 9
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe089
ses0: da4,pass5: Element descriptor: 'Slot10'
ses0: da4,pass5: SAS Device Slot Element: 1 Phys at Slot 10
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe08a
ses0: da11,pass12: Element descriptor: 'Slot11'
ses0: da11,pass12: SAS Device Slot Element: 1 Phys at Slot 11
ses0:  phy 0: SATA device
ses0:  phy 0: parent 5003048001abe0bf addr 5003048001abe08b

I have no idea what they mean and if they point to a real problem,
but everything continues just normaly.
I don't even know if those are the events that got the MFI driver
into permanent troubles.
ZFS is still happy with all the drives after those log messages.

-- 
B.Walter <bernd at bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.