On Sep 27, 2012, at 6:33 PM, Nikolay Denev <ndenev@gmail.com> wrote:
> Hi All,
>
> With the help of Chuck Tuffli, I'm now able to use CTL to export a zvol
over FC to a Linux host:
>
> LUN Backend Size (Blocks) BS Serial Number Device ID
> 0 block 4185915392 512 FBSDZFS001 ORA_ASM_01
> lun_type=0
> num_threads=14
> file=/dev/zvol/tank/oracle_asm_01
> 1 block 4185915392 512 FBSDZFS002 ORA_ASM_02
> lun_type=0
> num_threads=14
> file=/dev/zvol/tank/oracle_asm_02
> 2 block 4185915392 512 FBSDZFS003 ORA_ASM_03
> lun_type=0
> num_threads=14
> file=/dev/zvol/tank/oracle_asm_03
> 3 block 4185915392 512 FBSDZFS004 ORA_ASM_04
> lun_type=0
> num_threads=14
> file=/dev/zvol/tank/oracle_asm_04
>
> Then we ran some tests using Oracle's ORION benchmark tool from the
Linux host.
> We ran one test which passed successfully,
> then I've just disabled zfs prefetch ->
"vfs.zfs.prefetch_disable=1"
> and rerun the test, which failed due to this error.
>
> On the FreeBSD side:
>
> (0:3:0:1): READ(10). CDB: 28 0 84 f9 58 0 0 4 0 0
> (0:3:0:1): Tag: 0x116220, Type: 1
> (0:3:0:1): CTL Status: SCSI Error
> (0:3:0:1): SCSI Status: Check Condition
> (0:3:0:1): SCSI sense: NOT READY asc:4b,0 (Data phase error)
>
> Linux reported :
>
> sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
> Add. Sense: Data phase error
>
> end_request: I/O error, dev sdr, sector 2230933504
> device-mapper: multipath: Failing path 65:16.
> sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready
> Add. Sense: Data phase error
>
> end_request: I/O error, dev sdr, sector 2230934528
>
> There are no other suspicious messages in dmesg.
>
> Also, ctladm dumpooa does not show anything.
> Here is dumpscructs output :
>
> CTL IID to WWPN map start:
> CTL IID to WWPN map end
> CTL Persistent Reservation information start:
> CTL Persistent Reservation information end
> CTL Frontends:
> Frontend CTL ioctl Type 4 pport 0 vport 0 WWNN 0 WWPN 0
> Frontend ctl2cam Type 8 pport 0 vport 0 WWNN 0x5000000995680700 WWPN
0x5000000995680702
> Frontend CTL internal Type 8 pport 0 vport 0 WWNN 0 WWPN 0
> Frontend isp0 Type 1 pport 0 vport 0 WWNN 0x20000024ff376b98 WWPN
0x21000024ff376b98
> isp0: max tagged openings: 4096, max dev openings: 4096
> isp0: max_ccbs: 20488, ccb_count: 79
> isp0: ccb_freeq is NOT empty
> isp0: alloc_queue.entries 0, alloc_openings 4096
> isp0: qfrozen_cnt:0:0:0:0:0
> (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs
> (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 freed)
> (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 returned
> (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs
> (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 freed)
> (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 returned
> (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs
> (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 freed)
> (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 returned
> (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs
> (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 freed)
> (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 returned
> isp1: max tagged openings: 4096, max dev openings: 4096
> isp1: max_ccbs: 20488, ccb_count: 1
> isp1: ccb_freeq is NOT empty
> isp1: alloc_queue.entries 0, alloc_openings 4096
> isp1: qfrozen_cnt:0:0:0:0:0
> (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs
> (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs
> (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs
> (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs
> (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned
> Frontend isp1 Type 1 pport 1 vport 0 WWNN 0x20000024ff376b99 WWPN
0x21000024ff376b99
> isp0: max tagged openings: 4096, max dev openings: 4096
> isp0: max_ccbs: 20488, ccb_count: 79
> isp0: ccb_freeq is NOT empty
> isp0: alloc_queue.entries 0, alloc_openings 4096
> isp0: qfrozen_cnt:0:0:0:0:0
> (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs
> (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 freed)
> (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 returned
> (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs
> (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 freed)
> (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 returned
> (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs
> (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 freed)
> (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 returned
> (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs
> (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 freed)
> (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 returned
> isp1: max tagged openings: 4096, max dev openings: 4096
> isp1: max_ccbs: 20488, ccb_count: 1
> isp1: ccb_freeq is NOT empty
> isp1: alloc_queue.entries 0, alloc_openings 4096
> isp1: qfrozen_cnt:0:0:0:0:0
> (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs
> (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs
> (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs
> (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned
> (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs
> (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed)
> (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned
> CTL Frontend information end
>
> zpool status is showing no errors.
>
> P.S.:
> The machine is 8 core 2.0Ghz Xeon E5-2650 with 196G of RAM
> Other things to note are that I'm running with
"hw.mfi.max_cmds=254"
> Also my zpool is on GELI hw encrypted disks: 24 JBOD/RAID0 drives on mfi,
each encrypted with GELI, and arranged in 3 raidz2 vdevs of 8 drives.
> And the machine also acts as a fairly loaded NFS server.
>
After a whole day of orion tests without problems, we started an Oracle ASM
instance from the Linux host and
again got an error, this time it was WRITE error :
(0:3:0:3): WRITE(10). CDB: 2a 0 1 5b 10 0 0 4 0 0
(0:3:0:3): Tag: 0x110940, Type: 1
(0:3:0:3): CTL Status: SCSI Error
(0:3:0:3): SCSI Status: Check Condition
(0:3:0:3): SCSI sense: NOT READY asc:4b,0 (Data phase error)
I've tried to track down this "Data phase error" in the CTL code
and it looks like it is something related to the isp(4) driver:
default:
/*
* XXX KDM the isp(4) driver doesn't really
* seem to send errors back for data
* transfers that I can tell. There is one
* case where it'll send CAM_REQ_CMP_ERR,
* but probably not that many more cases.
* So set a generic data phase error here,
* like the SXP driver sets.
*/
io->io_hdr.port_status = 0xbad1;
ctl_set_data_phase_error(&io->scsiio);
/*
* XXX KDM figure out residual.
*/
break;
But I can't find error statistics for the isp(4) driver anywhere. Any ideas
how this can be debugged?