Has anyone found a set of configuration settings for the mps driver (LSI SAS series cards) that gets the following situation under control? I have a machine here with one of these cards and a SAS expander. The expander has 16 ports, the card has 8 -- one connector attached to the expander. So I have 20 potential drives associated with this adapter. The issue is that the driver makes utterly no sense when it comes to assigning drive designations. In theory it should enumerate in series; that is, target 0 on the card should be da0, target 1 da1, etc. This assumes there are no gaps in the attached drives; if there are then things may move around, which is why some people wire drives. Ok so far, but it doesn't behave that way! This is the excerpt of what it returns: root at Dbms-10:/boot # dmesg|grep mps mps0: <LSI SAS2008> port 0xc000-0xc0ff mem 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> ses0 at mps0 bus 0 scbus0 target 8 lun 0 da0 at mps0 bus 0 scbus0 target 0 lun 0 da1 at mps0 bus 0 scbus0 target 1 lun 0 da2 at mps0 bus 0 scbus0 target 2 lun 0 da3 at mps0 bus 0 scbus0 target 3 lun 0 *da4 at mps0 bus 0 scbus0 target 6 lun 0** **da5 at mps0 bus 0 scbus0 target 7 lun 0* Those last two are the drives the card identifies as being on targets 0 and 1 when looked at in the BIOS. The other four drives (0-3) are on the SAS expander -- and there's a two drive "gap" in the targets identified as well which has zippo for logic associated with it. Even more-oddly if I swap the plugs -- that is, plug the two boot disks into the OTHER connector (putatively targets 4-7) and put the SAS expander on the 0-3 port connector the BIOS *does* show the target shift but the machine, when it boots, still places those two drives on targets 6 and 7! There is nothing in /boot/loader.conf or /boot/device.hints that relates to these devices at all. I would like to wire down the various ports so Drive Bay #x corresponds to da[x] but how can you when you don't know what order they will come up in predicated on either the physical port they're occupying or, for that matter, what the card's BIOS shows them as? I looked in the driver's man page and saw nothing related to this; I get around it by labeling the drives and then using "glabel list" to see what da[x] number it grabbed if I need to pull a disk while the machine is up, but this is a rather-unfriendly way to handle that. -- Karl Denninger karl at denninger.net <mailto:karl at denninger.net> /The Market Ticker/ -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2711 bytes Desc: S/MIME Cryptographic Signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140929/b193dd87/attachment.bin>
I seem to remember detection being made parallel for speed, as there
can be lots of drives. The Avago guys may be able to shed more light.
Regards
Steve
----- Original Message -----
From: "Karl Denninger" <karl at denninger.net>
To: <freebsd-stable at freebsd.org>
Sent: Monday, September 29, 2014 9:04 PM
Subject: MPS
Has anyone found a set of configuration settings for the mps driver (LSI
SAS series cards) that gets the following situation under control?
I have a machine here with one of these cards and a SAS expander. The
expander has 16 ports, the card has 8 -- one connector attached to the
expander. So I have 20 potential drives associated with this adapter.
The issue is that the driver makes utterly no sense when it comes to
assigning drive designations. In theory it should enumerate in series;
that is, target 0 on the card should be da0, target 1 da1, etc. This
assumes there are no gaps in the attached drives; if there are then
things may move around, which is why some people wire drives.
Ok so far, but it doesn't behave that way!
This is the excerpt of what it returns:
root at Dbms-10:/boot # dmesg|grep mps
mps0: <LSI SAS2008> port 0xc000-0xc0ff mem
0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3
mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd
mps0: IOCCapabilities:
1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
ses0 at mps0 bus 0 scbus0 target 8 lun 0
da0 at mps0 bus 0 scbus0 target 0 lun 0
da1 at mps0 bus 0 scbus0 target 1 lun 0
da2 at mps0 bus 0 scbus0 target 2 lun 0
da3 at mps0 bus 0 scbus0 target 3 lun 0
*da4 at mps0 bus 0 scbus0 target 6 lun 0**
**da5 at mps0 bus 0 scbus0 target 7 lun 0*
Those last two are the drives the card identifies as being on targets 0
and 1 when looked at in the BIOS. The other four drives (0-3) are on
the SAS expander -- and there's a two drive "gap" in the targets
identified as well which has zippo for logic associated with it.
Even more-oddly if I swap the plugs -- that is, plug the two boot disks
into the OTHER connector (putatively targets 4-7) and put the SAS
expander on the 0-3 port connector the BIOS *does* show the target shift
but the machine, when it boots, still places those two drives on targets
6 and 7!
There is nothing in /boot/loader.conf or /boot/device.hints that relates
to these devices at all.
I would like to wire down the various ports so Drive Bay #x corresponds
to da[x] but how can you when you don't know what order they will come
up in predicated on either the physical port they're occupying or, for
that matter, what the card's BIOS shows them as?
I looked in the driver's man page and saw nothing related to this; I get
around it by labeling the drives and then using "glabel list" to see
what da[x] number it grabbed if I need to pull a disk while the machine
is up, but this is a rather-unfriendly way to handle that.
--
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sep 29, 2014, at 10:04 PM, Karl Denninger wrote: Ok so far, but it doesn't behave that way! There is a tool that, at least with the backplanes I am using, can successfully identify a physical slot. It's called SAS2IRCU and it's available for FreeBSD. This is a sample output. At least I can use it to identify a disk belonging to a ZFS pool. Just find the serial number with a camcontrol inq, and find it in the sas2ircu output. Convoluted ritual, but it works for me. Borja. # sas2ircu 0 display LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18) Copyright (c) 2009-2013 LSI Corporation. All rights reserved. Read configuration has been initiated for controller 0 - ------------------------------------------------------------------------ Controller information - ------------------------------------------------------------------------ Controller type : SAS2008 BIOS version : 7.35.00.00 Firmware version : 18.00.00.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 255 Concurrent commands supported : 3432 Slot : 3 Segment : 0 Bus : 17 Device : 0 Function : 0 RAID Support : No - ------------------------------------------------------------------------ IR Volume information - ------------------------------------------------------------------------ - ------------------------------------------------------------------------ Physical device information - ------------------------------------------------------------------------ Initiator at ID #0 Device is a Hard disk Enclosure # : 2 Slot # : 16 SAS Address : 5000c50-0-05b5-ce25 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02W5L GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 2 Slot # : 17 SAS Address : 5005076-0-3e8e-81a2 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08549F GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 18 SAS Address : 5005076-0-3e8e-81a3 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08548T GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 19 SAS Address : 5005076-0-3e8e-81a4 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08568E GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 20 SAS Address : 5005076-0-3e8e-81a5 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08547X GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 21 SAS Address : 5005076-0-3e8e-81a6 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08518Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 22 SAS Address : 5005076-0-3e8e-81a7 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08556K GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Enclosure services device Enclosure # : 2 Slot # : 255 SAS Address : 5005076-0-3e8e-81b9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device Device is a Hard disk Enclosure # : 3 Slot # : 0 SAS Address : 5005076-0-3e8e-86e9 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08550R GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 1 SAS Address : 5005076-0-3e8e-86ea State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08911Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 2 SAS Address : 5005076-0-3e8e-86eb State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA Model Number : Samsung SSD 840 Firmware Revision : BB0Q Serial No : S1D9NEADA08811L GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 13 SAS Address : 5000c50-0-05b5-e531 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02STR GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 14 SAS Address : 5000c50-0-05b5-d489 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD02TV1 GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 15 SAS Address : 5000c50-0-05b5-f0ad State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE Model Number : ST9146803SS Firmware Revision : FS03 Serial No : 3SD03F4C GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Enclosure services device Enclosure # : 3 Slot # : 255 SAS Address : 5005076-0-3e8e-86f9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device - ------------------------------------------------------------------------ Enclosure information - ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:07ba2100 Numslots : 8 StartSlot : 0 Enclosure# : 2 Logical ID : 50050760:3e8e81a0 Numslots : 25 StartSlot : 0 Enclosure# : 3 Logical ID : 50050760:3e8e86e0 Numslots : 25 StartSlot : 0 - ------------------------------------------------------------------------ SAS2IRCU: Command DISPLAY Completed Successfully. SAS2IRCU: Utility Completed Successfully. root at elibm:/home/borjam # -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iEYEARECAAYFAlQqeSYACgkQULpVo4XWgJ/XqwCgwLMTxspPTzODkucKh/IDisdD yCsAnR9BM0xEiu6mmgFqIWPZsUIUdfN3 =rC8r -----END PGP SIGNATURE-----