zfs discuss - Feb 2011 - Format returning bogus controller info

The hardware: SuperMicro 847A chassis (3 drive bays in 4U) -- A means
there are 9 SFF-8087 ports on the backplanes, each controlling 4 drives;
no expanders here.
SuperMicro X8DTH-6F motherboard with integrated LSI 2008 SAS chipset,
flashed to IT firmware, connected to one backplane port.
Four LSI 9211-8i SAS controllers, flashed to IT firmware, each connected
to two backplane ports

The OS: OpenSolaris b134, installed off a USB stick created using the
instructions at 
<http://blogs.sun.com/clayb/entry/creating_opensolaris_usb_sticks_is>

The problem:
 While trying to add drives one at a time so I can identify them for later
use, I noticed two interesting things: the controller information is
unlike any I''ve seen before, and out of nine disks added after the boot
drive all nine are attached to c12 -- and no single controller has more
than eight ports.

The output of format:
locadmin at bigdawg2:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c10t2d0 <DEFAULT cyl 9965 alt 2 hd 224 sec 56>
          /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 4/disk at p2,0
       1. c12t5000CCA222DDD7BAd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222ddd7ba
       2. c12t5000CCA222DEAFE6d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222deafe6
       3. c12t5000CCA222DF3CECd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222df3cec
       4. c12t5000CCA222DF8FBEd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222df8fbe
       5. c12t5000CCA222DF92A0d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222df92a0
       6. c12t5000CCA222E0A3DEd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222e0a3de
       7. c12t5000CCA222E006B6d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222e006b6
       8. c12t5000CCA222E046B7d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222e046b7
       9. c12t5000CCA222E0533Fd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          /scsi_vhci/disk at g5000cca222e0533f
Specify disk (enter its number): ^C


Any suggestions?
--
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On 02/26/11 17:21, Dave Pooser wrote:
>   While trying to add drives one at a time so I can identify them for later
> use, I noticed two interesting things: the controller information is
> unlike any I''ve seen before, and out of nine disks added after the
boot
> drive all nine are attached to c12 -- and no single controller has more
> than eight ports.
on your system, c12 is the mpxio virtual controller; any disk which is 
potentially multipath-able (and that includes the SAS drives) will 
appear as a child of the virtual controller (rather than appear as the 
child of two or more different physical controllers).

see stmsboot(1m) for information on how to turn that off if you don''t 
need multipathing and don''t like the longer device names.

						- Bill

On 2/26/11 7:43 PM, "Bill Sommerfeld" <sommerfeld at
hamachi.org> wrote:
>On your system, c12 is the mpxio virtual controller; any disk which is
>potentially multipath-able (and that includes the SAS drives) will
>appear as a child of the virtual controller (rather than appear as the
>child of two or more different physical controllers).
Hmm... That makes sense, except that my drives are all SATA because I''m
cheap^H^H^H fiscally conservative.  :^)

''stmsboot -L'' displayed no mappings, but I went ahead and
tried stmsboot
-d to disable multipathing; after reboot instead of seeing nine disks on a
single controller I now see ten different controllers (in a machine that
has four PCI controllers and one motherboard controller):

locadmin at bigdawg2:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c10t2d0 <DEFAULT cyl 9965 alt 2 hd 224 sec 56>
          /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 4/disk at p2,0
       1. c13t5000CCA222DF92A0d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0
       2. c14t5000CCA222DF8FBEd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
       3. c15t5000CCA222E006B6d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 8/disk at
w5000cca222e006b6,0
       4. c16t5000CCA222DDD7BAd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at 2/disk at
w5000cca222ddd7ba,0
       5. c17t5000CCA222DF3CECd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 20/disk at
w5000cca222df3cec,0
       6. c18t5000CCA222DEAFE6d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 2/disk at
w5000cca222deafe6,0
       7. c19t5000CCA222E0A3DEd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 4/disk at
w5000cca222e0a3de,0
       8. c20t5000CCA222E046B7d0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 8/disk at
w5000cca222e046b7,0
       9. c21t5000CCA222E0533Fd0 <DEFAULT cyl 60798 alt 2 hd 255 sec 252>
          
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 20/disk at
w5000cca222e0533f,0


So now I''m more baffled than I started. Any other suggestions will be
gratefully accepted...

-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On 27/02/11 05:24 PM, Dave Pooser wrote:
> On 2/26/11 7:43 PM, "Bill Sommerfeld"<sommerfeld at
hamachi.org>  wrote:
>
>> On your system, c12 is the mpxio virtual controller; any disk which is
>> potentially multipath-able (and that includes the SAS drives) will
>> appear as a child of the virtual controller (rather than appear as the
>> child of two or more different physical controllers).
>
> Hmm... That makes sense, except that my drives are all SATA because
I''m
> cheap^H^H^H fiscally conservative.  :^)
They''re attached to a SAS hba, which is doing translations for them
using SATL - SAS to ATA Translation Layer.
> ''stmsboot -L'' displayed no mappings,
this is because mpt_sas(7d) controllers - which you have - are using
MPxIO by default. Running stmsboot -L will only show mappings if you''ve
enabled or disabled MPxIO....

 > but I went ahead and tried stmsboot
> -d to disable multipathing;
... and now you have disabled MPxIO, stmsboot -L should show mappings.
> after reboot instead of seeing nine disks on a
> single controller I now see ten different controllers (in a machine that
> has four PCI controllers and one motherboard controller):
This is a side effect of how your expanders are configured to operate
on your motherboard.

> locadmin at bigdawg2:~# format
> Searching for disks...done
>
>
> AVAILABLE DISK SELECTIONS:
>         0. c10t2d0<DEFAULT cyl 9965 alt 2 hd 224 sec 56>
>            /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 4/disk at
p2,0
>         1. c13t5000CCA222DF92A0d0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0
>         2. c14t5000CCA222DF8FBEd0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
>         3. c15t5000CCA222E006B6d0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 8/disk at
w5000cca222e006b6,0
>         4. c16t5000CCA222DDD7BAd0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at 2/disk at
w5000cca222ddd7ba,0
>         5. c17t5000CCA222DF3CECd0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 20/disk at
w5000cca222df3cec,0
>         6. c18t5000CCA222DEAFE6d0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 2/disk at
w5000cca222deafe6,0
>         7. c19t5000CCA222E0A3DEd0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 4/disk at
w5000cca222e0a3de,0
>         8. c20t5000CCA222E046B7d0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 8/disk at
w5000cca222e046b7,0
>         9. c21t5000CCA222E0533Fd0<DEFAULT cyl 60798 alt 2 hd 255 sec
252>
>
> /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 20/disk at
w5000cca222e0533f,0
>
>
> So now I''m more baffled than I started. Any other suggestions will
be
> gratefully accepted...
If you''re lucky, your expanders and the enclosure that they''re
configured into will show up with one or more SES targets. If
that''s the case, you might be able to see bay numbers with the
fmtopo command - when you run it as root:

# /usr/lib/fm/fmd/fmtopo -V

If this doesn''t work for you, then you''ll have to resort to
the
tried and tested use of dd to /dev/null for each disk, and see
which lights blink.


James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog

On 2/27/11 5:15 AM, "James C. McPherson" <jmcp at
opensolaris.org> wrote:
>On 27/02/11 05:24 PM, Dave Pooser wrote:
>>On 2/26/11 7:43 PM, "Bill Sommerfeld"<sommerfeld at
hamachi.org>  wrote:
>>
>>>On your system, c12 is the mpxio virtual controller; any disk which
is
>>>potentially multipath-able (and that includes the SAS drives) will
>>>appear as a child of the virtual controller (rather than appear as
the
>>>child of two or more different physical controllers).
>>
>>Hmm... That makes sense, except that my drives are all SATA because
I''m
>>cheap^H^H^H fiscally conservative.  :^)
>
>They''re attached to a SAS hba, which is doing translations for them
>using SATL - SAS to ATA Translation Layer.
Yeah, but they''re still not multipathable, are they?
>>''stmsboot -L'' displayed no mappings,
>
>this is because mpt_sas(7d) controllers - which you have - are using
>MPxIO by default. Running stmsboot -L will only show mappings if
you''ve
>enabled or disabled MPxIO....
>
>> but I went ahead and tried stmsboot
>>-d to disable multipathing;
>
>... and now you have disabled MPxIO, stmsboot -L should show mappings.
Nope:
locadmin at bigdawg2:~# stmsboot -L
stmsboot: MPXIO disabled
>>after reboot instead of seeing nine disks on a
>>single controller I now see ten different controllers (in a machine that
>>has four PCI controllers and one motherboard controller):
>
>This is a side effect of how your expanders are configured to operate
>on your motherboard.
But there shouldn''t be any expanders in the system-- the front
backplane
has six SFF-8087 ports to control 24 drives, and the rear backplane has
three more SFF-8087 ports to control 12 more drives. Each of those ports
is connected directly to an SFF-8087 port on an LSI 9211-8i controller,
except that the ninth port is connected to the integrated LSI 2008
controller on the motherboard.
>If you''re lucky, your expanders and the enclosure that
they''re
>configured into will show up with one or more SES targets. If
>that''s the case, you might be able to see bay numbers with the
>fmtopo command - when you run it as root:
>
># /usr/lib/fm/fmd/fmtopo -V
>
>If this doesn''t work for you, then you''ll have to resort
to the
>tried and tested use of dd to /dev/null for each disk, and see
>which lights blink.
I can live with that-- but I really want to know what (real, not virtual)
controllers disks are connected to; I want to build 3 8-disk RAIDz2 vdevs
now (with room for a fourth for expansion later) and I really want to make
sure each of those vdevs has fewer than three disks per controller so a
single controller failure can degrade my vdevs but not kill them.

Probably my next step is going to be to take a look with Nexenta Core or
FreeBSD (or maybe SolEx11 for a temporary eval) and see if either of those
gives me a saner view, but other suggestions would be appreciated.
--
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com <http://www.alfordmedia.com/>

> I can live with that-- but I really want to know what (real, not virtual)
> controllers disks are connected to; I want to build 3 8-disk RAIDz2 vdevs
> now (with room for a fourth for expansion later) and I really want to make
> sure each of those vdevs has fewer than three disks per controller so
> a single controller failure can degrade my vdevs but not kill them.
I cannot but agree. On Linux and Windoze (haven''t tested FreeBSD),
drives connected to an LSI9211 show up in the correct order, but not on
OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a mapping between
device name and slot, since that relies on the SES hardware being properly
supported. The answer I''ve got for this issue is, it''s not an
issue, since it''s that way by design etc. This doesn''t make
sense when Linux/Windows show the drives in the correct order. IMHO this looks
more like a design flaw in the driver code

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On 2/27/11 11:18 AM, "Roy Sigurd Karlsbakk" <roy at
karlsbakk.net> wrote:
>I cannot but agree. On Linux and Windoze (haven''t tested FreeBSD),
drives
>connected to an LSI9211 show up in the correct order, but not on
>OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a mapping
between
>device name and slot, since that relies on the SES hardware being
>properly supported. The answer I''ve got for this issue is,
it''s not an
>issue, since it''s that way by design etc. This doesn''t
make sense when
>Linux/Windows show the drives in the correct order. IMHO this looks more
>like a design flaw in the driver code
Especially since the SAS3081 cards work as expected. I guess I''ll start
looking for some more of the 3Gb SAS controllers and chalk the 9211s up as
a failed bit.
--
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com <http://www.alfordmedia.com/>

On 28/02/11 07:30 AM, Dave Pooser wrote:
> On 2/27/11 11:18 AM, "Roy Sigurd Karlsbakk"<roy at
karlsbakk.net>  wrote:
>
>> I cannot but agree. On Linux and Windoze (haven''t tested
FreeBSD), drives
>> connected to an LSI9211 show up in the correct order, but not on
>> OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a mapping
between
>> device name and slot, since that relies on the SES hardware being
>> properly supported. The answer I''ve got for this issue is,
it''s not an
>> issue, since it''s that way by design etc. This
doesn''t make sense when
>> Linux/Windows show the drives in the correct order. IMHO this looks
more
>> like a design flaw in the driver code
>
> Especially since the SAS3081 cards work as expected. I guess I''ll
start
> looking for some more of the 3Gb SAS controllers and chalk the 9211s up as
> a failed bit.
LSISAS3081 cards use a different driver, mpt(7d), which still also
supports parallel SCSI and has different firmware to what you have
in your 9211 card. The 9211 card is also 2nd generation SAS, not 1st
generation like the 3081.

Personally, having worked on the mpt_sas(7d) project, I''m disappointed
that you believe the card and its driver are "a failed bit".


James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog

On 28/02/11 03:18 AM, Roy Sigurd Karlsbakk wrote:
>> I can live with that-- but I really want to know what (real, not
>> virtual) controllers disks are connected to; I want to build 3 8-disk
>> RAIDz2 vdevs now (with room for a fourth for expansion later) and I
>> really want to make sure each of those vdevs has fewer than three
>> disks per controller so a single controller failure can degrade my
>> vdevs but not kill them.
>
> I cannot but agree. On Linux and Windoze (haven''t tested FreeBSD),
> drives connected to an LSI9211 show up in the correct order, but not on
> OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a mapping
between
> device name and slot, since that relies on the SES hardware being
> properly supported. The answer I''ve got for this issue is,
it''s not an
> issue, since it''s that way by design etc. This doesn''t
make sense when
> Linux/Windows show the drives in the correct order. IMHO this looks
> more like a design flaw in the driver code

In Solaris-land we have different design requirements for drivers
when compared with Linux and Windows. As to what Linux and Windows
do inside their drivers, I cannot say - I don''t have their source.

It is possible that since Linux and Windows are fairly closely tied
to the PC architecture, that perhaps they do some bios calls to
try to figure out "correct order" mappings.


James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog

On 28/02/11 02:08 AM, Dave Pooser wrote:
> On 2/27/11 5:15 AM, "James C. McPherson"<jmcp at
opensolaris.org>  wrote:
>
>> On 27/02/11 05:24 PM, Dave Pooser wrote:
>>> On 2/26/11 7:43 PM, "Bill Sommerfeld"<sommerfeld at
hamachi.org>   wrote:
>>>
>>>> On your system, c12 is the mpxio virtual controller; any disk
which is
>>>> potentially multipath-able (and that includes the SAS drives)
will
>>>> appear as a child of the virtual controller (rather than appear
as the
>>>> child of two or more different physical controllers).
>>>
>>> Hmm... That makes sense, except that my drives are all SATA because
I''m
>>> cheap^H^H^H fiscally conservative.  :^)
>>
>> They''re attached to a SAS hba, which is doing translations for
them
>> using SATL - SAS to ATA Translation Layer.
>
> Yeah, but they''re still not multipathable, are they?
Not _physically_ multipathable, but the driver and the chip combine
their efforts to make this opaque to you. That way you, the user, only
have to worry about one path.
>>> ''stmsboot -L'' displayed no mappings,
>>
>> this is because mpt_sas(7d) controllers - which you have - are using
>> MPxIO by default. Running stmsboot -L will only show mappings if
you''ve
>> enabled or disabled MPxIO....
>>
>>> but I went ahead and tried stmsboot
>>> -d to disable multipathing;
>>
>> ... and now you have disabled MPxIO, stmsboot -L should show mappings.
>
> Nope:
> locadmin at bigdawg2:~# stmsboot -L
> stmsboot: MPXIO disabled
Sorry, forgot one step (it''s been a while since I actually disabled
MPxIO): you have to re-enable MPxIO :-)

>>> after reboot instead of seeing nine disks on a
>>> single controller I now see ten different controllers (in a machine
that
>>> has four PCI controllers and one motherboard controller):
>>
>> This is a side effect of how your expanders are configured to operate
>> on your motherboard.
>
> But there shouldn''t be any expanders in the system-- the front
backplane
> has six SFF-8087 ports to control 24 drives, and the rear backplane has
> three more SFF-8087 ports to control 12 more drives. Each of those ports
> is connected directly to an SFF-8087 port on an LSI 9211-8i controller,
> except that the ninth port is connected to the integrated LSI 2008
> controller on the motherboard.
I misread your initial email, sorry.

So your disks are connected to separate PHYs on the HBA, by virtue
of their cabling. You can see this for yourself by looking at the
iport at xx element in the physical paths:


1. c13t5000CCA222DF92A0d0
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0

2. c14t5000CCA222DF8FBEd0
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0

The "xx" part is a bitmask, starting from 0, which gives you an
indication of which PHY the device is attached to.

Your disk #1 above is connected to iport at 10, which is PHY #4 when
you have x1 ports:


PHY  iport@
0	1
1	2
2	4
3	8
4	10
5	20
6	40
7	80

otoh, if you have x4 "wide" ports:

PHY	iport at 0
0	f	(these are really 0f, but we drop the leading 0)
1	f
2	f
3	f
4	f0
5	f0
6	f0
7	f0


>> If you''re lucky, your expanders and the enclosure that
they''re
>> configured into will show up with one or more SES targets. If
>> that''s the case, you might be able to see bay numbers with the
>> fmtopo command - when you run it as root:
>>
>> # /usr/lib/fm/fmd/fmtopo -V
>>
>> If this doesn''t work for you, then you''ll have to
resort to the
>> tried and tested use of dd to /dev/null for each disk, and see
>> which lights blink.
>
> I can live with that-- but I really want to know what (real, not virtual)
> controllers disks are connected to; I want to build 3 8-disk RAIDz2 vdevs
> now (with room for a fourth for expansion later) and I really want to make
> sure each of those vdevs has fewer than three disks per controller so a
> single controller failure can degrade my vdevs but not kill them.
With the information above about the PHY/iport relationship, I
hope you can now see better what your physical layout is. Also,
please remember that using MPxIO means you have a single virtual
controller, and the driver stack handles the translation to physical
for you so you don''t have to worry about that aspect. Of course,
if you want to worry about it, feel free.
> Probably my next step is going to be to take a look with Nexenta Core or
> FreeBSD (or maybe SolEx11 for a temporary eval) and see if either of those
> gives me a saner view, but other suggestions would be appreciated.
Not sure where FreeBSD is up to, nor Nexenta Core. If Nexenta Core
includes changes from snv_118 onwards, then you''ll have pretty much
the same mpt_sas(7d) driver that S11x has.


James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog

On 2/27/11 4:07 PM, "James C. McPherson" <jmcp at
opensolaris.org> wrote:
>I misread your initial email, sorry.
No worries-- I probably could have written it more clearly.
>So your disks are connected to separate PHYs on the HBA, by virtue
>of their cabling. You can see this for yourself by looking at the
>iport at xx element in the physical paths:
>
>1. c13t5000CCA222DF92A0d0
>/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0
>
>2. c14t5000CCA222DF8FBEd0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
>
>The "xx" part is a bitmask, starting from 0, which gives you an
>indication of which PHY the device is attached to.
>
>Your disk #1 above is connected to iport at 10, which is PHY #4 when
>you have x1 ports:
>
>
>PHY  iport@
>0    1
>1    2
>2    4
>3    8
>4    10
>5    20
>6    40
>7    80
OK, bear with me for a moment because I''m feeling extra dense this
evening.

The PHY tells me which port on the HBA I''m connected to. What tells me
which HBA? That''s the information I care most about, and if that
information is contained up there I''ll do a happy dance and head on in
to
the office to start building zpools.

>With the information above about the PHY/iport relationship, I
>hope you can now see better what your physical layout is. Also,
>please remember that using MPxIO means you have a single virtual
>controller, and the driver stack handles the translation to physical
>for you so you don''t have to worry about that aspect. Of course,
>if you want to worry about it, feel free.
Well, I want to make sure that a single controller failure can''t cause
any
of my RAIDz2 vdevs to fault. I know I can do that manually by building the
vdevs in such a way that no more than two drives are on a single
controller. If the virtual controller is smart enough to do that
automagically-- when I''m using SATA disks and a backplane that
doesn''t
support multipathing-- then I have no complaints and I owe you a beer or
three the next time you''re in the Dallas area. But that seems unlikely
to
me, and so I think I have to worry about it. I''d love to be wrong,
though!
>Personally, having worked on the mpt_sas(7d) project, I''m
disappointed
>that you believe the card and its driver are "a failed bit".

I''d like to revise and extend my remarks and replace that with "a
suboptimal choice for this project." In fact, if I can''t make this
work my
backup plan is to take some of my storage towers that have only one HBA,
put the 9211s in them and grab the LSISAS3081 cards out of those towers
for this beast. So those cards will still get productive use -- not a
failed bit, at worst just not serving the purpose I had in mind.
-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On 28/02/11 12:46 PM, Dave Pooser wrote:
> On 2/27/11 4:07 PM, "James C. McPherson"<jmcp at
opensolaris.org>  wrote:
...
>> PHY  iport@
>> 0    1
>> 1    2
>> 2    4
>> 3    8
>> 4    10
>> 5    20
>> 6    40
>> 7    80
>
> OK, bear with me for a moment because I''m feeling extra dense this
evening.
>
> The PHY tells me which port on the HBA I''m connected to. What
tells me
> which HBA?
The devinfo path that''s reported in the non-MPxIO output from
format < /dev/null   (see below)

 > That''s the information I care most about, and if
that
> information is contained up there I''ll do a happy dance and head
on in to
> the office to start building zpools.

I''ve arranged these by devinfo path:

1st controller

c10t2d0                /pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at
4/disk at p2,0
c15t5000CCA222E006B6d0 
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 8/disk at
w5000cca222e006b6,0
c13t5000CCA222DF92A0d0 
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0
c12t5000CCA222E0533Fd0 
/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 20/disk at
w5000cca222e0533f,0

The most likely reason why you''re seeing a c10t2d0 is because the
disk is failing to respond in the required fashion for a particular
SCSI INQUIRY command when the disk is attached to the system.


2nd controller
c16t5000CCA222DDD7BAd0 
/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at 2/disk at
w5000cca222ddd7ba,0


3rd controller
c14t5000CCA222DF8FBEd0 
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
c18t5000CCA222DEAFE6d0 
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 2/disk at
w5000cca222deafe6,0
c19t5000CCA222E0A3DEd0 
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 4/disk at
w5000cca222e0a3de,0
c20t5000CCA222E046B7d0 
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 8/disk at
w5000cca222e046b7,0
c17t5000CCA222DF3CECd0 
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 20/disk at
w5000cca222df3cec,0


>> With the information above about the PHY/iport relationship, I
>> hope you can now see better what your physical layout is. Also,
>> please remember that using MPxIO means you have a single virtual
>> controller, and the driver stack handles the translation to physical
>> for you so you don''t have to worry about that aspect. Of
course,
>> if you want to worry about it, feel free.
>
> Well, I want to make sure that a single controller failure can''t
cause any
> of my RAIDz2 vdevs to fault. I know I can do that manually by building the
> vdevs in such a way that no more than two drives are on a single
> controller. If the virtual controller is smart enough to do that
> automagically-- when I''m using SATA disks and a backplane that
doesn''t
> support multipathing-- then I have no complaints and I owe you a beer or
> three the next time you''re in the Dallas area. But that seems
unlikely to
> me, and so I think I have to worry about it. I''d love to be wrong,
though!
No, the controller doesn''t do that for you. That''s one bit of
AI that
we haven''t quite got to just yet. Unless you wanted to spend $LOTS on
a full frame HDS beast ;-)
>> Personally, having worked on the mpt_sas(7d) project, I''m
disappointed
>> that you believe the card and its driver are "a failed bit".
> I''d like to revise and extend my remarks and replace that with
"a
> suboptimal choice for this project."
Not knowing your other requirements for the project, I''ll settle
for this version :)

[snip]


hth,
James
--
Oracle
http://www.jmcp.homeunix.com/blog

On 2/27/11 10:06 PM, "James C. McPherson" <jmcp at
opensolaris.org> wrote:
>I''ve arranged these by devinfo path:
>
>1st controller
>
>c10t2d0           
>/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 4/disk at p2,0
>c15t5000CCA222E006B6d0
>/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 8/disk at
w5000cca222e006b6,0
>c13t5000CCA222DF92A0d0
>/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 10/disk at
w5000cca222df92a0,0
>c12t5000CCA222E0533Fd0
>/pci at 0,0/pci8086,340a at 3/pci1000,72 at 0/iport at 20/disk at
w5000cca222e0533f,0
>
>The most likely reason why you''re seeing a c10t2d0 is because the
>disk is failing to respond in the required fashion for a particular
>SCSI INQUIRY command when the disk is attached to the system.
That''s an inexpensive SSD used as the boot disk, so it''s
different enough
from the other devices I can''t say I''m stunned that it behaves
differently.
>2nd controller
>c16t5000CCA222DDD7BAd0
>/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at 2/disk at
w5000cca222ddd7ba,0
>3rd controller
>c14t5000CCA222DF8FBEd0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
>c18t5000CCA222DEAFE6d0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 2/disk at
w5000cca222deafe6,0
>c19t5000CCA222E0A3DEd0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 4/disk at
w5000cca222e0a3de,0
>c20t5000CCA222E046B7d0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 8/disk at
w5000cca222e046b7,0
>c17t5000CCA222DF3CECd0
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 20/disk at
w5000cca222df3cec,0
So I mentioned I''m dense tonight, right? Is the key there where it says
340<x>@<y>, so each controller will have a different letter
associated
with it and a different number after the @? (That is, presumably in this
system there''s a 340b at 4 and a 340d at 6 if I add more drives and try
''format''
again?)
>>I''d like to revise and extend my remarks and replace that with
"a
>>suboptimal choice for this project."
>Not knowing your other requirements for the project, I''ll settle
>for this version :)

Actually at this point I think I have to re-revise it to "just fine for
this project had I brains enough to comprehend the output of
''format''."
:^)
-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On 28/02/11 02:51 PM, Dave Pooser wrote:
> On 2/27/11 10:06 PM, "James C. McPherson"<jmcp at
opensolaris.org>  wrote:
...
>> 2nd controller
>> c16t5000CCA222DDD7BAd0
>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0/iport at 2/disk at
w5000cca222ddd7ba,0
>
>
>> 3rd controller
>> c14t5000CCA222DF8FBEd0
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 1/disk at
w5000cca222df8fbe,0
>> c18t5000CCA222DEAFE6d0
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 2/disk at
w5000cca222deafe6,0
>> c19t5000CCA222E0A3DEd0
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 4/disk at
w5000cca222e0a3de,0
>> c20t5000CCA222E046B7d0
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 8/disk at
w5000cca222e046b7,0
>> c17t5000CCA222DF3CECd0
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0/iport at 20/disk at
w5000cca222df3cec,0
>
> So I mentioned I''m dense tonight, right? Is the key there where it
says
> 340<x>@<y>, so each controller will have a different letter
associated
> with it and a different number after the @? (That is, presumably in this
> system there''s a 340b at 4 and a 340d at 6 if I add more drives
and try ''format''
> again?)
So you''ve got two controllers

/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0
and
/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0

which are in different slots on your motherboard and connected to
different PCI Express Root Ports - which should help with transfer
rates amongst other things. Have a look at /usr/share/hwdata/pci.ids
for 340[0-9a-f] after the line which starts with 8086.

The addresses are hexadecimal, and based on the probe order for each
slot, as well as on the PCI Express device id. Figuring out what the
unit address (after the "@") is for this part of the device path is
deep magic that''s contained within the nexus driver for the bus type;
in this case, pcie.

Your 2nd controller is attached to
340c  5520/X58 I/O Hub PCI Express Root Port 5

while the 3rd is attached to
340e  5520/5500/X58 I/O Hub PCI Express Root Port 7

You mention 340b and 340d - these are
340b  5520/X58 I/O Hub PCI Express Root Port 4
and
340d  5520/X58 I/O Hub PCI Express Root Port 6



Now, just to make things more complex, you can have different root nodes
as well as pci-to-pci-X and pci-to-pci-E bridges, and depending on what
your BIOS thinks is going on, you''ll see different nexus(es) in use.

If you want to dig into this more, I suggest that we take the discussion
offline and that you send me a copy of the output from "prtconf -v"
and
"fmtopo -dV" run as root.


cheers,
James
--
Oracle
http://www.jmcp.homeunix.com/blog

On Feb 27, 2011, at 9:18 AM, Roy Sigurd Karlsbakk wrote:
>> I can live with that-- but I really want to know what (real, not
virtual)
>> controllers disks are connected to; I want to build 3 8-disk RAIDz2
vdevs
>> now (with room for a fourth for expansion later) and I really want to
make
>> sure each of those vdevs has fewer than three disks per controller so
>> a single controller failure can degrade my vdevs but not kill them.
> 
> I cannot but agree. On Linux and Windoze (haven''t tested FreeBSD),
drives connected to an LSI9211 show up in the correct order, but not on
OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a mapping between
device name and slot, since that relies on the SES hardware being properly
supported. The answer I''ve got for this issue is, it''s not an
issue, since it''s that way by design etc. This doesn''t make
sense when Linux/Windows show the drives in the correct order. IMHO this looks
more like a design flaw in the driver code
lsiutil can help. The question is whether the physical labels or silkscreen
match
the "slot" as reported by lsiutil.
 -- richard

On 2/27/11 11:13 PM, "James C. McPherson" <jmcp at
opensolaris.org> wrote:
>/pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0
>and
>/pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0
>
>which are in different slots on your motherboard and connected to
>different PCI Express Root Ports - which should help with transfer
>rates amongst other things. Have a look at /usr/share/hwdata/pci.ids
>for 340[0-9a-f] after the line which starts with 8086.
That''s the information I needed; I now have the drives allocated across
multiple controllers for the fault-tolerance I was looking for.

Thanks for all your help-- not only can I fully, unequivocally retract my
"failed bit" crack, but I just ordered two more of these cards for my
next
project!  :^)
--
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com <http://www.alfordmedia.com/>

> > I cannot but agree. On Linux and Windoze (haven''t tested
FreeBSD),
> > drives connected to an LSI9211 show up in the correct order, but not
> > on OI/osol/S11ex (IIRC), and fmtopo doesn''t always show a
mapping
> > between device name and slot, since that relies on the SES hardware
> > being properly supported. The answer I''ve got for this issue
is,
> > it''s not an issue, since it''s that way by design
etc. This doesn''t
> > make sense when Linux/Windows show the drives in the correct order.
> > IMHO this looks more like a design flaw in the driver code
> 
> lsiutil can help. The question is whether the physical labels or
> silkscreen match
> the "slot" as reported by lsiutil.
Last I checked, it didn''t help much. IMHO we need a driver that can
display the drives in the order they''re plugged in. Like Windoze. Like
Linux. Like FreeBSD. I really don''t understand what should be so hard
to do it like the others. As one said "I don''t have their
sources", both Linux and FreeBSD are OSS software, so the source should be
ready quite easuly.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On  1/03/11 03:00 AM, Dave Pooser wrote:
> On 2/27/11 11:13 PM, "James C. McPherson"<jmcp at
opensolaris.org>  wrote:
>
>> /pci at 0,0/pci8086,340c at 5/pci1000,3020 at 0
>> and
>> /pci at 0,0/pci8086,340e at 7/pci1000,3020 at 0
>>
>> which are in different slots on your motherboard and connected to
>> different PCI Express Root Ports - which should help with transfer
>> rates amongst other things. Have a look at /usr/share/hwdata/pci.ids
>> for 340[0-9a-f] after the line which starts with 8086.
>
> That''s the information I needed; I now have the drives allocated
across
> multiple controllers for the fault-tolerance I was looking for.
>
> Thanks for all your help-- not only can I fully, unequivocally retract my
> "failed bit" crack, but I just ordered two more of these cards
for my next
> project!  :^)
I''m glad I could help.


cheers,
James
--
Oracle
http://www.jmcp.homeunix.com/blog

On  1/03/11 07:02 AM, Roy Sigurd Karlsbakk wrote:
...
> Last I checked, it didn''t help much. IMHO we need a driver that
can
 > display the drives in the order they''re plugged in. Like Windoze.
 > Like Linux. Like FreeBSD. I really don''t understand what should
be
 > so hard to do it like the others. As one said "I don''t have
their
 > sources", both Linux and FreeBSD are OSS software, so the source
 > should be ready quite easuly.

The difference is in the license. While I could look at what a
BSD-licensed driver does, I do not want to look at a GPL-licensed
driver and create problems for myself or my employer.


James C. McPherson
--
Oracle
http://www.jmcp.homeunix.com/blog

On 2/28/11 4:23 PM, "Garrett D''Amore" <garrett at
nexenta.com> wrote:
>Drives are ordered in the order they are *enumerated* when they *first*
>show up in the system.  *Ever*.
Is the same true of controllers? That is, will c12 remain c12 or
/pci at 0,0/pci8086,340c at 5 remain /pci at 0,0/pci8086,340c at 5 even if other
controllers are active?
-- 
Dave Pooser, ACSA
Manager of Information Services
Alford Media  http://www.alfordmedia.com

On Mon, Feb 28, 2011 at 9:39 PM, Dave Pooser <dave.zfs at alfordmedia.com>
wrote:
> Is the same true of controllers? That is, will c12 remain c12 or
> /pci at 0,0/pci8086,340c at 5 remain /pci at 0,0/pci8086,340c at 5 even if
other
> controllers are active?
You can rebuild the device tree if it bothers you. There are some
(outdated) instructions here:
http://spiralbound.net/blog/2005/12/21/rebuilding-the-solaris-device-tree
. I think you can do this all with a new boot environment, rather than
boot from a CD.

-B

-- 
Brandon High : bhigh at freaks.com

(Dave P...I sent this yesterday, but it bounced on your email address)

A small comment from me would be to create some test pools and replace
devices in the pools to see if device names remain the same or change
during these operations.

If the device names change and the pools are unhappy, retest similar
operations while the pools'' are exported.

I''ve seen enough controller/device numbering wreak havoc on pool
availability that I''m automatically paranoid when I see the controller
numbering that you started with.

Thanks,

Cindy



On 02/28/11 22:39, Dave Pooser wrote:
> On 2/28/11 4:23 PM, "Garrett D''Amore" <garrett at
nexenta.com> wrote:
> 
>> Drives are ordered in the order they are *enumerated* when they *first*
>> show up in the system.  *Ever*.
> 
> Is the same true of controllers? That is, will c12 remain c12 or
> /pci at 0,0/pci8086,340c at 5 remain /pci at 0,0/pci8086,340c at 5 even if
other
> controllers are active?

Surprised that one of the most approachable outputs for any customer to
use which would enable simple identification/resolution of many
of these discussions didn''t come up, namely:

	cfgadm -al

for a reasonable physical mapping in which SAS/SATA drives are
relatively easy to map out by ID and of course by association
with their distinct physical controller paths.

And if you want to relate this view to the underlying physical
path to the device, then

	cfgadm -alv

will clarify the physical mapping.

Both commands can take <controller>::<target> arguments if you 
want to restrict the voluminous output somewhat.

BTW, you also should be looking to

	mpathadm show LU <device>

to successfully decode the virtual device entries.

Craig

On 1 Mar 2011, at 16:10, Cindy Swearingen wrote:
> (Dave P...I sent this yesterday, but it bounced on your email address)
> 
> A small comment from me would be to create some test pools and replace
> devices in the pools to see if device names remain the same or change
> during these operations.
> 
> If the device names change and the pools are unhappy, retest similar
> operations while the pools'' are exported.
> 
> I''ve seen enough controller/device numbering wreak havoc on pool
> availability that I''m automatically paranoid when I see the
controller
> numbering that you started with.
> 
> Thanks,
> 
> Cindy
> 
> 
> 
> On 02/28/11 22:39, Dave Pooser wrote:
>> On 2/28/11 4:23 PM, "Garrett D''Amore" <garrett at
nexenta.com> wrote:
>>> Drives are ordered in the order they are *enumerated* when they
*first*
>>> show up in the system.  *Ever*.
>> Is the same true of controllers? That is, will c12 remain c12 or
>> /pci at 0,0/pci8086,340c at 5 remain /pci at 0,0/pci8086,340c at 5 even
if other
>> controllers are active?
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-- 
Craig Morgan
Cinnabar Solutions Ltd

t: +44 (0)791 338 3190
f: +44 (0)870 705 1726
e: craig at cinnabar-solutions.com
w: www.cinnabar-solutions.com

> On 02/28/11 22:39, Dave Pooser wrote:
> > On 2/28/11 4:23 PM, "Garrett D''Amore" <garrett
at nexenta.com>
> wrote:
> >
> >> Drives are ordered in the order they are *enumerated* when they
*first*
> >> show up in the system.  *Ever*.
> >
> > Is the same true of controllers? That is, will c12 remain c12 or
> > /pci at 0,0/pci8086,340c at 5 remain /pci at 0,0/pci8086,340c at 5
even if other
> > controllers are active?
Well, it will so long as the contents of /etc/path_to_inst, /dev/cfg,
and /dev/[r]dsk remain untouched. 

If one is feeling intrepid (or annoyed that the SATA boot drives you
want to be c0 are showing up as c4 because the USB controllers are
getting enumerated first), one can modify this. For SAS controllers,
selectively nuking the contents of /dev/dsk/cX, renaming /dev/cfg/cX to
/dev/cfg/cY, updating the archive, and rebooting -r works rather well.
For ICH10s, I''ve discovered that if one writes a for-loop that renames
/dev/[r]dsk/c1t0d0s0 -> c0t0d0s0 while otherwise cleansing out unwanted
entries, the devfsadm reconfig process honors the naming implied by
existing symlinks and re-numbers the USB controllers accordingly.

Do *not* try this at ... well, actually, try it at home long before you
try this on your production box, as this is also a very good way to make
your machine completely unbootable - and I wouldn''t mention that
you''re
doing this to anyone at S.uh.Oracle... 

(Maybe there''s an easier way. I don''t know it, though.)

-bacon
who is incredibly bugged by the idea of the boot drive being anything
BUT c0t0d0s0....

> Date: Mon, 28 Feb 2011 22:02:37 +0100 (CET)
> From: Roy Sigurd Karlsbakk <roy at karlsbakk.net>
> 
> > > I cannot but agree. On Linux and Windoze (haven''t tested
FreeBSD),
> > > drives connected to an LSI9211 show up in the correct order, but
not
> > > on OI/osol/S11ex (IIRC), and fmtopo doesn''t always show
a mapping
> > > between device name and slot, since that relies on the SES
hardware
> > > being properly supported. The answer I''ve got for this
issue is,
> > > it''s not an issue, since it''s that way by
design etc. This doesn''t
> > > make sense when Linux/Windows show the drives in the correct
order.
> > > IMHO this looks more like a design flaw in the driver code
> > lsiutil can help. The question is whether the physical labels or 
> > silkscreen match the "slot" as reported by lsiutil.
> 
> Last I checked, it didn''t help much. IMHO we need a driver that
can
> display the drives in the order they''re plugged in. Like Windoze.
> Like Linux. Like FreeBSD. I really don''t understand what should be
> so hard to do it like the others. As one said "I don''t have
their
> sources", both Linux and FreeBSD are OSS software, so the source
> should be ready quite easuly.
OK, so I''m a little behind on this, but...

Yes, it is nice to know controller order - well, maybe. That assume the
LSI hasn''t decided to remap targets because you''ve moved
drives around -
the 1068/78 supported target persistence, but the LSI2008 doesn''t.
Nothing''s guaranteed. 

The SES driver is, I presume, a fairly generic driver for knowing about
enclosures. sd(7) on the other hand doesn''t really know crap, other
than
"this is the DDI path to what I want to talk to" from which it can
infer
a target, but it doesn''t know that that target means anything, and in
my
humble experience (having a whole pile of SMC hardware on LSI
controllers), it really has NFC - and it doesn''t need to, since
it''s
just the bit controlling the drive - what does it care what slot it''s
in?

You can use lsiutil advanced commands to query the expander chip and get
the topology, but that''s a PITA. 

The best way to deal with it IMHO - assuming you''re serious about the
need and willing to pay a small fee - is to go to www.santools.com and
buy a license for smartmon-ux (no relation to smartmontools). One
command, and it will generate a complete topology of your system,
mapping drive serial number to WWN to DDI path to enclosure to slot in
one nice big file. (It can also do lots of other nifty things but
that''s
not the point. However, being able to map /dev/dsk/cXtYdZsA -> encBslotC
and toggle LEDs is rather important when you''ve got a few hundred
drives
strung to a couple boxes.

And at least for SMC 847s, the slots do match the silkscreen (or the
schema in the manual).

-bacon
(it''s amazing how many drives you can string off a box, with a little
effort...)

> > Last I checked, it didn''t help much. IMHO we need a driver
that can
> > display the drives in the order they''re plugged in. Like
Windoze.
> > Like Linux. Like FreeBSD. I really don''t understand what
should be
> > so hard to do it like the others. As one said "I don''t
have their
> > sources", both Linux and FreeBSD are OSS software, so the source
> > should be ready quite easuly.
> 
> The difference is in the license. While I could look at what a
> BSD-licensed driver does, I do not want to look at a GPL-licensed
> driver and create problems for myself or my employer.
It''s no problem reusing methodics from a GPL driver in CDDL, but the
code must be rewritten. Also, copying trivial parts won''t break the
license, but then, I guess the Solaris driver API is quite different from the
one on Linux. Also, perhaps the FreeBSD driver works better at this?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Thu, 2011-03-03 at 21:02 +0100, Roy Sigurd Karlsbakk
wrote:
> > > Last I checked, it didn''t help much. IMHO we need a
driver that can
> > > display the drives in the order they''re plugged in. Like
Windoze.
> > > Like Linux. Like FreeBSD. I really don''t understand what
should be
> > > so hard to do it like the others. As one said "I
don''t have their
> > > sources", both Linux and FreeBSD are OSS software, so the
source
> > > should be ready quite easuly.
> > 
> > The difference is in the license. While I could look at what a
> > BSD-licensed driver does, I do not want to look at a GPL-licensed
> > driver and create problems for myself or my employer.
> 
> It''s no problem reusing methodics from a GPL driver in CDDL, but
the code must be rewritten. Also, copying trivial parts won''t break the
license, but then, I guess the Solaris driver API is quite different from the
one on Linux. Also, perhaps the FreeBSD driver works better at this?
> 
What you''re not realizing is that FreeBSD and Linux were designed with
a
different set of assumptions.  The basic assumption there is that you
won''t disturb the probe order, either by adding hardware, or by having
hardware that simply is non-deterministic in probing.

Solaris doesn''t do that.  It enumerates the hardware the first time it
sees it, and then remembers the "physical" path to that disk as an
alias, usually both in /etc/path_to_instance and in the symbolic links
that populate /dev/.

The failure here is in your assumption that the Linux and *BSD behavior
is correct.  It is correct in a small system that doesn''t have to
concern itself with fallout from changes in the probe order due to e.g.
dynamic reconfiguration, but in enterprise systems where devices and
buses can be hot swapped, it fails miserably.  The Solaris behavior is
simply better, once you learn to accept it.

	- Garrett