Based on recommendations from this list, I asked the company that built
my box to use an LSI SAS3081E controller.
The first problem I noticed was that the drive-numbers were ordered
incorrectly. That is, given that my system has 24 bays (6 rows, 4
bays/row), the drive numbers from top-to-bottom & left-to-right were 6,
1, 0, 2, 4, 5 - even though when the system boots, each drive is scanned
in perfect order (I can tell by watching the LEDs blink).
I contacted LSI tech support and they explained:
<start response>
SAS treats device IDs differently than SCSI. LSI SAS controllers
"remember" devices in the order they were discovered by the
controller.
This memory is persistent across power cycles. It is based on the world
wide name (WWN) given uniquely to every SAS device. This allows your
boot device to remain your boot device no matter where it migrates in
the SAS topology.
In order to clear the memory of existing devices you need at least one
device that will not be present in your final configuration. Re-boot
the machine and enter the LSI configuration utility (CTRL-C). Then find
your way to SAS Topology. To see "more" options, press CTRL-M.
Choose
the option to clear all non-present device IDs. This clears the
persistent memory of all devices not present at that time. Exchange the
drives. The system will now remember the order it finds the drives
after the next boot cycle.
<end response>
Sure enough, I was able to physical reorder my drives so they were 0, 1,
2, 4, 5, 6 - so, appearantly, the company that put my system together
moved the drives around after they were initially scanned. But where is
3? (answer below). Then I tried another test:
1. make first disk blink
# run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10
10+0 records in
10+0 records out
2. pull disk ''0'' out and replace it with a brand new disk
# run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10
dd: /dev/dsk/c2t0d0p0: open: No such file or directory
3. scratch head and try again with ''3'' (I had previously
cleared the
LSI''s controllers memory)
# run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10
10+0 records in
10+0 records out
So, it seems my SAS controller is being too smart for its own good - it
tracks the drives themselves, not the drive-bays. If I hot-swap a brand
new drive into a bay, Solaris will see it as a new disk, not a
replacement for the old disk. How can ZFS support this? I asked the
LSI tech support again and got:
<start quote>
I don''t have the knowledge to answer that, so I''ll just say
this: most vendors, including Sun, set up the SAS HBA to use
"enclosure/slot" naming, which means that if a drive is
swapped, it does NOT get a new name (after all, the enclosure
and slot did not change).
<end quote>
So, now I turn to you... Here some information about my system:
Specs:
Motherboard: SuperMicro H8DME-2 Rev 2.01
- BIOS: AMI v2.58
HBA: LSI SAS3081E (SN: P068170707) installed in Slot #5
- LSI Configuration Utility v6.16.00.00 (2007-05-07)
Backplane: CI-Design 12-6412-01BR
HBA connected to BP via two SFF-8087-SFF-8087 cables
OS: SXCE b74
Details:
* Chassis has 24 SAS/SATA bays
* There are 6 backplanes - one for each *row* of drives
* I currently have only 6 drives installed (see pic)
* The LSI card is plugged into backplanes 1 & 2
* The LSI card is NOT configured to do any RAID - its only
JBOD as I''m using Solaris''s ZFS (software-RAID)
Question:
* I only plan to use SATA drives, would using a SATA controller like
Supermicro''s
AOC-SAT2-MV8 help?
Thanks again,
Kent
James C. McPherson
2007-Dec-12 11:01 UTC
[zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Kent, I''m one of the team that works on Solaris'' mpt driver, which we recently enhanced to deliver mpxio support with SAS. I have a bit of knowledge about your issue :-) Kent Watsen wrote:> Based on recommendations from this list, I asked the company that built > my box to use an LSI SAS3081E controller. > > The first problem I noticed was that the drive-numbers were ordered > incorrectly. That is, given that my system has 24 bays (6 rows, 4 > bays/row), the drive numbers from top-to-bottom & left-to-right were 6, > 1, 0, 2, 4, 5 - even though when the system boots, each drive is scanned > in perfect order (I can tell by watching the LEDs blink). > > I contacted LSI tech support and they explained: > > <start response> > SAS treats device IDs differently than SCSI. LSI SAS controllers > "remember" devices in the order they were discovered by the controller. > This memory is persistent across power cycles. It is based on the world > wide name (WWN) given uniquely to every SAS device. This allows your > boot device to remain your boot device no matter where it migrates in > the SAS topology. > > In order to clear the memory of existing devices you need at least one > device that will not be present in your final configuration. Re-boot > the machine and enter the LSI configuration utility (CTRL-C). Then find > your way to SAS Topology. To see "more" options, press CTRL-M. Choose > the option to clear all non-present device IDs. This clears the > persistent memory of all devices not present at that time. Exchange the > drives. The system will now remember the order it finds the drives > after the next boot cycle. > <end response>Firstly, yes, the LSI SAS hbas do use persistent mapping, with a "logical target id" by default. This is where the hba does the translation between the physical disk device''s SAS address (which you''ll see in "prtconf -v" as the devid), and an essentially arbitrary target number which gets passed up to the OS - in this case Solaris. The support person @ LSI was correct about deleting all those mappings. Yes, the controller is being smart and tracking the actual device rather than a particular bay/slot mapping. This isn''t so bad, mostly. The effect for you is that you can''t assume that the replaced device is going to have the same target number as the old one (in fact, I''d call that quite unlikely) so you''ll have to see what the new device name is by checking your dmesg or iostat -En output.> Sure enough, I was able to physical reorder my drives so they were 0, 1, > 2, 4, 5, 6 - so, appearantly, the company that put my system together > moved the drives around after they were initially scanned. But where is > 3? (answer below). Then I tried another test: > > 1. make first disk blink > > # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 > 10+0 records in > 10+0 records out > > 2. pull disk ''0'' out and replace it with a brand new disk > > # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 > dd: /dev/dsk/c2t0d0p0: open: No such file or directory > > 3. scratch head and try again with ''3'' (I had previously cleared the > LSI''s controllers memory) > > # run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10 > 10+0 records in > 10+0 records out > > So, it seems my SAS controller is being too smart for its own good - it > tracks the drives themselves, not the drive-bays. If I hot-swap a brand > new drive into a bay, Solaris will see it as a new disk, not a > replacement for the old disk. How can ZFS support this? I asked the > LSI tech support again and got: > > <start quote> > I don''t have the knowledge to answer that, so I''ll just say > this: most vendors, including Sun, set up the SAS HBA to use > "enclosure/slot" naming, which means that if a drive is > swapped, it does NOT get a new name (after all, the enclosure > and slot did not change). > <end quote>Now here''s where things get murky. At this point in time at least (it may change!) Solaris'' mpt driver uses LSI''s logical target id mapping method. This is *NOT* an enclosure/slot naming method - at least, not from the OS'' point of view. Additionally, unless you''re using an actual real SCSI Enclosure Services (ses) device, there''s no enclosure to provide enclosure/slot mapping with either. Since mpt uses logical target id, therefore the target id which Solaris sees _will definitely change_ if you swap a disk. (I''m a tad annoyed that the LSI support person appears to have made an assumption based on a total lack of understanding about how Solaris'' mpt driver works). (My assumption here is that you''re using Solaris'' mpt(7d) driver rather than LSI''s itmpt driver) So how do you use your system and its up-to-24 drives with ZFS? (a) ensure that you note what Solaris''s idea of the target id is when you replace a drive, then (b) use "zpool replace" to tell ZFS what to do with the new device in your enclosure. I hope the above helps you along the way... but I''m sure you''ll have followup questions, so please don''t hesitate to ask either directly or to the list. best regards, James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Wow, how fortunate for me that you are on this list! I guess I do have a follow-up question... If each new drive gets a new id when plugged into the system - and I learn to discover that drive''s id using dmesg or iostat and use `zfs replace ` correctly - when a drive fails, what will it take for me to physically find it. I''m hoping there is a command, like dd, that I can use to make that drive''s LED blink, but I don''t know if I can trust that `dd` will work at all when the drive is failed! Since I don''t have enclosure-services, does that mean my best option is to manually track id-to-bay mappings? (envision a clip-board hanging on rack) Given that manually tracking shifting ids doesn''t sound appealing to me, would using a SATA controller like the AOC-SAT2-MV8 resolve the issue? Given that I currently only have one LSI HBA - I''d need to get 2 more for all 24 drives ---or--- I could get 3 of these SATA controllers plus 6 discrete-to-8087 reverse breakout cables. Going down the LSI-route would cost about $600 while going down the AOC-SAT2-MV8 would cost about $400. I understand that the SATA controllers are less performant, but I''d gladly exchange some performance that I''m likely to never need to simplify my administrative overhead... Thanks, Kent James C. McPherson wrote:> > Hi Kent, > I''m one of the team that works on Solaris'' mpt driver, which we > recently enhanced to deliver mpxio support with SAS. I have a bit > of knowledge about your issue :-) > > Kent Watsen wrote: >> Based on recommendations from this list, I asked the company that >> built my box to use an LSI SAS3081E controller. >> >> The first problem I noticed was that the drive-numbers were ordered >> incorrectly. That is, given that my system has 24 bays (6 rows, 4 >> bays/row), the drive numbers from top-to-bottom & left-to-right were >> 6, 1, 0, 2, 4, 5 - even though when the system boots, each drive is >> scanned in perfect order (I can tell by watching the LEDs blink). >> >> I contacted LSI tech support and they explained: >> >> <start response> >> SAS treats device IDs differently than SCSI. LSI SAS controllers >> "remember" devices in the order they were discovered by the controller. >> This memory is persistent across power cycles. It is based on the world >> wide name (WWN) given uniquely to every SAS device. This allows your >> boot device to remain your boot device no matter where it migrates in >> the SAS topology. >> >> In order to clear the memory of existing devices you need at least one >> device that will not be present in your final configuration. Re-boot >> the machine and enter the LSI configuration utility (CTRL-C). Then find >> your way to SAS Topology. To see "more" options, press CTRL-M. Choose >> the option to clear all non-present device IDs. This clears the >> persistent memory of all devices not present at that time. Exchange the >> drives. The system will now remember the order it finds the drives >> after the next boot cycle. <end response> > > > Firstly, yes, the LSI SAS hbas do use persistent mapping, > with a "logical target id" by default. This is where the > hba does the translation between the physical disk device''s > SAS address (which you''ll see in "prtconf -v" as the devid), > and an essentially arbitrary target number which gets passed > up to the OS - in this case Solaris. > > The support person @ LSI was correct about deleting all those > mappings. > > Yes, the controller is being smart and tracking the actual > device rather than a particular bay/slot mapping. This isn''t > so bad, mostly. The effect for you is that you can''t assume > that the replaced device is going to have the same target > number as the old one (in fact, I''d call that quite unlikely) > so you''ll have to see what the new device name is by checking > your dmesg or iostat -En output. > > > > >> Sure enough, I was able to physical reorder my drives so they were 0, >> 1, 2, 4, 5, 6 - so, appearantly, the company that put my system >> together moved the drives around after they were initially scanned. >> But where is 3? (answer below). Then I tried another test: >> >> 1. make first disk blink >> >> # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 >> 10+0 records in >> 10+0 records out >> >> 2. pull disk ''0'' out and replace it with a brand new disk >> >> # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 >> dd: /dev/dsk/c2t0d0p0: open: No such file or directory >> >> 3. scratch head and try again with ''3'' (I had previously cleared >> the LSI''s controllers memory) >> >> # run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10 >> 10+0 records in >> 10+0 records out >> >> So, it seems my SAS controller is being too smart for its own good - >> it tracks the drives themselves, not the drive-bays. If I hot-swap a >> brand new drive into a bay, Solaris will see it as a new disk, not a >> replacement for the old disk. How can ZFS support this? I asked the >> LSI tech support again and got: >> >> <start quote> >> I don''t have the knowledge to answer that, so I''ll just say >> this: most vendors, including Sun, set up the SAS HBA to use >> "enclosure/slot" naming, which means that if a drive is >> swapped, it does NOT get a new name (after all, the enclosure >> and slot did not change). >> <end quote> > > Now here''s where things get murky. > > At this point in time at least (it may change!) Solaris'' mpt > driver uses LSI''s logical target id mapping method. This is > *NOT* an enclosure/slot naming method - at least, not from the > OS'' point of view. Additionally, unless you''re using an actual > real SCSI Enclosure Services (ses) device, there''s no enclosure > to provide enclosure/slot mapping with either. > > Since mpt uses logical target id, therefore the target id which > Solaris sees _will definitely change_ if you swap a disk. > > (I''m a tad annoyed that the LSI support person appears to have > made an assumption based on a total lack of understanding about > how Solaris'' mpt driver works). > > (My assumption here is that you''re using Solaris'' mpt(7d) driver > rather than LSI''s itmpt driver) > > > So how do you use your system and its up-to-24 drives with ZFS? > > (a) ensure that you note what Solaris''s idea of the target id > is when you replace a drive, then > > (b) use "zpool replace" to tell ZFS what to do with the new > device in your enclosure. > > > I hope the above helps you along the way... but I''m sure > you''ll have followup questions, so please don''t hesitate > to ask either directly or to the list. > > > > best regards, > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > >
Hi Kent:
I have run into the same problem before, and have worked with LSI and SUN
support to fix it. LSI calls this "persistant drive mapping", and
here is how to clear it
1) obtain the latest version of the program "lsiutil" from LSI. They
don''t seem to have the Solaris versions on their website, but I got it
by email when entering a ticket into their support system. I know that they
have a version for Solaris x86 (and I believe a Sparc version also). The
version I currently have is: "LSI Logic MPT Configuration Utility, Version
1.52, September 7, 2007"
2) Execute the lsiutil program on your target box.
a) first it will ask you to select which card to use (I have multiple cards
in my machine, don''t know if it will ask if you only have 1 card in
your box)
b) then you need to select option 15 (it is a hidden option, not shown on
the menu)
c) then you select option 10 (Clear all persistant mappings)
d) then option 0 multiple times to get out of the program
e) I normally than reboot the box, and the next time it comes up, the drives
are back in order.
e) or (instead of rebooting) option 99, to reset the chip (causes new
mappings to be established), then option 8 (to verify lower target IDs), then
"devfsadm". After devfsadm completes, lsiutil option 42 should
display valid device names (in /dev/rdsk), and "format" should find
the devices so that you can label them.
Hope this helps. I happened to need it last night again (I normally have to run
it after re-imaging a box, assuming that I don''t want to save the data
that was on those drives).
Paul Jochum
This message posted from opensolaris.org
Hi Paul, Already in my LSI Configuration Utility I have an option to clear the persistent mapping for drives not present, but then the card resumes its normal persistent-mapping logic. What I really want is to disable to persistent mapping logic completely - is the `lsiutil` doing that for you? Thanks, Kent Paul Jochum wrote:> Hi Kent: > > I have run into the same problem before, and have worked with LSI and SUN support to fix it. LSI calls this "persistant drive mapping", and here is how to clear it > > 1) obtain the latest version of the program "lsiutil" from LSI. They don''t seem to have the Solaris versions on their website, but I got it by email when entering a ticket into their support system. I know that they have a version for Solaris x86 (and I believe a Sparc version also). The version I currently have is: "LSI Logic MPT Configuration Utility, Version 1.52, September 7, 2007" > > 2) Execute the lsiutil program on your target box. > a) first it will ask you to select which card to use (I have multiple cards in my machine, don''t know if it will ask if you only have 1 card in your box) > b) then you need to select option 15 (it is a hidden option, not shown on the menu) > c) then you select option 10 (Clear all persistant mappings) > d) then option 0 multiple times to get out of the program > e) I normally than reboot the box, and the next time it comes up, the drives are back in order. > e) or (instead of rebooting) option 99, to reset the chip (causes new mappings to be established), then option 8 (to verify lower target IDs), then "devfsadm". After devfsadm completes, lsiutil option 42 should display valid device names (in /dev/rdsk), and "format" should find the devices so that you can label them. > > Hope this helps. I happened to need it last night again (I normally have to run it after re-imaging a box, assuming that I don''t want to save the data that was on those drives). > > Paul Jochum > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > >
Hi Kent:
What the lsiutil does for me is clear the persistent mapping for all
of the drives on a card. I don''t know of a way to disable the mapping
completely (but that does sound like a nice option). Since SUN is
reselling this card now (that is how I got my cards), I wonder if they
can put in a request to LSI to provide this enhancement?
Paul
Kent Watsen wrote:>
> Hi Paul,
>
> Already in my LSI Configuration Utility I have an option to clear the
> persistent mapping for drives not present, but then the card resumes
> its normal persistent-mapping logic. What I really want is to disable
> to persistent mapping logic completely - is the `lsiutil` doing that
> for you?
>
> Thanks,
> Kent
>
>
>
> Paul Jochum wrote:
>> Hi Kent:
>>
>> I have run into the same problem before, and have worked with LSI and
>> SUN support to fix it. LSI calls this "persistant drive
mapping",
>> and here is how to clear it
>>
>> 1) obtain the latest version of the program "lsiutil" from
LSI. They
>> don''t seem to have the Solaris versions on their website, but
I got
>> it by email when entering a ticket into their support system. I know
>> that they have a version for Solaris x86 (and I believe a Sparc
>> version also). The version I currently have is: "LSI Logic MPT
>> Configuration Utility, Version 1.52, September 7, 2007"
>>
>> 2) Execute the lsiutil program on your target box.
>> a) first it will ask you to select which card to use (I have
>> multiple cards in my machine, don''t know if it will ask if you
only
>> have 1 card in your box)
>> b) then you need to select option 15 (it is a hidden option, not
>> shown on the menu)
>> c) then you select option 10 (Clear all persistant mappings)
>> d) then option 0 multiple times to get out of the program
>> e) I normally than reboot the box, and the next time it comes up,
>> the drives are back in order.
>> e) or (instead of rebooting) option 99, to reset the chip (causes
>> new mappings to be established), then option 8 (to verify lower
>> target IDs), then "devfsadm". After devfsadm completes,
lsiutil
>> option 42 should display valid device names (in /dev/rdsk), and
>> "format" should find the devices so that you can label them.
>>
>> Hope this helps. I happened to need it last night again (I normally
>> have to run it after re-imaging a box, assuming that I don''t
want to
>> save the data that was on those drives).
>>
>> Paul Jochum
>>
>>
>> This message posted from opensolaris.org
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>>
>>
>>
James C. McPherson wrote:> > Now here''s where things get murky. > > At this point in time at least (it may change!) Solaris'' mpt > driver uses LSI''s logical target id mapping method. This is > *NOT* an enclosure/slot naming method - at least, not from the > OS'' point of view. Additionally, unless you''re using an actual > real SCSI Enclosure Services (ses) device, there''s no enclosure > to provide enclosure/slot mapping with either. > > Since mpt uses logical target id, therefore the target id which > Solaris sees _will definitely change_ if you swap a disk. >JMCP, For Sun systems, we have 3 LEDs on the drives: 1. Ready to remove (blue) 2. Service required (amber) 3. OK/Activity (green) So there must be a way to set the ready to remove LED from Solaris. In the old days, we could use luxadm(1m). Does that still work, or is there some new equivalent? -- richard
On Wed, Dec 12, 2007 at 09:22:07AM -0800, Richard Elling wrote:> > For Sun systems, we have 3 LEDs on the drives: > 1. Ready to remove (blue) > 2. Service required (amber) > 3. OK/Activity (green) > > So there must be a way to set the ready to remove LED from Solaris. > In the old days, we could use luxadm(1m). Does that still work, or is > there some new equivalent? >For x86 systems, you can use ipmitool to manipulate the led state (ipmitool sunoem led ...). On older galaxy systems, you can only set the fail LED (''io.hdd0.led''), as the ok2rm LED is not physically connected to anything. On newer systems, you can set both the ''fail'' and ''okrm'' LEDs. You cannot change the activity LED except by manually sending the ''set sensor reading'' IPMI command (not available via impitool). For external enclosures, you''ll need a SES control program. Both of these problems are being worked on under the FMA sensor framework to create a unified view through libtopo. Until that''s complete, you''ll be stuck using ad hoc methods. - Eric -- Eric Schrock, FishWorks http://blogs.sun.com/eschrock
Kent Watsen wrote:> Given that manually tracking shifting ids doesn''t sound appealing to > me, would using a SATA controller like the AOC-SAT2-MV8 resolve the > issue? Given that I currently only have one LSI HBA - I''d need to get 2 > more for all 24 drives ---or--- I could get 3 of these SATA controllers > plus 6 discrete-to-8087 reverse breakout cables. Going down the > LSI-route would cost about $600 while going down the AOC-SAT2-MV8 would > cost about $400. I understand that the SATA controllers are less > performant, but I''d gladly exchange some performance that I''m likely to > never need to simplify my administrative overhead... >So, I picked up an AOC-SAT2-MV8 off eBay for not too much and then I got a 4xSATA to one SFF-8087 cable to connect it to one one my six backplanes. But, as fortune would have it, the cable I bought has SATA connectors that are physically too big to plug into the AOC-SAT2-MV8 - since the AOC-SAT2-MV8 stacks two SATA connectors on top of each other... Any chance anyone knows of a 4xSATA to SFF-8087 reverse breakout cable that will plug into the AOC-SAT2-MV8? Alternatively, anyone know how hard it would be to splice the SATA cables that came with the AOC-SAT2-MV8 into the cable I bought? Also, the cable I got actually had 5 cables going into the SFF-8087 - 4 SATA and another that is rectangular and has holes for either 7 or 8 pins (not sure if its for 7 or 8 pins as one of the holes appears a bit different). Any idea what this fifth cable is for? Thanks, Kent
Eric Schrock wrote:> For x86 systems, you can use ipmitool to manipulate the led state > (ipmitool sunoem led ...). On older galaxy systems, you can only set the > fail LED (''io.hdd0.led''), as the ok2rm LED is not physically connected > to anything. On newer systems, you can set both the ''fail'' and ''okrm'' > LEDs. You cannot change the activity LED except by manually sending the > ''set sensor reading'' IPMI command (not available via impitool). > > For external enclosures, you''ll need a SES control program. > > Both of these problems are being worked on under the FMA sensor > framework to create a unified view through libtopo. Until that''s > complete, you''ll be stuck using ad hoc methods. >Hi Eric, I''ve looked at your blog and have tried your suggestions, but I need a little more advice. I am on an x86 system running SXCE svn_74 - the system has 6 SAS backplanes but, according to the integrator, no "real" scsi enclosure services. according to the man page, I found that I could use `sdr elist generic` to list LEDs, but that command doesn''t return any output: # ipmitool sdr elist generic # /* there was no output */ Is it not returning sensor ids because I don''t have real scsi enclosure services? Is there anything I can do or am I doomed to ad hoc methods forever? Thanks, Kent
James C. McPherson
2007-Dec-16 11:12 UTC
[zfs-discuss] LSI SAS3081E = unstable drive numbers?
Kent Watsen wrote:> Eric Schrock wrote: >> For x86 systems, you can use ipmitool to manipulate the led state >> (ipmitool sunoem led ...). On older galaxy systems, you can only set the >> fail LED (''io.hdd0.led''), as the ok2rm LED is not physically connected >> to anything. On newer systems, you can set both the ''fail'' and ''okrm'' >> LEDs. You cannot change the activity LED except by manually sending the >> ''set sensor reading'' IPMI command (not available via impitool). >> >> For external enclosures, you''ll need a SES control program. >> >> Both of these problems are being worked on under the FMA sensor >> framework to create a unified view through libtopo. Until that''s >> complete, you''ll be stuck using ad hoc methods. >> > Hi Eric, > > I''ve looked at your blog and have tried your suggestions, but I need a > little more advice. > > I am on an x86 system running SXCE svn_74 - the system has 6 SAS > backplanes but, according to the integrator, no "real" scsi enclosure > services. according to the man page, I found that I could use `sdr > elist generic` to list LEDs, but that command doesn''t return any output: > > # ipmitool sdr elist generic > # /* there was no output */ > > Is it not returning sensor ids because I don''t have real scsi enclosure > services? Is there anything I can do or am I doomed to ad hoc methods > forever?The version of ipmitool which we''re shipping with snv has this comment at the start of libipmi.h :: (usr/src/lib/libipmi/common/libipmi.h) /* * Private interfaces for communicating with attached services over IPMI. This * library is designed for system software communicating with Sun-supported * service processors over /dev/bmc. It is not a generic IPMI library. * * Documentation references refer to "Intelligent Platform Management Interface * Specification Second Generation v2.0", document revision 1.0 with Februrary * 15, 2006 Markup from "IPMI v2.0 Addenda, Errata, and Clarifications Revision * 3". */ Further, unless you have a /dev/bmc device you won''t get anything back from the shipped ipmitool. Unfortunately that means you are stuck - for the moment at least - with ad hoc measures. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Paul Jochum wrote:> What the lsiutil does for me is clear the persistent mapping for > all of the drives on a card.Since James confirms that I''m doomed to ad hoc methods tracking device-ids to bays, I''m interested in knowing if somehow your ability to clear the persistent mapping for *all* of the drives on the card somehow gets you to a stable-state? Seems to me that, if the tool is run from the BIOS, like the LSI Configuration Utility, then it kind defeats the uptime that I''m trying to achieve by having a RAID system in the first place... Is it, by chance, a user land program?> I don''t know of a way to disable the mapping completely (but that does > sound like a nice option). Since SUN is reselling this card now (that > is how I got my cards), I wonder if they can put in a request to LSI > to provide this enhancement?Yes, please! - can someone from SUN please request LSI to add a feature to selectively disable their card''s persistent drive mapping feature? Thanks, Kent
Hi Kent:
So, in using the lsiutil utility, what do I find, but the following
option: (this was under the hidden option 15)
12. Change (enable/disable) persistence
I have not had a chance to try it, but it says the default is
Enabled. Let me know if you try it.
Paul
Kent Watsen wrote:>
>
>
> Paul Jochum wrote:
>> What the lsiutil does for me is clear the persistent mapping for
>> all of the drives on a card.
> Since James confirms that I''m doomed to ad hoc methods tracking
> device-ids to bays, I''m interested in knowing if somehow your
ability
> to clear the persistent mapping for *all* of the drives on the card
> somehow gets you to a stable-state? Seems to me that, if the tool is
> run from the BIOS, like the LSI Configuration Utility, then it kind
> defeats the uptime that I''m trying to achieve by having a RAID
system
> in the first place... Is it, by chance, a user land program?
>
>> I don''t know of a way to disable the mapping completely (but
that
>> does sound like a nice option). Since SUN is reselling this card now
>> (that is how I got my cards), I wonder if they can put in a request
>> to LSI to provide this enhancement?
> Yes, please! - can someone from SUN please request LSI to add a
> feature to selectively disable their card''s persistent drive
mapping
> feature?
>
>
> Thanks,
> Kent
>
>
>
>
>
Kent Watsen
2007-Dec-18 13:28 UTC
[zfs-discuss] aoc-sat2-mv8 (was: LSI SAS3081E = unstable drive numbers?)
Kent Watsen wrote:> So, I picked up an AOC-SAT2-MV8 off eBay for not too much and then I got > a 4xSATA to one SFF-8087 cable to connect it to one one my six > backplanes. But, as fortune would have it, the cable I bought has SATA > connectors that are physically too big to plug into the AOC-SAT2-MV8 - > since the AOC-SAT2-MV8 stacks two SATA connectors on top of each other... >As a temporary solution, I hooked up the reverse breakout cable using ports 1, 3, 5, and 7 on the aoc-sat2-mv8 - the cables fit this way because its using only one port from each stack. Anyway, the good news is that drives showed up in Solaris right away and their IDs are stable between hot-swaps and reboots. So I''ll be keeping the aoc-sat2-mv8 (anybody want a SAS3081E?) I''ve already ordered more cables for the aoc-sat2-mv8 and will report which ones work when I get them Thanks, Kent
Kent Watsen wrote:> > Kent Watsen wrote: >> So, I picked up an AOC-SAT2-MV8 off eBay for not too much and then I got >> a 4xSATA to one SFF-8087 cable to connect it to one one my six >> backplanes. But, as fortune would have it, the cable I bought has SATA >> connectors that are physically too big to plug into the AOC-SAT2-MV8 - >> since the AOC-SAT2-MV8 stacks two SATA connectors on top of each other... > As a temporary solution, I hooked up the reverse breakout cable using > ports 1, 3, 5, and 7 on the aoc-sat2-mv8 - the cables fit this way > because its using only one port from each stack. Anyway, the good news > is that drives showed up in Solaris right away and their IDs are stable > between hot-swaps and reboots. So I''ll be keeping the aoc-sat2-mv8This is progress, I''m glad to hear it.> (anybody want a SAS3081E?)MEMEMEMEMEMEMEMEMEMEMEME !!! :-)> I''ve already ordered more cables for the aoc-sat2-mv8 and will report > which ones work when I get themThat will be very good info to have - there''s too little information and personal experience surrounding the SAS cabling world as yet. cheers, James C. McPherson -- Solaris kernel software engineer, system admin and troubleshooter http://www.jmcp.homeunix.com/blog http://blogs.sun.com/jmcp Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson
On Tue, 18 Dec 2007, James C. McPherson wrote: ... snip .....> That will be very good info to have - there''s too little information > and personal experience surrounding the SAS cabling world as yet.Here''s a couple of resources: SAS Integrators Guide: wget http://www.lsi.com/documentation/storage/megaraid/SAS_IG.pdf You can always get good advice from: http://www.cs-electronics.com/ Regards, Al Hopper Logical Approach Inc, Plano, TX. al at logical-approach.com Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/ Graduate from "sugar-coating school"? Sorry - I never attended! :)
Al Hopper wrote:> On Tue, 18 Dec 2007, James C. McPherson wrote: > > ... snip ..... >> That will be very good info to have - there''s too little information >> and personal experience surrounding the SAS cabling world as yet. > > Here''s a couple of resources: > > SAS Integrators Guide: > wget http://www.lsi.com/documentation/storage/megaraid/SAS_IG.pdf > > You can always get good advice from: > http://www.cs-electronics.com/thanks for the links Al, duly bookmarked. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Kent Watsen wrote:> I''ve already ordered more cables for the aoc-sat2-mv8 and will report > which ones work when I get themI''ve tried the following four cables: 1. "CBL-SFF8087OCR-05M" from http://store.3ware.com <http://store.3ware.com/status.cfm?confirmation=200712172508AMC821> 2. "2136476-XXXXMM" from www.techcable.com 3. iSAS-7P73/.5m from http://www.cs-electronics.com/ 4. *"4 SATA Discrete to SFF-8087 Reverse Breakout" from *http://tekramonline.com Using the numbers assigned above: Fits AOC-SAT2-MV8 miniSAS Connector Sheathing ----------------- ----------------- --------- 1 Yes well made None 2 Yes ok None 3 Yes hard to disconnect 1/2 way 4 No ok 1/2 way I thought #3 was going to be my favorite, but it was nearly impossible to get off the first time, after which I crimped the prongs a bit and then it worked better, but the bad experience discourages me from ordering more of them In the end, I decided to go with #1. But since the sprawl of cables is unsightly. I plan to sheath them myself Kent
Did anybody ever find out if this option worked? Does setting this hidden option mean that drives are always listed and named according to the order they are physically connected? Ross> Hi Kent: > > So, in using the lsiutil utility, what do I find, > but the following option: (this was under the hidden option 15) > 12. Change (enable/disable) persistence > > I have not had a chance to try it, but it says > the default is enabled. Let me know if you try it. > > PaulThis message posted from opensolaris.org