I know that this has been well-discussed already, but it''s been a few months - WD caviars with mpt/mpt_sas generating lots of retryable read errors, spitting out lots of beloved " Log info 31080000 received for target" messages, and just generally not working right. (SM 836EL1 and 836TQ chassis - though I have several variations on theme depending on date of purchase: 836EL2s, 846s and 847s - sol10u8, 1.26/1.29/1.30 LSI firmware on LSI retail 3801 and 3081E controllers. Not that it works any better on the brace of 9211-8is I also tried these drives on.) Before signing up for the list, I "accidentally" bought a wad of caviar black 2TBs. No, they are new enough to not respond to WDTLER.EXE, and yes, they are generally unhappy with my boxen. I have them "working" now, running direct-attach off 3 3081E-Rs with breakout cables in the SC836TQ (passthru backplane) chassis, set up as one pool of 2 6+2 raidz2 vdevs (16 drives total), but they still toss the occasional error and performance is, well, abysmal - zpool scrub runs at about a third the speed of the 1TB cudas that they share the machine with, in terms of iostat reported ops/sec or bytes/sec. They don''t want to work in an expander chassis at all - spin up the drives and connect them and they''ll run great for a while, then after about 12 hours they start throwing errors. (Cycling power on the enclosure does seem to reset them to run for another 12 hours, but...) I''ve caved in and bought a brace of replacement cuda XTs, and I am currently going to resign these drives to other lesser purposes (attached to si3132s and ICH10 in a box to be used to store backups, running Windoze). It''s kind of a shame, because their single-drive performance is quite good - I''ve been doing single-drive tests in another chassis against cudas and constellations, and they seem quite a bit faster except on random-seek. Have I missed any changes/updates in the situation? Thanks, -bacon -- This message posted from opensolaris.org
On Fri, Jun 18, 2010 at 9:53 AM, Jeff Bacon <bacon at twinight.org> wrote:> I know that this has been well-discussed already, but it''s been a few months - WD caviars with mpt/mpt_sas generating lots of retryable read errors, spitting out lots of beloved " Log info 31080000 received for target" messages, and just generally not working right. > > (SM 836EL1 and 836TQ chassis - though I have several variations on theme depending on date of purchase: 836EL2s, 846s and 847s - sol10u8, 1.26/1.29/1.30 LSI firmware on LSI retail 3801 and 3081E controllers. Not that it works any better on the brace of 9211-8is I also tried these drives on.) > > Before signing up for the list, I "accidentally" bought a wad of caviar black 2TBs. No, they are new enough to not respond to WDTLER.EXE, and yes, they are generally unhappy with my boxen. I have them "working" now, running direct-attach off 3 3081E-Rs with breakout cables in the SC836TQ (passthru backplane) chassis, set up as one pool of 2 6+2 raidz2 vdevs (16 drives total), but they still toss the occasional error and performance is, well, abysmal - zpool scrub runs at about a third the speed of the 1TB cudas that they share the machine with, in terms of iostat reported ops/sec or bytes/sec. They don''t want to work in an expander chassis at all - spin up the drives and connect them and they''ll run great for a while, then after about 12 hours they start throwing errors. (Cycling power on the enclosure does seem to reset them to run for another 12 hours, but...) > > I''ve caved in and bought a brace of replacement cuda XTs, and I am currently going to resign these drives to other lesser purposes (attached to si3132s and ICH10 in a box to be used to store backups, running Windoze). It''s kind of a shame, because their single-drive performance is quite good - I''ve been doing single-drive tests in another chassis against cudas and constellations, and they seem quite a bit faster except on random-seek. > > Have I missed any changes/updates in the situation?I''m been getting very bad performance out of a LSI 9211-4i card (mpt_sas) with Seagate Constellation 2TB SAS disks, SM SC846E1 and Intel X-25E/M SSDs. Long story short, I/O will hang for over 1 minute at random under heavy load. Swapping the 9211-4i for a MegaRAID 8888ELP (mega_sas) improves performance by 30-40% instantly and there are no hangs anymore so I''m guessing it''s something related to the mpt_sas driver. I submitted bug #6963321 a few minutes ago (not available yet). -- Giovanni Tirloni gtirloni at sysdroid.com
> > Have I missed any changes/updates in the situation? > > I''m been getting very bad performance out of a LSI 9211-4i card > (mpt_sas) with Seagate Constellation 2TB SAS disks, SM SC846E1 and > Intel X-25E/M SSDs. Long story short, I/O will hang for over 1 minute > at random under heavy load.Hm. That I haven''t seen. Is this hang as in some drive hangs up with iostat busy% at 100 and nothing else happening (can''t talk to a disk) or a hang as perceived by applications under load? What''s your read/write mix, and what are you using for CPU/mem? How many drives? I wonder if maybe your SSDs are flooding the channel. I have a (many) 847E2 chassis, and I''m considering putting in a second pair of controllers and splitting the drives front/back so it''s 24/12 vs all 36 on one pair.> Swapping the 9211-4i for a MegaRAID 8888ELP (mega_sas) improves > performance by 30-40% instantly and there are no hangs anymore so I''m > guessing it''s something related to the mpt_sas driver.Well, I sorta hate to swap out all of my controllers (bother, not to mention the cost) but it''d be nice to have raidutil/lsiutil back.
On Wed, Jun 23, 2010 at 10:14 AM, Jeff Bacon <bacon at walleyesoftware.com> wrote:>> > Have I missed any changes/updates in the situation? >> >> I''m been getting very bad performance out of a LSI 9211-4i card >> (mpt_sas) with Seagate Constellation 2TB SAS disks, SM SC846E1 and >> Intel X-25E/M SSDs. Long story short, I/O will hang for over 1 minute >> at random under heavy load. > > Hm. That I haven''t seen. Is this hang as in some drive hangs up with > iostat busy% at 100 and nothing else happening (can''t talk to a disk) or > a hang as perceived by applications under load? > > What''s your read/write mix, and what are you using for CPU/mem? How many > drives?I''m using iozone to get some performance numbers and I/O hangs when it''s doing the writing phase. This pool has: 18 x 2TB SAS disks as 9 data mirrors 2 x 32GB X-25E as log mirror 1 x 160GB X-160M as cache iostat shows "2" I/O operations active and SSDs at 100% busy when it''s stuck. There are timeout messages when this happens: Jun 23 00:05:51 osol-x8-hba scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:05:51 osol-x8-hba Disconnected command timeout for Target 11 Jun 23 00:05:51 osol-x8-hba scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:05:51 osol-x8-hba Log info 0x31140000 received for target 11. Jun 23 00:05:51 osol-x8-hba scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Jun 23 00:05:51 osol-x8-hba scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:05:51 osol-x8-hba Log info 0x31140000 received for target 11. Jun 23 00:05:51 osol-x8-hba scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Jun 23 00:11:51 osol-x8-hba scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:11:51 osol-x8-hba Disconnected command timeout for Target 11 Jun 23 00:11:51 osol-x8-hba scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:11:51 osol-x8-hba Log info 0x31140000 received for target 11. Jun 23 00:11:51 osol-x8-hba scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc Jun 23 00:11:51 osol-x8-hba scsi: [ID 365881 kern.info] /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): Jun 23 00:11:51 osol-x8-hba Log info 0x31140000 received for target 11. Jun 23 00:11:51 osol-x8-hba scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc> I wonder if maybe your SSDs are flooding the channel. I have a (many) > 847E2 chassis, and I''m considering putting in a second pair of > controllers and splitting the drives front/back so it''s 24/12 vs all 36 > on one pair.My plan is to use the newest SC846E26 chassis with 2 cables but right now what I''ve available for testing is the SC846E1. I like the fact that SM uses the LSI chipsets in their backplanes. It''s been a good experience so far.>> Swapping the 9211-4i for a MegaRAID 8888ELP (mega_sas) improves >> performance by 30-40% instantly and there are no hangs anymore so I''m >> guessing it''s something related to the mpt_sas driver. > > Well, I sorta hate to swap out all of my controllers (bother, not to > mention the cost) but it''d be nice to have raidutil/lsiutil back.As much as I would like to blame faulty hardware for this issue, I only pointed out that using the MegaRAID doesn''t show the problem because that''s what I''ve been using without any issues in this particular setup. This system will be available to me for quite some time, so if anyone wants all kinds of tests to understand what''s happening, I would be happy to provide those. -- Giovanni Tirloni gtirloni at sysdroid.com
Gack, that''s the same message we''re seeing with the mpt controller with SATA drives. I''ve never seen it with a SAS drive before . Has anyone noticed a trend of 2TB SATA drives en-masse not working well with the LSI SASx28/x36 expander chips? I can seemingly reproduce it on demand - hook > 4 2TB disks to one of my supermicro chassis, spin up the array, and beat on it. (The last part is optional; merely hooking up the WD Caviar Blacks and attempting an import is sometimes sufficient.) Sun guys, I''ve got piles of hardware, if you want a testbed you got it.> > What''s your read/write mix, and what are you using for CPU/mem? Howmany> > drives? > > I''m using iozone to get some performance numbers and I/O hangs > when it''s doing the writing phase. > > This pool has: > > 18 x 2TB SAS disks as 9 data mirrors > 2 x 32GB X-25E as log mirror > 1 x 160GB X-160M as cache > > iostat shows "2" I/O operations active and SSDs at 100% busy when > it''s stuck. > > There are timeout messages when this happens: > > Jun 23 00:05:51 osol-x8-hba scsi: [ID 107833 kern.warning] > WARNING: > /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): > Jun 23 00:05:51 osol-x8-hba Disconnected command timeout for > Target 11 > Jun 23 00:05:51 osol-x8-hba scsi: [ID 365881 kern.info] > /pci at 0,0/pci8086,340a at 3/pci1000,3010 at 0 (mpt_sas0): > Jun 23 00:05:51 osol-x8-hba Log info 0x31140000 received fortarget> 11.
> I''m using iozone to get some performance numbers and I/O hangs when > it''s doing the writing phase. > > This pool has: > > 18 x 2TB SAS disks as 9 data mirrors > 2 x 32GB X-25E as log mirror > 1 x 160GB X-160M as cache > > iostat shows "2" I/O operations active and SSDs at 100% busy when > it''s stuck.Interesting. Have a SM 847E2 chassis with 33 constellation 2TB SAS and 3 vertex LE 100G, dual-connected across a pair of 9211-8is, sol10u8 with may patchset, and it runs like a champ - left several bonnie++ processes running on it for three days straight thrashing the pool, not even a blip. (the rear and front backplanes are separately cabled to the controllers.) (that''s with load-balance="none", in deference to Josh Simon''s observations - not really willing to lock the paths because I want the auto-failover. I''m going to be dropping in another pair of 9211-4is and connecting the back 12 drives to them since I have the PCIe slots, though it''s probably not especially necessary.) I wonder if the expander chassis work better if you''re running with the dual-expander-chip backplane? So far all of my testing with the 2TB SAS drives have been with single-expander-chip backplanes. Hm, might have to give that a try; it never came up simply because both of my dual-expander-chip-backplane JBODs were filled and in use, which just recently changed.> My plan is to use the newest SC846E26 chassis with 2 cables but right > now what I''ve available for testing is the SC846E1.Agreed. I just got my first 847E2 chassis in today - been waiting for months for them to be available, and I''m not entirely sure there''s any real stock (sorta like SM''s quad-socket Magny-Cours boards - a month ago, they didn''t even have any boards in the USA available for RMA, they got one batch in and sold it in a week or so).> >> Swapping the 9211-4i for a MegaRAID 8888ELP (mega_sas) improves > >> performance by 30-40% instantly and there are no hangs anymore soI''m> >> guessing it''s something related to the mpt_sas driver.Wait. The mpt_sas driver by default uses scsi_vhci, and scsi_vhci by default does load-balance round-robin. Have you tried setting load-balance="none" in scsi_vhci.conf? -bacon -- This message posted from opensolaris.org
On Wed, Jun 23, 2010 at 2:43 PM, Jeff Bacon <bacon at walleyesoftware.com> wrote:>> >> Swapping the 9211-4i for a MegaRAID 8888ELP (mega_sas) improves >> >> performance by 30-40% instantly and there are no hangs anymore so > I''m >> >> guessing it''s something related to the mpt_sas driver. > > Wait. The mpt_sas driver by default uses scsi_vhci, and scsi_vhci by > default does load-balance round-robin. Have you tried setting > load-balance="none" in scsi_vhci.conf?That didn''t help. -- Giovanni Tirloni gtirloni at sysdroid.com