I have a strange problem involving changes in large file on a mirrored zpool in Open solaris snv96. We use it at storage in a VMware ESXi lab environment. All virtual disk files gets corrupted when changes are made within the files (when running the machine that is). The "sad" thing is that I''ve created about ~200Gb of random data in large files and even modified those files without any problem (using dd with skip and conv=notrunc options). I''ve copied the files within the pool and over the network on all network interfaces on the machine - without problems. It''s just those .vmdk files that gets corrupted. The hardware is an Opteron desktop machine with a SIL3114 sata interface. Personally I have exactly the same interface at home with the same setup without problem. Only the other hardware differs (disks and so on). The disks are WD7500AACS, which is those with variable rotation speed 5400-7200. Could it be the disks? Could it be the disk controller or the rest of the hardware?? I should mention that the controller has been flashed with a non-raid bios. I could provide more information if needed! Is there anyone that have any ideas or suggestions? Some output: bash-3.00# zpool status -vx pool: testing state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub completed with 1 errors on Wed Sep 24 16:59:13 2008 config: NAME STATE READ WRITE CKSUM testing ONLINE 0 0 16 mirror ONLINE 0 0 16 c0d1 ONLINE 0 0 51 c1d1 ONLINE 0 0 54 errors: Permanent errors have been detected in the following files: /testing/ZFS-problem/ZFS-problem-flat.vmdk Regards Mikael
It''s almost certainly the SIL3114 controller. Google "SIL3114 data corruption" -- it''s nasty. Jeff On Thu, Sep 25, 2008 at 07:50:01AM +0200, Mikael Karlsson wrote:> I have a strange problem involving changes in large file on a mirrored > zpool in > Open solaris snv96. > We use it at storage in a VMware ESXi lab environment. All virtual disk > files gets > corrupted when changes are made within the files (when running the > machine that is). > > The "sad" thing is that I''ve created about ~200Gb of random data in > large files and > even modified those files without any problem (using dd with skip and > conv=notrunc options). > I''ve copied the files within the pool and over the network on all > network interfaces > on the machine - without problems. > > It''s just those .vmdk files that gets corrupted. > > The hardware is an Opteron desktop machine with a SIL3114 sata > interface. Personally I have exactly > the same interface at home with the same setup without problem. Only the > other hardware differs (disks and so on). > > The disks are WD7500AACS, which is those with variable rotation speed > 5400-7200. Could it > be the disks? Could it be the disk controller or the rest of the > hardware?? I should mention that the > controller has been flashed with a non-raid bios. > > I could provide more information if needed! Is there anyone that have > any ideas or suggestions? > > > Some output: > > bash-3.00# zpool status -vx > pool: testing > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scrub: scrub completed with 1 errors on Wed Sep 24 16:59:13 2008 > config: > > NAME STATE READ WRITE CKSUM > testing ONLINE 0 0 16 > mirror ONLINE 0 0 16 > c0d1 ONLINE 0 0 51 > c1d1 ONLINE 0 0 54 > > errors: Permanent errors have been detected in the following files: > > /testing/ZFS-problem/ZFS-problem-flat.vmdk > > > Regards > > Mikael > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 09/24/08 10:57 PM, Jeff Bonwick wrote:> It''s almost certainly the SIL3114 controller. > Google "SIL3114 data corruption" -- it''s nasty. >I''ve also in the past had the misfortune of experiencing Silicon Image. My corruption was with other file types and not even ZFS. Silicon Image is something I do not wish on my enemies. No attempt to acknowledge or recall defective silicon. No interest in customer data loss. Well, this customer has no further interest in Silicon Image. I refuse to acknowledge that they exist.> Jeff > > On Thu, Sep 25, 2008 at 07:50:01AM +0200, Mikael Karlsson wrote: > >> I have a strange problem involving changes in large file on a mirrored >> zpool in >> Open solaris snv96. >> We use it at storage in a VMware ESXi lab environment. All virtual disk >> files gets >> corrupted when changes are made within the files (when running the >> machine that is). >> >> The "sad" thing is that I''ve created about ~200Gb of random data in >> large files and >> even modified those files without any problem (using dd with skip and >> conv=notrunc options). >> I''ve copied the files within the pool and over the network on all >> network interfaces >> on the machine - without problems. >> >> It''s just those .vmdk files that gets corrupted. >> >> The hardware is an Opteron desktop machine with a SIL3114 sata >> interface. Personally I have exactly >> the same interface at home with the same setup without problem. Only the >> other hardware differs (disks and so on). >> >> The disks are WD7500AACS, which is those with variable rotation speed >> 5400-7200. Could it >> be the disks? Could it be the disk controller or the rest of the >> hardware?? I should mention that the >> controller has been flashed with a non-raid bios. >> >> I could provide more information if needed! Is there anyone that have >> any ideas or suggestions? >> >> >> Some output: >> >> bash-3.00# zpool status -vx >> pool: testing >> state: ONLINE >> status: One or more devices has experienced an error resulting in data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore the >> entire pool from backup. >> see: http://www.sun.com/msg/ZFS-8000-8A >> scrub: scrub completed with 1 errors on Wed Sep 24 16:59:13 2008 >> config: >> >> NAME STATE READ WRITE CKSUM >> testing ONLINE 0 0 16 >> mirror ONLINE 0 0 16 >> c0d1 ONLINE 0 0 51 >> c1d1 ONLINE 0 0 54 >> >> errors: Permanent errors have been detected in the following files: >> >> /testing/ZFS-problem/ZFS-problem-flat.vmdk >> >> >> Regards >> >> Mikael >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/57994c8c/attachment.html>
>>>>> "np" == Neal Pollack <Neal.Pollack at Sun.COM> writes:np> No attempt to acknowledge or recall defective silicon. No np> interest in customer data loss. Well, this customer has no np> further interest in Silicon Image. I refuse to acknowledge np> that they exist. 1. too bad Sil is the only ones selling chips on PCI cards that have source code for their drivers. 2. if the .vmdk''s were stored in ZFS why was the corruption not flagged as a CKSUM error? I think the problem''s probably elsewhere. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/254ba978/attachment.bin>
On Thu, Sep 25, 2008 at 13:38, Miles Nordin <carton at ivy.net> wrote:> 1. too bad Sil is the only ones selling chips on PCI cards that have > source code for their drivers.Indeed, it is too bad. But I''d rather have a working closed blob than a driver that is Free Software for a device that is faulty. Ideals are very nice, but broken hardware isn''t.> 2. if the .vmdk''s were stored in ZFS why was the corruption not > flagged as a CKSUM error? I think the problem''s probably > elsewhere.They were. From the OP:> NAME STATE READ WRITE CKSUM > testing ONLINE 0 0 16 > mirror ONLINE 0 0 16 > c0d1 ONLINE 0 0 51 > c1d1 ONLINE 0 0 54Will
>>>>> "wm" == Will Murnane <will.murnane at gmail.com> writes:wm> I''d rather have a working closed blob than a driver that is wm> Free Software for a device that is faulty. Ideals are very wm> nice, but broken hardware isn''t. except, 1. part of the reason the closed Solaris drivers are (also) broken, IMHO, is that they''re closed, so highly-invested competent people can''t fix them if they happen to be on the wrong side of the wall. 2. Linux has open drivers for the Marvell chip that work better than Sun''s closed driver, so there''s an existence proof that (a) you can have both freedom and working drivers. It''s a false trade-off, like ``civic freedom vs. security.'''' And (b) the appologist''s claim ``we can only make half-working broken drivers, fine for Linux but nothing up to our high standards <cough!>, unless we sign an NDA'''' is bunk. The people who refused to sign(*) now have a better driver, not a worse one. that''s why I suggest running solaris as a domU of Linux may be the best plan until the SATA stack is no longer broken, or at least not broken on some limited subset of hardware, like for example the hardware that Sun sells. 3. The position is incredibly short-sighted. Imagine the quality of driver we''d have right now if _everyone_ refused to sign that damned paper, not just the Linux people. We would have a better driver. It would be open, too, but open or not it would be better. 4. there are missing features like NCQ, hotplug, port-multiplier support, all highly relevant to ZFS, for which we will have to wait longer because we''ve accepted closed drivers. 5. The Sil 3124 chip works fine on Linux. I have not tried the 3114, but at least on Linux it is part of libata, their SATA framework, not supported in remedial PATA mode, so it''s at least more of a first-class driver in Linux than in Solaris, if not simply a better one. But to be honest I don''t wish for a driver for every chip---I''m not trying to ``convert'''' machines, I buy them specifically for the task. I just want an open driver that works well for some fairly-priced card I can actually buy. I''m willing to fight the OEM problem: http://www.openbsd.org/papers/brhard2007/mgp00022.html by purchasing systems in a complicated way, with lots of add-in cards, at higher cost. I will buy whatever card I''m told. But so far the track record is not good. I still have one of those bunk Supermicro Marvell cards sitting on the shelf. so this is a really big gap: On Linux I get an open driver that works well for almost any chip, in the stable branch of RedHat/CentOS, and I get the source code for the exact same stable supported version I''m using not just source for the development version. And many of the drivers support advanced features: NCQ, hotplug, PMP, smartctl not panicing the system. On Solaris there is one closed driver (LSI) and one open driver (AHCI) that works sort-of well but not as well as Linux, and doesn''t support advanced features. The open driver isn''t obtainable as an add-on card and doesn''t support port multipliers, so your port density is really limited. The closed driver is an expensive card (largely because of the _cables_?!), and brand-new/unproven/not-even-available-in-Sol10. If there _is_ an open vs. closed trade-off, the track record so far suggests a different trade-off than what you suggest: you can have closed drivers if you really want them, but they''ll be more broken than the open ones. --- (*) The Linux Marvell driver does come with source, but this presentation: http://www.openbsd.org/papers/opencon06-drivers/mgp00025.html says some Linux guys sign NDA''s to get documentation they use to write GPL drivers. not sure how they can publish source code without ``disclosing'''', but the OpenBSD people are saying (1) code is not a substitute for documentation, (2) craven Linux developers are lowering the bar and making open documentation for writing openbsd drivers unobtainable. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/dcdb275a/attachment.bin>
Miles Nordin wrote: ...> On Solaris there is one closed driver (LSI) and one open driver > (AHCI) that works sort-of well but not as well as Linux, and > doesn''t support advanced features. The open driver isn''t > obtainable as an add-on card and doesn''t support port multipliers, > so your port density is really limited. The closed driver is an > expensive card (largely because of the _cables_?!), and > brand-new/unproven/not-even-available-in-Sol10.Hi Miles, I assume you''re referring to mpt(7d) here? Since we started shipping it at all, with Solaris _8_, it''s definitely been available in Solaris 10. We''ve backported (to Solaris 10 Update releases) a massive number of new features for it. Is there a specific thing you''re not seeing in the Solaris 10 version? (and which patch rev are you running?) James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
c> 2. if the .vmdk''s were stored in ZFS why was the corruption not c> flagged as a CKSUM error? wm> They were. From the OP: > NAME STATE READ WRITE CKSUM > testing ONLINE 0 0 16 > mirror ONLINE 0 0 16 > c0d1 ONLINE 0 0 51 > c1d1 ONLINE 0 0 54 oops. ok good. if oyu want to try a Sil3124, this is the one I bought for Linux: http://www.newegg.com/Product/Product.aspx?Item=N82E16816124008 that''s 32-bit 5V PCI. there are 64-bit pcix versions and PCIe (via PCIe-PCIX bridge) versions from addonics. If you try it let me know how well it works. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/26e50b22/attachment.bin>
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes:jcm> I assume you''re referring to mpt(7d) here? jcm> Since we started shipping it at all, with Solaris _8_, it''s jcm> definitely been available in Solaris 10. no, I was mistaken then. My perhaps mistaken understanding though was that there''s a scsi_vhci sort of driver for the LSI card in the Ultra {20,25}---I''m talking about whichever was the SPARC Ultra, not the x86 thing stupidly called Ultra <nn>. This driver made the card look like a SCSI host adapter and made SATA disks attach through the sd driver, sort oflike a RAID card in JBOD mode, to work around the problem that the SATA framework is not endian-clean. Then there is the LSISAS3801E which people here have been buying, about which I have these notes: -----8<-----> The driver for LSI''s MegaRAID SAS card is "mega_sas" which > was integrated into snv_88. It''s planned for backporting to > a Solaris 10 update.There is also a BSD-licensed driver for that hardware, called "mfi". It''s available from http://www.itee.uq.edu.au/~dlg/mfi -----8<----- and AIUI this is a SATA framework driver. so, althouigh this new fashionable card may be FusionMPT-ish, it doesn''t use the mpt driver, and it uses a different branch of the rest of the storage stack than the mpt driver. Am I right? The important short-sighted thing to me for using the SATA stack instead of doing SCSI translation inside the proprietary driver, would be that smartctl works and SATA DVD burners and stuff work, but from what I''ve heard here smartctl is not completely working on any card. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/714818a8/attachment.bin>
> > > But to be honest I don''t wish for a driver for every chip---I''m > not trying to ``convert'''' machines, I buy them specifically for > the task. I just want an open driver that works well for some > fairly-priced card I can actually buy. I''m willing to fight the > OEM problem: > > http://www.openbsd.org/papers/brhard2007/mgp00022.html > > by purchasing systems in a complicated way, with lots of add-in > cards, at higher cost. I will buy whatever card I''m told. But so > far the track record is not good. I still have one of those bunk > Supermicro Marvell cards sitting on the shelf. >So what''s wrong with this card? http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080925/29383c82/attachment.html>
Miles Nordin wrote:>>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: > > jcm> I assume you''re referring to mpt(7d) here? > > jcm> Since we started shipping it at all, with Solaris _8_, it''s > jcm> definitely been available in Solaris 10. > > no, I was mistaken then. > > My perhaps mistaken understanding though was that there''s a scsi_vhci > sort of driver for the LSI card in the Ultra {20,25}---I''m talking > about whichever was the SPARC Ultra, not the x86 thing stupidly called > Ultra <nn>. This driver made the card look like a SCSI host adapter > and made SATA disks attach through the sd driver, sort oflike a RAID > card in JBOD mode, to work around the problem that the SATA framework > is not endian-clean.Well yes, that''s mpt(7d) as delivered into NV build 63, and backported to Solaris 10 Update 5, found in patch 125081-14 and 125082-14. We''ve got support for both SAS (1064/E, 1068/E and 1078) and Parallel SCSI (1030) chips from LSI in that driver. SATA disks will always show up when attached to a SAS HBA, because that''s one of the requirements of the SAS specification. It was designed in to the spec from the very beginning.> Then there is the LSISAS3801E which people here have been buying, about > which I have these notes:I think you might actually be referring to the LSI SAS3801-R> -----8<----- >> The driver for LSI''s MegaRAID SAS card is "mega_sas" which >> was integrated into snv_88. It''s planned for backporting to >> a Solaris 10 update. > > There is also a BSD-licensed driver for that hardware, called > "mfi". It''s available from > > http://www.itee.uq.edu.au/~dlg/mfi > -----8<----- > > and AIUI this is a SATA framework driver. so, althouigh this new > fashionable card may be FusionMPT-ish, it doesn''t use the mpt driver, > and it uses a different branch of the rest of the storage stack than > the mpt driver. > Am I right?You''re correct in that it''s not using mpt, but mega_sas or mfi. But it''s not a SATA framework driver.> The important short-sighted thing to me for using the SATA stack > instead of doing SCSI translation inside the proprietary driver, would > be that smartctl works and SATA DVD burners and stuff work, but from > what I''ve heard here smartctl is not completely working on any card.I can''t comment on smartctl (or smartmontools) since it''s not something that I''ve used. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
>>>>> "t" == Tim <tim at tcsac.net> writes:t> http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm I''m not sure. A different thing is wrong with it depending on what driver attaches to it. I can''t tell for sure because this page: http://linuxmafia.com/faq/Hardware/sas.html says the LSI SAS 3800 series uses a 1068E chip, and James says (1) 1068E is supported by mpt, (2) LSI SAS 3800 uses mega_sas. so, I don''t know which for that card, which means I don''t know which for this card. If it''s mpt: * does not come with source according to: http://www.openbsd.org/papers/opencon06-drivers/mgp00024.html http://www.opensolaris.org/os/about/no_source/ If it''s mega_sas: * does not come with source * driver is new and unproven. We believed the Marvell driver was good for the first few months too, the same amount of experience we have with mega_sas. * not sure if it''s available in stable solaris. In either case: * may require expensive cables Uncertain problems: * might not support hotplug * might not support NCQ * probably doesn''t support port multipliers * probably doesn''t support smartctl * none of these features can be fixed by the community without source. all are available with cheaper cards on Linux, and on Linux both mptsas and megaraid_sas come with source as far as I can tell maintained by dell and lsi, though might not support the above features. HTH, HAND. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080926/6a92d15a/attachment.bin>
On Thu, Sep 25, 2008 at 18:51, Tim <tim at tcsac.net> wrote:> So what''s wrong with this card? > http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfmIf you have a UIO slot (many recent Supermicro boards do) then it''s a fine choice. But if you have a non-Supermicro board, you may be in for a shock when you get it---it''s swapped left for right, compare it to a regular pci-e card. It won''t fit in a standard case. AIUI it is a standard pci express slot, just shifted over a bit so the backwards slot cover fits into normal cases, so perhaps you could try fastening a normal slot cover to it and using it in a normal pci-e slot... but that doesn''t sound particularly elegant, and would take up the slot on the other side of it as well. Will
On Thu, Sep 25, 2008 at 21:59, Miles Nordin <carton at ivy.net> wrote:>>>>>> "wm" == Will Murnane <will.murnane at gmail.com> writes: > > wm> I''d rather have a working closed blob than a driver that is > wm> Free Software for a device that is faulty. Ideals are very > wm> nice, but broken hardware isn''t. > > except, > > 1. part of the reason the closed Solaris drivers are (also) broken, > IMHO, is that they''re closed, so highly-invested competent people > can''t fix them if they happen to be on the wrong side of the wall.I agree this is an issue. But as I said, I''d rather have a working closed driver than a broken open one.> 2. Linux has open drivers for the Marvell chip that work better than > Sun''s closed driver (snip)That''s not my experience. I bought my Marvell card around 2005, and at that point I used Linux drivers. Drivers for the card at that point did not support DMA, but were fairly reliable. In late 2006 or so, DMA support was finally added, so I gleefully installed a new kernel and was happy. Until I realized that my data was corrupt. This is for a home system, so I didn''t have checksums for the data before the corruption, but I started to hear glitches in music playback. At that point I switched to Solaris, and was very glad for the drivers that didn''t cause corruption---and the filesystem that could tell me when things went wrong. I did have a problem with disks falling off the card, so I posted to the storage-discuss mailing list [2]. Despite being on the wrong side of the wall, the drivers were updated fairly soon thereafter, and my problem was solved [1]. The system worked quite well for me in this instance.> 3. The position is incredibly short-sighted. Imagine the quality of > driver we''d have right now if _everyone_ refused to sign that > damned paper, not just the Linux people. We would have a better > driver. It would be open, too, but open or not it would be > better.Not necessarily. Suppose that the corporation making the hardware released its own drivers, for Windows and Linux, say, and didn''t release specs to anyone else, even under NDA conditions. Then nobody gets "good" drivers (ones that correctly use all the features the hardware has). I agree that having complete hardware specs is a very helpful thing to make drivers. But they''re not strictly necessary, as the Linux/BSD folks have shown.> 4. there are missing features like NCQ, hotplug, port-multiplier > support, all highly relevant to ZFS, for which we will have to > wait longer because we''ve accepted closed drivers.That''s true. But honestly, I don''t see those features (with the exception of hot-plug) as being all that necessary. Port multipliers are uncommon and don''t perform as well as they could, and NCQ seems to me to be something the OS could do better than the drive firmware.> 5. The Sil 3124 chip works fine on Linux. I have not tried the 3114, > but at least on Linux it is part of libata, their SATA framework, > not supported in remedial PATA mode, so it''s at least more of a > first-class driver in Linux than in Solaris, if not simply a > better one.IMHO, attempting to make SilImage controllers work well is lipstick on a pig. Working around the bugs in the hardware is not worth the effort.> I just want an open driver that works well for some > fairly-priced card I can actually buy.This I can agree with. Despite my objections to "free" drivers being inherently better than "closed" ones, I do like the idea of being able to have a completely transparent machine, where I can inspect every piece of software. I would be more than happy to buy such hardware were it available, but in the interim I will continue to suggest and buy LSI''s products, which are not free but which have good drivers for them.> The open driver isn''t > obtainable as an add-on cardThe ICH series would indeed be nice to see as an addon card of some sort.> If there _is_ an open vs. closed trade-off, the track record so > far suggests a different trade-off than what you suggest: you can > have closed drivers if you really want them, but they''ll be more > broken than the open ones.That may be the case in the larger picture, but in my experience I''ve seen otherwise. Will [1]: http://www.mail-archive.com/zfs-discuss at opensolaris.org/msg14188.html [2]: http://osdir.com/ml/os.solaris.opensolaris.storage.general/2007-08/msg00054.html
On Fri, Sep 26, 2008 at 1:02 PM, Will Murnane <will.murnane at gmail.com>wrote:> On Thu, Sep 25, 2008 at 18:51, Tim <tim at tcsac.net> wrote: > > So what''s wrong with this card? > > http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm > If you have a UIO slot (many recent Supermicro boards do) then it''s a > fine choice. But if you have a non-Supermicro board, you may be in > for a shock when you get it---it''s swapped left for right, compare it > to a regular pci-e card. It won''t fit in a standard case. AIUI it is > a standard pci express slot, just shifted over a bit so the backwards > slot cover fits into normal cases, so perhaps you could try fastening > a normal slot cover to it and using it in a normal pci-e slot... but > that doesn''t sound particularly elegant, and would take up the slot on > the other side of it as well. > > Will >This is not a UIO card. It''s a standard PCI-E card. What the description is telling you is that you can combine it with a UIO card to add raid functionality as there is none built-in. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080926/7137edfa/attachment.html>
On Fri, Sep 26, 2008 at 12:29 PM, Miles Nordin <carton at ivy.net> wrote:> >>>>> "t" == Tim <tim at tcsac.net> writes: > > t> > http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm > > I''m not sure. A different thing is wrong with it depending on what > driver attaches to it. I can''t tell for sure because this page: > > http://linuxmafia.com/faq/Hardware/sas.html > > says the LSI SAS 3800 series uses a 1068E chip, and James says (1) > 1068E is supported by mpt, (2) LSI SAS 3800 uses mega_sas. so, I > don''t know which for that card, which means I don''t know which for > this card. > > If it''s mpt: > > * does not come with source according to: > > http://www.openbsd.org/papers/opencon06-drivers/mgp00024.html > http://www.opensolaris.org/os/about/no_source/ > > If it''s mega_sas: > > * does not come with source > > * driver is new and unproven. We believed the Marvell driver was > good for the first few months too, the same amount of experience we > have with mega_sas. > > * not sure if it''s available in stable solaris. >Someone''s already gotten it working, if they''re watching I''m sure they''ll pipe up on what driver it uses.> > > In either case: > > * may require expensive cables >Nope, cables are standardized. I''m not sure what your definition of "expensive" is but I believe they were roughly 15$ for a SAS>>4sata ports.> > Uncertain problems: > > * might not support hotplug > > * might not support NCQ > > * probably doesn''t support port multipliers > > * probably doesn''t support smartctl > > * none of these features can be fixed by the community without > source. all are available with cheaper cards on Linux, and on > Linux both mptsas and megaraid_sas come with source as far as I can > tell maintained by dell and lsi, though might not support the above > features. > > > HTH, HAND. >I know it supports hotplug and NCQ. Can''t say smartctl was ever on my list of important features so I haven''t bothered to research if it does. I''m also not sure what good port multipliers are going to do you in this instance... the cables it uses already support 4 SATA drives per physical card port. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080926/23192280/attachment.html>
On Fri, Sep 26, 2008 at 21:51, Tim <tim at tcsac.net> wrote:> This is not a UIO card. It''s a standard PCI-E card. What the description > is telling you is that you can combine it with a UIO card to add raid > functionality as there is none built-in.Not so. The description [1] mentions that this is UIO, and says only that it negotiates pci-e link speeds, not that it fits in a pci express slot. UIO is pci express, but the slots are positioned differently from pci-e ones. Compare this to the picture of an equivalent LSI card [2]. The pictures are similar, but compare the position of the bracket. The components are mounted on the wrong sides. Take a look at a UIO board [3]: the PCI-X slot is shared with the blue UIO slot on the left side, like PCI and ISA slots used to be shared. This is why the components are backwards. Will [1]: http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm [2]: http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/megaraid_sas_8208elp/index.html [3]: http://www.supermicro.com/products/motherboard/Xeon1333/5400/X7DWE.cfm
On Fri, Sep 26, 2008 at 5:07 PM, Will Murnane <will.murnane at gmail.com>wrote:> On Fri, Sep 26, 2008 at 21:51, Tim <tim at tcsac.net> wrote: > > This is not a UIO card. It''s a standard PCI-E card. What the > description > > is telling you is that you can combine it with a UIO card to add raid > > functionality as there is none built-in. > Not so. The description [1] mentions that this is UIO, and says only > that it negotiates pci-e link speeds, not that it fits in a pci > express slot. UIO is pci express, but the slots are positioned > differently from pci-e ones. > > Compare this to the picture of an equivalent LSI card [2]. The > pictures are similar, but compare the position of the bracket. The > components are mounted on the wrong sides. Take a look at a UIO board > [3]: the PCI-X slot is shared with the blue UIO slot on the left side, > like PCI and ISA slots used to be shared. This is why the components > are backwards. > > Will > > [1]: > http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm > [2]: > http://www.lsi.com/storage_home/products_home/internal_raid/megaraid_sas/megaraid_sas_8208elp/index.html > [3]: > http://www.supermicro.com/products/motherboard/Xeon1333/5400/X7DWE.cfm >Well, there''s people that have it working in a PCI-E slot, so I don''t know what to tell you. http://www.opensolaris.org/jive/thread.jspa?messageID=272283񂞛 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080926/3fa1e5cd/attachment.html>
Tim wrote:> On Fri, Sep 26, 2008 at 12:29 PM, Miles Nordin <carton at ivy.net > <mailto:carton at ivy.net>> wrote: > >>>>> "t" == Tim <tim at tcsac.net <mailto:tim at tcsac.net>> writes: > t> > http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm > I''m not sure. A different thing is wrong with it depending on what > driver attaches to it. I can''t tell for sure because this page: > > http://linuxmafia.com/faq/Hardware/sas.html > > says the LSI SAS 3800 series uses a 1068E chip, and James says (1) > 1068E is supported by mpt, (2) LSI SAS 3800 uses mega_sas. so, I > don''t know which for that card, which means I don''t know which for > this card.There are several LSI cards which use the 1068 and 1068E chip. Some of these use mpt(7d), some use mega_sas(7d). It all depends on the firmware of the card, basically. You could also have a look at the PCI IDs database at http://pciids.sourceforge.net to see what the name to pci vid/did mapping is. That provides a fairly good indicator of whether you''ll need mpt(7d) or mega_sas(7d).> If it''s mpt: > > * does not come with source according to: > > http://www.openbsd.org/papers/opencon06-drivers/mgp00024.html > http://www.opensolaris.org/os/about/no_source/ > > If it''s mega_sas: > > * does not come with source > > * driver is new and unproven. We believed the Marvell driver was > good for the first few months too, the same amount of experience we > have with mega_sas. > > * not sure if it''s available in stable solaris. > Someone''s already gotten it working, if they''re watching I''m sure > they''ll pipe up on what driver it uses.http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/mega_sas We''ve got this driver into Solaris 10 Update 6. I''m still keen to find out from Miles why mega_sas is "new and unproven" given that it''s been in NV since build 88. Miles - if you''re seeing problems with it, please let us know so that we can fix them. If you don''t tell us, how will we ever know?> In either case: > > * may require expensive cables > Nope, cables are standardized. I''m not sure what your definition of > "expensive" is but I believe they were roughly 15$ for a SAS>>4sata ports.If you want to get an external SAS cable (particularly if it''s got the InfiniBand-style SF8088 connector), then that might cost you a bit. If you just want to connect devices internally, then I would expect the cables to be somewhat cheaper. Either way, with more and more volume of cards and devices on the market, the pricing for cables should decrease too. [snip] James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: >>>>> "t" == Tim <tim at tcsac.net> writes:jcm> find out from Miles why mega_sas is "new and unproven" jcm> given that it''s been in NV since build 88. This has degenerated to an argument over definition, so if you like I can retract ``new and unproven'''' and replace my comment with: * it has not yet been in any stable release of solaris * it is more new and unproven than any other solaris SATA driver * it''s a lot more new and unproven than the Marvell driver is/was when it is/was causing major problems jcm> Miles - if you''re seeing problems with it, please let us know jcm> so that we can fix them. If you don''t tell us, how will we jcm> ever know? yeah, I''m not seeing problems, because I''m not using it. But I do not need to use it to point out the three things above, nor the lack of source. c> * may require expensive cables t> Nope, cables are standardized. I''m not sure what your t> definition of "expensive" is but I believe they were roughly t> 15$ for a SAS>>4sata ports. plain SATA is $1/cable so $15/4cable is fine to my view. If you could post again where you buy them for that price, the thread might be more useful to people not so captivated by our bickering. :) t> I''m also not sure what good port multipliers are going to do t> you in this instance... this seems obvious to me: however many drives you can use without port multipliers, they let you use five times that many. The point of source is that, even if you or Sun doesn''t think port multipliers or smartctl are important, someone else with a different opinion has enough software freedom to add the support himself. I do think PMP support is too bad, but not having enough software freedom to add it is, to my view, unacceptable under current ``market forces.'''' jm> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/mega_sas so, this means there _is_ source, under BSD license? This is the whole driver or one of those ``shims''''? I thought the BSD-licensed driver was unbundled mfi available from: http://www.itee.uq.edu.au/~dlg/mfi/ and that the bundled mega_sas didn''t include source. Is it newly opened-up, or is that not the complete source, or I misunderstood your 2008-07-26 post? I guess I am not the only one who can''t easily tell what''s open and what isn''t, or someone would have corrected me sooner. If it''s not fully open, again it''s hard to tell. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080928/1fbf26c9/attachment.bin>
Miles Nordin wrote:>>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: >>>>>> "t" == Tim <tim at tcsac.net> writes:.... [snip argument over definitions] ....> jm> http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/mega_sas > > so, this means there _is_ source, under BSD license? This is the > whole driver or one of those ``shims''''?Yes, there is source, and as you note, it is indeed provided under a BSD license. It''s the whole driver. I really don''t know why you thought that the source didn''t exist, let alone not exist in an open form. If you read through that code, you should be able to see that it isn''t a shim, either. I don''t know why you would think it might be a shim.> I thought the BSD-licensed driver was unbundled mfi available from: > > http://www.itee.uq.edu.au/~dlg/mfi/ > > and that the bundled mega_sas didn''t include source. Is it newly > opened-up, or is that not the complete source, or I misunderstood your > 2008-07-26 post? > > I guess I am not the only one who can''t easily tell what''s open and > what isn''t, or someone would have corrected me sooner. If it''s not > fully open, again it''s hard to tell.The source at the url you mention is David Gwynne''s port of the *BSD mfi driver to Solaris. Can I assume that my "2008-07-26 post" was in fact two messages that were sent to you and cc''d to zfs-discuss: http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049605.html and http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049607.html If so, what''s the problem? Both drivers are Open. Both drivers are provided under BSD licenses. Both drivers are, to the best of my knowledge, "the complete source". Only one has been integrated into OpenSolaris. And finally, no, you won''t get source for the version which is integrated into a Solaris 10 Update. It shouldn''t be too different from the version which is in OpenSolaris, but nobody has ever claimed otherwise. Is this the heart of the problem? James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes:jcm> Can I assume that my "2008-07-26 post" was in fact two jcm> messages that were sent to you and cc''d to zfs-discuss: jcm> http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049605.html jcm> and jcm> http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049607.html Yeah, the second. I thought your second message was saying there''s no source for mega_sas. In context, we were following up to an osnews forum posting in which some angry person was saying there was no source for mega_sas, which you didn''t deny and then offered the alternate BSD driver, so I got confused. jcm> If you read through that code, you should be able to see jcm> that it isn''t a shim, either. no, I can''t read code that well. If I wanted to determine that for sure without asking others I''d have to understand the build procedure really well and watch the thing build, or else rely on some license/legal framework like the ``Taints kernel'''' message emitted by Linux when loading proprietary modules. jcm> I don''t know why you would think it might be a shim. nothing personal about Solaris. people everywhere are getting screwed (or at least confused) by shim drivers. Now that free software is more mainstream, the attacks on it have become incredibly creative, and to my view often very effective. jcm> And finally, no, you won''t get source for the version which is jcm> integrated into a Solaris 10 Update. It shouldn''t be too jcm> different from the version which is in OpenSolaris, but nobody jcm> has ever claimed otherwise. Is this the heart of the problem? I think it''s important, and good that you brought it up. So, the source in hg & src.opensolaris.org never includes the Sol10 stable branch? I''ve always wondered that and never been able to figure out out. At some point, not necessarily disclosed, the sources currently on mercurial will be taken back inside the wall where they''ll have the bugs thrashed out of them before they become the next release of stable Solaris? On Linux, the GPL ensures that you can always get source even for the stable version of the OS on which you''ve bought support, not just the development version. Nothing can be ``taken back inside.'''' The difference is no big deal if your goal is to work on big bugs, or contribute new drivers and subsystems to Sun. If your goals are those of a modest sysadmin, to fix your own small bugs locally, or disable some ZFS sanity check to recover a pool that won''t import, no source for the stable version does matter. But that wasn''t what I was thinking in this thread. I just thought there was no source for mega_sas. so... great! I guess the comparison I settled on was Solaris mega_sas vs. Solaris domU/Linux {anychip,Sil3124}. I had some other concerns, most resolved by other posters'' reports and a few not, but no need to repeat them. If one''s been burned enough though, he can always imagine more concerns---if it turns out this card has some 1MByte ROM chip running a whole proprietary vxworks image, so that even with a free software driver you''re separated from the SATA interface so much that you can''t implement new QoS ideas or NCQ or PMP support or fix USCSI/smartctl or make a comstar Target Mode, while on cheaper junky chips you can do those things....well, at least there''s still Sil3124 source too. I would say about all my speculative concerns, ``try it and see,'''' but part of the point I see in this process, is that mainstream users should exercise some vigilance to keep space open for a few others on the fringe who may be particularly brilliant, motivated, and creative, but without others'' vigilance hamstrung by having no truly free platform. And by the time you''ve ``tried it and seen,'''' you''ve already been fooled into handing over the cash, wasted a bunch of time doing work for free without achieving freedom, gotten burned, and the OEM''s have moved on to the next chip. I only really want one card that works well and includes source and is obtainable in the ballpark of <motherboard cost> * 3, so if this card is it, great. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080929/2a0944b3/attachment.bin>
Miles Nordin wrote:>>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: > > jcm> Can I assume that my "2008-07-26 post" was in fact two > jcm> messages that were sent to you and cc''d to zfs-discuss: > jcm> http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049605.html > jcm> and > jcm> http://mail.opensolaris.org/pipermail/zfs-discuss/2008-July/049607.html > > Yeah, the second. I thought your second message was saying there''s no > source for mega_sas. In context, we were following up to an osnews > forum posting in which some angry person was saying there was no > source for mega_sas, which you didn''t deny and then offered the > alternate BSD driver, so I got confused.Ah, ok. That explains things. I remember that posting, and really not wanting to get involved :) [snip]> jcm> I don''t know why you would think it might be a shim. > > nothing personal about Solaris. people everywhere are getting > screwed (or at least confused) by shim drivers. Now that free > software is more mainstream, the attacks on it have become incredibly > creative, and to my view often very effective.True, true. To the best of my knowledge, we just don''t do stunts like that with OpenSolaris (or Solaris, for that matter). Personally, I find the shim concept to be inelegant and almost always The Wrong Way To Do It(tm).> jcm> And finally, no, you won''t get source for the version which is > jcm> integrated into a Solaris 10 Update. It shouldn''t be too > jcm> different from the version which is in OpenSolaris, but nobody > jcm> has ever claimed otherwise. Is this the heart of the problem? > > I think it''s important, and good that you brought it up. So, the > source in hg & src.opensolaris.org never includes the Sol10 stable > branch? I''ve always wondered that and never been able to figure out > out. At some point, not necessarily disclosed, the sources currently > on mercurial will be taken back inside the wall where they''ll have the > bugs thrashed out of them before they become the next release of > stable Solaris?Ummm ... kinda yes kinda no. It''s more like a fishbone diagram: (use a monospace font here) NV builds..X..............Y..............Z.............. \ \ \ S10 Update A S10 Update B S10 Update C In very general terms, at certain points we take changes that have appeared in an NV build (X, Y or Z), and backport some or all of them to the Solaris 10 Update gate. Generally those changes will be bugfixes, but there are some RFEs which get backported as well. I wouldn''t say that the sources are "taken back inside the wall" as such, because to me that implies trying to Close previously Opened code - which we don''t do. Oh, and we thrash out the bugs in NV first - that''s the Release Currently Under Development(tm, etc).> On Linux, the GPL ensures that you can always get source even for the > stable version of the OS on which you''ve bought support, not just the > development version. Nothing can be ``taken back inside.'''' The > difference is no big deal if your goal is to work on big bugs, or > contribute new drivers and subsystems to Sun. If your goals are those > of a modest sysadmin, to fix your own small bugs locally, or disable > some ZFS sanity check to recover a pool that won''t import, no source > for the stable version does matter.Yeah, different story to how we''re doing things with OpenSolaris, due mainly to the engineering history and methodology.> But that wasn''t what I was thinking in this thread. I just thought > there was no source for mega_sas. so... great!:-)> I guess the comparison I settled on was Solaris mega_sas vs. Solaris > domU/Linux {anychip,Sil3124}. I had some other concerns, most > resolved by other posters'' reports and a few not, but no need to > repeat them.righto. Personally, I''d go with mega_sas - I''ve been involved with the project for quite a while so I''ve got a bit of bias :)> If one''s been burned enough though, he can always imagine more > concerns---if it turns out this card has some 1MByte ROM chip running > a whole proprietary vxworks image, so that even with a free software > driver you''re separated from the SATA interface so much that you can''t > implement new QoS ideas or NCQ or PMP support or fix USCSI/smartctl or > make a comstar Target Mode, while on cheaper junky chips you can do > those things....well, at least there''s still Sil3124 source too.True enough. Does VxWorks fit in 1Mb? I''ve truly got no idea about it.> I would say about all my speculative concerns, ``try it and see,'''' but > part of the point I see in this process, is that mainstream users > should exercise some vigilance to keep space open for a few others on > the fringe who may be particularly brilliant, motivated, and creative, > but without others'' vigilance hamstrung by having no truly free > platform. And by the time you''ve ``tried it and seen,'''' you''ve > already been fooled into handing over the cash, wasted a bunch of time > doing work for free without achieving freedom, gotten burned, and the > OEM''s have moved on to the next chip.Agreed. I''ve been burnt by that myself - most recently with the Marvell 88e8040 chip on the motherboard of my Dell laptop.> I only really want one card that works well and includes source and is > obtainable in the ballpark of <motherboard cost> * 3, so if this card > is it, great.I do think it could fit your requirements. cheers, James -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Miles Nordin
2009-Mar-13 20:02 UTC
[zfs-discuss] reboot when copying large amounts of data
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes:jcm> the mpt(7D) driver supports that card. Then I am apparently stuck with a closed-source driver again, and again by surprise. I bought it because I thought you said 1068E was supported by mega_sas: >> http://www.osnews.com/thread?317113 jcm> The driver for LSI''s MegaRAID SAS card is "mega_sas" which was jcm> integrated into snv_88. It''s planned for backporting to a jcm> Solaris 10 update. but apparently it was wishful thinking on my part: jcm> There are several LSI cards which use the 1068 and 1068E chip. jcm> Some of these use mpt(7d), some use mega_sas(7d). It all jcm> depends on the firmware of the card, basically. You could also jcm> have a look at the PCI IDs database at jcm> http://pciids.sourceforge.net to see what the name to pci jcm> vid/did mapping is. That provides a fairly good indicator of jcm> whether you''ll need mpt(7d) or mega_sas(7d). I downloaded the pci.ids file from sourceforge, but I do not understand how to tell which cards have the proprietary closed-source driver, even if I somehow know the PCI ID of the card before I buy it? (pci.ids lists only chip number in the description, and (a) LSI does not disclose the chip-to-card model association, and apparently the same chip can have different drivers so chip number isn''t enough). Is there some file in OpenSolaris against which I can cross-reference this? or...really, just use instead of pci.ids, since only the PCI ID not the description is enough to uniquely identify the card. Is it possible to change the firmware and thus the driver binding without changing the PCI ID so that I have to worry about manufacturers doing that, or can I really count on the PCI ID alone to tell me which driver will run the card? I''m worried that for example adding an iButton to unlock RAID5 on a supermicro card will change the driver attachment. Also does anyone know which cards work on SPARC? None? All? I know the SATA framework is x86 only, but AIUI none of the LSI cards actually use the SATA framework (so, presumably things like hd/smartmontools will not work, but at least the card has got a better shot of working on SPARC). I cannot test the AOC-USAS-L8i card I have in SPARC because it is PCIe, and I have only a PCI-X SPARC. Does anyone have an affordable card which is working well with the open-source mega_sas driver? It seems we are still without any open-source SATA card that works well. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/66b92fed/attachment.bin>
Miles Nordin
2009-Mar-13 20:33 UTC
[zfs-discuss] reboot when copying large amounts of data
>>>>> "c" == Miles Nordin <carton at Ivy.NET> writes:c> Is there some file in OpenSolaris against which I can c> cross-reference this? or...really, just use instead of c> pci.ids, since only the PCI ID not the description is enough I found these: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/intel/os/driver_aliases http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sparc/os/driver_aliases but they don''t seem to be complete. mega_sas is not listed in either. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/17948a7d/attachment.bin>
On Fri, Mar 13, 2009 at 3:33 PM, Miles Nordin <carton at ivy.net> wrote:> >>>>> "c" == Miles Nordin <carton at Ivy.NET> writes: > > c> Is there some file in OpenSolaris against which I can > c> cross-reference this? or...really, just use instead of > c> pci.ids, since only the PCI ID not the description is enough > > I found these: > > > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/intel/os/driver_aliases > > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sparc/os/driver_aliases > > but they don''t seem to be complete. mega_sas is not listed in either. >You can manually create an entry in /etc/driver_aliases to force a driver to bind to a specific card, regardless of firmware. Whether it will work or not isn''t apparent. What is apparent is that it won''t be supported by anyone though :) ANYWAYS, a bit of research goes a long ways :) You''ll find the various pci device ID''s towards the bottom. http://src.opensolaris.org/source/xref/zfs-crypto/phase2/usr/src/pkgdefs/SUNWmegasas/postinstall?&r=6542 Looks like there''s a TON of supported cards. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/ca56a855/attachment.html>
On Fri, Mar 13, 2009 at 3:33 PM, Miles Nordin <carton at ivy.net> wrote:> >>>>> "c" == Miles Nordin <carton at Ivy.NET> writes: > > c> Is there some file in OpenSolaris against which I can > c> cross-reference this? or...really, just use instead of > c> pci.ids, since only the PCI ID not the description is enough > > I found these: > > > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/intel/os/driver_aliases > > http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/sparc/os/driver_aliases > > but they don''t seem to be complete. mega_sas is not listed in either. > >Oh, and for people too lazy to google, here''s a start: http://pci-ids.ucw.cz/read/PC/1000/0060 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/76a76b71/attachment.html>
Miles Nordin
2009-Mar-13 22:00 UTC
[zfs-discuss] reboot when copying large amounts of data
>>>>> "t" == Tim <tim at tcsac.net> writes:t> http://src.opensolaris.org/source/xref/zfs-crypto/phase2/usr/src/pkgdefs/SUNWmegasas/postinstall?&r=6542 thanks. t> Looks like there''s a TON of supported cards. They are all 1078 cards though. James mentioned mega_sas supports some 1068E cards depending on the firmware? The 1078 seem to have rather large built-in write caches and thus be expensive and unnecessary for people who want to do raidz/slog. For example a lot of the cardsd are PERC with 4 ports for $500---costs more than the drives you plug it into. Are there any cheap cards for mega_sas? any word on which cards work on SPARC? t> http://pci-ids.ucw.cz/read/PC/1000/0060 or else the whole list at pciids.sourceforge.net in the mail i quoted -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/07a7f37d/attachment.bin>
James C. McPherson
2009-Mar-13 22:30 UTC
[zfs-discuss] reboot when copying large amounts of data
On Fri, 13 Mar 2009 18:00:04 -0400 Miles Nordin <carton at Ivy.NET> wrote:> >>>>> "t" == Tim <tim at tcsac.net> writes: > > t> http://src.opensolaris.org/source/xref/zfs-crypto/phase2/usr/src/pkgdefs/SUNWmegasas/postinstall?&r=6542 > > thanks. > > t> Looks like there''s a TON of supported cards. > > They are all 1078 cards though. James mentioned mega_sas supports > some 1068E cards depending on the firmware?Yes, that''s correct. Not very convenient, I''m afraid.> The 1078 seem to have rather large built-in write caches and thus be > expensive and unnecessary for people who want to do raidz/slog. For > example a lot of the cardsd are PERC with 4 ports for $500---costs > more than the drives you plug it into. Are there any cheap cards for > mega_sas?I don''t know, sorry. I''ve only ever seen Sun and Dell-branded MegaRAID SAS cards, and I have no idea about pricing.> any word on which cards work on SPARC? > > t> http://pci-ids.ucw.cz/read/PC/1000/0060 > > or else the whole list at pciids.sourceforge.net in the mail i quotedWe haven''t got mega_sas on SPARC at this point either. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Miles Nordin
2009-Mar-14 02:46 UTC
[zfs-discuss] reboot when copying large amounts of data
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes:jcm> We haven''t got mega_sas on SPARC at this point either. The card Blake found: http://www.provantage.com/lsi-logic-lsi00117~7LSIG03X.htm http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3080xr/index.html any idea if that''s got a shot of working on SPARC (with mpt i presume, or is there a fourth lsi driver)? or is it impossible to tell the chip/firmware of 3080X? or is it not going to happen period without Forth firmware? -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090313/c3a312d1/attachment.bin>
James C. McPherson
2009-Mar-14 07:05 UTC
[zfs-discuss] reboot when copying large amounts of data
On Fri, 13 Mar 2009 22:46:19 -0400 Miles Nordin <carton at Ivy.NET> wrote:> >>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: > > jcm> We haven''t got mega_sas on SPARC at this point either. > > The card Blake found: > > http://www.provantage.com/lsi-logic-lsi00117~7LSIG03X.htm > http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3080xr/index.html > > any idea if that''s got a shot of working on SPARC (with mpt i presume, > or is there a fourth lsi driver)? or is it impossible to tell the > chip/firmware of 3080X? or is it not going to happen period without > Forth firmware?The lsi.com has a link to their driver for that card, which is itmpt. That *generally* means that it''ll work fine with the mpt driver - on either sparc or x86/x64. I''ve not come across that particular model before, so I can''t say for sure. As for "will it work on sparc" - yes, I would expect so. BUT without fcode it won''t be bootable. That''s what you are really asking, isn''t it? It turns out you can change between IT and IR firmware using LSI''s sasflash utility, and if I''m reading other parts of http://www.lsi.com/DistributionSystem/AssetDocument/support/downloads/SASFlash_Readme.txt correctly, then you should be able to flash fcode onto the card if it isn''t there already. Have a look at the other downloads on the LSI url you mentioned. But no, sorry, no guarantees. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
Miles Nordin
2009-Mar-14 17:57 UTC
[zfs-discuss] reboot when copying large amounts of data
>>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes:jcm> As for "will it work on sparc" - yes, I would expect so. BUT jcm> without fcode it won''t be bootable. That''s what you are really jcm> asking, isn''t it? just if it will work. I want to add a slog. jcm> It turns out you can change between IT and IR firmware using jcm> LSI''s sasflash utility, is there a reason to use one mode or the other? itmpt sounds like it only works with IT but that''s a WAG and probably wrong. Is there a way to tell which mode mpt is using right now? Does mpt behave differently depending on mode? Thanks for the pointer to raidctl---I got this out of it: -----8<----- jacko at dharavi:~$ pfexec raidctl -l -g 0.2.0 2 Disk Vendor Product Firmware Capacity Status HSP ---------------------------------------------------------------------------- 0.2.0 ATA WDC WD10EADS-00 1A01 931.5G GOOD N/A GUID:50014ee202720bc0 jacko at dharavi:~$ pfexec raidctl -l 2 Controller Type Version ---------------------------------------------------------------- c2 LSI_1068E 1.22.01.00 jacko at dharavi:~$ -----8<----- I was hoping to see an -IT or -IR suffix on the version number but there isn''t one. In the README for sasflash, it looks like SASFLASH will not tell you what kind of firmware you currently have, either, just let you choose what kind to load. In the ^C peecee boot widget, it shows my card''s firmware version ending in -IT, so I think I have IT firmware. If I do, then Will''s speculation that IT means the card acts as a SATA target is an incorrect guess because it''s acting like a normal card right now. but on SPARC I''d have no way of seeing this -IT suffix, and I don''t know if I trust it anyway without understanding what the different versions are supposed to do differently. so, is there any way to tell from mpt in what mode the card''s running, or what the consequence of that is? It sounds like it''s possible one could buy a 1068E card from another vendor and end up with -IR firmware instead, without changing chip number or necessarily even PCI ID, so if we are trying to predict which cards behave how, it''s worth trying to figure out. I did try to RTFM. On OpenSolaris I get this loveliness: jacko at dharavi:~$ man mpt No manual entry for mpt. on SXCE I do at least have a man page, but there''s no mention of IT, IR, SR inside it. jcm> Have a look at the other downloads on the LSI url you jcm> mentioned. oops. yeah, a proprietary sparc driver is there! That''s not exactly, pow, we''re definitely done, because their driver is for Sol10 not nevada. But hopefully that means the card will not need some firmware-provided ``initialization'''' and might work with the sparc mpt included in Nevada! As for the closed-source LSI tool ''lsiutil'' mentioned in this thread: http://www.opensolaris.org/jive/message.jspa?messageID=184811#184811 A hidden undocumented option 15 -> option 12 in lsiutil, to make drive mapping based on port number rather than some LSI-GUID/drive-serial as it supposedly is by default. My AOC-USAS-L8i card is already behaving port-based, though: Solaris device names corresponding to physical ports even when I move disks around. I guess it''s other LSI cards that have the problem mentioned in the thread. there is a lsiutil binary inside the SPARC itmpt_sparc_5.07.04.zip file, so hopefully it''s the same one the thread says is hard to find. I guess I''m a little closer to understanding the LSI cards, though I still don''t know for sure what chips are on what card models (one can sort of guess), nor what IT, IR, SR mean. I''m still kind of pissed to be passing around warez too. Option 15, 12 wouldn''t be secret if we had source to lsiutil. We would have less problems with whether the driver is for Sol10 or Nevada. The firmware on this card is way too big, and there does not seem to be any SAS card for less than $100/port with an open-source driver that works well. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 304 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090314/a24a23c8/attachment.bin>
On Sat, Mar 14, 2009 at 12:57 PM, Miles Nordin <carton at ivy.net> wrote:> >>>>> "jcm" == James C McPherson <James.McPherson at Sun.COM> writes: > > jcm> As for "will it work on sparc" - yes, I would expect so. BUT > jcm> without fcode it won''t be bootable. That''s what you are really > jcm> asking, isn''t it? > > just if it will work. I want to add a slog. > > jcm> It turns out you can change between IT and IR firmware using > jcm> LSI''s sasflash utility, > > is there a reason to use one mode or the other? itmpt sounds like it > only works with IT but that''s a WAG and probably wrong. Is there a > way to tell which mode mpt is using right now? Does mpt behave > differently depending on mode? > > Thanks for the pointer to raidctl---I got this out of it: > > -----8<----- > jacko at dharavi:~$ pfexec raidctl -l -g 0.2.0 2 > Disk Vendor Product Firmware Capacity Status HSP > > ---------------------------------------------------------------------------- > 0.2.0 ATA WDC WD10EADS-00 1A01 931.5G GOOD N/A > GUID:50014ee202720bc0 > jacko at dharavi:~$ pfexec raidctl -l 2 > Controller Type Version > ---------------------------------------------------------------- > c2 LSI_1068E 1.22.01.00 > jacko at dharavi:~$ > -----8<----- > > I was hoping to see an -IT or -IR suffix on the version number but > there isn''t one. In the README for sasflash, it looks like SASFLASH > will not tell you what kind of firmware you currently have, either, > just let you choose what kind to load. > > In the ^C peecee boot widget, it shows my card''s firmware version > ending in -IT, so I think I have IT firmware. If I do, then Will''s > speculation that IT means the card acts as a SATA target is an > incorrect guess because it''s acting like a normal card right now. > > but on SPARC I''d have no way of seeing this -IT suffix, and I don''t > know if I trust it anyway without understanding what the different > versions are supposed to do differently. so, is there any way to tell > from mpt in what mode the card''s running, or what the consequence of > that is? > > It sounds like it''s possible one could buy a 1068E card from another > vendor and end up with -IR firmware instead, without changing chip > number or necessarily even PCI ID, so if we are trying to predict > which cards behave how, it''s worth trying to figure out. > > I did try to RTFM. On OpenSolaris I get this loveliness: > > jacko at dharavi:~$ man mpt > No manual entry for mpt. > > on SXCE I do at least have a man page, but there''s no mention of IT, > IR, SR inside it. >IR==integrated RAID. This is a stripped down version of raid. It only does 1/0/1E. No 5 or 6. See: http://www.lsi.com/storage_home/products_home/host_bus_adapters/software_solutions/integrated_raid/ IT==initiator. In other words it should function as a regular non-RAID adapter or as a target. Initiator vs. target depends entirely on the driver bound to the card. See: http://www.supermicro.com/manuals/other/LSI%20MegaRAID_Configuration_for_the_LSI_1068_Controller.pdf SR==software RAID. I believe this is nearly the same as IR mode, but Supermicro for instance has an add-in card that will allow it to do RAID-5 as well. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090314/0c89f438/attachment.html>