zfs discuss - Jan 2010 - Is LSI SAS3081E-R suitable for a ZFS NAS ?

According to various posts the LSI SAS3081E-R seems to work well with
OpenSolaris.
But I''ve got pretty chilled-out from my recent problems with
Areca-1680''s.

Could anyone please confirm that the LSI SAS3081E-R works well ?
Is hotplug supported ?

Anything else I should know before buying one of these cards ?

Thanks,
Arnaud

>According to various posts the LSI SAS3081E-R seems to work well 
>with OpenSolaris.
>But I''ve got pretty chilled-out from my recent problems with
Areca-1680''s.
>
>Could anyone please confirm that the LSI SAS3081E-R works well ?
>Is hotplug supported ?
It works well in Solaris 10 including hotplugging.
-- 

Maurice Volaski, maurice.volaski at einstein.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University

On 11/01/10 11:57 PM, Arnaud Brand wrote:
>
> According to various posts the LSI SAS3081E-R seems to work well with
OpenSolaris.
> But I''ve got pretty chilled-out from my recent problems with
Areca-1680''s.
>
> Could anyone please confirm that the LSI SAS3081E-R works well ?
> Is hotplug supported ?
>
> Anything else I should know before buying one of these cards ?

These cards work very well with OpenSolaris, and attach using
the mpt(7d) driver - supports hotplugging and MPxIO too.


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

Thanks for your answer.

I asked primarily because of the mpt timeout issues I saw on the list.

I never experienced timeouts with my (personal) usas-l8i (lsi 1068e) but feared
this issue might cause some problems with 3081.

Anyway, thanks again.

Arnaud

-----Message d''origine-----
De?: James.McPherson at Sun.COM [mailto:James.McPherson at Sun.COM] De la part
de James C. McPherson
Envoy??: lundi 11 janvier 2010 23:05
??: Arnaud Brand
Cc?: zfs-discuss at opensolaris.org
Objet?: Re: [zfs-discuss] Is LSI SAS3081E-R suitable for a ZFS NAS ?

On 11/01/10 11:57 PM, Arnaud Brand wrote:
>
> According to various posts the LSI SAS3081E-R seems to work well with
OpenSolaris.
> But I''ve got pretty chilled-out from my recent problems with
Areca-1680''s.
>
> Could anyone please confirm that the LSI SAS3081E-R works well ?
> Is hotplug supported ?
>
> Anything else I should know before buying one of these cards ?

These cards work very well with OpenSolaris, and attach using
the mpt(7d) driver - supports hotplugging and MPxIO too.


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

As in they work without any possibility of mpt timeout issues?  I''m at
my wits end with a machine right now that has an integrated 1068E and is dying
almost hourly at this point.

If I could spend three hundred dollars or so and have my problems magically go
away, I''d love to pull the trigger on one of these SAS3081E-R cards
right now.

Let me know if I can provide any more feedback on my existing 1068E problems.  I
followed all the threads, but nothing apparently really fixes the problem.  I
was running 2009.06 (111b I think) and I just updated to 131 earlier today with
more of the same problems.  My ZFS pool is in such a sad state, it takes an hour
or so at most with all the resilvering happening before the machine goes to
lunch.  All of this on a SuperMicro motherboard and chassis with port expanders.
-- 
This message posted from opensolaris.org

It may depend on the firmware you''re running. We''ve got a
SAS1068E based
card in Dell R710 at the moment, connected to an external SAS JBOD, and 
we did have problems with the as shipped firmware.

However we''ve upgraded that, and _so far_ haven''t had further
issues. I
didn''t do the upgrade myself, so I''m not sure what it shipped
with, but
we''re currently running:


Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 1

Current active firmware version is 011a0000 (1.26.00)
Firmware image''s version is MPTFW-01.26.00.00-IT
   LSI Logic
x86 BIOS image''s version is MPTBIOS-6.24.00.00 (2008.07.01)
FCode image''s version is MPT SAS FCode Version 1.00.49 (2007.09.21)


We''ve got multiple ISCSI and NFS clients all running disk benchmarks in
loops to produce load, as well as repeated scrubs on the server, and 
we''ve been running several days now without fault.We''re going
to give it
several more days before we sign it off as stable tho!

Regards,
     Tristan.

On 26/01/2010 10:47 AM, Mark Nipper wrote:
> As in they work without any possibility of mpt timeout issues? 
I''m at my wits end with a machine right now that has an integrated
1068E and is dying almost hourly at this point.
>
> If I could spend three hundred dollars or so and have my problems magically
go away, I''d love to pull the trigger on one of these SAS3081E-R cards
right now.
>
> Let me know if I can provide any more feedback on my existing 1068E
problems.  I followed all the threads, but nothing apparently really fixes the
problem.  I was running 2009.06 (111b I think) and I just updated to 131 earlier
today with more of the same problems.  My ZFS pool is in such a sad state, it
takes an hour or so at most with all the resilvering happening before the
machine goes to lunch.  All of this on a SuperMicro motherboard and chassis with
port expanders.
>

I don''t claim to understand all the nitty gritty details but there
seems to be a difference between the supermicro motherboard integrated 1068E
controllers and the HBA based controllers in that when I had a machine built
with a motherboard based controller it wouldn''t properly talk to/work
with the LSI-SAS expander in the supermicro chassis. Using an 1068e based HBA
works fine and works well with osol.

Jonathan
-- 
This message posted from opensolaris.org

> It may depend on the firmware you''re running. We''ve
> got a SAS1068E based 
> card in Dell R710 at the moment, connected to an
> external SAS JBOD, and 
> we did have problems with the as shipped firmware.
Well, I may have misspoke.  I just spent a good portion of yesterday upgrading
to the latest firmware myself (downloaded from SuperMicro''s FTP,
version 1.26.00 also; after I figured out I had to pass the -o option to mptutil
to force the flash since it was complaining about a mismatched card or some
such) and I thought that the machine had locked up again later in the day
yesterday because I couldn''t ssh into the machine.

To my surprise though, I was able to log into the machine just fine this morning
directly on the console from the command line.  It seems the snv_125 bug with
/dev/ptmx bit me (the "error: /dev/ptmx: Permission denied" problem
that required me tracking down the release notes for snv_125 to figure out the
problem) and the server was happy otherwise.  More importantly, the zpool
activity had all finished and I have three clean spares again!  Normally this
amount of I/O would have totally killed the machine!

So somewhere between upgrading the firmware to the latest version and upgrading
to snv_131, it looks like the problem may have actually been addressed. 
I''m guardedly optimistic at this point, given the previous problems
I''ve had so far with this on-board controller.

Interesting to hear someone else with the same chip but on an expansion card has
no problems (but was with the on-board chip).
-- 
This message posted from opensolaris.org

> Thanks for your answer.
> 
> I asked primarily because of the mpt timeout issues I
> saw on the list.
Hi Arnaud,

I am looking into the LSI SAS 3081 as well. My current understanding with mpt
issues is that the "sticky" part of these problems is rather related
to multipath features, that is using port multipliers or sas expanders. Avoiding
these one should be fine.
I am quite a newbie though. Just judging from what I read here.

Regards,

Tonmaus
-- 
This message posted from opensolaris.org

On 28/01/10 09:36 PM, Tonmaus wrote:
>> Thanks for your answer.
>>
>> I asked primarily because of the mpt timeout issues I
>> saw on the list.
>
> Hi Arnaud,
>
> I am looking into the LSI SAS 3081 as well. My current understanding
> with mpt issues is that the "sticky" part of these problems is
rather
>  related to multipath features, that is using port multipliers or sas
 > expanders. Avoiding these one should be fine.
> I am quite a newbie though. Just judging from what I read here.
I do not think that you are reading the data correctly.

The issues that we have seen via this list and storage-discuss
have implicated downrev firmware on cards, and the various different
disk drives that people choose to attach to those cards.

The use of SAS expanders with mpt-based cards is *not* an issue.
The use of MPxIO with mpt-based cards is *not* an issue.

Both MPxIO and SAS Expanders are an essential part of the whole
picture that is the SS7000 appliance series, as well as the
J4x00 series.

Personally, I''m quite happy with the LSISAS3081E that I have
installed in my system, with the attached 320Gb consumer-grade
SATA2 disks.



James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

Hi James
> I do not think that you are reading the data
> correctly.
> 
> The issues that we have seen via this list and
> storage-discuss
> have implicated downrev firmware on cards, and the
> various different
> disk drives that people choose to attach to those
> cards.
Thanks for pointing that out. I have indeed noticed such reports but I
didn''t see any specific plans or ack''s to take these issues
from mpt. Thus the question is if these reports justify the assumption that
there was anything wrong with mpt in general.
> 
> The use of SAS expanders with mpt-based cards is
> *not* an issue.
> The use of MPxIO with mpt-based cards is *not* an
> issue.
I didn''t want to make the point of denying some of mpt''s core
features. I just saw a couple of reports that involved SAS Expanders,
specifically those based on LSI silicon, reported under the "mpt
problem" umbrella, and I understood that these issues were quite obstinate.
> Personally, I''m quite happy with the LSISAS3081E that
> I have
> installed in my system, with the attached 320Gb
> consumer-grade
> SATA2 disks.
> 
Excellent. That''s encouraging. I am planning a similar configuration,
with WD RE3 1 TB disks though.

Regards,

Tonmaus
-- 
This message posted from opensolaris.org

My experience was different again.
I have the same timeout issues with both the LSI and Supermicro cards in IT
mode.
IR mode on the Supermicro card didn''t solve the problem, but seems to
have reduced it .
Server has 1 x 16 bay chassis and 1 x 24 bay chassis (both use expander)
test pool has 24 x WD10EARS in 6 disk vdev sets, 1 on 16 bay and 2 on 24 bay.

Mark
-- 
This message posted from opensolaris.org

Are you using the latest IT mode firmware? (1.26.00 I think, listed above and
without checking mine using AOC-USAS-L8i which uses same controller)

Also, I noticed you''re using ''EARS'' series drives.
Again, I''m not sure if the WD10EARS drives suffer from a problem
mentioned in these posts, but it might be worth looking into -- especially the
last link:

1. On synology site, seems like older 4-platter 1.5TB EADS OK (WD15EADS-00R6B0),
but newer 3 platter EADS have problems (WD15EADS-00P8B0):
http://forum.synology.com/enu/viewtopic.php?f=151&t=19131&sid=c1c446863595a5addb8652a4af2d09ca
2. A mac user has problems with WD15EARS-00Z5B1:
http://community.wdc.com/t5/Desktop/WD-1-5TB-Green-drives-Useful-as-door-stops/td-p/1217/page/2
(WD 1.5TB Green drives - Useful as door stops)
http://community.wdc.com/t5/Desktop/WDC-WD15EARS-00Z5B1-awful-performance/m-p/5242
(WDC WD15EARS-00Z5B1 awful performance)

Cheers,
Simon

http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
-- 
This message posted from opensolaris.org

Are you using the latest IT mode firmware? (1.26.00 I think, listed above and
without checking mine using AOC-USAS-L8i which uses same controller)

Also, I noticed you''re using ''EARS'' series drives.
Again, I''m not sure if the WD10EARS drives suffer from a problem
mentioned in these posts, but it might be worth looking into -- especially the
last link:

1. On synology site, seems like older 4-platter 1.5TB EADS OK (WD15EADS-00R6B0),
but newer 3 platter EADS have problems (WD15EADS-00P8B0):
http://forum.synology.com/enu/viewtopic.php?f=151&t=19131&sid=c1c446863595a5addb8652a4af2d09ca
2. A mac user has problems with WD15EARS-00Z5B1:
http://community.wdc.com/t5/Desktop/WD-1-5TB-Green-drives-Useful-as-door-stops/td-p/1217/page/2
(WD 1.5TB Green drives - Useful as door stops)
http://community.wdc.com/t5/Desktop/WDC-WD15EARS-00Z5B1-awful-performance/m-p/5242
(WDC WD15EARS-00Z5B1 awful performance)

Cheers,
Simon

http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
-- 
This message posted from opensolaris.org

> Also, I noticed you''re using ''EARS'' series
drives.
> Again, I''m not sure if the WD10EARS drives suffer
> from a problem mentioned in these posts, but it might
> be worth looking into -- especially the last link:
Aren''t the EARS drives the first ones using 4k sectors? Does
OpenSolaris support that properly yet? From what I''ve read using the
512-byte emulation mode in the drives is not good for performance (lots of
read/modify/write), though I don''t know whether that could cause these
kind of problems.
-- 
This message posted from opensolaris.org

> Aren''t the EARS drives the first ones using 4k
> sectors? Does OpenSolaris support that properly yet?
> From what I''ve read using the 512-byte emulation mode
> in the drives is not good for performance (lots of
> read/modify/write), though I don''t know whether that
> could cause these kind of problems.
Yes, I think the ''EARS'' series are the first WD consumer
drives using the new 4K sector ''Advanced Formatting'':
http://www.wdc.com/wdproducts/library/SpecSheet/ENG/2879-701229.pdf :
"Advanced Formatting
Technology being adopted by WD and
other drive manufacturers to increase
media format efficiencies, thus
enabling larger drive capacities.
(WDxxEARS and WDxxxxAARS
models only)"

I saw some discussion of performance and alignment, but it was in relation to
Linux, perhaps it doesn''t relate to Solaris?:
http://www.gossamer-threads.com/lists/linux/kernel/1039141

Cheers,
Simon
-- 
This message posted from opensolaris.org

I do not know if this has anything to do with some of the problems
you''re seeing.  however, there is definitely an issue using /some/ LSI
controllers on Supermicro boards.  I just rencetly had an issue with using
LSI3442 (the next gen over the 1068E chipset) on a Supermicro MB (X8DTN-6). 
Anyhow, if the LSI board is in the system, the drives are recognized and
everything works.  HOWEVER, if you have to use the SATA CD-rom/DVD drive - it
will not boot from the drive.   For whatever strange reason, supermico and I
determined that it just didn''t like the LSI SAS Controller in the
system which caused this.  We have no idea why a SAS controller would affect a
SATA boot. . .

Another issue we have found is the latest x8dtn board does not play nice with
the latest release of Solaris (at least for us).  It will work fine up to
snx129.  However 131 (or upgrading from 129 to 131) will fail on boot after
GRUB.  It''ll just cycle/reboot at that stage over and over.

Not sure what the issue is between 129 and 131 but that is what we''ve
found on our Supermicro MB and the latest OpenSolaris version.
-- 
This message posted from opensolaris.org

I''m looking into the alignment implications for the WD10EARS disks.
It may explain my issues.

I seem to recall boot issues in some of the LSI release notes affecting other
boot devices. I think it takes over boot responsibility.
I''ve encountered this sort of issue over the years with many scsi
cards.

Mark.
-- 
This message posted from opensolaris.org

Update:

For the WD10EARS, the blocks appear to be aligned on the 4k boundary when zfs
uses the whole disk (whole disk as EFI partition).

Part      Tag    Flag     First Sector         Size         Last Sector
 0        usr    wm                256       931.51Gb          1953508750
 
 calc    256*512/4096=32

Mark.
-- 
This message posted from opensolaris.org

Mark Bennett <mark.bennett at public.co.nz> writes:
> Update:
>
> For the WD10EARS, the blocks appear to be aligned on the 4k boundary
> when zfs uses the whole disk (whole disk as EFI partition).
>
> Part      Tag    Flag     First Sector         Size         Last Sector
>  0        usr    wm                256       931.51Gb          1953508750
>  
>  calc    256*512/4096=32
I''m afraid this isn''t enough.  if you enable compression, any
ZFS write
can be unaligned.  also, if you''re using raid-z with an odd number of
data disks, some of (most of?) your stripes will be misaligned.

ZFS needs to use 4096 octets as the basic block to fully exploit
the performance of these disks.
-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game

See also PSARC 2008/769 which considers 4 KB blocks for the entire OS
in a phased approach.
http://arc.opensolaris.org/caselog/PSARC/2008/769/inception.materials/design_doc
 -- richard

Hey Mark,
        I spent *so* many hours looking for that firmware.  Would you please
post the link?  Did the firmware dl you found come with fcode? Running blade
2000 here (SPARC).

Thx
Jake

On Jan 26, 2010 11:52 AM, "Mark Nipper" <nipsy at bitgnome.net>
wrote:
> It may depend on the firmware you''re running. We''ve
> got a SAS1068E based
> card in Dell R710 at...
Well, I may have misspoke.  I just spent a good portion of yesterday
upgrading to the latest firmware myself (downloaded from SuperMicro''s
FTP,
version 1.26.00 also; after I figured out I had to pass the -o option to
mptutil to force the flash since it was complaining about a mismatched card
or some such) and I thought that the machine had locked up again later in
the day yesterday because I couldn''t ssh into the machine.

To my surprise though, I was able to log into the machine just fine this
morning directly on the console from the command line.  It seems the snv_125
bug with /dev/ptmx bit me (the "error: /dev/ptmx: Permission denied"
problem
that required me tracking down the release notes for snv_125 to figure out
the problem) and the server was happy otherwise.  More importantly, the
zpool activity had all finished and I have three clean spares again!
 Normally this amount of I/O would have totally killed the machine!

So somewhere between upgrading the firmware to the latest version and
upgrading to snv_131, it looks like the problem may have actually been
addressed.  I''m guardedly optimistic at this point, given the previous
problems I''ve had so far with this on-board controller.

Interesting to hear someone else with the same chip but on an expansion card
has no problems (but was with the on-board chip).

-- 
This message posted from opensolaris.org
_______________________________________________

zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zf...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100131/e6ab1193/attachment.html>

On 31 Jan 2010, Jacob Ritorto wrote:
>    I spent *so* many hours looking for that firmware.? Would you
>    please post the link?? Did the firmware dl you found come with fcode?
>    Running blade 2000 here (SPARC).
	Well, I can''t say for sure it''s the right firmware for
your device.  I grabbed the LSI firmware from SuperMicro''s FTP
site at:
---
ftp://ftp.supermicro.com/driver/SAS/LSI/1064_1068/IT/Firmware/B3/L8i/1.26.01/

which appears to be a little newer than even what I grabbed
before.

	I''m not familiar with fcode, so I can''t say whether it
did.  The two things that actually flashed for me were the
firmware image for the 1068E itself and the MegaSAS BIOS/ROM
image.  Both version numbers changed once I had done the update.

	Now having said all of that, things look a bit flaky
still on my end.  I''m running a scrub which started earlier today
and seems to be hung.  I''m seeing a status of:
---
  pool: san
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using ''zpool upgrade''.  Once this is
done, the
        pool will no longer be accessible on older software versions.
 scrub: scrub in progress for 20h16m, 21.68% done, 73h15m to go
config:

        NAME         STATE     READ WRITE CKSUM
        san          ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c7t5d0   ONLINE       0     0     0
            c7t6d0   ONLINE       0     0     0
            c7t7d0   ONLINE       0     0     0
            c7t8d0   ONLINE       0     0     0  2.69M repaired
            c7t9d0   ONLINE       0     0     0
            c7t10d0  ONLINE       0     0     0
          raidz2-1   ONLINE       0     0     0
            c7t11d0  ONLINE       0     0     0
            c7t12d0  ONLINE       0     0     0
            c7t13d0  ONLINE       0     0     0
            c7t14d0  ONLINE       0     0     0
            c7t15d0  ONLINE       0     0     0
            c7t16d0  ONLINE       0     0     0
          raidz2-2   ONLINE       0     0     0
            c7t17d0  ONLINE       0     0     0
            c7t18d0  ONLINE       0     0     0
            c7t19d0  ONLINE       0     0     0
            c7t20d0  ONLINE       0     0     0
            c7t21d0  ONLINE       0     0     0
            c7t22d0  ONLINE       0     0     0
        spares
          c7t23d0    AVAIL
          c7t24d0    AVAIL
          c7t25d0    AVAIL

errors: No known data errors
---

and errors like this:
---
Jan 31 20:15:21 goliath scsi: [ID 243001 kern.warning] WARNING: /pci at
0,0/pci1002,5a20 at c/pci15d9,a511 at 0 (mpt0):
Jan 31 20:15:21 goliath         mpt_handle_event: IOCStatus=0x8000,
IOCLogInfo=0x31123000
Jan 31 20:15:23 goliath scsi: [ID 365881 kern.info] /pci at 0,0/pci1002,5a20 at
c/pci15d9,a511 at 0 (mpt0):
Jan 31 20:15:23 goliath         Log info 0x31123000 received for target 13.
---

	With no problems at all, that scrub I don''t think takes
nearly that long (I think it was less than 12 hours previously)
and the percentage is barely moving, although it is increasing.

	Even still, the exported volumes still appear to be
working via iSCSI for now, which is a plus.

-- 
Mark Nipper
nipsy at bitgnome.net (XMPP)
+1 979 575 3193

I did see that and confirmed the support has made it into the 130 release
I''m testing with.

However, the WD10EARS does not expose 4k sectors to the outside world, so it is
not identified as supporting it.
Correct alignment, to ensure best performance of the internal translation, seems
to be the only option available.

Mark.
-- 
This message posted from opensolaris.org

The WD10EARS disks don''t work well.

I had too many issues with timeouts that disappeared when replacing them with
ST32000542AS drives.

My next challenge is to get the LSI 3081 to boot off the disk I want it to, and
then to get multipath functional.

Has anyone else had issues with the LSI IT mode firmware changing the disk order
so the bootable disk is no longer the one booted from with expanders?
It boots with only two disks installed(bootable zfs mirror). Add some more and
the target "boot disk" moves to one of them.

Mark.
-- 
This message posted from opensolaris.org

The problems you had with the x8dtn, did the affect the x8dtn+-o that you know
of, or just the -6?

I''m thinking of building a system around this due to the PCI-X so I can
use the AOC-SAT2-MV8.

Any thoughts?

Thanks.
-- 
This message posted from opensolaris.org