zfs discuss - Apr 2007 - Experience with Promise Tech. arrays/jbod''s?

Greetings,

In looking for inexpensive JBOD and/or RAID solutions to use with ZFS,
I''ve
run across the recent "VTrak" SAS/SATA systems from Promise
Technologies,
specifically their E-class and J-class series:

E310f FC-connected RAID:
  http://www.promise.com/product/product_detail_eng.asp?product_id=175

E310s SAS-connected RAID:
  http://www.promise.com/product/product_detail_eng.asp?product_id=176

J300s SAS-connected JBOD:
  http://www.promise.com/product/product_detail_eng.asp?product_id=163


The prices seem very reasonable as compared to Sun, HP, EMC, etc. You can
buy bare drives (SAS or SATA) and plug them into the chassis.  If
you''re
using dual RAID controllers with SATA drives, they sell a dual-port adapter
that goes in the canister with each disk drive.

Has anyone tried these with Solaris yet?  The docs claim Solaris-ready,
but I''m particularly curious to know if the E310f supports
Sun''s MPXIO
(multipathing) drivers in the dual-controller configuration.

How about the SAS-connected chassis''?  What SAS HBA''s are
people using
with Solaris-10?

I''m mostly hoping to hear if these products are decent, or junk, or
somewhere
in between.

Thanks and regards,

Marion

Marion Hakanson wrote:
> In looking for inexpensive JBOD and/or RAID solutions to use with ZFS,
I''ve
> run across the recent "VTrak" SAS/SATA systems from Promise
Technologies,
> specifically their E-class and J-class series:
> 
> E310f FC-connected RAID:
>   http://www.promise.com/product/product_detail_eng.asp?product_id=175
> 
> E310s SAS-connected RAID:
>   http://www.promise.com/product/product_detail_eng.asp?product_id=176
> 
> J300s SAS-connected JBOD:
>   http://www.promise.com/product/product_detail_eng.asp?product_id=163
> 
> 
> The prices seem very reasonable as compared to Sun, HP, EMC, etc. You can
> buy bare drives (SAS or SATA) and plug them into the chassis.  If
you''re
> using dual RAID controllers with SATA drives, they sell a dual-port adapter
> that goes in the canister with each disk drive.
> 
> Has anyone tried these with Solaris yet?  The docs claim Solaris-ready,
> but I''m particularly curious to know if the E310f supports
Sun''s MPXIO
> (multipathing) drivers in the dual-controller configuration.
This looks similar to the recently announced Sun StorageTek 2500 Low Cost
Array product line.
http://www.sun.com/storagetek/disk_systems/workgroup/2500/

multi-path support is coming later (sorry, I don''t know the details)
> How about the SAS-connected chassis''?  What SAS HBA''s are
people using
> with Solaris-10?
http://www.sun.com/storagetek/storage_networking/hba/sas/specs.xml
  -- richard

Hello Richard,

Thursday, April 19, 2007, 10:41:58 PM, you wrote:

RE> Marion Hakanson wrote:
>> In looking for inexpensive JBOD and/or RAID solutions to use with ZFS,
I''ve
>> run across the recent "VTrak" SAS/SATA systems from Promise
Technologies,
>> specifically their E-class and J-class series:
>> 
>> E310f FC-connected RAID:
>>   http://www.promise.com/product/product_detail_eng.asp?product_id=175
>> 
>> E310s SAS-connected RAID:
>>   http://www.promise.com/product/product_detail_eng.asp?product_id=176
>> 
>> J300s SAS-connected JBOD:
>>   http://www.promise.com/product/product_detail_eng.asp?product_id=163
>> 
>> 
>> The prices seem very reasonable as compared to Sun, HP, EMC, etc. You
can
>> buy bare drives (SAS or SATA) and plug them into the chassis.  If
you''re
>> using dual RAID controllers with SATA drives, they sell a dual-port
adapter
>> that goes in the canister with each disk drive.
>> 
>> Has anyone tried these with Solaris yet?  The docs claim Solaris-ready,
>> but I''m particularly curious to know if the E310f supports
Sun''s MPXIO
>> (multipathing) drivers in the dual-controller configuration.
RE> This looks similar to the recently announced Sun StorageTek 2500 Low Cost
RE> Array product line.
RE> http://www.sun.com/storagetek/disk_systems/workgroup/2500/

RE> multi-path support is coming later (sorry, I don''t know the
details)

MPxIO for SAS went into b63 - see
http://www.opensolaris.org/os/community/on/flag-days/pages/2007041203/

Now the question is if it will make s10u4?

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Robert Milkowski wrote:
> RE> This looks similar to the recently announced Sun StorageTek 2500 Low
Cost
> RE> Array product line.
> RE> http://www.sun.com/storagetek/disk_systems/workgroup/2500/
> 
> RE> multi-path support is coming later (sorry, I don''t know the
details)
> 
> MPxIO for SAS went into b63 - see
http://www.opensolaris.org/os/community/on/flag-days/pages/2007041203/
Nice catch... timing is everything :-)
I''ll infer from this that the SAS HBA from Sun is based on the mpt
driver which
works with LSI controllers.
  -- richard

Robert Milkowski wrote:
...
> RE> multi-path support is coming later (sorry, I don''t know the
details)
> 
> MPxIO for SAS went into b63 - see
http://www.opensolaris.org/os/community/on/flag-days/pages/2007041203/
> 
> Now the question is if it will make s10u4?
We''re working on that right now. The backport has been
assigned to me and we''re hoping to get patches out
real soon now. No committment on actual dates though.



James C. McPherson
--
Solaris kernel software engineer
Sun Microsystems

Richard Elling wrote:
...
> Nice catch... timing is everything :-)
> I''ll infer from this that the SAS HBA from Sun is based on the mpt
> driver which works with LSI controllers.
yes.


James C. McPherson
--
Solaris kernel software engineer
Sun Microsystems

Hello James,

Friday, April 20, 2007, 12:53:04 AM, you wrote:

JCM> Robert Milkowski wrote:
JCM> ...
>> RE> multi-path support is coming later (sorry, I don''t know
the details)
>> 
>> MPxIO for SAS went into b63 - see
http://www.opensolaris.org/os/community/on/flag-days/pages/2007041203/
>> 
>> Now the question is if it will make s10u4?
JCM> We''re working on that right now. The backport has been
JCM> assigned to me and we''re hoping to get patches out
JCM> real soon now. No committment on actual dates though.


Thanks.

-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Richard.Elling at Sun.COM said:
> This looks similar to the recently announced Sun StorageTek 2500 Low Cost
> Array product line.
http://www.sun.com/storagetek/disk_systems/workgroup/2500/
Wonder how I missed those.  Oh, probably because you can''t see them
on store.sun.com/shop.sun.com.  On papger, there are some differences
between this product and the VTrak E310f:
  - VTrak has RAID-6;  ST-2500 does not.
  - VTrak can take SATA disks;  ST-2500 lists only 15k rpm SAS disks.
  - VTrak can take up to 4 expansion shelves;  ST-2500 takes only 2.
  - VTrak has 3-yr advance-return warranty;  ST-2500 has 1-yr on-site.

Thanks for the SAS HBA and SAS MPXIO information.  Now, anyone have any
experience with the VTrak E-series arrays?  Does it work with Sun''s
fiber channel MPXIO drivers?

Regards,

Marion

Marion Hakanson wrote:
> Richard.Elling at Sun.COM said:
>> This looks similar to the recently announced Sun StorageTek 2500 Low
Cost
>> Array product line.
http://www.sun.com/storagetek/disk_systems/workgroup/2500/
> 
> Wonder how I missed those.  Oh, probably because you can''t see
them
> on store.sun.com/shop.sun.com.  On papger, there are some differences
> between this product and the VTrak E310f:
>   - VTrak has RAID-6;  ST-2500 does not.
that''s not necesarily a bad thing. "raid-6" is an
interpretable feature,
not one that different vendors agree on a definition for.
>   - VTrak can take SATA disks;  ST-2500 lists only 15k rpm SAS disks.
I think that might be a different model.

...
> Thanks for the SAS HBA and SAS MPXIO information.  Now, anyone have any
> experience with the VTrak E-series arrays?  Does it work with
Sun''s
> fiber channel MPXIO drivers?
The scsi_vhci multipathing driver doesn''t just work with Sun''s
FC stack, it also works with SAS (at least, it does in snv_63
and ... soon .. with patches for s10).

Multipathing support tends to depend on table entries and
manufacturer-specific information if the array isn''t active/active.
So ... unless the VTrak array is active/active in your configuration,
then you probably won''t see any multipathing capabilities. If you
want them added you should log an RFE requesting that support
be added.


cheers,
James C. McPherson
--
Solaris kernel software engineer
Sun Microsystems

James.McPherson at Sun.COM said:
> The scsi_vhci multipathing driver doesn''t just work with
Sun''s FC stack, it
> also works with SAS (at least, it does in snv_63 and ... soon .. with
patches
> for s10). 
Yes, it''s nice to see that''s coming;  And that FC & SAS
are "the same".
But I''m at S10U3 right now.

> Multipathing support tends to depend on table entries and
manufacturer-specifi
> c information if the array isn''t active/active. So ... unless the
VTrak array
> is active/active in your configuration, then you probably won''t
see any
> multipathing capabilities. If you want them added you should log an RFE
> requesting that support be added. 
I have experience making S10 MPXIO work with 3rd-party arrays (HDS);  This
VTrak model can be configured to do active/active in a mode which looks
like what Sun calls "symmetric".  But my experience tells me that
mileage
may vary.  As I haven''t bought anything yet, I was hoping to hear from
someone who has already tried it.

> >   - VTrak can take SATA disks;  ST-2500 lists only 15k rpm SAS disks.
> I think that might be a different model.
Understood.  Perhaps I''m odd, but the fact that the same VTrak model
can
take either SAS or SATA disks, depending on needs, is a good feature.  Our
customers are researchers, and such flexibility is a plus.  Too bad
it''s
so hard to get pricing on the ST-2500 unit.  Here''s one reference I
found
which gives a idea:
http://www.storageperformance.org/results/b00022_Sun-ST2540-Mirroring_SPC2_full
-disclosure.pdf

Thanks for the note,

Marion

On Fri, 20 Apr 2007, James C. McPherson wrote:
> Richard Elling wrote:
> ...
> > Nice catch... timing is everything :-)
> > I''ll infer from this that the SAS HBA from Sun is based on
the mpt
> > driver which works with LSI controllers.
>
> yes.
As of Solaris 10 update 2, this combo did -not- work:

 - Sun T1000
 - Promise vtrak J300s (12 SATA drives presented as SAS)
 - LSI SAS3442E-R

See http://groups.google.com/group/comp.unix.solaris/msg/c8ad66278a5bf817
for more info.

I would be -delighted- to find that this is no longer the case.
After failing to get anywhere with the above setup, we punted and
went with Linux... which is better than what it replaced, but
I''d much rather be using ZFS than XFS or ext3!

The J300s itself isn''t too bad.  We have 24 of them and
they''ve
been in service for the past 6 months w/o problems; two of them
were DOA though and had to be RMA''d.  We had (and are still having)
a very hard time getting rails for them, so keep that in mind.

James

James W. Abendschan wrote:
> On Fri, 20 Apr 2007, James C. McPherson wrote:
> 
>> Richard Elling wrote:
>> ...
>>> Nice catch... timing is everything :-)
>>> I''ll infer from this that the SAS HBA from Sun is based on
the mpt
>>> driver which works with LSI controllers.
>> yes.
> 
> As of Solaris 10 update 2, this combo did -not- work:
> 
>  - Sun T1000
>  - Promise vtrak J300s (12 SATA drives presented as SAS)
>  - LSI SAS3442E-R
> 
> See http://groups.google.com/group/comp.unix.solaris/msg/c8ad66278a5bf817
> for more info.
> 
> I would be -delighted- to find that this is no longer the case.
> After failing to get anywhere with the above setup, we punted and
> went with Linux... which is better than what it replaced, but
> I''d much rather be using ZFS than XFS or ext3!
> 
> The J300s itself isn''t too bad.  We have 24 of them and
they''ve
> been in service for the past 6 months w/o problems; two of them
> were DOA though and had to be RMA''d.  We had (and are still
having)
> a very hard time getting rails for them, so keep that in mind.
Hi James,
thankyou for the pointer to that newsgroup article.

Since s10u2 we''ve fixed numerous issues with the mpt
driver on both Sparc and x86/x64. For instance, in
the latest version (125037-02 sparc / 125038-02 x86)
these items are addressed:


6368089 mpt misinterprets integrated raid events with reason code 
VOLUME_STATUS_CHANGED
6408660 mpt/scsi panicked due to general protection fault
6418521 sd driver with Solaris 10, LSI 1064 SAS HBA can''t detect more
than
12 drives
6457857 x86 showing mpt0: unknown event e received running bonnie/dbench on 
HW raid


If you have the opportunity to try out s10u3 + the
appropriate mpt patch I hope you''d see much better
behaviour.


cheers,
James C. McPherson
--
Solaris kernel software engineer
Sun Microsystems

On April 19, 2007 4:45:20 PM -0700 Marion Hakanson <hakansom at ohsu.edu> 
wrote:
>   - VTrak can take SATA disks;  ST-2500 lists only 15k rpm SAS disks.
I''d be surprised (and disappointed) if the 2500 can''t accept a
SATA disk.
> Thanks for the SAS HBA and SAS MPXIO information.  Now, anyone have any
> experience with the VTrak E-series arrays?  Does it work with
Sun''s
> fiber channel MPXIO drivers?
Isn''t MPXIO support by HBA and hard drive identification (not by the
enclosure)?  At least I don''t see how the enclosure should matter, as
long as it has 2 active paths.  So if you add the drive vendor info
into /kernel/drv/scsi_vhci.conf it should work.

-frank

Hello Frank,

Friday, April 20, 2007, 4:50:07 AM, you wrote:

FC> On April 19, 2007 4:45:20 PM -0700 Marion Hakanson <hakansom at
ohsu.edu>
FC> wrote:
>>   - VTrak can take SATA disks;  ST-2500 lists only 15k rpm SAS disks.
FC> I''d be surprised (and disappointed) if the 2500 can''t
accept a SATA disk.
>> Thanks for the SAS HBA and SAS MPXIO information.  Now, anyone have any
>> experience with the VTrak E-series arrays?  Does it work with
Sun''s
>> fiber channel MPXIO drivers?
FC> Isn''t MPXIO support by HBA and hard drive identification (not by
the
FC> enclosure)?  At least I don''t see how the enclosure should
matter, as
FC> long as it has 2 active paths.  So if you add the drive vendor info
FC> into /kernel/drv/scsi_vhci.conf it should work.

In a so called symmetric mode it should work as you described.
But many entry level and midsize arrays aren''t actually symmetric and
they have to be treated specifically.

And when it comes to SAS - Solaris mpt driver didn''t work with MPxIO
prior to b63.


-- 
Best regards,
 Robert                            mailto:rmilkowski at task.gda.pl
                                       http://milek.blogspot.com

Thanks to all for the helpful comments and questions.


fcusack at fcusack.com said:
> Isn''t MPXIO support by HBA and hard drive identification (not by
the
> enclosure)?  At least I don''t see how the enclosure should matter,
as long as
> it has 2 active paths.  So if you add the drive vendor info into
/kernel/drv/
> scsi_vhci.conf it should work. 
If the enclosure is JBOD, then yes, the drives would be the targets of MPXIO.
But for a RAID enclosure, it''s the RAID controller which speaks SCSI,
adds and
removes LUN''s, etc.  The three different arrays I''ve used have
all had settings
where you specify what kind of alternate-path "protocol" to speak to
the
various hosts involved.


rmilkowski at task.gda.pl said:
> In a so called symmetric mode it should work as you described. But many
entry
> level and midsize arrays aren''t actually symmetric and they have
to be
> treated specifically. 
This matches my limited experience.  What Sun calls "asymmetric" seems
to
match what some array vendors call "active/active with LUN affinity"
(or
"LUN ownership").  MPXIO "knows" about such asymmetric
arrays, but some
arrays don''t speak the right protocol (T10 ALUA), and there''s
so far no
way to manually tell MPXIO to do the asymmetric thing with them.

For example, our low-end HDS array looks to MPXIO as if it''s symmetric,
since
both controllers show their configured LUN''s all the time.  But only
one
controller can do I/O to a given LUN at one time, and the array takes a long
while to swap ownership between controllers, so MPXIO''s default
round-robin
load balancing yields terrible performance.  The workaround is to manually
set load-balancing to "none" and hope MPXIO uses the controller that
you
were wanting to be primary.

And some people wonder why I prefer NAS over SAN...(:-).

Regards,

Marion