zfs discuss - Feb 2007 - solaris - ata over ethernet - zfs - HPC

Hi,

I''d like to consider using the coraid products with solaris and ZFS but
I need them to work with x86_64 on on generic opteron/amd compatible 
hardware.   Currently the AOE driver is beta for sparc only.  I am 
planning to use the ZFS file system so the raid hardware in the coraid 
device will not be used as recommended for ZFS.   Only the direct access 
over ethernet to the disks will be used.  The installation will be part 
of new HPC cluster for computational chemistry and biology.

Does this seem like a good idea?  I am not a storage expert and am 
attempting to create a scalable distributed storage cluster for an HPC 
cluster.  I expect have a limited budget and commodity hardware, except 
for the coraid box.

Relevant links:
http://en.wikipedia.org/wiki/Ata_over_ethernet
http://www.coraid.com/
http://www.coraid.com/support/solaris/


All comments are welcome.
Thank you,
Kevin



-- 
Kevin C. Abbey
System Administrator
Rutgers University - BioMaPS Institute

Email: kabbey at biomaps.rutgers.edu


Hill Center - Room 279
110 Frelinghuysen Road
Piscataway, NJ  08854
Phone and Voice mail: 732-445-3288   


Wright-Rieman Laboratories Room 201
610 Taylor Rd.
Piscataway, NJ  08854-8087
Phone: 732-445-2069
Fax: 732-445-5958

On Mon, 5 Feb 2007, Kevin Abbey wrote:
> Hi,
>
> I''d like to consider using the coraid products with solaris and
ZFS but
> I need them to work with x86_64 on on generic opteron/amd compatible
> hardware.   Currently the AOE driver is beta for sparc only.  I am
                            ^^^

AOE ??
> planning to use the ZFS file system so the raid hardware in the coraid
> device will not be used as recommended for ZFS.   Only the direct access
> over ethernet to the disks will be used.  The installation will be part
> of new HPC cluster for computational chemistry and biology.
>
> Does this seem like a good idea?  I am not a storage expert and am
> attempting to create a scalable distributed storage cluster for an HPC
> cluster.  I expect have a limited budget and commodity hardware, except
> for the coraid box.
>
> Relevant links:
> http://en.wikipedia.org/wiki/Ata_over_ethernet
> http://www.coraid.com/
> http://www.coraid.com/support/solaris/
>
>
> All comments are welcome.
> Thank you,
> Kevin
>
Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
             OpenSolaris Governing Board (OGB) Member - Feb 2006

I''ve got a similar setup in a small ISP I''m helping out.

Right now, they use the AOE driver for Linux on both client and server
(they don''t use a CoRaid box, rather a standard Linux device). It
actually works quite well, including failover and hot-migration, when
used in conjunction with EVMS. Remember, this is an iSCSI competitor,
not a FC SAN replacement.

Personally, I think a nice solution is to get an old 1GBps FiberChannel
JBOD (say an EMC Clariion or Dell 650F/660F), two QLogic Dual-attach
HBAs (say the QLA2202 or QLA2212), and build a little dual-homed FC
setup. Think kind of thing has very good driver support in S10 and NV,
and with ZFS, works really nicely. Stuff like that is cheap, and easy to
find, and darned standard.  


--- This section is RE: the Solaris AOE Server Daemon ---

We''ve considered looking at porting the AOE _server_ module to Solaris,
especially since the Solaris loopback driver (/dev/lofi) is _much_ more
stable than the loopback module in Linux (the Linux loopback module is a
stellar piece of crap). 

The only currently available piece of software for Solaris from Co-Raid
is the AOE _client_, not the AOE _server_.  

Probably, the best option is to take a look at doing the port of the
Linux (or more likely, the FreeBSD) AOE server module to Solaris.

Alternately, something I''d like to explore is running the Linux AOE
module in a BrandZ container.

----</end>


-Erik



On Mon, 2007-02-05 at 19:27 -0500, Kevin Abbey wrote:
> Hi,
> 
> I''d like to consider using the coraid products with solaris and
ZFS but
> I need them to work with x86_64 on on generic opteron/amd compatible 
> hardware.   Currently the AOE driver is beta for sparc only.  I am 
> planning to use the ZFS file system so the raid hardware in the coraid 
> device will not be used as recommended for ZFS.   Only the direct access 
> over ethernet to the disks will be used.  The installation will be part 
> of new HPC cluster for computational chemistry and biology.
> 
> Does this seem like a good idea?  I am not a storage expert and am 
> attempting to create a scalable distributed storage cluster for an HPC 
> cluster.  I expect have a limited budget and commodity hardware, except 
> for the coraid box.
> 
> Relevant links:
> http://en.wikipedia.org/wiki/Ata_over_ethernet
> http://www.coraid.com/
> http://www.coraid.com/support/solaris/
> 
> 
> All comments are welcome.
> Thank you,
> Kevin
> 
> 
> 
-- 
Erik Trimble
Java System Support
Mailstop:  usca14-102
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

Kevin Abbey wrote:
> Does this seem like a good idea?  I am not a storage expert and am 
> attempting to create a scalable distributed storage cluster for an HPC 
> cluster. 
An AOE/ZFS/NFS setup doesn''t sound scalable or distributed; your
ZFS/NFS
server may turn out to be a bottleneck.

Wes Felter - wesley at felter.org

>We''ve considered looking at porting the AOE _server_ module to
Solaris,
>especially since the Solaris loopback driver (/dev/lofi) is _much_ more
>stable than the loopback module in Linux (the Linux loopback module is a
>stellar piece of crap). 
ok, it`s quite old and probably not the most elegant solution - but unstable?
could you underline that somehow?
i use the loopback module for years and never had a problem.
anyway - it`s getting a competitor: bugfixed version of dm-loop device-mapper
target has just been posted on dm-devel today.

roland
 
 
This message posted from opensolaris.org

roland wrote:
>> We''ve considered looking at porting the AOE _server_ module to
Solaris,
>> especially since the Solaris loopback driver (/dev/lofi) is _much_ more
>> stable than the loopback module in Linux (the Linux loopback module is
a
>> stellar piece of crap). 
>>     
>
> ok, it`s quite old and probably not the most elegant solution - but
unstable?
> could you underline that somehow?
> i use the loopback module for years and never had a problem.
> anyway - it`s getting a competitor: bugfixed version of dm-loop
device-mapper target has just been posted on dm-devel today.
>
> roland
>   
Thanks for the note about the device mapper loopback thing. I''ve banged
on the standard loopback module, and it is indeed only really suitable 
for casual use in Linux.   It has really horrid performance when you try 
to do something like mount a large file and use that as a filesystem (as 
in under Xen for guests), and actually frequently has either just 
crapped out or caused data corruption when used under significant load.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

AoE and ZFS do not work, as the coraid do not support this yet:

REF: http://www.coraid.com/support/solaris/aoe-1.3.1/doc/aoe-guide.html
Search for ZFS

---
Pascal Gauthier
http://www.nihilisme.ca/
 
 
This message posted from opensolaris.org

A new driver was released recently that has support for ZFS; check the Coraid
website for details.

We have a Coraid at work that we are testing and hope to (eventually) put on our
production network. We''re running Solaris 9, so I''m not sure
how comparable our results are with your situation. Anyway, we have ours
configured with 4 RAID-5 volumes across 12 disks. The main reason that
it''s not being used in production yet, is that we have to get anywhere
close to their advertised throughput. We''re getting around 30 MB/sec
reads and around 23 MB/sec writes. The Coraid is attached to via a cross-over
cable to a V240. Last I knew, the Coraid development team was aware of
throughput issues on Solaris, and was working to improve their drivers. We have
not yet tested with jumbo frames; I would expect that that would improve things
somewhat.

-Andrew
 
 
This message posted from opensolaris.org