CentOS - Nov 2005 - ATA-over-Ethernet v's iSCSI

Nick,

What are you planning on running over the shared connection? Database, eMail,
File Shares? How many users? How much data? What is your I/O profile?

I've worked with 'enterprise' storage most of my career either as a
consumer, adviser or provider - can't comment on AoE other than to suggest
you look at what are the business & technical goals, how they solve it and
what is your risk profile against the business need of the system. (I'm
currently working as an adviser)

When you have an opportunity to chase into "enterprise support" of new
disk is usually comes down to insufficient time for testing, or known problems
(heat, torque, variability in sample, failure rate & edge-case
incompatibility with previously certified products are normal)

You have hinted with the concern over TCP offload that you may have higher-end
performance needs and that this system carries a high business value and needs a
lower risk solution.

Remember risk is a cost.

regards Dave
www.hornfordassociates.com
>Nick Bryant
>Subject: [CentOS] ATA-over-Ethernet v's iSCSI
>
>Just wondering if anyone out there has any opinions or better still
>experience with Coraid's AoE products in a centos environment?
>
>We're looking at developing a very basic active/standby 2 node cluster
>which will need a shared storage component. It would be a huge bonus if
>other servers could also use the storage device. Originally I was looking
>at the Dell/EMC iSCSI solution as it's a cheaper solution than fibre
>channel.
>However, the performance issues without using a TCP offload HBA are a bit
>of a concern. 
>
>Then I found the Coraid (www.coraid.com) products based on the open
>standard AoE protocol. It's got a number of benefits including: price,
>less protocol overhead for the server and the ability to use any disks
>where as "enterprise" approved products form the likes of Dell/Sun
etc
>only support 250gb sata disks at the moment.
>
>I guess my concern is that it's a new technology that's not been
widely
>adopted so far and all the issues that go along with that. 
>
>Any options or feedback would be really helpful.
>
>Cheers,
>
>Nick
>
>_______________________________________________
>CentOS mailing list
>CentOS at centos.org
>http://lists.centos.org/mailman/listinfo/centos
>  
>

Nick Bryant wrote:  
> Just wondering if anyone out there has any opinions or
> better still experience with Coraid's AoE products in a
> centos environment?
I have been bombarded with CORAID marketing in various Linux
groups over the last few months.  This is because CORAID
smartly hit the Linux tradeshow circuit, which is definitely
the best way to get free "word of mouth" advertising.

I have seen their products.
I have read their technical specifications.
And I have come to one conclusion.

AoE is vendor marketing.
It is NOT SAN of any sort.
It relies 100% on server-wide (e.g., GFS) coherency.
Or you merely use it as dedicated storage (even if you slice
for different servers).

- The "efficiency" argument

Sure, it has less overhead than iSCSI.  That means if you put
in a cheap, $50 GbE card or use the on-motherboard GbE NIC,
you're going to get better performance.

But anyone who is serious about a SAN puts in a $500 iSCSI
HBA GbE, which are very affordable now.

- The "feature" reality

AoE has virtually _no_ features.  It's "dumb storage" and
that's that.  iSCSI is intelligent.  How intelligent your
target acts is up to you (and the cost).  It's multi-vendor
and there are complete standards, from host to target --
especially intelligent targets.  AoE just a simplistic block
interface that relies on 100% software coherency.
> We're looking at developing a very basic active/standby
> 2 node cluster which will need a shared storage component.
AoE is _not_ it then.  AoE does _not_ allow shared storage. 
You must slice the array for each system so they are
_independent_.  It is _not_ a SAN.  It does not have
multi-targetting features, only segmented target capability.

They talk GFS.  It's GFS in the worst absolute setup -- 100%
software, 0% intelligent target.  ;->
> It would be a huge bonus if other servers could also use
> the storage device.
Now it can do that.  You just slice your array for whatever
servers.
> Originally I was looking at the Dell/EMC iSCSI solution as
> it's a cheaper solution than fibre channel.
Have you considered Serial Attached SCSI (SAS)?
Most people haven't heard of it, so be sure to read my blog
from 3 months back:  
http://thebs413.blogspot.com/2005/08/serial-storage-is-future.html

12-24Gibps (1.2-2.4GiBps after 10/8 encoding) using a 4 or 8
channel trunk, which is what external target solutions are
using.  SAS is SCSI-2 over Serial (physically, basically same
as SATA, with twisted pair which SATA-IO also requires). 
It's very scalable and flexable and _damn_fast_.  In a
nutshell, SAS can use ...
- Internal SATA drives
- Internal SAS drives
- External SAS drives
- External SAS enclosures (with SAS drives)
- External SAS subsystems (with SATA or SAS drives)
- External SAS hubs (intelligent multi-targetting)

So what's the catch of SAS?  Same as SCSI:  
1.  Few people go for the multi-target options
2.  Shorter distance (8m ~ 25')

#2 is not an issue if you're providing storage for the
closet.  That was always the bonus with multi-target,
intelligent target, SCSI, before higher-speed FC-AL was
available.

#1 is where SAS is still "getting off the ground."  It
leverages existing SCSI-2, so the multi-targetting is there. 
But the vendor products are still coming out.

So the verdict is still out on SAS as a SAN solution merely
because of the limited products available right now.  But
it's definitely far more affordable than FC-AL, leverages
everything learned with multi-target SCSI-2 and is a heck of
a lot better for the closet than iSCSI (where distance isn't
a factor).
> However, the performance issues without using a TCP
> offload HBA are a bit of a concern. 
If you go iSCSI, you _must_ go with a HBA.  Heck, I would
argue that you would probably want a HBA for layer-2 Ethernet
too, although the layer-3/4 traffic is far worse.

But HBAs start at $500 these days.  That's chump change. 
There are some outstanding iSCSI HBAs under $1,000, so that
shouldn't deter you.

An intelligent, multi-targetted subsystem is where the cost
is going to be.  And that's where multi-target SAS devices,
once they become more commonplace, should be significantly
cheaper than iSCSI (before figuring disk cost).
> Then I found the Coraid (www.coraid.com) products based on
> the open standard AoE protocol.
It's _their_ standard.  And its _empty_.  It has _no_
SAN-level code.  There is _no_ multi-targeting logic.  You
have to slice the array -- only 1 connection per end-user
volume.
> It's got a number of benefits including: price,
So does SAS, and it uses the proven SCSI-2 protocol for
multi-targetting.
> less protocol overhead for the server
SCSI-2 is better than layer-2 Ethernet, let alone designed
for storage.  ;->
> and the ability to use any disks where as "enterprise"
> approved products form the likes of Dell/Sun etc
> only support 250gb sata disks at the moment.
Not true!  There are 400 and 500GB near-line 24x7 disks from
Seagate and Hitachi, respectively.

[ Sounds like someone fed you marketing. ;-]

In fact, one of Hitachi's big partners is Copan Systems, and
they have the 500GB drives in their VTL solution.
> I guess my concern is that it's a new technology that's
> not been widely adopted so far and all the issues that go
> along with that.
That's the _least_ of your concerns.  AoE has _nothing_ in it
from a SAN perspective.

I sure wish CORAID was more forthcoming on that.

At the same time, whenever I mention multi-targetting the
same volume, it finally shuts up the marketeers.  It's all
hype, 0 SAN substance.


Dave Hornford <OSD at HornfordAssociates.com>
wrote:
> What are you planning on running over the shared
> connection? Database, eMail, File Shares? How many users?
> How much data? What is your I/O profile?
Agreed.  If you can afford the latency and lower DTR, iSCSI
will do.  If you need maximum performance, really investigate
SAS.
> I've worked with 'enterprise' storage most of my career
> either as a consumer, adviser or provider - can't comment
> on AoE other than to suggest you look at what are the
> business & technical goals, how they solve it and what is
> your risk profile against the business need of the system.
> (I'm currently working as an adviser)
AoE is _not_ a SAN solution.
That 1 statement remove any consideration.

At the same time, I haven't deployed a multi-targetted SAS
solution yet.  The boards are out, the drives are out, the
enclosures are out, and some subsystem products exist.  Just
because it leverages existing SCSI-2 multi-targetting doesn't
mean someone has a well-designed, well-respected, intelligent
SAS multi-targettable/sharable solution yet.
> When you have an opportunity to chase into "enterprise
> support" of new disk is usually comes down to insufficient
> time for testing, or known problems (heat, torque,
> variability in sample, failure rate & edge-case
> incompatibility with previously certified products are
> normal)
AoE is designed for 1 thing -- centralized, segmented
storage.  It does that well.  However, it is _not_ a SAN
standard.  It does _not_ support multi-targetting of the same
volume.

So that means it's no different than if you had local storage
using 100% software (e.g., via GFS) to synchronize.  It is
"dumb".
> You have hinted with the concern over TCP offload that you
> may have higher-end performance needs and that this system
> carries a high business value and needs a lower risk
> solution. 
Agreed.  If you only need storage in the same closet (same
25'), see what intelligent, multi-target SAS solutions are
available.
> Remember risk is a cost.
Hmmm, I don't know why I haven't formulated that statement
before.  I talk at length with clients about "risk analysis,"
yet that simple statement says it all.



-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith at ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)

[ I'm resending this to the list ]

Nick Bryant wrote:  
> Just wondering if anyone out there has any opinions or
> better still experience with Coraid's AoE products in a
> centos environment?
I have been bombarded with CORAID marketing in various Linux
groups over the last few months.  This is because CORAID
smartly hit the Linux tradeshow circuit, which is definitely
the best way to get free "word of mouth" advertising.

I have seen their products.
I have read their technical specifications.
And I have come to one conclusion.

AoE is vendor marketing.
It is NOT SAN of any sort.
It relies 100% on server-wide (e.g., GFS) coherency.
Or you merely use it as dedicated storage (even if you slice
for different servers).

- The "efficiency" argument

Sure, it has less overhead than iSCSI.  That means if you put
in a cheap, $50 GbE card or use the on-motherboard GbE NIC,
you're going to get better performance.

But anyone who is serious about a SAN puts in a $500 iSCSI
HBA GbE, which are very affordable now.

- The "feature" reality

AoE has virtually _no_ features.  It's "dumb storage" and
that's that.  iSCSI is intelligent.  How intelligent your
target acts is up to you (and the cost).  It's multi-vendor
and there are complete standards, from host to target --
especially intelligent targets.  AoE just a simplistic block
interface that relies on 100% software coherency.
> We're looking at developing a very basic active/standby
> 2 node cluster which will need a shared storage component.
AoE is _not_ it then.  AoE does _not_ allow shared storage. 
You must slice the array for each system so they are
_independent_.  It is _not_ a SAN.  It does not have
multi-targetting features, only segmented target capability.

They talk GFS.  It's GFS in the worst absolute setup -- 100%
software, 0% intelligent target.  ;->
> It would be a huge bonus if other servers could also use
> the storage device.
Now it can do that.  You just slice your array for whatever
servers.
> Originally I was looking at the Dell/EMC iSCSI solution as
> it's a cheaper solution than fibre channel.
Have you considered Serial Attached SCSI (SAS)?
Most people haven't heard of it, so be sure to read my blog
from 3 months back:  
http://thebs413.blogspot.com/2005/08/serial-storage-is-future.html

12-24Gibps (1.2-2.4GiBps after 10/8 encoding) using a 4 or 8
channel trunk, which is what external target solutions are
using.  SAS is SCSI-2 over Serial (physically, basically same
as SATA, with twisted pair which SATA-IO also requires). 
It's very scalable and flexable and _damn_fast_.  In a
nutshell, SAS can use ...
- Internal SATA drives
- Internal SAS drives
- External SAS drives
- External SAS enclosures (with SAS drives)
- External SAS subsystems (with SATA or SAS drives)
- External SAS hubs (intelligent multi-targetting)

So what's the catch of SAS?  Same as SCSI:  
1.  Few people go for the multi-target options
2.  Shorter distance (8m ~ 25')

#2 is not an issue if you're providing storage for the
closet.  That was always the bonus with multi-target,
intelligent target, SCSI, before higher-speed FC-AL was
available.

#1 is where SAS is still "getting off the ground."  It
leverages existing SCSI-2, so the multi-targetting is there. 
But the vendor products are still coming out.

So the verdict is still out on SAS as a SAN solution merely
because of the limited products available right now.  But
it's definitely far more affordable than FC-AL, leverages
everything learned with multi-target SCSI-2 and is a heck of
a lot better for the closet than iSCSI (where distance isn't
a factor).
> However, the performance issues without using a TCP
> offload HBA are a bit of a concern. 
If you go iSCSI, you _must_ go with a HBA.  Heck, I would
argue that you would probably want a HBA for layer-2 Ethernet
too, although the layer-3/4 traffic is far worse.

But HBAs start at $500 these days.  That's chump change. 
There are some outstanding iSCSI HBAs under $1,000, so that
shouldn't deter you.

An intelligent, multi-targetted subsystem is where the cost
is going to be.  And that's where multi-target SAS devices,
once they become more commonplace, should be significantly
cheaper than iSCSI (before figuring disk cost).
> Then I found the Coraid (www.coraid.com) products based on
> the open standard AoE protocol.
It's _their_ standard.  And its _empty_.  It has _no_
SAN-level code.  There is _no_ multi-targeting logic.  You
have to slice the array -- only 1 connection per end-user
volume.
> It's got a number of benefits including: price,
So does SAS, and it uses the proven SCSI-2 protocol for
multi-targetting.
> less protocol overhead for the server
SCSI-2 is better than layer-2 Ethernet, let alone designed
for storage.  ;->
> and the ability to use any disks where as "enterprise"
> approved products form the likes of Dell/Sun etc
> only support 250gb sata disks at the moment.
Not true!  There are 400 and 500GB near-line 24x7 disks from
Seagate and Hitachi, respectively.

[ Sounds like someone fed you marketing. ;-]

In fact, one of Hitachi's big partners is Copan Systems, and
they have the 500GB drives in their VTL solution.
> I guess my concern is that it's a new technology that's
> not been widely adopted so far and all the issues that go
> along with that.
That's the _least_ of your concerns.  AoE has _nothing_ in it
from a SAN perspective.

I sure wish CORAID was more forthcoming on that.

At the same time, whenever I mention multi-targetting the
same volume, it finally shuts up the marketeers.  It's all
hype, 0 SAN substance.


Dave Hornford <OSD at HornfordAssociates.com>
wrote:
> What are you planning on running over the shared
> connection? Database, eMail, File Shares? How many users?
> How much data? What is your I/O profile?
Agreed.  If you can afford the latency and lower DTR, iSCSI
will do.  If you need maximum performance, really investigate
SAS.
> I've worked with 'enterprise' storage most of my career
> either as a consumer, adviser or provider - can't comment
> on AoE other than to suggest you look at what are the
> business & technical goals, how they solve it and what is
> your risk profile against the business need of the system.
> (I'm currently working as an adviser)
AoE is _not_ a SAN solution.
That 1 statement remove any consideration.

At the same time, I haven't deployed a multi-targetted SAS
solution yet.  The boards are out, the drives are out, the
enclosures are out, and some subsystem products exist.  Just
because it leverages existing SCSI-2 multi-targetting doesn't
mean someone has a well-designed, well-respected, intelligent
SAS multi-targettable/sharable solution yet.
> When you have an opportunity to chase into "enterprise
> support" of new disk is usually comes down to insufficient
> time for testing, or known problems (heat, torque,
> variability in sample, failure rate & edge-case
> incompatibility with previously certified products are
> normal)
AoE is designed for 1 thing -- centralized, segmented
storage.  It does that well.  However, it is _not_ a SAN
standard.  It does _not_ support multi-targetting of the same
volume.

So that means it's no different than if you had local storage
using 100% software (e.g., via GFS) to synchronize.  It is
"dumb".
> You have hinted with the concern over TCP offload that you
> may have higher-end performance needs and that this system
> carries a high business value and needs a lower risk
> solution. 
Agreed.  If you only need storage in the same closet (same
25'), see what intelligent, multi-target SAS solutions are
available.
> Remember risk is a cost.
Hmmm, I don't know why I haven't formulated that statement
before.  I talk at length with clients about "risk analysis,"
yet that simple statement says it all.


-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith at ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)