crossbow discuss - Jun 2009 - [Fwd: RDMA & Virtual Nics Possible ?]

Gary,

It depends on what RDMA transport protocol and link layer you are 
talking about.  Currently there are two RDMA options available in the 
market, InfiniBand (IB) and iWARP (RDMA over TCP/IP), and there are more 
on the way.

At the moment, IB transport operates only over IB physical links, but IB 
transport over Ethernet (DCE/CEE/DCBE) is being standardized.  It will 
be implemented both in IB hardware engines (HCAs) and in SW.  We expect 
that these will be implemented in Solaris in a way that is consistent 
with Crossbow VNIC virtualization.

iWARP is not currently supported in Solaris.  iWARP is typically 
implemented in HW as an RDMA engine on top of a TCP Offload Engine 
(TOE).  TOE is not supported in Solaris and it is hard to imagine how it 
could be integrated with Crossbow VNICs.  It is possible to implement 
iWARP in SW and that could be integrated with Crossbow VNICs, but we 
have no plans to do so.

So the answer is that there is currently no way to do RDMA through 
Crossbow VNICs.

Carl

> -------- Original Message --------
> Subject: [crossbow-discuss] RDMA & Virtual Nics Possible ?
> Date: Thu, 18 Jun 2009 15:32:00 -0700 (PDT)
> From: gary mazz <garym at oedata.com>
> To: crossbow-discuss at opensolaris.org
> 
> I''m cross-posting to the crossbow discussion group
> 
> -g
> 
> Can anyone tell me or point me in the right direction
>  to find out whether RDMA works though crossbow
>  virtual adapters ?
> 
>  cheers,
>  gary

gary mazzaferro wrote:
> Hi Carl,
> 
>  Thanks for the breakdown. 
> 
> Hmmm, Using hw rma kind of defeats the purpose rma in the same box. The 
> standardization efforts for IB over ethernet is still a ways off. I 
> personally don''t see the sense in it since IB is 30% lower cost
than
> 10ge and IB is already running 4x with 8x not far off.
Gary, its not quite clear from your original question or comment above 
what it is that you are asking for. Perhaps you can clarify? I 
originally thought you were asking about RDMA over ETH, but now you seem 
to be asking for a VNIC over IB.
> Since IB is already supported in the kernel and the IB drivers are 
> pretty stable, shouldn''t it be fairly straight forward to
implement the
> sw copy with crossbow polling ? It could be done using raw IP or UDP 
> until PF_PACKET is in production. The virtual switch can just pass 
> pointers around at that point.
As of OpenSolaris 2009.06, the IPoIB driver (ibd) is a crossbow-enabled 
driver, but does have a VNIC abstraction, but that could be done if you 
were so inclined to do it. But we are taking a slightly different 
approach and are working on a leaner EoIB driver that will allow for 
VNICs/VSwitches to provide more efficient host-to-host (including 
DomU-to-DomU) and host-to-bridged-ethernet connectivity.
> 
> Maybe you can set me straight on one other issue. I though I had seen a 
> presentation where the presenter mentioned easily developing new 
> protocol stacks for opensolaris. Am I imaging that ?
Depending on what you are trying to do, you should be able to get a lot 
of mileage from the IPoIB driver today as it has been significantly 
improved for 2009.06 and performance is on par with Linux OFED.

Greg
> 
> -gary
> 
> On Thu, Jun 18, 2009 at 6:25 PM, Carl Hensler <carl.hensler at sun.com 
> <mailto:carl.hensler at sun.com>> wrote:
> 
>     Gary,
> 
>     It depends on what RDMA transport protocol and link layer you are
>     talking about.  Currently there are two RDMA options available in
>     the market, InfiniBand (IB) and iWARP (RDMA over TCP/IP), and there
>     are more on the way.
> 
>     At the moment, IB transport operates only over IB physical links,
>     but IB transport over Ethernet (DCE/CEE/DCBE) is being standardized.
>      It will be implemented both in IB hardware engines (HCAs) and in
>     SW.  We expect that these will be implemented in Solaris in a way
>     that is consistent with Crossbow VNIC virtualization.
> 
>     iWARP is not currently supported in Solaris.  iWARP is typically
>     implemented in HW as an RDMA engine on top of a TCP Offload Engine
>     (TOE).  TOE is not supported in Solaris and it is hard to imagine
>     how it could be integrated with Crossbow VNICs.  It is possible to
>     implement iWARP in SW and that could be integrated with Crossbow
>     VNICs, but we have no plans to do so.
> 
>     So the answer is that there is currently no way to do RDMA through
>     Crossbow VNICs.
> 
>     Carl
> 
> 
>         -------- Original Message --------
>         Subject: [crossbow-discuss] RDMA & Virtual Nics Possible ?
>         Date: Thu, 18 Jun 2009 15:32:00 -0700 (PDT)
>         From: gary mazz <garym at oedata.com <mailto:garym at
oedata.com>>
>         To: crossbow-discuss at opensolaris.org
>         <mailto:crossbow-discuss at opensolaris.org>
> 
>         I''m cross-posting to the crossbow discussion group
> 
>         -g
> 
>         Can anyone tell me or point me in the right direction
>          to find out whether RDMA works though crossbow
>          virtual adapters ?
> 
>          cheers,
>          gary
> 
> 
>

That should have been

"...IPoIB driver .... does NOT have a VNIC abstraction..."

Greg Lavender wrote:
> gary mazzaferro wrote:
>> Hi Carl,
>>
>>  Thanks for the breakdown.
>> Hmmm, Using hw rma kind of defeats the purpose rma in the same box. 
>> The standardization efforts for IB over ethernet is still a ways off. 
>> I personally don''t see the sense in it since IB is 30% lower
cost than
>> 10ge and IB is already running 4x with 8x not far off.
> 
> Gary, its not quite clear from your original question or comment above 
> what it is that you are asking for. Perhaps you can clarify? I 
> originally thought you were asking about RDMA over ETH, but now you seem 
> to be asking for a VNIC over IB.
> 
>> Since IB is already supported in the kernel and the IB drivers are 
>> pretty stable, shouldn''t it be fairly straight forward to
implement
>> the sw copy with crossbow polling ? It could be done using raw IP or 
>> UDP until PF_PACKET is in production. The virtual switch can just pass 
>> pointers around at that point.
> 
> As of OpenSolaris 2009.06, the IPoIB driver (ibd) is a crossbow-enabled 
> driver, but does have a VNIC abstraction, but that could be done if you 
> were so inclined to do it. But we are taking a slightly different 
> approach and are working on a leaner EoIB driver that will allow for 
> VNICs/VSwitches to provide more efficient host-to-host (including 
> DomU-to-DomU) and host-to-bridged-ethernet connectivity.
> 
>>
>> Maybe you can set me straight on one other issue. I though I had seen 
>> a presentation where the presenter mentioned easily developing new 
>> protocol stacks for opensolaris. Am I imaging that ?
> 
> Depending on what you are trying to do, you should be able to get a lot 
> of mileage from the IPoIB driver today as it has been significantly 
> improved for 2009.06 and performance is on par with Linux OFED.
> 
> Greg
> 
>>
>> -gary
>>
>> On Thu, Jun 18, 2009 at 6:25 PM, Carl Hensler <carl.hensler at
sun.com
>> <mailto:carl.hensler at sun.com>> wrote:
>>
>>     Gary,
>>
>>     It depends on what RDMA transport protocol and link layer you are
>>     talking about.  Currently there are two RDMA options available in
>>     the market, InfiniBand (IB) and iWARP (RDMA over TCP/IP), and there
>>     are more on the way.
>>
>>     At the moment, IB transport operates only over IB physical links,
>>     but IB transport over Ethernet (DCE/CEE/DCBE) is being
standardized.
>>      It will be implemented both in IB hardware engines (HCAs) and in
>>     SW.  We expect that these will be implemented in Solaris in a way
>>     that is consistent with Crossbow VNIC virtualization.
>>
>>     iWARP is not currently supported in Solaris.  iWARP is typically
>>     implemented in HW as an RDMA engine on top of a TCP Offload Engine
>>     (TOE).  TOE is not supported in Solaris and it is hard to imagine
>>     how it could be integrated with Crossbow VNICs.  It is possible to
>>     implement iWARP in SW and that could be integrated with Crossbow
>>     VNICs, but we have no plans to do so.
>>
>>     So the answer is that there is currently no way to do RDMA through
>>     Crossbow VNICs.
>>
>>     Carl
>>
>>
>>         -------- Original Message --------
>>         Subject: [crossbow-discuss] RDMA & Virtual Nics Possible ?
>>         Date: Thu, 18 Jun 2009 15:32:00 -0700 (PDT)
>>         From: gary mazz <garym at oedata.com <mailto:garym at
oedata.com>>
>>         To: crossbow-discuss at opensolaris.org
>>         <mailto:crossbow-discuss at opensolaris.org>
>>
>>         I''m cross-posting to the crossbow discussion group
>>
>>         -g
>>
>>         Can anyone tell me or point me in the right direction
>>          to find out whether RDMA works though crossbow
>>          virtual adapters ?
>>
>>          cheers,
>>          gary
>>
>>
>>
> 
>