Gluster users - Jul 2008 - Gluster on tcp ip compared to infiniband

Hi,

 I have seen the benchmarks on the gluster website and i wondered if 
there is a big difference between infiniband and tcp/ip performances. 
Anyone can share some stats on this i do not know the infiniband 
technology at all.

regards,
Jean.

Yes, there is a significant performance difference between Infiniband
and GigE. Infiniband's latency is 1us and bandwidth is 20Gbps (40 Gbps
capable already) . GigE's latency is between 30 to 100us at 1Gbps.
Infiniband has become affordable. For example, Flextronics 8-port
Infiniband switch costs $2,824. Flextronics 24-port Infiniband DDR
switch costs $7,609.  Cisco, Voltaire IB switches are slightly more
expensive, but still way too cheaper than 10GigE.

Here is a good summary:
http://www.mellanox.com/applications/IB_performance.php

When you use Infiniband, GlusterFS uses RDMA (remote direct memory
access) for communication. Meaning, TCP/IP and local/remote
CPUs are bypassed.

10GigE is still not mature and very expensive.

Hope this helps.
--
Anand Babu Periasamy
GPG Key ID: 0x62E15A31
Blog [http://ab.freeshell.org]
The GNU Operating System [http://www.gnu.org]
Z RESEARCH Inc [http://www.zresearch.com]



Jean Spirat wrote:
> Hi,
> 
>  I have seen the benchmarks on the gluster website and i wondered if 
> there is a big difference between infiniband and tcp/ip performances. 
> Anyone can share some stats on this i do not know the infiniband 
> technology at all.
> 
> regards,
> Jean.
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>

Jean Spirat wrote:
> Hi,
>
>  I have seen the benchmarks on the gluster website and i wondered if 
> there is a big difference between infiniband and tcp/ip performances. 
> Anyone can share some stats on this i do not know the infiniband 
> technology at all.
>   
Hi,

Some preliminary numbers I have for a 4x SDR IB network, single
disk (RAID 5, 4+1, SAS, 300G 15K disks), single user:

transport        read    write   notes
---------        ----    -----   -----
local            ~254     ~255   for a 100G file, not compensating for 
caching
                                 take them for what their worth :)
nfs              158     76      not gluster, of course!
tcp (ipoib)      171     118     no optimizations
sdp              211     108     no optimizations
sdp              210     128     iot 4, 32MB
verbs            218     150     no optimizations
verbs            220     150     recv/send size 1048675, recv/send-count 16


John