Jagga Soorma
2010-Feb-11 00:48 UTC
[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused
Hi Guys,
I have setup a new cluster with a infiniband interconnect and I am a bit
confused with the performance I am getting as far as transfer rates. Is
there something that I am missing here? Shouldn''t the transfer rate
over
the ib interface be much faster and not the same compared with the 1G
(bonded mode=6) interface?
(I am new to infiniband but this does not seem right. My routing is also
setup correctly. Not sure what I might be missing here):
--
hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103
:/var/tmp/
SLES-11-DVD-x86_64-GM-DVD1.iso
100% 2749MB 62.5MB/s 00:44
real 0m43.882s
user 0m29.482s
sys 0m6.736s
hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103-ib
:/var/tmp/
SLES-11-DVD-x86_64-GM-DVD1.iso
100% 2749MB 80.9MB/s 00:34
real 0m35.757s
user 0m24.498s
sys 0m6.292s
hpc101:/var/tmp # netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt
Iface
10.0.250.0 0.0.0.0 255.255.255.0 U 0 0 0 ib0
128.137.126.0 0.0.0.0 255.255.255.0 U 0 0 0
bond0
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 128.137.126.253 0.0.0.0 UG 0 0 0
bond0
hpc101:/var/tmp # ifconfig ib0
ib0 Link encap:UNSPEC HWaddr
80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:10.0.250.5 Bcast:10.0.250.255 Mask:255.255.255.0
inet6 addr: fe80::223:7dff:ff93:bf4d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:493685 errors:0 dropped:0 overruns:0 frame:0
TX packets:890402 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:22848079 (21.7 Mb) TX bytes:7349365631 (7008.9 Mb)
--
Thanks in advance for your assistance.
Regards,
-J
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100210/af8a2d75/attachment.html
Erik Froese
2010-Feb-11 05:00 UTC
[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused
Jagga, I think this is more a function of scp than than ib in general. Have you tried using the HPN-SSH patches? http://www.psc.edu/networking/projects/hpn-ssh/ You could also try using apache to serve the iso over HTTP to see if SCP is the bottleneck. Also I think IPoIB should be able to use a much larger MTU. It may help for large files. Do this on both machines: ifconfig ib0 mtu?65520 To benchmark the raw infiniband performance see the man pages for ib_read_bw and ib_write_bw Erik On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <jagga13 at gmail.com> wrote:> > Hi Guys, > > I have setup a new cluster with a infiniband interconnect and I am a bit confused with the performance I am getting as far as transfer rates.? Is there something that I am missing here?? Shouldn''t the transfer rate over the ib interface be much faster and not the same compared with the 1G (bonded mode=6) interface? > > (I am new to infiniband but this does not seem right.? My routing is also setup correctly.? Not sure what I might be missing here): > > -- > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103:/var/tmp/ > SLES-11-DVD-x86_64-GM-DVD1.iso???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 100% 2749MB? 62.5MB/s?? 00:44 > > real??? 0m43.882s > user??? 0m29.482s > sys??? 0m6.736s > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso root at hpc103-ib:/var/tmp/ > SLES-11-DVD-x86_64-GM-DVD1.iso???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? 100% 2749MB? 80.9MB/s?? 00:34 > > real??? 0m35.757s > user??? 0m24.498s > sys??? 0m6.292s > > hpc101:/var/tmp # netstat -rn > Kernel IP routing table > Destination???? Gateway???????? Genmask???????? Flags?? MSS Window? irtt Iface > 10.0.250.0????? 0.0.0.0???????? 255.255.255.0?? U???????? 0 0????????? 0 ib0 > 128.137.126.0?? 0.0.0.0???????? 255.255.255.0?? U???????? 0 0????????? 0 bond0 > 127.0.0.0?????? 0.0.0.0???????? 255.0.0.0?????? U???????? 0 0????????? 0 lo > 0.0.0.0???????? 128.137.126.253 0.0.0.0???????? UG??????? 0 0????????? 0 bond0 > > hpc101:/var/tmp # ifconfig ib0 > ib0?????? Link encap:UNSPEC? HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00 > ????????? inet addr:10.0.250.5? Bcast:10.0.250.255? Mask:255.255.255.0 > ????????? inet6 addr: fe80::223:7dff:ff93:bf4d/64 Scope:Link > ????????? UP BROADCAST RUNNING MULTICAST? MTU:2044? Metric:1 > ????????? RX packets:493685 errors:0 dropped:0 overruns:0 frame:0 > ????????? TX packets:890402 errors:0 dropped:0 overruns:0 carrier:0 > ????????? collisions:0 txqueuelen:256 > ????????? RX bytes:22848079 (21.7 Mb)? TX bytes:7349365631 (7008.9 Mb) > -- > > Thanks in advance for your assistance. > > Regards, > -J > > _______________________________________________ > Lustre-discuss mailing list > Lustre-discuss at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-discuss >
Peter Kjellstrom
2010-Feb-11 10:09 UTC
[Lustre-discuss] Infiniband VS 1GiG Transfer rates. Confused
On Thursday 11 February 2010, Erik Froese wrote:> Jagga, > I think this is more a function of scp than than ib in general. Have > you tried using the HPN-SSH patches? > http://www.psc.edu/networking/projects/hpn-ssh/ > > You could also try using apache to serve the iso over HTTP to see if > SCP is the bottleneck.Iperf and netpipe are examples of tools made for this purpose. All this is, however, IPoIB which is _not_ what you''d typically use to run IB at high speed. Running Lustre on IB is typically done on verbs (lnet o2ib, not tcp). /Peter> Also I think IPoIB should be able to use a much larger MTU. It may > help for large files. > > Do this on both machines: ifconfig ib0 mtu?65520 > To benchmark the raw infiniband performance see the man pages for > ib_read_bw and ib_write_bw > > Erik > > On Wed, Feb 10, 2010 at 7:48 PM, Jagga Soorma <jagga13 at gmail.com> wrote: > > Hi Guys, > > > > I have setup a new cluster with a infiniband interconnect and I am a bit > > confused with the performance I am getting as far as transfer rates.? Is > > there something that I am missing here?? Shouldn''t the transfer rate over > > the ib interface be much faster and not the same compared with the 1G > > (bonded mode=6) interface? > > > > (I am new to infiniband but this does not seem right.? My routing is also > > setup correctly.? Not sure what I might be missing here): > > > > -- > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > > root at hpc103:/var/tmp/ > > SLES-11-DVD-x86_64-GM-DVD1.iso??????????????????????????????????????????? > >?????????????????????????????????????????????????????????????????????????? > >??????????????????????????????????????????????????????? 100% 2749MB? > > 62.5MB/s?? 00:44 > > > > real??? 0m43.882s > > user??? 0m29.482s > > sys??? 0m6.736s > > > > hpc101:/var/tmp # time scp SLES-11-DVD-x86_64-GM-DVD1.iso > > root at hpc103-ib:/var/tmp/ > > SLES-11-DVD-x86_64-GM-DVD1.iso??????????????????????????????????????????? > >?????????????????????????????????????????????????????????????????????????? > >??????????????????????????????????????????????????????? 100% 2749MB? > > 80.9MB/s?? 00:34 > > > > real??? 0m35.757s > > user??? 0m24.498s > > sys??? 0m6.292s... -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part. Url : http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100211/8f4eae4c/attachment.bin