thr3ads.net - Gluster users - [Gluster-users] infiniband speed [Jan 2011]

If this information is useful, please help other people find it:
Share via:

Christopher Hawkins

2011-Jan-11 00:43 UTC

[Gluster-users] infiniband speed

I am testing Infiniband for the first time. It seems that I should be able to
get a lot more speed than I am with some pretty basic tests. Maybe someone
running Infiniband can confirm that what I am seeing is way out of line, and/or
help diagnose?

I have two systems connected using 3.1.2qa3. With 3.1.1 infiniband wouldn't
even start, it gave an error about unable to intialize rdma. But with the latest
version and an upgrade to OFED 1.5.2, everything starts up with no errors and I
can create a volume and mount it.

The underlying Infiniband seems ok, and a basic ibv_rc_pingpong test shows I can
move data pretty fast:
81920000 bytes in 0.23 seconds = 2858.45 Mbit/sec
10000 iters in 0.23 seconds = 22.93 usec/iter

So now I have two volumes created, one that uses tcp over a gig-e link and one
that uses rdma. I mount them and do some file copy tests... And they are almost
exactly the same? What?

gluster volume info

Volume Name: test2_volume
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: bravo:/cluster/shadow/test2
Brick2: backup:/cluster/shadow/test2

Volume Name: test_volume
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: rdma
Bricks:
Brick1: bravo:/cluster/shadow/test
Brick2: backup:/cluster/shadow/test

mount:
glusterfs#localhost:/test_volume on /mnt/test type fuse
(rw,allow_other,default_permissions,max_read=131072)
glusterfs#localhost:/test2_volume on /mnt/test2 type fuse
(rw,allow_other,default_permissions,max_read=131072)


time cp files.tar /mnt/test2/

real    0m11.159s
user    0m0.123s
sys     0m1.244s

files.tar is single file, 390MB, so this about 35MB/s. Fine for gig-e. 
----------------------------

time cp files.tar /mnt/test/

real    0m5.656s
user    0m0.116s
sys     0m0.962s

69MB/s... ehhh. Faster at least. On a few runs, this was not any faster at all.
Maybe a cache effect?
----------------------------

time cp -av /usr/src/kernels /mnt/test2/
real    0m49.605s
user    0m0.681s
sys     0m2.593s

kernels dir is 34MB of small files. The low latency of IB should really show an
improvement here I thought.
-----------------------------

time cp -av /usr/src/kernels /mnt/test/

real    0m56.046s
user    0m0.625s
sys     0m2.675s

It took LONGER? That can't be right. 
------------------------------

And finally, this error is appearing in the rdma mount log every 3 seconds on
both nodes:

[2011-01-10 19:46:56.728127] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:46:59.738291] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:02.748260] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:05.758256] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:08.768299] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:11.778308] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:14.788356] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:17.798381] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)
[2011-01-10 19:47:20.808413] E [rdma.c:4428:tcp_connect_finish]
test_volume-client-1: tcp connect to  failed (Connection refused)

But there are no restrictions in the config. Everything is allow *. So my
questions are, can anyone else tell me what kind of basic file copy performance
they see using IB? And what can I do to troubleshoot?

Thanks List and Devs, 

Chris

Joe Landman

2011-Jan-11 03:26 UTC

head link

[Gluster-users] infiniband speed

On 01/10/2011 07:43 PM, Christopher Hawkins wrote:> I am testing Infiniband for the first time. It seems that I should be
> able to get a lot more speed than I am with some pretty basic tests.
> Maybe someone running Infiniband can confirm that what I am seeing is
> way out of line, and/or help diagnose?
What is the underlying disk speed?

Try this:

# (on bravo)
dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k

This will write a 20GB file to the same partition. We need to see how
fast that write is (outside of cache)

Do the same test on the other machine.

Infiniband won't make slow disks fast. So unless you are already
getting a gigabyte or two per second (we do on our units), adding
Infiniband won't get you any significant additional performance.

Put another way, you are bound by the slowest aspect of your storage system.

What are your underlying disks? Are these individual drives? RAID?
What sort of RAID if it is a RAID? How is it organized?

We tune our systems pretty hard, so we start with 2+GB/s for TB sized
files before we ever touch the next stack up. Each additional stack you
traverse takes performance away (you lose it in stack inefficiency). So
you should start out with as fast a system as you possibly can, so you
don't lose too much performance.

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615

Christopher Hawkins

2011-Jan-11 06:01 UTC

head link

[Gluster-users] infiniband speed

Thanks Joe, you nailed it. These are just test machines and in each case, just a
single 10k scsi drive. That is the throughput bottleneck... I was not able to
get more than 70MB/s sustained.

Chris

----- "Joe Landman" <landman at scalableinformatics.com> wrote:
> On 01/10/2011 07:43 PM, Christopher Hawkins wrote:
> > I am testing Infiniband for the first time. It seems that I should
> be
> > able to get a lot more speed than I am with some pretty basic
> tests.
> > Maybe someone running Infiniband can confirm that what I am seeing
> is
> > way out of line, and/or help diagnose?
> 
> What is the underlying disk speed?
> 
> Try this:
> 
> 	# (on bravo)
> 	dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k
> 
> This will write a 20GB file to the same partition.  We need to see how
> 
> fast that write is (outside of cache)
> 
> Do the same test on the other machine.
> 
> Infiniband won't make slow disks fast.  So unless you are already 
> getting a gigabyte or two per second (we do on our units), adding 
> Infiniband won't get you any significant additional performance.
> 
> Put another way, you are bound by the slowest aspect of your storage
> system.
> 
> What are your underlying disks?  Are these individual drives?  RAID? 
> What sort of RAID if it is a RAID?  How is it organized?
> 
> We tune our systems pretty hard, so we start with 2+GB/s for TB sized
> 
> files before we ever touch the next stack up.  Each additional stack
> you 
> traverse takes performance away (you lose it in stack inefficiency). 
> So 
> you should start out with as fast a system as you possibly can, so you
> 
> don't lose too much performance.
> 
> -- 
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics, Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>         http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Gluster users - Jan 2011 - infiniband speed

[Gluster-users] infiniband speed

[Gluster-users] infiniband speed

[Gluster-users] infiniband speed