Gluster users - Aug 2013 - Writing performance gets slower when write threadsare many

-----Original Mail-----
From: Jung Young Seok [jung.youngseok at gmail.com]
Sent: 22.08.13 - 17:19:32
To: gluster-users at gluster.org

Subject: [Gluster-users] Writing performance gets slower when write threadsare
many
> Dear gluster users,
>
> We have set up? glusterfs 3.4 on cent 6. 2 Gluster FS servers are
> replicated each (2brick x 2).
> Glusterfs is mounted on WAS as fuse.
>
> We realized when there are more thread to write files on gluster fs, it
> gets slower on writing.
>
> We've set 250 thread to write 1M file each repeatedly. In this case we
get
> under 30 TPS. No NW bandwidth problem.
> Is there any way to avoid performance degradation when there are lots of
> write threads?
That depends on a lot of factors - what interconnect are you using? 1GE? 10GE?
RDMA? What are your bricks built on? SATA? SSD? RAID?

My first guess would be that interconnect latency and a kernel maximum for some
connection count could be the problem.
>
> Look forward any help.
> Thanks.
>
> Regards,
> Youngseok jung

2013/8/23 Maik Kulbe <info at linux-web-development.de>
> -----Original Mail-----
> From: Jung Young Seok [jung.youngseok at gmail.com]
> Sent: 22.08.13 - 17:19:32
> To: gluster-users at gluster.org
>
> Subject: [Gluster-users] Writing performance gets slower when write
> threadsare many
>
>
>  Dear gluster users,
>>
>> We have set up  glusterfs 3.4 on cent 6. 2 Gluster FS servers are
>> replicated each (2brick x 2).
>> Glusterfs is mounted on WAS as fuse.
>>
>> We realized when there are more thread to write files on gluster fs, it
>> gets slower on writing.
>>
>> We've set 250 thread to write 1M file each repeatedly. In this case
we get
>> under 30 TPS. No NW bandwidth problem.
>> Is there any way to avoid performance degradation when there are lots
of
>> write threads?
>>
>
> That depends on a lot of factors - what interconnect are you using? 1GE?
> 10GE? RDMA? What are your bricks built on? SATA? SSD? RAID?
>
> My first guess would be that interconnect latency and a kernel maximum for
> some connection count could be the problem.

Thank you for the comments,

We use 1GE for storage network. When I checked the net stream using iftop,
it never reached 1G. (only was under 500Mbps. Since we have 2 glusterfs
servers, 250Mb for first brick 250Mb for second brick)
When I scp a file to Gluster servers from WAS directly ,  it reached 80MB/s
each. (used less encrypted option)

About HDD,
First server is on SAS 10k RAID1.
Second server is on SATA 10k RAID1.
Those are for test. We'll use SATA RAID1 both finally to reduce cost.

We are using CentOS 6.4. Since we use vanilla installation, most of kernel
parameters are defaults.
If you give me any option that I can test, it would be great.

Thank you.

>
>> Look forward any help.
>> Thanks.
>>
>> Regards,
>> Youngseok jung
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130823/89840999/attachment.html>