Gluster users - Aug 2012 - Gluster-users Digest, Vol 51, Issue 46

4. Re: kernel parameters for improving gluster writes on millions of small
writes (long) (Harry Mangalam)

Harry, You are correct, Glusterfs throughput with small write transfer sizes is
a client-side problem, here are workarounds that at least some applications
could use.

1) NFS client is one workaround, since it buffers writes using the kernel buffer
cache.

2) If your app does not have a configurable I/O size, but it lets you write to
stdout, you can try piping your output to stdout and letting dd aggregate your
I/O to the filesystem for you.  In this example we triple single-thread write
throughput for 4-KB I/O requests in this example.

[root at perf56 ~]# mount -t glusterfs perf66-10ge:/repl2 /mnt/repl2

[root at perf56 ~]# dd if=/dev/zero of=/mnt/repl2/a.dd bs=4k count=1024k
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 55.8191 s, 76.9 MB/s

[root at perf56 ~]# dd if=/dev/zero bs=4k count=1024k | dd of=/mnt/repl2/a.dd
bs=1024k
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB) copied, 20.6882 s, 208 MB/s
0+58116 records in
0+58116 records out
4294967296 bytes (4.3 GB) copied, 19.9023 s, 216 MB/s

3) If your program is written in "C" and it uses stdio.h, you can
probably do setvbuf() "C" RTL call to increase buffer size to
something greater than 8 KB, which is the default in gcc-4.4.

http://en.cppreference.com/w/c/io/setvbuf

Hi Ben,

Thanks for the expert advice.

On Fri, Aug 3, 2012 at 2:35 PM, Ben England <bengland at redhat.com>
wrote:
> 4. Re: kernel parameters for improving gluster writes on millions of small
> writes (long) (Harry Mangalam)
>
> Harry, You are correct, Glusterfs throughput with small write transfer
> sizes is a client-side problem, here are workarounds that at least some
> applications could use.
>
Not to be impertinent nor snarky, but why is the gluster client written in
this way and is that a high priority for fixing?  It seems that
caching/buffering is one of the great central truths of computer science in
general.  Is there a countering argument for not doing this?

1) NFS client is one workaround, since it buffers writes using the
kernel
> buffer cache.
>
Yes, I tried this and I find the same thing.  One thing I am unclear about
tho is whether you can set up and run 1 NFS server per gluster server node.
 ie my glusterfs runs on 4 servers - could I connect clients to each one
using a round robin selection or other load/bandwidth balancing approach?
 I've read opinions that seem to support both yes and no.

>
> 2) If your app does not have a configurable I/O size, but it lets you
> write to stdout, you can try piping your output to stdout and letting dd
> aggregate your I/O to the filesystem for you.  In this example we triple
> single-thread write throughput for 4-KB I/O requests in this example.

I agree again - I wrote up this for the gluster 'hints'
<http://goo.gl/NyMXO>
using gzip (other utilities seem to work as well, as do named pipes for
handling more complex output options.
>
[nice examples deleted]
>
> 3) If your program is written in "C" and it uses stdio.h, you can
probably
> do setvbuf() "C" RTL call to increase buffer size to something
greater than
> 8 KB, which is the default in gcc-4.4.
>
Most of our users are not programmers and so this is not an option in most
cases.
>
> http://en.cppreference.com/w/c/io/setvbuf
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>


-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120803/ecf6bd29/attachment.html>