Gluster users - Jan 2013 - Question regarding write performance issues

Hi to all Gluster experts,

I have a cluster of 10 machines exposing a volume into which 12 other
machines do many writes of large files (~100-300MB each).
In general I'm very happy with gluster. It's a great solution, and is
quite
stable (thanks for the great work!).

However, I have a problem which I was unable to solve yet, nor find any
solution to in the documentation or on this list archive.

When the client machines write locally, and then just copy the files they
created to the gluster mount - everything works great.
When the client machines write directly to the gluster mounted volume I get
a huge performance hit.
In one specific test case the difference was 20 minutes for the copy and 8
hours for the direct write.

I tried to set the iocache attributes of write-behind-window and
flush-behind, but to no avail.

I will very much appreciate your help in solving this problem.

Thanks,
Ayelet
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130116/a6e3c6d8/attachment.html>

Just a guess, but how are the writes being done?  If they're being written
in
zillions of tiny writes, then what you may be seeing is described here:
<http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl>
and the following stanza on named pipes.

This is often the case with the large files being used in NGS/HTS where the 
fasta/fastq files are composed of millions of short (60-100 char) lines of 
characters and are typically written line-by-line.

hjm



On Wednesday, January 16, 2013 02:47:37 PM Ayelet Shemesh
wrote:
> Hi to all Gluster experts,
> 
> I have a cluster of 10 machines exposing a volume into which 12 other
> machines do many writes of large files (~100-300MB each).
> In general I'm very happy with gluster. It's a great solution, and
is quite
> stable (thanks for the great work!).
> 
> However, I have a problem which I was unable to solve yet, nor find any
> solution to in the documentation or on this list archive.
> 
> When the client machines write locally, and then just copy the files they
> created to the gluster mount - everything works great.
> When the client machines write directly to the gluster mounted volume I get
> a huge performance hit.
> In one specific test case the difference was 20 minutes for the copy and 8
> hours for the direct write.
> 
> I tried to set the iocache attributes of write-behind-window and
> flush-behind, but to no avail.
> 
> I will very much appreciate your help in solving this problem.
> 
> Thanks,
> Ayelet
---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
"Something must be done. [X] is something. Therefore, we must do it."
Bruce Schneier, on American response to just about anything.