Gluster users - May 2020 - GlusterFS saturates server disk IO due to write brick temporary file to ".glusterfs" directory

Hi,

I have GridFTP + a network speedup solution in network + GlusterFS as a
file system component in a disk-to-disk data transferring scenario. For
glusterfs, I start with creating bricks inside /dev/sda1 filesystem. During
the file transfer (12 GB), it seems that glusterfs tries to write a
temporary file into the .glusterfs directory at a very high rate and
essentially the disk IO rate of the server host is ~100%. This leads the
file transfer performance unpredictable as file transfer(i.e., GridFTP)
would sometimes wait and send nothing due to the disk busy of glusterfs
server. As as test, I change the brick location to tmpfs (in memory) then
the disk I/O won't be saturated, this makes sense b/c the file writing
would happen in memory in this case, but this is not disk-to-disk transfer
anymore so this can't be an effective workaround in my case.

I wonder if there is any way I can turn off glusterfs to write temporary
files into the .gluster directory during the file transfer? Based on my
reading so far, I understand that this is a glusterfs design decision for
some recovering purpose, just in my case the writing rate is too fast and
saturates the disk IO and makes my disk-to-disk performance unstable.

My glusterfs version is: glusterfs 3.7.6 built on Dec 25 2015 20:50:46. I
am glad to post more volume or other setup details if necessary.

Thanks,
Qing
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200529/b892a17a/attachment.html>

Hi Qing,

I think that it is not  possible, yet  someone more experience can offer a 
solution.

You can play with gluster  settings to reduce I/O load  by reducing the thread 
count  tunables:

server.event-threads
client.event-threads
performance.io-thread-count

For details  check this  article (it's  for small files tuning,  but 
describes  the options above):
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements

Best  Regards,
Strahil Nikolov

?? 29 ??? 2020 ?. 22:25:29 GMT+03:00, Qing Wang <qw at g.clemson.edu>
??????:
>Hi,
>
>I have GridFTP + a network speedup solution in network + GlusterFS as a
>file system component in a disk-to-disk data transferring scenario. For
>glusterfs, I start with creating bricks inside /dev/sda1 filesystem.
>During
>the file transfer (12 GB), it seems that glusterfs tries to write a
>temporary file into the .glusterfs directory at a very high rate and
>essentially the disk IO rate of the server host is ~100%. This leads
>the
>file transfer performance unpredictable as file transfer(i.e., GridFTP)
>would sometimes wait and send nothing due to the disk busy of glusterfs
>server. As as test, I change the brick location to tmpfs (in memory)
>then
>the disk I/O won't be saturated, this makes sense b/c the file writing
>would happen in memory in this case, but this is not disk-to-disk
>transfer
>anymore so this can't be an effective workaround in my case.
>
>I wonder if there is any way I can turn off glusterfs to write
>temporary
>files into the .gluster directory during the file transfer? Based on my
>reading so far, I understand that this is a glusterfs design decision
>for
>some recovering purpose, just in my case the writing rate is too fast
>and
>saturates the disk IO and makes my disk-to-disk performance unstable.
>
>My glusterfs version is: glusterfs 3.7.6 built on Dec 25 2015 20:50:46.
>I
>am glad to post more volume or other setup details if necessary.
>
>Thanks,
>Qing