Geoffrey Letessier
2015-Aug-04 23:12 UTC
[Gluster-users] Cascading errors and very bad write performance
Hello, Since the problem motioned previously (all errors noticed in brick log files), i notice a very very bad performance: i can note my write performance divided by 4 than previously -knowing it was not so good before. Now, a write of a 33GB file, my write throughput is around 150MBs (with Infiniband), before it was around 550-600MBs; and this, both with RDMA and TCP protocol. During this test, more than 40 000 error lines (as the following) were added to the brick log files. [2015-08-04 22:34:27.337622] E [dict.c:1418:dict_copy_with_ref] (-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60) [0x7f021c6f7410] -->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88) [0x7f021c6f7188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4) [0x7f0229cba674] ) 0-dict: invalid argument: dict [Argument invalide] All brick log files are in attachments. Thanks in advance for all your help and fix, Best, Geoffrey PS: question: is it possible to easily downgrade GlusterFS to a previous version from 3.7 (for example: v3.5)? ------------------------------------------------------ Geoffrey Letessier Responsable informatique & ing?nieur syst?me UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique Institut de Biologie Physico-Chimique 13, rue Pierre et Marie Curie - 75005 Paris Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150805/84d119ff/attachment-0002.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: bricks-logs.tgz Type: application/octet-stream Size: 2985130 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150805/84d119ff/attachment-0001.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150805/84d119ff/attachment-0003.html>
Geoffrey Letessier
2015-Aug-05 08:40 UTC
[Gluster-users] Cascading errors and very bad write performance
Hello,
In addition, knowing I have reactivated the log (brick-log-level = INFO not
CRITICAL) only for the file creation duration (i.e. a few minutes), do you have
noticed the log sizes and the number of lines inside:
# ls -lh storage*
-rw------- 1 letessier staff 18M 5 ao? 00:54
storage1__export-brick_home-brick1-data.log
-rw------- 1 letessier staff 2,1K 5 ao? 00:54
storage1__export-brick_home-brick2-data.log
-rw------- 1 letessier staff 15M 5 ao? 00:56
storage2__export-brick_home-brick1-data.log
-rw------- 1 letessier staff 2,1K 5 ao? 00:54
storage2__export-brick_home-brick2-data.log
-rw------- 1 letessier staff 47M 5 ao? 00:55
storage3__export-brick_home-brick1-data.log
-rw------- 1 letessier staff 2,1K 5 ao? 00:54
storage3__export-brick_home-brick2-data.log
-rw------- 1 letessier staff 47M 5 ao? 00:55
storage4__export-brick_home-brick1-data.log
-rw------- 1 letessier staff 2,1K 5 ao? 00:55
storage4__export-brick_home-brick2-data.log
# wc -l storage*
55381 storage1__export-brick_home-brick1-data.log
17 storage1__export-brick_home-brick2-data.log
41636 storage2__export-brick_home-brick1-data.log
17 storage2__export-brick_home-brick2-data.log
270360 storage3__export-brick_home-brick1-data.log
17 storage3__export-brick_home-brick2-data.log
270358 storage4__export-brick_home-brick1-data.log
17 storage4__export-brick_home-brick2-data.log
637803 total
If the let brick-log-level to INFO, the brick log files in each server will
consume all my /var partition capacity within only a few hours/days?
Thanks in advance,
Geoffrey
------------------------------------------------------
Geoffrey Letessier
Responsable informatique & ing?nieur syst?me
UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
Institut de Biologie Physico-Chimique
13, rue Pierre et Marie Curie - 75005 Paris
Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
Le 5 ao?t 2015 ? 01:12, Geoffrey Letessier <geoffrey.letessier at cnrs.fr>
a ?crit :
> Hello,
>
> Since the problem motioned previously (all errors noticed in brick log
files), i notice a very very bad performance: i can note my write performance
divided by 4 than previously -knowing it was not so good before.
> Now, a write of a 33GB file, my write throughput is around 150MBs (with
Infiniband), before it was around 550-600MBs; and this, both with RDMA and TCP
protocol.
>
> During this test, more than 40 000 error lines (as the following) were
added to the brick log files.
> [2015-08-04 22:34:27.337622] E [dict.c:1418:dict_copy_with_ref]
(-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(server_resolve_inode+0x60)
[0x7f021c6f7410]
-->/usr/lib64/glusterfs/3.7.3/xlator/protocol/server.so(resolve_gfid+0x88)
[0x7f021c6f7188] -->/usr/lib64/libglusterfs.so.0(dict_copy_with_ref+0xa4)
[0x7f0229cba674] ) 0-dict: invalid argument: dict [Argument invalide]
>
>
> All brick log files are in attachments.
>
> Thanks in advance for all your help and fix,
> Best,
> Geoffrey
>
> PS: question: is it possible to easily downgrade GlusterFS to a previous
version from 3.7 (for example: v3.5)?
>
> ------------------------------------------------------
> Geoffrey Letessier
> Responsable informatique & ing?nieur syst?me
> UPR 9080 - CNRS - Laboratoire de Biochimie Th?orique
> Institut de Biologie Physico-Chimique
> 13, rue Pierre et Marie Curie - 75005 Paris
> Tel: 01 58 41 50 93 - eMail: geoffrey.letessier at ibpc.fr
> <bricks-logs.tgz>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150805/4cb49373/attachment.html>