Julio Guevara
2017-Jun-13 20:43 UTC
[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume
I'm having a hard time trying to get a gluster volume up and running. I have setup other gluster volumes on other systems without much problems but this one is killing me. The gluster vol was created with the command: gluster volume create mariadb_gluster_volume laeft-dccdb01p:/export/mariadb/brick I had to lower frame-timeout since the system would become unresponsive until the frame failed by timeout: gluster volume set mariadb_gluster_volume networking.frame-timeout 5 running gluster version: glusterfs 3.8.12 The workload i'm using is: sysbench --test=fileio --file-total-size=4G --file-num=64 prepare sysbench version: sysbench 0.4.12-5.el6 kernel version: 2.6.32-696.1.1.el6 centos: 6.8 Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I get the error that is shown on the log output. It is a constant issue, I can reproduce it when I start increasing the --file-num for sysbench above 3. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170613/6f52705b/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: all.logs Type: application/octet-stream Size: 4795 bytes Desc: not available URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170613/6f52705b/attachment.obj>
Julio Guevara
2017-Jun-14 15:27 UTC
[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume
Also, this is the profile output of this Volume:
gluster> volume profile mariadb_gluster_volume info cumulative
Brick: laeft-dccdb01p.core.epay.us.loc:/export/mariadb_backup/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 0 0
0
No. of Writes: 83391 465750
102911
Block Size: 131072b+
No. of Reads: 33
No. of Writes: 8551
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 26
FORGET
0.00 0.00 us 0.00 us 0.00 us 33
RELEASE
0.00 0.00 us 0.00 us 0.00 us 92
RELEASEDIR
0.00 102.25 us 70.00 us 155.00 us 4
SETXATTR
0.00 159.80 us 97.00 us 207.00 us 5
RMDIR
0.00 266.75 us 121.00 us 675.00 us 4
SETATTR
0.00 100.62 us 29.00 us 700.00 us 16
INODELK
0.00 84.33 us 29.00 us 477.00 us 33
FLUSH
0.00 68.16 us 34.00 us 165.00 us 92
OPENDIR
0.01 88.35 us 21.00 us 608.00 us 92
STAT
0.01 754.04 us 262.00 us 10104.00 us 25
CREATE
0.02 169.04 us 27.00 us 997.00 us 179
READDIRP
0.04 150.90 us 43.00 us 1867.00 us 365
LOOKUP
0.04 16330.75 us 297.00 us 46360.00 us 4
MKDIR
0.18 7896.70 us 71.00 us 256814.00 us 33
READ
0.68 466.93 us 19.00 us 1848.00 us 2119
STATFS
2.39 151339.17 us 227.00 us 540998.00 us 23
UNLINK
5.49 320155.28 us 2035.00 us 1273394.00 us 25
FSYNC
31.28 69.00 us 30.00 us 11447.00 us 660603
WRITE
59.84 10899586.88 us 5827342.00 us 13921169.00 us 8
OPEN
Duration: 71425 seconds
Data Read: 4325376 bytes
Data Written: 29195534336 bytes
As you can see OPEN Fop take the most amount of time and they normally
timeout, even with default value for networking.frame-timeout
Thanks
On Tue, Jun 13, 2017 at 3:43 PM, Julio Guevara <julioguevara150 at
gmail.com>
wrote:
> I'm having a hard time trying to get a gluster volume up and running. I
> have setup other gluster volumes on other systems without much problems but
> this one is killing me.
>
> The gluster vol was created with the command:
> gluster volume create mariadb_gluster_volume laeft-dccdb01p:/export/
> mariadb/brick
>
> I had to lower frame-timeout since the system would become unresponsive
> until the frame failed by timeout:
> gluster volume set mariadb_gluster_volume networking.frame-timeout 5
>
> running gluster version: glusterfs 3.8.12
>
> The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> --file-num=64 prepare
>
> sysbench version: sysbench 0.4.12-5.el6
>
> kernel version: 2.6.32-696.1.1.el6
>
> centos: 6.8
>
> Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I
> get the error that is shown on the log output.
>
> It is a constant issue, I can reproduce it when I start increasing the
> --file-num for sysbench above 3.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170614/c0b377d0/attachment.html>
Ben Turner
2017-Jun-15 04:11 UTC
[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume
----- Original Message -----> From: "Julio Guevara" <julioguevara150 at gmail.com> > To: gluster-users at gluster.org > Sent: Tuesday, June 13, 2017 4:43:06 PM > Subject: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume > > I'm having a hard time trying to get a gluster volume up and running. I have > setup other gluster volumes on other systems without much problems but this > one is killing me. > > The gluster vol was created with the command: > gluster volume create mariadb_gluster_volume > laeft-dccdb01p:/export/mariadb/brick > > I had to lower frame-timeout since the system would become unresponsive until > the frame failed by timeout: > gluster volume set mariadb_gluster_volume networking.frame-timeout 5 > > running gluster version: glusterfs 3.8.12 > > The workload i'm using is: sysbench --test=fileio --file-total-size=4G > --file-num=64 prepare > > sysbench version: sysbench 0.4.12-5.el6 > > kernel version: 2.6.32-696.1.1.el6 > > centos: 6.8 > > Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups I > get the error that is shown on the log output. > > It is a constant issue, I can reproduce it when I start increasing the > --file-num for sysbench above 3.It looks like you may be seeing a crash. If you look at /var/log/messages on all of the clients / servers do you see any crashes / seg faults / ABRT messages in the log? If so can you open a BZ with the core / other info here? Here is an example of a crash on one of the bricks: http://lists.gluster.org/pipermail/gluster-users/2016-February/025460.html My guess is something is happening client sidesince we don't see anything in the server logs, check the client mount log(/var/log/glusterfs/<mount-point>.log and the messages file on your client. Also check messages on the servers. If you see anything shoot us out the info and lets get a BZ open, if not maybe someone else on the list has some other ideas. -b> > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users