similar to: glusterfs crash when the one of replicate node restart

I am trying to run GlusterFS on only my internal interfaces. I have setup two bricks and have a replicated volume that is started. Everything works fine when I run with no transport.socket.bind-address defined in the /etc/glusterfs/glusterd.vol file, but when I add it I get: Transport endpoint is not connected My configuration looks like this: volume management type mgmt/glusterd

Hi Atin/Team, We are using gluster-3.7.6 with setup of two brick and while restart of system I have seen that the glusterd daemon is getting failed from start. At the time of analyzing the logs from etc-glusterfs.......log file I have received the below logs [2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version

I'm hoping that someone here can point me the right direction to help me solve a problem I am having. I've got 3 gluster peers and for some reason glusterd sill not start on one of them. All are running glusterfs version 3.4.0-8.el6 on Centos 6.4 (2.6.32-358.el6.x86_64). In /var/log/glusterfs/etc-glusterfs-glusterd.vol.log I see this error repeated 36 times (alternating between brick-0

Hello all. I have a working replica 2 cluster (4 nodes) up and running happily over Amazon EC2. My end goal is to create AMIs of each machine and then quickly reproduce the same, but new, cluster from those AMIs. Essentially, I'd like a cluster "template". -Assigned original instances' Elastic IPs to new machines to reduce resolution issues. -Passwordless SSH works on initial

Hi Niels, No problem we wil try to backport that patch on 3.7.6. Could you please let me know in which release Gluster community is going to provide this patch and date of that release? Regards, Abhishek On Wed, May 31, 2017 at 10:05 PM, Niels de Vos <ndevos at redhat.com> wrote: > On Wed, May 31, 2017 at 04:08:06PM +0530, ABHISHEK PALIWAL wrote: > > We are using 3.7.6 and on

I had looked at the logs shared by Victor privately and it seems to be there is a N/W glitch in the cluster which is causing the glusterd to lose its connection with other peers and as a side effect to this, lot of rpc requests are getting bailed out resulting glusterd to end up into a stale lock and hence you see that some of the commands failed with "another transaction is in progress or

I've run into a problem with Gluster stability with the RDMA transport. Below is a description of the environment, a simple script that can replicate the problem, and log files from my test system. I can work around the problem by using the TCP transport over IPoIB but would like some input onto what may be making the RDMA transport fail in this case. ===== Symptoms ===== - Error from test

Thanks for the reply. What would be the best course of action? The data on the volume isn?t important right now but I?m worried when our setup goes to production we don?t have the same situation and really need to recover our Gluster setup. I?m assuming that to redo is to delete everything in the /var/lib/glusterd directory on each of the nodes and recreate the volume again. Essentially

On Thu, 29 Jun 2017 at 22:51, Victor Nomura <victor at mezine.com> wrote: > Thanks for the reply. What would be the best course of action? The data > on the volume isn?t important right now but I?m worried when our setup goes > to production we don?t have the same situation and really need to recover > our Gluster setup. > > > > I?m assuming that to redo is to

On Thu, Jun 01, 2017 at 01:03:25PM +0530, ABHISHEK PALIWAL wrote: > Hi Niels, > > No problem we wil try to backport that patch on 3.7.6. > > Could you please let me know in which release Gluster community is going to > provide this patch and date of that release? It really depends on when someone has time to work on it. Our releases are time based, and will happen even when a

Could you attach glusterd.log and cmd_history.log files from all the nodes? On Wed, Jun 21, 2017 at 11:40 PM, Victor Nomura <victor at mezine.com> wrote: > Hi All, > > > > I?m fairly new to Gluster (3.10.3) and got it going for a couple of months > now but suddenly after a power failure in our building it all came crashing > down. No client is able to connect after

Hi All, I'm fairly new to Gluster (3.10.3) and got it going for a couple of months now but suddenly after a power failure in our building it all came crashing down. No client is able to connect after powering back the 3 nodes I have setup. Looking at the logs, it looks like there's some sort of "Lock" placed on the volume which prevents all the clients from connecting to

Specifically, I must stop glusterfs-server service on the other nodes in order to perform any gluster commands on any node. From: Victor Nomura [mailto:victor at mezine.com] Sent: July-04-17 9:41 AM To: 'Atin Mukherjee' Cc: 'gluster-users' Subject: RE: [Gluster-users] Gluster failure due to "0-management: Lock not released for <volumename>" The nodes have

By any chance are you having any redundant peer entries in /var/lib/glusterd/peers directory? Can you please share the content of this folder from all the nodes? On Tue, Jul 4, 2017 at 11:55 PM, Victor Nomura <victor at mezine.com> wrote: > Specifically, I must stop glusterfs-server service on the other nodes in > order to perform any gluster commands on any node. > > > >

Hi guys, we have a rebalance running on eight bricks since July and this is what the status looks like right now: ===Tue Oct 18 13:45:01 CST 2011 ==== rebalance step 1: layout fix in progress: fixed layout 223623 There are roughly 8T photos in the storage,so how long should this rebalance take? What does the number (in this case) 22362 represent? Our gluster infomation: Repository

I just built a pair of AWS Red Hat 6.5 instances to create a gluster replicated pair file system. I can install everything, peer probe, and create the volume, but as soon as I try to start the volume, glusterd dumps core. The tail of the log after the crash: +------------------------------------------------------------------------------+ [2014-04-28 21:49:18.102981] I

Hi Phil, we?d the same Problem, try to compile with debug options. Yes this sounds strange but it help?s when u are using SLES, the glusterd works ok and u can start to work with it. just put exportCFLAGS='-g3 -O0' between %build and %configure in the glusterfs spec file. But be warned don?t use it with important data especially when u are planing to use the replication feature,

Hi all, I have an issue where our volume will not start from any node. When attempting to start the volume it will eventually return: Error: Request timed out For some time after that, the volume is locked and we either have to wait or restart Gluster services. In the gluserd.log, it shows the following: [2017-12-15 18:00:12.423478] I [glusterd-utils.c:5926:glusterd_brick_start]

Hello, We have a cluster with two nodes, "sg" and "br", which were running GlusterFS 9.1, installed via the Ubuntu package manager. We updated the Ubuntu packages on "sg" to version 9.6, and now have big problems. The "br" node is still on version 9.1. Running "gluster volume status" on either host gives "Error : Request timed out". On

On Sat, Dec 16, 2017 at 12:45 AM, Matt Waymack <mwaymack at nsgdv.com> wrote: > Hi all, > > > > I have an issue where our volume will not start from any node. When > attempting to start the volume it will eventually return: > > Error: Request timed out > > > > For some time after that, the volume is locked and we either have to wait > or restart