Gluster users - Jun 2014 - socket.c:2161:socket_connect_finish (Connection refused)

Hi,
I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was 
running fine until this morning. In a fuse mount we were having write 
issues. Creating and deleting files became an issue all of a sudden 
without any new changes to the cluster.

In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting 
this:

[2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 
0-sr_vol01-client-13: changing port to 49156 (from 0)
[2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 
0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 
failed (Connection refused)

# gluster volume status sr_vol01
shows that two bricks of the 18 are offline.

rebalance fails

Iptables was stopped on all nodes

If I cd into the two bricks which are offline according to the gluster v 
status, I can read/write without any problems... The disks are clearly 
fine. They are mounted, they are available.

I cannot find much info online about the error.
Does anyone have an idea what could be wrong?
How can I get the two bricks back online?

Cheers,
Olav

Olav,
      Check logs of the bricks to see why the bricks went down.

Pranith

On 06/11/2014 04:02 AM, Olav Peeters wrote:
> Hi,
> I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was 
> running fine until this morning. In a fuse mount we were having write 
> issues. Creating and deleting files became an issue all of a sudden 
> without any new changes to the cluster.
>
> In /var/log/glusterfs/glustershd.log every couple of seconds I'm 
> getting this:
>
> [2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 
> 0-sr_vol01-client-13: changing port to 49156 (from 0)
> [2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 
> 0-sr_vol01-client-13: connection to 
> ip-of-one-of-the-gluster-nodes:49156 failed (Connection refused)
>
> # gluster volume status sr_vol01
> shows that two bricks of the 18 are offline.
>
> rebalance fails
>
> Iptables was stopped on all nodes
>
> If I cd into the two bricks which are offline according to the gluster 
> v status, I can read/write without any problems... The disks are 
> clearly fine. They are mounted, they are available.
>
> I cannot find much info online about the error.
> Does anyone have an idea what could be wrong?
> How can I get the two bricks back online?
>
> Cheers,
> Olav
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users