Gluster users - Oct 2016 - Unable to reset gluster node after crash as filesystem ran out of space

Hi all.

We had a strange issue with Gluster 3.8.4 under RHEL 7.2.



Initially, the partition storing the Gluster bricks ran out of space. We
tried recovering after expanding the underlying partition. Eventually we
decided to ?reset? Gluster, create the volume again from scratch. I tried
purging gluster by running something like:





yum remove ?y ?glusterfs*?

rm -rf /var/lib/glusterd

rm ?rf /etc/gluster*



Reinstalling gluster:



yum install ?y glusterfs-server

systemctl start glusterd



Now a simple peer probe operation crashes the daemon:



$ gluster peer probe <any-valid-hostname>



Connection failed. Please check if gluster daemon is operational.





Looks like there is something I missed in the filesystem.





On a clean gluster install on another host,



$ gluster peer probe <any-valid-hostname>



peer probe: failed: Probe returned with Transport endpoint is not connected



Which is expected.



So somehow my clean install is not clean any more. This host I can rebuild,
but would be good to know the issue if this occurs on a host that cannot
easily be rebuilt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161011/f5b7f434/attachment.html>

On Tue, Oct 11, 2016 at 11:52 AM, Abeer Mahendroo <abeerm at gmail.com>
wrote:
> Hi all.
>
> We had a strange issue with Gluster 3.8.4 under RHEL 7.2.
>
>
>
> Initially, the partition storing the Gluster bricks ran out of space. We
> tried recovering after expanding the underlying partition. Eventually we
> decided to ?reset? Gluster, create the volume again from scratch. I tried
> purging gluster by running something like:
>
>
>
>
>
> yum remove ?y ?glusterfs*?
>
> rm -rf /var/lib/glusterd
>
> rm ?rf /etc/gluster*
>
>
>
> Reinstalling gluster:
>
>
>
> yum install ?y glusterfs-server
>
> systemctl start glusterd
>
>
>
> Now a simple peer probe operation crashes the daemon:
>
>
>
> $ gluster peer probe <any-valid-hostname>
>
>
>
> Connection failed. Please check if gluster daemon is operational.
>
This typically indicates that glusterd is not running. Could you check if
glusterd instance is running on this node? If not any error message on
glusterd log file?

>
>
>
>
> Looks like there is something I missed in the filesystem.
>
>
>
>
>
> On a clean gluster install on another host,
>
>
>
> $ gluster peer probe <any-valid-hostname>
>
>
>
> peer probe: failed: Probe returned with Transport endpoint is not connected
>
Same question here, is glusterd running on the host which you are trying to
probe? Are the firewalld/iptables rules clean?

>
> Which is expected.
>
>
>
> So somehow my clean install is not clean any more. This host I can
> rebuild, but would be good to know the issue if this occurs on a host that
> cannot easily be rebuilt.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


-- 

--Atin
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161017/76913e11/attachment.html>