Gluster users - Oct 2016 - gluster brick daemon segfaulted in pairs

Hi,

We are running a distributed replicated volume: 16 pairs of bricks (rep count
2), 2 nodes.

On Friday, 2 pairs of brick daemons seg-faulted within minutes of each other,
leading to 2 subvolumes down (no replicas left).  We tried to bring them up
again by doing a "volume start force?, which worked, but roughly 4 hours
later this happened again, but to two other pairs of bricks.

There is nothing of note in brick logs for the downed bricks, except that it
just suddenly stops logging.  In the other logs (nfs, glusterhd, etc), we simply
start seeing errors saying ?All sub volumes down? for those replicates.

We are running GluserFS 3.8.2 on Ubuntu 16.04.

I do have a couple of core dumps preserved by apport.  Any ideas?  Should I file
this straight into bugzilla?

Thanks,
Jackie
-- 
 

The information in this email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful.

On Monday 24 October 2016, Jackie Tung <jackie at drive.ai> wrote:
> Hi,
>
> We are running a distributed replicated volume: 16 pairs of bricks (rep
> count 2), 2 nodes.
>
> On Friday, 2 pairs of brick daemons seg-faulted within minutes of each
> other, leading to 2 subvolumes down (no replicas left).  We tried to bring
> them up again by doing a "volume start force?, which worked, but
roughly 4
> hours later this happened again, but to two other pairs of bricks.
>
> There is nothing of note in brick logs for the downed bricks, except that
> it just suddenly stops logging.  In the other logs (nfs, glusterhd, etc),
> we simply start seeing errors saying ?All sub volumes down? for those
> replicates.
>
> We are running GluserFS 3.8.2 on Ubuntu 16.04.
>
> I do have a couple of core dumps preserved by apport.  Any ideas?  Should
> I file this straight into bugzilla?

Filing a bug with coredumps attached would be ideal, have you got a chance
to look at the backtraces of these coredumps? If not please provide the
backtrace too, sometimes developers can straight away identify the issues
looking at the backtraces.

> Thanks,
> Jackie
> --
>
>
> The information in this email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <javascript:;>
> http://www.gluster.org/mailman/listinfo/gluster-users


-- 
--Atin
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20161025/ae265098/attachment.html>