Gluster users - Jun 2015 - consistency between nodes?

Hi all,

I'm having some trouble where different nodes in my volume have differing
ideas as to the state of the system.  This is a 4-node (one brick per node)
cluster, with replica 2.

I have identified two differing symptoms, I am hoping of the same problem.

The first symptom is that two of the four nodes don't show files in some of
the directories of the volume, which are present in the other two nodes.

The second symptom is that I am having some compile commands (but not all)
error with "No space left on device" when there is plenty of space on
every
brick.

The second symptom seems harder to track down than the first.  Presumably
my nodes are not properly synchronized in some way.  How would I track
something like this down?

A side note:  I have had rebalance fix-layout seemingly hang with no
evidence of progress, and a plain rebalance took a long time and I
eventually stopped it.  I created the cluster two nodes at a time, with the
understanding that I would be able to add pairs of nodes without problems,
but it doesn't seem to be working out that way so far.

Any suggestions would be appreciated! As with my last question, I'm not
sure what further information will be helpful, and will be happy to provide
more information!

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150619/ffa6b351/attachment.html>

Could you paste output of :

gluster peer status from all the nodes?

gluster volume status

gluster --version

Atin

Sent from one plus one
On Jun 20, 2015 4:12 AM, "David Roundy" <roundyd at
physics.oregonstate.edu>
wrote:
>
> Hi all,
>
> I'm having some trouble where different nodes in my volume have
differing
ideas as to the state of the system.  This is a 4-node (one brick per node)
cluster, with replica 2.
>
> I have identified two differing symptoms, I am hoping of the same problem.
>
> The first symptom is that two of the four nodes don't show files in
some
of the directories of the volume, which are present in the other two
nodes.
>
> The second symptom is that I am having some compile commands (but not
all) error with "No space left on device" when there is plenty of
space on
every brick.
>
> The second symptom seems harder to track down than the first.  Presumably
my nodes are not properly synchronized in some way.  How would I track
something like this down?
>
> A side note:  I have had rebalance fix-layout seemingly hang with no
evidence of progress, and a plain rebalance took a long time and I
eventually stopped it.  I created the cluster two nodes at a time, with the
understanding that I would be able to add pairs of nodes without problems,
but it doesn't seem to be working out that way so
far.
>
> Any suggestions would be appreciated! As with my last question, I'm not
sure what further information will be helpful, and will be happy to provide
more information!
>
> David
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150620/5a7c4eed/attachment.html>