Gluster users - May 2015 - Write operations failing on clients

Geesh Alex I don't have much for you.  Are all writes failing or just writes
to certain places?  Anyone else have any input on what could cause this?  To me
it sounds like rebalance was run, there was a problem and now writes are
problematic.  Is an accurate problem description?

-b

----- Original Message -----
> From: "Alex" <alex.m at icecat.biz>
> To: gluster-users at gluster.org
> Sent: Friday, May 1, 2015 5:47:29 AM
> Subject: Re: [Gluster-users] Write operations failing on clients
> 
> 
> > 
> > Are your files split brained:
> > 
> > gluster v heal img info split-brain
> > 
> > I see alot of problem with your self heal daemon connecting:
> 
> As far as I can see nodes are not split brained:
> 
> # gluster v heal img info split-brain
> Gathering list of split brain entries on volume img has been successful
> 
> Brick gluster1:/var/gl/images
> Number of entries: 0
> 
> Brick gluster2:/var/gl/images
> Number of entries: 0
> 
> Brick gluster3:/var/gl/images
> Number of entries: 0
> 
> Brick gluster4:/var/gl/images
> Number of entries: 0
> 
> Brick gluster5:/var/gl/images
> Number of entries: 0
> 
> Brick gluster6:/var/gl/images
> Number of entries: 0
> 
> > $ service glusterd stop
> > $ killall glusterfs
> > $ killall glusterfsd
> > $ ps aux | grep glu  <- Make sure evertyhing is actually cleaned up
> 
> Yes, I actually did this in the first place with problematic nodes.
> Unfortunately it did'nt help. CPU load came back in about 3-4 minutes.
> 
> > Have you recently run a rebalance?
> 
> Rebalance was running when the problem occur and I stopped it to see if it
> caused problems. I try to run it again.
> 
> > Are you having trouble access those directories?  It looks like the
fix
> layout failed for those two.
> 
> I can access those dirs via gluster-client:
> 
> # grep gluster /etc/fstab
> gluster1:/img   /media       glusterfs       defaults,_netdev        0 1
> 
> # ls -la /media/www/ | wc -l
> 47
> 
> /www/thumbs have excessive amount of files so i just stat something inside:
> # ls -l /media/www/thumbs/1000025.jpg
> -rw-r--r-- 1 apache apache 4365 Oct  8  2009 /media/www/thumbs/1000025.jpg
> 
> Everything looks fine.
> 
> Thank you,
> Alex
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>

Ben Turner <bturner at ...> writes:

> Are all writes failing or just writes to certain places?
When problem occur I've done basic testing such as creating directory and
successfully copied small file to it. Looks like writes only to some places
caused problem.
> To me it sounds like rebalance was run, there was a problem and now
> writes are problematic.  Is an accurate problem description?
I'm not sure about rebalance as source of problem. But it definitely was
running when writes begin to fail.

Okay, I understand that it difficult to help with such problem via mail.
Thank you anyway.

Alex