Gluster users - Jan 2016 - Unexpected behaviour adding a third server

We've a simple two-server one volume arrangement, replicating ~340k files
(15GB) between our web servers.

The servers are in AWS, sat in different availability zones. One of the
operations for this weekend is to add another pair of machines, one in each
AZ.

I've deployed the same OS image of the gluster server (3.6) and was under
the impression I could add a brick to the existing replica simply by
issuing the below:

gluster volume add-brick volume1 replica 3 pd-wfe3:/gluster-store

And then presumably would add the fourth server by repeating the above with
"replica 4" and the fourth server name.

The operation appeared to succeed, the brick appears alongside the others:

Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: pd-wfe1:/gluster-store
Brick2: pd-wfe2:/gluster-store
Brick3: pd-wfe3:/gluster-store

but almost immediately pd-wfe1 crept up to 100% CPU with the gluster
processes, and nginx began timing out serving content from the volume.

The glusterfs-glusterd-vol log is filled with this error at pd-wfe1:

[2016-01-23 08:43:28.459215] W [socket.c:620:__socket_rwv] 0-management:
readv on /var/run/c8bc2f99e7584cb9cf077c4f98d1db2e.socket failed (Invalid
argument)

while I see this error for the log named by the mount point:

[2016-01-23 08:43:28.986379] W [client-rpc-fops.c:306:client3_3_mkdir_cbk]
2-volume1-client-2: remote operation failed: Permission denied. Path: (null)

Does anyone have any suggestions how to proceed? I would appreciate any
input on this one.

Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160123/34e040a8/attachment.html>

On 01/23/2016 02:17 PM, Steve Spence wrote:
> We've a simple two-server one volume arrangement, replicating ~340k 
> files (15GB) between our web servers.
>
> The servers are in AWS, sat in different availability zones. One of 
> the operations for this weekend is to add another pair of machines, 
> one in each AZ.
>
> I've deployed the same OS image of the gluster server (3.6) and was 
> under the impression I could add a brick to the existing replica 
> simply by issuing the below:
>
> gluster volume add-brick volume1 replica 3 pd-wfe3:/gluster-store
>
> And then presumably would add the fourth server by repeating the above 
> with "replica 4" and the fourth server name.
>
> The operation appeared to succeed, the brick appears alongside the others:
>
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: pd-wfe1:/gluster-store
> Brick2: pd-wfe2:/gluster-store
> Brick3: pd-wfe3:/gluster-store
>
> but almost immediately pd-wfe1 crept up to 100% CPU with the gluster 
> processes, and nginx began timing out serving content from the volume.
Could you disable client-side healing?
"gluster volume set <volname> cluster.entry-self-heal off
"gluster volume set <volname> cluster.data-self-heal off
"gluster volume set <volname> cluster.metadata-self-heal off

We are in the process of making this experience smooth for 3.8. by 
introducing throttling of self-heal traffic, automatic healing.
+Anuradha,
        Could you give him the steps he need to perform after doing 
add-brick until the patch you sent is merged?

Pranith
>
> The glusterfs-glusterd-vol log is filled with this error at pd-wfe1:
>
> [2016-01-23 08:43:28.459215] W [socket.c:620:__socket_rwv] 
> 0-management: readv on 
> /var/run/c8bc2f99e7584cb9cf077c4f98d1db2e.socket failed (Invalid argument)
>
> while I see this error for the log named by the mount point:
>
> [2016-01-23 08:43:28.986379] W 
> [client-rpc-fops.c:306:client3_3_mkdir_cbk] 2-volume1-client-2: remote 
> operation failed: Permission denied. Path: (null)
>
> Does anyone have any suggestions how to proceed? I would appreciate 
> any input on this one.
>
> Steve
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160125/a1b305b3/attachment.html>