Gluster users - Jul 2017 - Problem manipulating files when a replica is down

We currently have a small pilot gluster setup as Replica 2 + 1 (arbiter)

As a DR test we decided to see that to see what would happen when taking
one of the replicas offline. For the first couple of hours everything
seemed fine and then one of my colleagues tried to a directory with files
via samba from a windows client by using robocopy.

At this point it starting to go wrong. The brick went offline, I could
force it online but when trying to manipulate files on the volume the brick
kept going offline.

Some Examples of trying to manipulate the files from the mounted gluster
volume:

rm -rf a Directory: "Directory not empty"

on an empty directory ls -la : "total 0" ( didn't show
"." or ".." )


After bringing the partner replica up the brick on the system that was
never shutdown kept going offline. In the end I ended up restoring the
volume from a gluster snapshot from the night before and then everything
seemed fine again.


This morning I was made aware that creating a new directory via samba
resulted in 5 New folders being created. Restarting glusterd has made the
issue go away for now.


This is on Centos 7.3 with gluster 3.10.3-1 and samba 4.4.4-14


Can anyone help ?

Thanks

*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
+44 1782 733063 <%2B44%201782%20733073>
________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170717/130489d2/attachment.html>

On 07/17/2017 02:40 PM, Gary Lloyd wrote:
> We currently have a small pilot gluster setup as Replica 2 + 1 (arbiter)
>
> As a DR test we decided to see that to see what would happen when 
> taking one of the replicas offline. For the first couple of hours 
> everything seemed fine and then one of my colleagues tried to a 
> directory with files via samba from a windows client by using robocopy.
>
> At this point it starting to go wrong. The brick went offline, I could 
> force it online but when trying to manipulate files on the volume the 
> brick kept going offline.
What do you mean by the brick going offline? The brick gets killed? What 
does the brick log say?
>
> Some Examples of trying to manipulate the files from the mounted 
> gluster volume:
>
> rm -rf a Directory: "Directory not empty"
>
> on an empty directory ls -la : "total 0" ( didn't show
"." or ".." )
>
>
> After bringing the partner replica up the brick on the system that was 
> never shutdown kept going offline. In the end I ended up restoring the 
> volume from a gluster snapshot from the night before and then 
> everything seemed fine again.
>
>
> This morning I was made aware that creating a new directory via samba 
> resulted in 5 New folders being created. Restarting glusterd has made 
> the issue go away for now.
Are you saying that each brick has the same directory created 5 times?
-Ravi
>
>
> This is on Centos 7.3 with gluster 3.10.3-1 and samba 4.4.4-14
>
>
> Can anyone help ?
>
> Thanks
>
> /Gary Lloyd/
> ________________________________________________
> I.T. Systems:Keele University
> Finance & IT Directorate
> Keele:Staffs:IC1 Building:ST5 5NB:UK
> +44 1782 733063 <tel:%2B44%201782%20733073>
> ________________________________________________
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170717/e1f891dc/attachment.html>