Gluster users - Apr 2015 - Disastrous performance with rsync to mounted Gluster volume.

On 2015-04-23 18:10, Joe Julian wrote:
> On 04/23/2015 04:41 PM, Ernie Dunbar wrote:
>> On 2015-04-23 12:58, Ben Turner wrote:
>> 
>>> 
>>> +1, lets nuke everything and start from a known good.  Those error
>>> messages make me think something is really wrong with how we are
>>> copying the data.  Gluster does NFS by default so you shouldn't
have
>>> have to reconfigure anything after you recreate the volume.
>>> 
>> 
>> Okay... this is a silly question. How do I do that? Deleting the 
>> volume doesn't affect the files in the underlying filesystem, and I
>> get the impression that trying to delete the files in the underlying 
>> filesystem without shutting down or deleting the volume would result 
>> in Gluster trying to write the files back where they
"belong".
>> 
>> Should I stop the volume, delete it, then delete the files and start 
>> from scratch, re-creating the volume?
> 
> That's what I would do.
> 
Well, apparently removing the .glusterfs directory from the brick is an 
exceptionally bad thing, and breaks gluster completely, rendering it 
inoperable. I'm going to have to post another thread about how to fix 
this mess now.

----- Original Message -----
> From: "Ernie Dunbar" <maillist at lightspeed.ca>
> To: "Gluster Users" <gluster-users at gluster.org>
> Sent: Friday, April 24, 2015 1:15:32 PM
> Subject: Re: [Gluster-users] Disastrous performance with rsync to mounted
Gluster volume.
> 
> On 2015-04-23 18:10, Joe Julian wrote:
> > On 04/23/2015 04:41 PM, Ernie Dunbar wrote:
> >> On 2015-04-23 12:58, Ben Turner wrote:
> >> 
> >>> 
> >>> +1, lets nuke everything and start from a known good.  Those
error
> >>> messages make me think something is really wrong with how we
are
> >>> copying the data.  Gluster does NFS by default so you
shouldn't have
> >>> have to reconfigure anything after you recreate the volume.
> >>> 
> >> 
> >> Okay... this is a silly question. How do I do that? Deleting the
> >> volume doesn't affect the files in the underlying filesystem,
and I
> >> get the impression that trying to delete the files in the
underlying
> >> filesystem without shutting down or deleting the volume would
result
> >> in Gluster trying to write the files back where they
"belong".
> >> 
> >> Should I stop the volume, delete it, then delete the files and
start
> >> from scratch, re-creating the volume?
> > 
> > That's what I would do.
> > 
> 
> Well, apparently removing the .glusterfs directory from the brick is an
> exceptionally bad thing, and breaks gluster completely, rendering it
> inoperable. I'm going to have to post another thread about how to fix
> this mess now.
You are correct and I would just start from scratch Ernie.  Creating a gluster
cluster is only about 3-4 commands and should only take a minute or two.  Also
with all the problems you are having I am not confident in your data integrity. 
All you need to do to clear EVERYTHING out is:

service glusterd stop
killall glusterfsd
killall glusterfs
sleep 1
for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks'
>/dev/null 2>&1;then rm -rf $file; fi; done
>From there restart the gluster service and recreate everything:
service glusterd restart
<make a new filesystem on your bricks, mount>
gluster peer probe <my peer>
gluster v create <my vol>
gluster v start <my vol>
gluster v info 
>From there mount the new volume on your system with the data  you want to
migrate:
mount -t nfs -o vers=3 <my vol> <my mount>
rsync <your rsync command>

This should get you where you need to be.  Before you start to migrate the data
maybe do a couple DDs and send me the output so we can get an idea of how your
cluster performs:

time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; sync`
echo 3 > /proc/sys/vm/drop_caches
dd if=<gluster mount> of=/dev/null bs=1024k count=1000

If you are using gigabit and glusterfs mounts with replica 2 you should get ~55
MB / sec writes and ~110 MB / sec reads.  With NFS you will take a bit of a hit
since NFS doesnt know where files live like glusterfs does.

-b
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>