Ben Turner
2015-Apr-24 18:03 UTC
[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.
----- Original Message -----> From: "Ernie Dunbar" <maillist at lightspeed.ca> > To: "Gluster Users" <gluster-users at gluster.org> > Sent: Friday, April 24, 2015 1:15:32 PM > Subject: Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume. > > On 2015-04-23 18:10, Joe Julian wrote: > > On 04/23/2015 04:41 PM, Ernie Dunbar wrote: > >> On 2015-04-23 12:58, Ben Turner wrote: > >> > >>> > >>> +1, lets nuke everything and start from a known good. Those error > >>> messages make me think something is really wrong with how we are > >>> copying the data. Gluster does NFS by default so you shouldn't have > >>> have to reconfigure anything after you recreate the volume. > >>> > >> > >> Okay... this is a silly question. How do I do that? Deleting the > >> volume doesn't affect the files in the underlying filesystem, and I > >> get the impression that trying to delete the files in the underlying > >> filesystem without shutting down or deleting the volume would result > >> in Gluster trying to write the files back where they "belong". > >> > >> Should I stop the volume, delete it, then delete the files and start > >> from scratch, re-creating the volume? > > > > That's what I would do. > > > > Well, apparently removing the .glusterfs directory from the brick is an > exceptionally bad thing, and breaks gluster completely, rendering it > inoperable. I'm going to have to post another thread about how to fix > this mess now.You are correct and I would just start from scratch Ernie. Creating a gluster cluster is only about 3-4 commands and should only take a minute or two. Also with all the problems you are having I am not confident in your data integrity. All you need to do to clear EVERYTHING out is: service glusterd stop killall glusterfsd killall glusterfs sleep 1 for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks' >/dev/null 2>&1;then rm -rf $file; fi; done>From there restart the gluster service and recreate everything:service glusterd restart <make a new filesystem on your bricks, mount> gluster peer probe <my peer> gluster v create <my vol> gluster v start <my vol> gluster v info>From there mount the new volume on your system with the data you want to migrate:mount -t nfs -o vers=3 <my vol> <my mount> rsync <your rsync command> This should get you where you need to be. Before you start to migrate the data maybe do a couple DDs and send me the output so we can get an idea of how your cluster performs: time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; sync` echo 3 > /proc/sys/vm/drop_caches dd if=<gluster mount> of=/dev/null bs=1024k count=1000 If you are using gigabit and glusterfs mounts with replica 2 you should get ~55 MB / sec writes and ~110 MB / sec reads. With NFS you will take a bit of a hit since NFS doesnt know where files live like glusterfs does. -b> _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >
Joe Julian
2015-Apr-24 18:43 UTC
[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.
On 04/24/2015 11:03 AM, Ben Turner wrote:> ----- Original Message ----- >> From: "Ernie Dunbar" <maillist at lightspeed.ca> >> To: "Gluster Users" <gluster-users at gluster.org> >> Sent: Friday, April 24, 2015 1:15:32 PM >> Subject: Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume. >> >> On 2015-04-23 18:10, Joe Julian wrote: >>> On 04/23/2015 04:41 PM, Ernie Dunbar wrote: >>>> On 2015-04-23 12:58, Ben Turner wrote: >>>> >>>>> +1, lets nuke everything and start from a known good. Those error >>>>> messages make me think something is really wrong with how we are >>>>> copying the data. Gluster does NFS by default so you shouldn't have >>>>> have to reconfigure anything after you recreate the volume. >>>>> >>>> Okay... this is a silly question. How do I do that? Deleting the >>>> volume doesn't affect the files in the underlying filesystem, and I >>>> get the impression that trying to delete the files in the underlying >>>> filesystem without shutting down or deleting the volume would result >>>> in Gluster trying to write the files back where they "belong". >>>> >>>> Should I stop the volume, delete it, then delete the files and start >>>> from scratch, re-creating the volume? >>> That's what I would do. >>> >> Well, apparently removing the .glusterfs directory from the brick is an >> exceptionally bad thing, and breaks gluster completely, rendering it >> inoperable. I'm going to have to post another thread about how to fix >> this mess now. > You are correct and I would just start from scratch Ernie. Creating a gluster cluster is only about 3-4 commands and should only take a minute or two. Also with all the problems you are having I am not confident in your data integrity. All you need to do to clear EVERYTHING out is: > > service glusterd stop > killall glusterfsd > killall glusterfs > sleep 1 > for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks' >/dev/null 2>&1;then rm -rf $file; fi; done > > From there restart the gluster service and recreate everything: > > service glusterd restart > <make a new filesystem on your bricks, mount> > gluster peer probe <my peer> > gluster v create <my vol> > gluster v start <my vol> > gluster v info > > From there mount the new volume on your system with the data you want to migrate: > > mount -t nfs -o vers=3 <my vol> <my mount> > rsync <your rsync command>And your rsync command should include, "--inplace".> > This should get you where you need to be. Before you start to migrate the data maybe do a couple DDs and send me the output so we can get an idea of how your cluster performs: > > time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; sync` > echo 3 > /proc/sys/vm/drop_caches > dd if=<gluster mount> of=/dev/null bs=1024k count=1000 > > If you are using gigabit and glusterfs mounts with replica 2 you should get ~55 MB / sec writes and ~110 MB / sec reads. With NFS you will take a bit of a hit since NFS doesnt know where files live like glusterfs does. > > -b > >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users