Joe Julian
2015-Apr-24 18:43 UTC
[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.
On 04/24/2015 11:03 AM, Ben Turner wrote:> ----- Original Message ----- >> From: "Ernie Dunbar" <maillist at lightspeed.ca> >> To: "Gluster Users" <gluster-users at gluster.org> >> Sent: Friday, April 24, 2015 1:15:32 PM >> Subject: Re: [Gluster-users] Disastrous performance with rsync to mounted Gluster volume. >> >> On 2015-04-23 18:10, Joe Julian wrote: >>> On 04/23/2015 04:41 PM, Ernie Dunbar wrote: >>>> On 2015-04-23 12:58, Ben Turner wrote: >>>> >>>>> +1, lets nuke everything and start from a known good. Those error >>>>> messages make me think something is really wrong with how we are >>>>> copying the data. Gluster does NFS by default so you shouldn't have >>>>> have to reconfigure anything after you recreate the volume. >>>>> >>>> Okay... this is a silly question. How do I do that? Deleting the >>>> volume doesn't affect the files in the underlying filesystem, and I >>>> get the impression that trying to delete the files in the underlying >>>> filesystem without shutting down or deleting the volume would result >>>> in Gluster trying to write the files back where they "belong". >>>> >>>> Should I stop the volume, delete it, then delete the files and start >>>> from scratch, re-creating the volume? >>> That's what I would do. >>> >> Well, apparently removing the .glusterfs directory from the brick is an >> exceptionally bad thing, and breaks gluster completely, rendering it >> inoperable. I'm going to have to post another thread about how to fix >> this mess now. > You are correct and I would just start from scratch Ernie. Creating a gluster cluster is only about 3-4 commands and should only take a minute or two. Also with all the problems you are having I am not confident in your data integrity. All you need to do to clear EVERYTHING out is: > > service glusterd stop > killall glusterfsd > killall glusterfs > sleep 1 > for file in /var/lib/glusterd/*; do if ! echo $file | grep 'hooks' >/dev/null 2>&1;then rm -rf $file; fi; done > > From there restart the gluster service and recreate everything: > > service glusterd restart > <make a new filesystem on your bricks, mount> > gluster peer probe <my peer> > gluster v create <my vol> > gluster v start <my vol> > gluster v info > > From there mount the new volume on your system with the data you want to migrate: > > mount -t nfs -o vers=3 <my vol> <my mount> > rsync <your rsync command>And your rsync command should include, "--inplace".> > This should get you where you need to be. Before you start to migrate the data maybe do a couple DDs and send me the output so we can get an idea of how your cluster performs: > > time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; sync` > echo 3 > /proc/sys/vm/drop_caches > dd if=<gluster mount> of=/dev/null bs=1024k count=1000 > > If you are using gigabit and glusterfs mounts with replica 2 you should get ~55 MB / sec writes and ~110 MB / sec reads. With NFS you will take a bit of a hit since NFS doesnt know where files live like glusterfs does. > > -b > >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://www.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users
Ernie Dunbar
2015-Apr-27 20:24 UTC
[Gluster-users] Disastrous performance with rsync to mounted Gluster volume.
On 2015-04-24 11:43, Joe Julian wrote:>> This should get you where you need to be. Before you start to migrate >> the data maybe do a couple DDs and send me the output so we can get an >> idea of how your cluster performs: >> >> time `dd if=/dev/zero of=<gluster-mount>/myfile bs=1024k count=1000; >> sync` >> echo 3 > /proc/sys/vm/drop_caches >> dd if=<gluster mount> of=/dev/null bs=1024k count=1000 >> >> If you are using gigabit and glusterfs mounts with replica 2 you >> should get ~55 MB / sec writes and ~110 MB / sec reads. With NFS you >> will take a bit of a hit since NFS doesnt know where files live like >> glusterfs does.After copying our data and doing a couple of very slow rsyncs, I did your speed test and came back with these results: 1048576 bytes (1.0 MB) copied, 0.0307951 s, 34.1 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=1024 bs=1024; sync 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.0298592 s, 35.1 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=1024 bs=1024; sync 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.0501495 s, 20.9 MB/s root at backup:/home/webmailbak# echo 3 > /proc/sys/vm/drop_caches root at backup:/home/webmailbak# # dd if=/mnt/testfile of=/dev/null bs=1024k count=1000 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124498 s, 84.2 MB/s Keep in mind that this is an NFS share over the network. I've also noticed that if I increase the count of those writes, the transfer speed increases as well: 2097152 bytes (2.1 MB) copied, 0.036291 s, 57.8 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=2048 bs=1024; sync 2048+0 records in 2048+0 records out 2097152 bytes (2.1 MB) copied, 0.0362724 s, 57.8 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=2048 bs=1024; sync 2048+0 records in 2048+0 records out 2097152 bytes (2.1 MB) copied, 0.0360319 s, 58.2 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=10240 bs=1024; sync 10240+0 records in 10240+0 records out 10485760 bytes (10 MB) copied, 0.127219 s, 82.4 MB/s root at backup:/home/webmailbak# dd if=/dev/zero of=/mnt/testfile count=10240 bs=1024; sync 10240+0 records in 10240+0 records out 10485760 bytes (10 MB) copied, 0.128671 s, 81.5 MB/s However, the biggest stumbling block for rsync seems to be changes to directories. I'm unsure about what exactly it's doing (probably changing last access times?) but these minor writes seem to take a very long time when normally they would not. Actual file copies (as in the very files that are actually new within those same directories) appear to take quite a lot less time than the directory updates. For example: # time rsync -av --inplace --whole-file --ignore-existing --delete-after gromm/* /mnt/gromm/ building file list ... done Maildir/ ## This part takes a long time. Maildir/.INBOX.Trash/ Maildir/.INBOX.Trash/cur/ Maildir/.INBOX.Trash/cur/1429836077.H817602P21531.pop.lightspeed.ca:2,S Maildir/.INBOX.Trash/tmp/ ## The previous three lines took nearly no time at all. Maildir/cur/ ## This takes a long time. Maildir/cur/1430160436.H952679P13870.pop.lightspeed.ca:2,S Maildir/new/ Maildir/tmp/ ## The previous lines again take no time at all. deleting Maildir/cur/1429836077.H817602P21531.pop.lightspeed.ca:2,S ## This delete did take a while. sent 1327634 bytes received 75 bytes 59009.29 bytes/sec total size is 624491648 speedup is 470.35 real 0m26.110s user 0m0.140s sys 0m1.596s So, rsync reports that it wrote 1327634 bytes at 59 kBytes/sec, and the whole operation took 26 seconds. To write 2 files that were around 20-30 kBytes each and delete 1. The last rsync took around 56 minutes, when normally such an rsync would have taken 5-10 minutes, writing over the network via ssh.