Hi, I have hard time understanding how volume rebalance works. I had 2 bricks: gluster3:/data & gluster4:/data It contains: on at gluster3:/data$ ls -lrat total 5117020 -rwx------ 1 root root 108693504 2011-09-28 19:38 VMware-tools-linux-8.6.0-425873.iso -rwx------ 1 root root 93786112 2011-09-28 19:38 VMware-tools-linux-8.3.7-341836.iso -rwx------ 1 root root 266371072 2011-09-28 19:39 VMware-VMvisor-InstallerCD-3.5.0_Update_5-207095.i386.iso -rwx------ 1 root root 304859136 2011-09-28 19:39 VMware-VMvisor-Installer-5.0.0-469512.x86_64.iso -rwx------ 1 root root 412487680 2011-09-28 19:39 VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso -rwx------ 1 root root 611405824 2011-09-28 19:40 ubuntu-8.04.3-server-amd64.iso -rwx------ 1 root root 686305280 2011-09-28 19:41 ubuntu-9.10-server-amd64.iso -rwx------ 1 root root 631246848 2011-09-28 19:41 ubuntu-8.04.3-server-i386.iso -rwx------ 1 root root 671686656 2011-09-28 19:42 ubuntu-9.10-server-i386.iso -rwx------ 1 root root 706330624 2011-09-28 19:42 ubuntu-11.04-server-amd64.iso -rwx------ 1 root root 13121536 2011-09-28 19:42 ubuntu-10.04-minimal-i386.iso -rwx------ 1 root root 733419520 2011-09-28 19:42 ubuntu-10.04-desktop-i386.iso drwxr-xr-x 22 root root 4096 2011-09-28 19:44 .. drwxr-xr-x 2 root root 4096 2011-09-28 20:04 . To my volume I add gluster1:/data & gluster2:/data gluster> volume add-brick test-vol gluster1:/data gluster2:/data Add Brick successful gluster> volume info Volume Name: test-vol Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster3:/data Brick2: gluster4:/data Brick3: gluster1:/data Brick4: gluster2:/data gluster> volume rebalance test-vol start starting rebalance on volume test-vol has been successful gluster> volume rebalance test-vol status rebalance step 2: data migration in progress gluster> volume rebalance test-vol status rebalance completed gluster> After the rebalance, if I look at gluster1: on at gluster1:/data$ ls -la total 1624168 drwxr-xr-x 2 root root 4096 2011-09-28 20:07 . drwxr-xr-x 22 root root 4096 2011-09-28 20:04 .. ---------T 1 root root 0 2011-09-28 20:06 ubuntu-8.04.3-server-i386.iso -rwx------ 1 root root 631246848 2011-09-28 20:07 .ubuntu-8.04.3-server-i386.iso.gfs631246848 ---------T 1 root root 0 2011-09-28 20:06 ubuntu-9.10-server-i386.iso -rwx------ 1 root root 671686656 2011-09-28 20:07 .ubuntu-9.10-server-i386.iso.gfs671686656 ---------T 1 root root 0 2011-09-28 20:06 VMware-tools-linux-8.3.7-341836.iso -rwx------ 1 root root 93786112 2011-09-28 20:07 .VMware-tools-linux-8.3.7-341836.iso.gfs93786112 ---------T 1 root root 0 2011-09-28 20:06 VMware-VMvisor-InstallerCD-3.5.0_Update_5-207095.i386.iso -rwx------ 1 root root 266371072 2011-09-28 20:06 .VMware-VMvisor-InstallerCD-3.5.0_Update_5-207095.i386.iso.gfs266371072 And bricks gluster3 and gluster4 still contain the exact same data. Why on the new bricks the files come with a name prefixed with a dot, why is there an empty file created with sticky bit? I would have expected the temporary dof ile to be renamed into the final name (sticky bit removed) and the original file on the old brick to be removed. Am I missing something? Best regards, Olivier
Hi, I just did another test this morning. After rebalancing, I get an error in the logs /var/log/gluster/etc-glusterfs-glusterd.vol.log about file size: [2011-09-29 09:36:39.703090] W [glusterd-rebalance.c:251:gf_glusterd_rebalance_move_data] 0-glusterfs: file sizes are not same : /etc/glusterd/mount/test-vol/ubuntu-9.10-server-i386.iso But from the NFS client, I checked the original file and the temporary copy, and they are the same: banyan<root>: md5 ubuntu-9.10-server-i386.iso MD5 (ubuntu-9.10-server-i386.iso) = 55618ad5f180692f9dac20cbff352634 banyan<root>: md5 .ubuntu-9.10-server-i386.iso.gfs671686656 MD5 (.ubuntu-9.10-server-i386.iso.gfs671686656) = 55618ad5f180692f9dac20cbff352634 banyan<root>: ls -lart |grep ubuntu-9.10-server-i386.iso -rw-r--r-- 1 root wheel 671686656 Sep 29 09:03 ubuntu-9.10-server-i386.iso -rw-r--r-- 1 root wheel 671686656 Sep 29 09:36 .ubuntu-9.10-server-i386.iso.gfs671686656 What is happening? TIA, Olivier