Samuele Giovanni Tonon
2010-Nov-03 17:20 UTC
[Gluster-users] expanding volumes produce weird results
hi, i've found a strange problem while expanding volumes: steps to do: 1) create a replica 2 volume gluster volume create ddvol replica 2 transport tcp \ 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 gluster volume start ddvol 2) create a client vol files using: glusterfs-volgen -n ddvol --num-replica=2 -t tcp 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 3) mount it using mount -t glusterfs /root/ddvol-tcp.vol /mnt/ 4) put some files in it and umount it: ls -la /mnt/ total 114008 drwxr-xr-x 5 root root 8192 Nov 3 18:03 . drwxr-xr-x 22 root root 4096 Nov 3 17:32 .. -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg -rw-r--r-- 1 root root 31162400 Nov 3 18:03 Scrat2.mp4 drwxr-xr-x 2 root root 8192 Nov 3 18:03 sc2 umount /mnt 4) add 2 more bricks and rebalance gluster volume add-brick ddvol 192.168.1.202:/public/dd2 \ 192.168.1.203:/public/dd3 gluster volume rebalance ddvol start 5) after rebalance has finished create a new client vol file and mount again: gluster volume rebalance ddvol status rebalance completed: rebalanced 94 files of size 50547334 (total files scanned 230) glusterfs-volgen -n ddvol --num-replica=2 -t tcp 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 192.168.1.202:/public/dd2 192.168.1.203:/public/dd3 6) mount and ls: mount -t glusterfs /root/ddvol-tcp.vol /mnt/ ls -la /mnt/ total 228100 drwxr-xr-x 5 root root 16384 Nov 3 18:05 . drwxr-xr-x 22 root root 4096 Nov 3 17:32 .. -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg drwxr-xr-x 2 root root 16384 Nov 3 18:05 sc2 -rw-r--r-- 1 root root 20977664 Nov 3 18:03 scryed.avi -rw-r--r-- 1 root root 20977664 Nov 3 18:03 scryed.avi as you can see only files are doubled, however : files inside sc2 are double too. if you start from the beginning with 4 nodes it won't happen. if you mount directly from one server you see normal files, this happen only after an "add-brick" action. you can delete one of those files but at the next mount you'll see them again. I'm mounting through client vol because i would like to implement a ha system (and pointing directly to a server would be a single point of failure) while being able to add more space "on the fly" if needed; i'm aware i need to add brick in "couple" and it's perfectly fine since until there aren't two servers down of the same couple i should be able to see my replica, i didn't expect this kind of behaviour . Is it well known ? i wasn't able to find any link to this issue. btw all was tested on ubuntu 10.10 64bit with package 3.1.0 md5sum glusterfs_3.1.0-1_amd64.deb 9e91f8a335f54c59bd771a0aea8e6ae1 /root/glusterfs_3.1.0-1_amd64.deb uname -a Linux ubuntu0 2.6.35-22-server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC 2010 x86_64 GNU/Linux Best Regards Samuele -- While various networks have become deeply rooted, and thoughts have been sent out as light and electrons in a singular direction, this era has yet to digitize/computerize to the degree necessary for individuals to become a singular complex entity. KOUKAKU KIDOUTAI Stand Alone Complex
Samuele - You don't need to create a client vol file with 3.1. Please delete it from the clients and follow these instructions to mount - http://www.gluster.com/community/documentation/index.php/Gluster_3.1:_Manually_Mounting_Volumes Thanks, Craig --> Craig Carl Gluster, Inc. Cell - (408) 829-9953 (California, USA) Gtalk - craig.carl at gmail.com From: "Samuele Giovanni Tonon" <samu at linuxasylum.net> To: "Gluster General Discussion List" <gluster-users at gluster.org> Sent: Wednesday, November 3, 2010 10:20:14 AM Subject: [Gluster-users] expanding volumes produce weird results hi, i've found a strange problem while expanding volumes: steps to do: 1) create a replica 2 volume gluster volume create ddvol replica 2 transport tcp \ 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 gluster volume start ddvol 2) create a client vol files using: glusterfs-volgen -n ddvol --num-replica=2 -t tcp 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 3) mount it using mount -t glusterfs /root/ddvol-tcp.vol /mnt/ 4) put some files in it and umount it: ls -la /mnt/ total 114008 drwxr-xr-x 5 root root 8192 Nov 3 18:03 . drwxr-xr-x 22 root root 4096 Nov 3 17:32 .. -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg -rw-r--r-- 1 root root 31162400 Nov 3 18:03 Scrat2.mp4 drwxr-xr-x 2 root root 8192 Nov 3 18:03 sc2 umount /mnt 4) add 2 more bricks and rebalance gluster volume add-brick ddvol 192.168.1.202:/public/dd2 \ 192.168.1.203:/public/dd3 gluster volume rebalance ddvol start 5) after rebalance has finished create a new client vol file and mount again: gluster volume rebalance ddvol status rebalance completed: rebalanced 94 files of size 50547334 (total files scanned 230) glusterfs-volgen -n ddvol --num-replica=2 -t tcp 192.168.1.200:/public/dd0 192.168.1.201:/public/dd1 192.168.1.202:/public/dd2 192.168.1.203:/public/dd3 6) mount and ls: mount -t glusterfs /root/ddvol-tcp.vol /mnt/ ls -la /mnt/ total 228100 drwxr-xr-x 5 root root 16384 Nov 3 18:05 . drwxr-xr-x 22 root root 4096 Nov 3 17:32 .. -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 2139951 Nov 3 18:02 03 - All Together Now.mp3 -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg -rw-r--r-- 1 root root 14133252 Nov 3 18:02 Cyborg 009_opening.mpg drwxr-xr-x 2 root root 16384 Nov 3 18:05 sc2 -rw-r--r-- 1 root root 20977664 Nov 3 18:03 scryed.avi -rw-r--r-- 1 root root 20977664 Nov 3 18:03 scryed.avi as you can see only files are doubled, however : files inside sc2 are double too. if you start from the beginning with 4 nodes it won't happen. if you mount directly from one server you see normal files, this happen only after an "add-brick" action. you can delete one of those files but at the next mount you'll see them again. I'm mounting through client vol because i would like to implement a ha system (and pointing directly to a server would be a single point of failure) while being able to add more space "on the fly" if needed; i'm aware i need to add brick in "couple" and it's perfectly fine since until there aren't two servers down of the same couple i should be able to see my replica, i didn't expect this kind of behaviour . Is it well known ? i wasn't able to find any link to this issue. btw all was tested on ubuntu 10.10 64bit with package 3.1.0 md5sum glusterfs_3.1.0-1_amd64.deb 9e91f8a335f54c59bd771a0aea8e6ae1 /root/glusterfs_3.1.0-1_amd64.deb uname -a Linux ubuntu0 2.6.35-22-server #35-Ubuntu SMP Sat Oct 16 22:02:33 UTC 2010 x86_64 GNU/Linux Best Regards Samuele -- While various networks have become deeply rooted, and thoughts have been sent out as light and electrons in a singular direction, this era has yet to digitize/computerize to the degree necessary for individuals to become a singular complex entity. KOUKAKU KIDOUTAI Stand Alone Complex _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users