Dmitry Antipov
2019-Nov-29 12:50 UTC
[Gluster-users] Healing completely loss file on replica 3 volume
I'm trying to manually garbage data on bricks (when the volume is stopped) and then check whether healing is possible. For example: Start: # glusterd --debug Bricks (on EXT4 mounted with 'rw,realtime'): # mkdir /root/data0 # mkdir /root/data1 # mkdir /root/data2 Volume: # gluster volume create gv0 replica 3 [local-ip]:/root/data0 [local-ip]:/root/data1 [local-ip]:/root/data2 force volume create: gv0: success: please start the volume to access data # gluster volume start gv0 volume start: gv0: success Mount: # mkdir /mnt/gv0 # mount -t glusterfs [local-ip]:/gv0 /mnt/gv0 WARNING: getfattr not found, certain checks will be skipped.. Create file: # openssl rand 65536 > /mnt/gv0/64K # md5sum /mnt/gv0/64K ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K Umount and down the volume: # umount /mnt/gv0 # gluster volume stop gv0 Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y volume stop: gv0: success Check data on bricks: # md5sum /root/data[012]/64K ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data0/64K ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data1/64K ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data2/64K Seems OK. Then garbage all: # openssl rand 65536 > /root/data0/64K # openssl rand 65536 > /root/data1/64K # openssl rand 65536 > /root/data2/64K # md5sum /root/data[012]/64K c69096d15007578dab95d9940f89e167 /root/data0/64K b85292fb60f1a1d27f1b0e3bc6bfdfae /root/data1/64K c2e90335cc2f600ddab5c53a992b2bb6 /root/data2/64K Restart the volume and start full heal: # gluster volume start gv0 volume start: gv0: success # /usr/glusterfs/sbin/gluster volume heal gv0 full Launching heal operation to perform full self heal on volume gv0 has been successful Use heal info commands to check status. Finally: # gluster volume heal gv0 info summary Brick [local-ip]:/root/data0 Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick [local-ip]:/root/data1 Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Brick [local-ip]:/root/data2 Status: Connected Total Number of entries: 0 Number of entries in heal pending: 0 Number of entries in split-brain: 0 Number of entries possibly healing: 0 Since all 3 copies are different from each other, majority voting is useless and data (IIUC) should be marked as split-brain at least. But I'm seeing just zeroes everywhere above. Why it is so? Thanks in advance, Dmitry
Strahil Nikolov
2019-Nov-30 15:13 UTC
[Gluster-users] Healing completely loss file on replica 3 volume
Hi Dmitry,
keep in mind that gluster doesn't run md5sum (or any SHAsum) against the
local copies on the bricks , in order to verify your data.
The simplest said, it relies on entries in gluster's
"/brick/.glusterfs" folder and the metadata (getfattr -d -m . -e hex
<file-path-on-brick>.) differences in order to detect and fight split
brain.In your test , you have overwritten the contents of the file , but
metadata is the old one and also there are no entries
in?"/brick/.glusterfs" - as last operation was successful?(all bricks
were online).
Your test is not properly testing the software and as far as I know it's not
expecting that someone will mess up with all 3 bricks :D.
Best Regards,Strahil Nikolov
? ?????, 29 ??????? 2019 ?., 14:55:56 ?. ???????+2, Dmitry Antipov
<dmantipov at yandex.ru> ??????:
I'm trying to manually garbage data on bricks (when the volume is
stopped) and then check whether healing is possible. For example:
Start:
# glusterd --debug
Bricks (on EXT4 mounted with 'rw,realtime'):
# mkdir /root/data0
# mkdir /root/data1
# mkdir /root/data2
Volume:
# gluster volume create gv0 replica 3 [local-ip]:/root/data0?
[local-ip]:/root/data1? [local-ip]:/root/data2 force
volume create: gv0: success: please start the volume to access data
# gluster volume start gv0
volume start: gv0: success
Mount:
# mkdir /mnt/gv0
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
WARNING: getfattr not found, certain checks will be skipped..
Create file:
# openssl rand 65536 > /mnt/gv0/64K
# md5sum /mnt/gv0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K
Umount and down the volume:
# umount /mnt/gv0
# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
y
volume stop: gv0: success
Check data on bricks:
# md5sum /root/data[012]/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data1/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data2/64K
Seems OK. Then garbage all:
# openssl rand 65536 > /root/data0/64K
# openssl rand 65536 > /root/data1/64K
# openssl rand 65536 > /root/data2/64K
# md5sum /root/data[012]/64K
c69096d15007578dab95d9940f89e167? /root/data0/64K
b85292fb60f1a1d27f1b0e3bc6bfdfae? /root/data1/64K
c2e90335cc2f600ddab5c53a992b2bb6? /root/data2/64K
Restart the volume and start full heal:
# gluster volume start gv0
volume start: gv0: success
# /usr/glusterfs/sbin/gluster volume heal gv0 full
Launching heal operation to perform full self heal on volume gv0 has been
successful
Use heal info commands to check status.
Finally:
# gluster volume heal gv0 info summary
Brick [local-ip]:/root/data0
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick [local-ip]:/root/data1
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick [local-ip]:/root/data2
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Since all 3 copies are different from each other, majority voting is useless
and data (IIUC) should be marked as split-brain at least. But I'm seeing
just
zeroes everywhere above. Why it is so?
Thanks in advance,
Dmitry
________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191130/31144ca4/attachment-0001.html>
Karthik Subrahmanya
2019-Dec-02 06:20 UTC
[Gluster-users] Healing completely loss file on replica 3 volume
Hi Dmitry, Answers inline. On Fri, Nov 29, 2019 at 6:26 PM Dmitry Antipov <dmantipov at yandex.ru> wrote:> I'm trying to manually garbage data on bricksFirst of all changing data directly on the backend is not recommended and is not supported. All the operations needs to be done from the client mount point. Only few special cases needs changing few data about the file directly on the backend.> (when the volume is > stopped) and then check whether healing is possible. For example: > > Start: > > # glusterd --debug > > Bricks (on EXT4 mounted with 'rw,realtime'): > > # mkdir /root/data0 > # mkdir /root/data1 > # mkdir /root/data2 > > Volume: > > # gluster volume create gv0 replica 3 [local-ip]:/root/data0 > [local-ip]:/root/data1 [local-ip]:/root/data2 force > volume create: gv0: success: please start the volume to access data > # gluster volume start gv0 > volume start: gv0: success > > Mount: > > # mkdir /mnt/gv0 > # mount -t glusterfs [local-ip]:/gv0 /mnt/gv0 > WARNING: getfattr not found, certain checks will be skipped.. > > Create file: > > # openssl rand 65536 > /mnt/gv0/64K > # md5sum /mnt/gv0/64K > ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K > > Umount and down the volume: > > # umount /mnt/gv0 > # gluster volume stop gv0 > Stopping volume will make its data inaccessible. Do you want to continue? > (y/n) y > volume stop: gv0: success > > Check data on bricks: > > # md5sum /root/data[012]/64K > ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data0/64K > ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data1/64K > ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data2/64K > > Seems OK. Then garbage all: > > # openssl rand 65536 > /root/data0/64K > # openssl rand 65536 > /root/data1/64K > # openssl rand 65536 > /root/data2/64K > # md5sum /root/data[012]/64K > c69096d15007578dab95d9940f89e167 /root/data0/64K > b85292fb60f1a1d27f1b0e3bc6bfdfae /root/data1/64K > c2e90335cc2f600ddab5c53a992b2bb6 /root/data2/64K > > Restart the volume and start full heal: > > # gluster volume start gv0 > volume start: gv0: success > # /usr/glusterfs/sbin/gluster volume heal gv0 full > Launching heal operation to perform full self heal on volume gv0 has been > successful > Use heal info commands to check status. > > Finally: > > # gluster volume heal gv0 info summary > > Brick [local-ip]:/root/data0 > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick [local-ip]:/root/data1 > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick [local-ip]:/root/data2 > Status: Connected > Total Number of entries: 0 > Number of entries in heal pending: 0 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Since all 3 copies are different from each other, majority voting is > useless > and data (IIUC) should be marked as split-brain at least. But I'm seeing > just > zeroes everywhere above. Why it is so? >Since the data is changed directly on the backend, gluster will not be knowing these changes. If the changes done from the client mount fails on some bricks, only those will be recognized and marked by gluster so that it can heal those when possible. Since this is a replica 3 volume and if you end up in split-brain when you are doing the operations on the mount pint, then that will be a bug. As far as this is considered it is not a bug or issue on the gluster side. HTH, Karthik> > Thanks in advance, > Dmitry > ________ > > Community Meeting Calendar: > > APAC Schedule - > Every 2nd and 4th Tuesday at 11:30 AM IST > Bridge: https://bluejeans.com/441850968 > > NA/EMEA Schedule - > Every 1st and 3rd Tuesday at 01:00 PM EDT > Bridge: https://bluejeans.com/441850968 > > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191202/638e3941/attachment.html>