thr3ads.net - Gluster users - [Gluster-users] Healing completely loss file on replica 3 volume [Dec 2019]

If this information is useful, please help other people find it:
Share via:

Dmitry Antipov

2019-Nov-29 12:50 UTC

[Gluster-users] Healing completely loss file on replica 3 volume

I'm trying to manually garbage data on bricks (when the volume is
stopped) and then check whether healing is possible. For example:

Start:

# glusterd --debug

Bricks (on EXT4 mounted with 'rw,realtime'):

# mkdir /root/data0
# mkdir /root/data1
# mkdir /root/data2

Volume:

# gluster volume create gv0 replica 3 [local-ip]:/root/data0 
[local-ip]:/root/data1  [local-ip]:/root/data2 force
volume create: gv0: success: please start the volume to access data
# gluster volume start gv0
volume start: gv0: success

Mount:

# mkdir /mnt/gv0
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
WARNING: getfattr not found, certain checks will be skipped..

Create file:

# openssl rand 65536 > /mnt/gv0/64K
# md5sum /mnt/gv0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K

Umount and down the volume:

# umount /mnt/gv0
# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
y
volume stop: gv0: success

Check data on bricks:

# md5sum /root/data[012]/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data1/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data2/64K

Seems OK. Then garbage all:

# openssl rand 65536 > /root/data0/64K
# openssl rand 65536 > /root/data1/64K
# openssl rand 65536 > /root/data2/64K
# md5sum /root/data[012]/64K
c69096d15007578dab95d9940f89e167  /root/data0/64K
b85292fb60f1a1d27f1b0e3bc6bfdfae  /root/data1/64K
c2e90335cc2f600ddab5c53a992b2bb6  /root/data2/64K

Restart the volume and start full heal:

# gluster volume start gv0
volume start: gv0: success
# /usr/glusterfs/sbin/gluster volume heal gv0 full
Launching heal operation to perform full self heal on volume gv0 has been
successful
Use heal info commands to check status.

Finally:

# gluster volume heal gv0 info summary

Brick [local-ip]:/root/data0
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data1
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data2
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Since all 3 copies are different from each other, majority voting is useless
and data (IIUC) should be marked as split-brain at least. But I'm seeing
just
zeroes everywhere above. Why it is so?

Thanks in advance,
Dmitry

Strahil Nikolov

2019-Nov-30 15:13 UTC

head link

[Gluster-users] Healing completely loss file on replica 3 volume

Hi Dmitry,
keep in mind that gluster doesn't run md5sum (or any SHAsum) against the
local copies on the bricks , in order to verify your data.
The simplest said, it relies on entries in gluster's
"/brick/.glusterfs" folder and the metadata (getfattr -d -m . -e hex
<file-path-on-brick>.) differences in order to detect and fight split
brain.In your test , you have overwritten the contents of the file , but
metadata is the old one and also there are no entries
in?"/brick/.glusterfs" - as last operation was successful?(all bricks
were online).
Your test is not properly testing the software and as far as I know it's not
expecting that someone will mess up with all 3 bricks :D.

Best Regards,Strahil Nikolov
    ? ?????, 29 ??????? 2019 ?., 14:55:56 ?. ???????+2, Dmitry Antipov
<dmantipov at yandex.ru> ??????:
 
 I'm trying to manually garbage data on bricks (when the volume is
stopped) and then check whether healing is possible. For example:

Start:

# glusterd --debug

Bricks (on EXT4 mounted with 'rw,realtime'):

# mkdir /root/data0
# mkdir /root/data1
# mkdir /root/data2

Volume:

# gluster volume create gv0 replica 3 [local-ip]:/root/data0?
[local-ip]:/root/data1? [local-ip]:/root/data2 force
volume create: gv0: success: please start the volume to access data
# gluster volume start gv0
volume start: gv0: success

Mount:

# mkdir /mnt/gv0
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
WARNING: getfattr not found, certain checks will be skipped..

Create file:

# openssl rand 65536 > /mnt/gv0/64K
# md5sum /mnt/gv0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K

Umount and down the volume:

# umount /mnt/gv0
# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n)
y
volume stop: gv0: success

Check data on bricks:

# md5sum /root/data[012]/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data1/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9? /root/data2/64K

Seems OK. Then garbage all:

# openssl rand 65536 > /root/data0/64K
# openssl rand 65536 > /root/data1/64K
# openssl rand 65536 > /root/data2/64K
# md5sum /root/data[012]/64K
c69096d15007578dab95d9940f89e167? /root/data0/64K
b85292fb60f1a1d27f1b0e3bc6bfdfae? /root/data1/64K
c2e90335cc2f600ddab5c53a992b2bb6? /root/data2/64K

Restart the volume and start full heal:

# gluster volume start gv0
volume start: gv0: success
# /usr/glusterfs/sbin/gluster volume heal gv0 full
Launching heal operation to perform full self heal on volume gv0 has been
successful
Use heal info commands to check status.

Finally:

# gluster volume heal gv0 info summary

Brick [local-ip]:/root/data0
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data1
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data2
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Since all 3 copies are different from each other, majority voting is useless
and data (IIUC) should be marked as split-brain at least. But I'm seeing
just
zeroes everywhere above. Why it is so?

Thanks in advance,
Dmitry
________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191130/31144ca4/attachment-0001.html>

Karthik Subrahmanya

2019-Dec-02 06:20 UTC

head link

[Gluster-users] Healing completely loss file on replica 3 volume

Hi Dmitry,

Answers inline.

On Fri, Nov 29, 2019 at 6:26 PM Dmitry Antipov <dmantipov at yandex.ru>
wrote:
> I'm trying to manually garbage data on bricks
First of all changing data directly on the backend is not recommended and
is not supported.  All the operations needs to be done from the client
mount point.
Only few special cases needs changing few data about the file directly on
the backend.
> (when the volume is
> stopped) and then check whether healing is possible. For example:
>
> Start:
>
> # glusterd --debug
>
> Bricks (on EXT4 mounted with 'rw,realtime'):
>
> # mkdir /root/data0
> # mkdir /root/data1
> # mkdir /root/data2
>
> Volume:
>
> # gluster volume create gv0 replica 3 [local-ip]:/root/data0
> [local-ip]:/root/data1  [local-ip]:/root/data2 force
> volume create: gv0: success: please start the volume to access data
> # gluster volume start gv0
> volume start: gv0: success
>
> Mount:
>
> # mkdir /mnt/gv0
> # mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
> WARNING: getfattr not found, certain checks will be skipped..
>
> Create file:
>
> # openssl rand 65536 > /mnt/gv0/64K
> # md5sum /mnt/gv0/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K
>
> Umount and down the volume:
>
> # umount /mnt/gv0
> # gluster volume stop gv0
> Stopping volume will make its data inaccessible. Do you want to continue?
> (y/n) y
> volume stop: gv0: success
>
> Check data on bricks:
>
> # md5sum /root/data[012]/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data0/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data1/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data2/64K
>
> Seems OK. Then garbage all:
>
> # openssl rand 65536 > /root/data0/64K
> # openssl rand 65536 > /root/data1/64K
> # openssl rand 65536 > /root/data2/64K
> # md5sum /root/data[012]/64K
> c69096d15007578dab95d9940f89e167  /root/data0/64K
> b85292fb60f1a1d27f1b0e3bc6bfdfae  /root/data1/64K
> c2e90335cc2f600ddab5c53a992b2bb6  /root/data2/64K
>
> Restart the volume and start full heal:
>
> # gluster volume start gv0
> volume start: gv0: success
> # /usr/glusterfs/sbin/gluster volume heal gv0 full
> Launching heal operation to perform full self heal on volume gv0 has been
> successful
> Use heal info commands to check status.
>
> Finally:
>
> # gluster volume heal gv0 info summary
>
> Brick [local-ip]:/root/data0
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick [local-ip]:/root/data1
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick [local-ip]:/root/data2
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Since all 3 copies are different from each other, majority voting is
> useless
> and data (IIUC) should be marked as split-brain at least. But I'm
seeing
> just
> zeroes everywhere above. Why it is so?
>Since the data is changed directly on the backend, gluster will not be
knowing these changes. If the changes done from the client mount fails on
some bricks, only those will be recognized and marked by gluster so that it
can heal those when possible. Since this is a replica 3 volume and if you
end up in split-brain when you are doing the operations on the mount pint,
then that will be a bug. As far as this is considered it is not a bug or
issue on the gluster side.

HTH,
Karthik
>
> Thanks in advance,
> Dmitry
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191202/638e3941/attachment.html>

Gluster users - Dec 2019 - Healing completely loss file on replica 3 volume

[Gluster-users] Healing completely loss file on replica 3 volume

[Gluster-users] Healing completely loss file on replica 3 volume

[Gluster-users] Healing completely loss file on replica 3 volume