thr3ads.net - Gluster users - [Gluster-users] Long post about gluster issue after 6.5 to 6.6 upgrade and recovery steps [Nov 2019]

If this information is useful, please help other people find it:
Share via:
Strahil Nikolov
2019-Nov-10 12:12 UTC
[Gluster-users] Long post about gluster issue after 6.5 to 6.6 upgrade and recovery steps

Hello Colleagues,
I want to share my experience and especially how I have recovered after a
situation where gluster refuses to heal several files of my oVirt Lab.
Most probably the situation was caused by the fact that I didn't check if
all files were healed before I started the upgrade on the next node , which in a
'replica 2 arbiter1' setup has caused multiple files missing/conflicting
and heal fails to happen.
Background:1. I have powered off my HostedEngine VM and made a snapshot of the
gluster volume2. Started the update , but without screen - I messed it up and
decided to revert from the snapshot3. Powered off the volume , restored from
snapshot and then started (and again snapshoted the volume)4. Upgrade of the
HostedEngine was successfull5. Upgraded the arbiter (ovirt3)6. I forgot to check
the heal status on the arbiter and upgraded ovirt1/gluster1 (which maybe was the
reason for the issue)7. After gluster1 was healed I saw that some files are left
for healing , but expected it will finish till ovirt2/gluster2 is patched8.
Sadly my assumption was not right and after the reboot of ovirt2/gluster2 I
noticed that some files never heal.
Symptoms:2 files per volume never heal (used 'full' mode) even after I
'stat'-ed every file/dir in the volume.Ovirt Dashboard reported multiple
errors (50+) that it cannot update the OVF metadata for volume/VM.
Here are my notes while I was recovering from the situation. As this is my LAB,
I shutdown all VMs (including the HostedEngine) as downtime was not an issue:
Heals never complete:

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
<gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8> ?
<gfid:95bc2cd2-8a1e-464e-a384-6b128780d370> ?
Status: Connected
Number of entries: 2

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
<gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8> ?
<gfid:95bc2cd2-8a1e-464e-a384-6b128780d370> ?
Status: Connected
Number of entries: 2


Mount to get the gfid to path relationship:
mount -t glusterfs -o aux-gfid-mount gluster1:/data_fast4 /mnt

# getfattr -n trusted.glusterfs.pathinfo -e text
/mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
getfattr: Removing leading '/' from absolute path names
# file: mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
 trusted.glusterfs.pathinfo="(<REPLICATE:data_fast4-replicate-0>
<POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt2.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a651
2-eaf6-4859-90cf-eeef2cc0cab8>
<POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt3.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8>)"

The local brick (that is supposed to be healed) is missing some data:

# ls -l
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
ls: ???? ?????? ??
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8:
???? ????? ???? ??? ??????????

Remote is OK:
# ssh gluster2 'ls -l
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
-rw-r--r--. 2 vdsm kvm 436 ?9 ??? 19,34
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8

Arbiter is also OK:
# ssh ovirt3 'ls -l
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
-rw-r--r--. 2 vdsm kvm 0 ?9 ??? 19,34
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8


Rsync the file/directory from a good brick to broken one:

# rsync -avP gluster2:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/
# ls -l
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
-rw-r--r--. 1 vdsm kvm 436 ?9 ??? 19,34
/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8

After a full heal we see our problematic file:

# gluster volume heal data_fast4 full
Launching heal operation to perform full self heal on volume data_fast4 has been
successful ?
Use heal info commands to check status.

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
?
Status: Connected
Number of entries: 1

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
?
Status: Connected
Number of entries: 1


This time the file is missing on both gluster1 (older version) and arbiter:

# cat
/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
CTIME=1558265783
DESCRIPTION={"Updated":true,"Size":256000,"Last
Updated":"Sat Nov 09 19:24:08 EET 2019","Storage
Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk
Description":"OVF_STORE"}
DISKTYPE=OVFS
DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
FORMAT=RAW
GEN=0
IMAGE=58e197a6-12df-4432-a643-298d40e44130
LEGALITY=LEGAL
MTIME=0
PUUID=00000000-0000-0000-0000-000000000000
SIZE=262144
TYPE=PREALLOCATED
VOLTYPE=LEAF
EOF

# ssh gluster2 cat
/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
CTIME=1558265783
DESCRIPTION={"Updated":true,"Size":256000,"Last
Updated":"Sat Nov 09 19:34:33 EET 2019","Storage
Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk
Description":"OVF_STORE"}
DISKTYPE=OVFS
DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
FORMAT=RAW
GEN=0
IMAGE=58e197a6-12df-4432-a643-298d40e44130
LEGALITY=LEGAL
MTIME=0
PUUID=00000000-0000-0000-0000-000000000000
SIZE=262144
TYPE=PREALLOCATED
VOLTYPE=LEAF
EOF

# ssh ovirt3 cat
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
cat:
/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta:
???? ????? ???? ??? ??????????

As we miss the same file on both a brick and arbiter we take another approach:
First remove on gluster1 the file to another name:

# mv
/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
/gluster_bricks/data_fast4
/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta_old

Then we copy the file from gluster2 (good brick)
# rsync -avP
gluster2:/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
/gluster_
bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
receiving incremental file list
535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
 ???????????436 100% ?425.78kB/s ???0:00:00 (xfr#1, to-chk=0/1)

sent 43 bytes ?received 571 bytes ?1,228.00 bytes/sec
total size is 436 ?speedup is 0.71

Run full heal:
# gluster volume heal data_fast4 full
Launching heal operation to perform full self heal on volume data_fast4 has been
successful ?
Use heal info commands to check status

# gluster volume heal data_fast4 info
Brick gluster1:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick gluster2:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0

Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
Status: Connected
Number of entries: 0


And ofcourse umount /mnt

I did the above for all oVirt Storage domains and rebooted all nodes
(simultaneously) after stopping the whole stack. This one should not be
neccessary , but I wanted to be sure that after power outage the cluster will be
operational again:
systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd supervdsmd sanlock
glusterd/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh

Verification:After the reboot , I have tried to set each oVirt storage domain in
'Maintenance' which confirms that engine can update the OVMF meta and
then set it back to Active. Without downtime , this will not be possible.

I hope this long post will help anyone .
PS: I have collected some data for some of the files, that I have ommited as
this e-mail is very long.
Best Regards,Strahil Nikolov


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191110/3aabd32b/attachment.html>
Gluster users - Nov 2019 - Long post about gluster issue after 6.5 to 6.6 upgrade and recovery steps

[Gluster-users] Long post about gluster issue after 6.5 to 6.6 upgrade and recovery steps
