thr3ads.net - Gluster users - [Gluster-users] Determine og force healing occurring [Jul 2016]

If this information is useful, please help other people find it:
Share via:

Pranith Kumar Karampuri

2016-Jul-16 02:10 UTC

[Gluster-users] Determine og force healing occurring

On Fri, Jul 15, 2016 at 5:20 PM, Jesper Led Lauridsen TS Infra server <
JLY at dr.dk> wrote:
> Hi,
>
> How do I determine in which log etc. that a healing is in progress or
> startet and how do I if not startet force it.
>
> Additional info, is that I have some problem with a volume if I execute
> 'gluster volume heal <volume> info' the command just hangs
but if I
> execute  'gluster volume heal <volume> info split-brain' it
return that no
> file are in split-brain. Yet there is and I have successfully recovered
> another one.
>
If the command hangs there is a chance that operations on the file may have
lead to stale locks. Could you give the output of statedump?
You can follow
https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ to
generat the files.

>
> I just have problem with this one. I can determine if there is a healing
> process running or not
>
> I have change 'trusted.afr.glu_rhevtst_dr2_data_01-client-1' to
> 0x000000000000000000000000 on the file located on glustertst03 and executed
> a 'ls -lrt' on the file on the gluster-mount.
>
> [root at glustertst04 ]# getfattr -d -m . -e hex
>
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
> getfattr: Removing leading '/' from absolute path names
> # file:
>
bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
> security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x00004c700000000000000000
> trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
> trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>
>
trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x00000001606dc000
> trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>
>  [root at glustertst03 ]# getfattr -d -m . -e hex
>
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
> getfattr: Removing leading '/' from absolute path names
> # file:
>
bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
> security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
> trusted.afr.dirty=0x000000270000000000000000
> trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x000000000000000000000000
> trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
> trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>
>
trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x0000000160662000
> trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>
> [root at glustertst04 ]# stat
>
/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>   File:
>
`/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>   Size: 21474836480     Blocks: 11548384   IO Block: 131072 regular file
> Device: 31h/49d Inode: 11517990069246079393  Links: 1
> Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)
> Access: 2016-07-15 13:33:47.860224289 +0200
> Modify: 2016-07-15 13:34:44.396125458 +0200
> Change: 2016-07-15 13:34:44.397125492 +0200
>
> [root at glustertst03 ]# stat
>
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>   File:
>
`/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>   Size: 21474836480     Blocks: 11547408   IO Block: 4096   regular file
> Device: fd02h/64770d    Inode: 159515      Links: 2
> Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)
> Access: 2016-07-13 08:33:00.000561984 +0200
> Modify: 2016-07-13 08:32:59.969561154 +0200
> Change: 2016-07-15 12:52:28.414192052 +0200
>
> Thanks
> Jesper
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160716/73eb919e/attachment.html>

Jesper Led Lauridsen TS Infra server

2016-Jul-16 16:23 UTC

head link

[Gluster-users] Determine og force healing occurring

On 07/16/2016 04:10 AM, Pranith Kumar Karampuri wrote:>
>
> On Fri, Jul 15, 2016 at 5:20 PM, Jesper Led Lauridsen TS Infra server
> <JLY at dr.dk <mailto:JLY at dr.dk>> wrote:
>
>     Hi,
>
>     How do I determine in which log etc. that a healing is in progress
>     or startet and how do I if not startet force it.
>
>     Additional info, is that I have some problem with a volume if I
>     execute 'gluster volume heal <volume> info' the command
just hangs
>     but if I execute  'gluster volume heal <volume> info
split-brain'
>     it return that no file are in split-brain. Yet there is and I have
>     successfully recovered another one.
>
>
> If the command hangs there is a chance that operations on the file may
> have lead to stale locks. Could you give the output of statedump?
> You can follow
> https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ to
> generat the files.
Thanks for you response. You are right there was a stale lock. But I am
sorry I booted all my cluster nodes, so I guess (without knowing) that
there is no reason to give you the output of a statedump?

What I can confirm and give of information is:
  * All the servers failed to reboot so I had to push the button. They
all failed with the message
     "Unmounting pipe file system: Cannot create link /etc/mtab~
     Perhaps there is a stale lock file?"
  * After 2 nodes had rebooted the command executed without any problem
and reported a couple off split-brain (Both Directory and Files)
  * strace the command showed that it was just looping, so basically the
command didn't hanging. It just couldn't finish.
  * I am using "glusterfs-3.6.2-1.el6.x86_64". But hoping to upgrade
to
3.6.9 this weekend.
  * The file I refereed to here. Now has the same output on both
replicas when getting getfattr information. The
grusted.afr.glu_rhevtst_dr2_data_01-client-[0,1] and trusted.afr.dirty
are now all zero
 >
>
>     I just have problem with this one. I can determine if there is a
>     healing process running or not
>
>     I have change 'trusted.afr.glu_rhevtst_dr2_data_01-client-1' to
>     0x000000000000000000000000 on the file located on glustertst03 and
>     executed a 'ls -lrt' on the file on the gluster-mount.
>
>     [root at glustertst04 ]# getfattr -d -m . -e hex
>    
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>     getfattr: Removing leading '/' from absolute path names
>     # file:
>    
bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>    
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
>     trusted.afr.dirty=0x000000000000000000000000
>     trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x00004c700000000000000000
>     trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
>     trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>    
trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x00000001606dc000
>     trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>
>      [root at glustertst03 ]# getfattr -d -m . -e hex
>    
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>     getfattr: Removing leading '/' from absolute path names
>     # file:
>    
bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>    
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
>     trusted.afr.dirty=0x000000270000000000000000
>     trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x000000000000000000000000
>     trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
>     trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>    
trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x0000000160662000
>     trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>
>     [root at glustertst04 ]# stat
>    
/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>       File:
>    
`/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>       Size: 21474836480     Blocks: 11548384   IO Block: 131072
>     regular file
>     Device: 31h/49d Inode: 11517990069246079393  Links: 1
>     Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/   
>      kvm)
>     Access: 2016-07-15 13:33:47.860224289 +0200
>     Modify: 2016-07-15 13:34:44.396125458 +0200
>     Change: 2016-07-15 13:34:44.397125492 +0200
>
>     [root at glustertst03 ]# stat
>    
/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>       File:
>    
`/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>       Size: 21474836480     Blocks: 11547408   IO Block: 4096 
>      regular file
>     Device: fd02h/64770d    Inode: 159515      Links: 2
>     Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/   
>      kvm)
>     Access: 2016-07-13 08:33:00.000561984 +0200
>     Modify: 2016-07-13 08:32:59.969561154 +0200
>     Change: 2016-07-15 12:52:28.414192052 +0200
>
>     Thanks
>     Jesper
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> -- 
> Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160716/8206d8d2/attachment.html>

Gluster users - Jul 2016 - Determine og force healing occurring

[Gluster-users] Determine og force healing occurring

[Gluster-users] Determine og force healing occurring