Hello ; I found many self-heal triggered log in every 10 minutes. Only one file , it 's gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f. Heal-failed and split-brain have not display anything. Does any problem in this file ? GlusterFS config: OS: 2.6.32-220.17.1.el6.x86_64 Scientific Linux release 6.2 (Carbon) # rpm -qa | grep glusterfs glusterfs-3.3.0-2.el6.x86_64 glusterfs-devel-3.3.0-2.el6.x86_64 glusterfs-fuse-3.3.0-2.el6.x86_64 glusterfs-geo-replication-3.3.0-2.el6.x86_64 glusterfs-rdma-3.3.0-2.el6.x86_64 glusterfs-server-3.3.0-2.el6.x86_64 glusterfs-debuginfo-3.3.0-2.el6.x86_64 gluster> volume info Volume Name: gvol1 Type: Distributed-Replicate Volume ID: a7d8ffdf-7296-404b-aeab-824ee853ec59 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 172.30.1.125:/export/data00 Brick2: 172.30.1.125:/export/data01 Brick3: 172.30.1.125:/export/data10 Brick4: 172.30.1.125:/export/data11 Options Reconfigured: features.limit-usage: /source:500GB features.quota: on performance.cache-refresh-timeout: 30 performance.io-thread-count: 32 nfs.disable: off cluster.min-free-disk: 5% performance.cache-size: 128MB gluster volume heal gvol1 info Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 1 /fs126/Graphite-monitor_vdb.qcow2 Brick 172.30.1.125:/export/data11 Number of entries: 1 /fs126/Graphite-monitor_vdb.qcow2 # gluster volume heal gvol1 info heal-failed Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 0 Brick 172.30.1.125:/export/data11 Number of entries: 0 gluster volume heal gvol1 info split-brain Heal operation on volume gvol1 has been successful Brick 172.30.1.125:/export/data00 Number of entries: 0 Brick 172.30.1.125:/export/data01 Number of entries: 0 Brick 172.30.1.125:/export/data10 Number of entries: 0 Brick 172.30.1.125:/export/data11 Number of entries: 0 Log detail: [2012-07-12 09:13:11.666417] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:13:11.666998] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:13:11.667132] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:13:11.667141] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:13:11.667589] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:13:11.668581] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.669322] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.669775] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:13:11.670408] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.770994] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:23:11.771358] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:23:11.771416] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:23:11.771898] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:23:11.772059] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:23:11.773074] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.773686] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.774094] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:23:11.774652] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.874474] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:33:11.874919] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:33:11.874978] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: d9b0c350-33ba-4090-ab08-f91f30dd661f: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:33:11.875505] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:33:11.875676] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:33:11.876613] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.877244] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.877646] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:33:11.878191] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.971727] I [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0: Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on gvol1-client-0 [2012-07-12 09:43:11.972004] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote operation failed: No such file or directory [2012-07-12 09:43:11.972066] E [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0: e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on gvol1-client-0 - No such file or directory [2012-07-12 09:43:11.972635] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:43:11.972924] I [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1: background data self-heal triggered. path: <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected pending operations [2012-07-12 09:43:11.973885] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.974451] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.974965] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no active sinks for performing self-heal on file <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> [2012-07-12 09:43:11.975772] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-gvol1-replicate-1: background data self-heal completed on <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f> -- Best Regards Homer Li
Pranith Kumar Karampuri
2012-Jul-12 04:35 UTC
[Gluster-users] One file incessant self heal
Homer,
Could you give the output of
getfattr -d -m . -e hex
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
getfattr -d -m . -e hex
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
and also 'stat' of these files.
Pranith.
----- Original Message -----
From: "Homer Li" <01jay.ly at gmail.com>
To: "gluster-users" <gluster-users at gluster.org>
Sent: Thursday, July 12, 2012 7:54:24 AM
Subject: [Gluster-users] One file incessant self heal
Hello ;
I found many self-heal triggered log in every 10 minutes.
Only one file , it 's gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f.
Heal-failed and split-brain have not display anything.
Does any problem in this file ?
GlusterFS config:
OS: 2.6.32-220.17.1.el6.x86_64 Scientific Linux release 6.2 (Carbon)
# rpm -qa | grep glusterfs
glusterfs-3.3.0-2.el6.x86_64
glusterfs-devel-3.3.0-2.el6.x86_64
glusterfs-fuse-3.3.0-2.el6.x86_64
glusterfs-geo-replication-3.3.0-2.el6.x86_64
glusterfs-rdma-3.3.0-2.el6.x86_64
glusterfs-server-3.3.0-2.el6.x86_64
glusterfs-debuginfo-3.3.0-2.el6.x86_64
gluster> volume info
Volume Name: gvol1
Type: Distributed-Replicate
Volume ID: a7d8ffdf-7296-404b-aeab-824ee853ec59
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 172.30.1.125:/export/data00
Brick2: 172.30.1.125:/export/data01
Brick3: 172.30.1.125:/export/data10
Brick4: 172.30.1.125:/export/data11
Options Reconfigured:
features.limit-usage: /source:500GB
features.quota: on
performance.cache-refresh-timeout: 30
performance.io-thread-count: 32
nfs.disable: off
cluster.min-free-disk: 5%
performance.cache-size: 128MB
gluster volume heal gvol1 info
Heal operation on volume gvol1 has been successful
Brick 172.30.1.125:/export/data00
Number of entries: 0
Brick 172.30.1.125:/export/data01
Number of entries: 0
Brick 172.30.1.125:/export/data10
Number of entries: 1
/fs126/Graphite-monitor_vdb.qcow2
Brick 172.30.1.125:/export/data11
Number of entries: 1
/fs126/Graphite-monitor_vdb.qcow2
# gluster volume heal gvol1 info heal-failed
Heal operation on volume gvol1 has been successful
Brick 172.30.1.125:/export/data00
Number of entries: 0
Brick 172.30.1.125:/export/data01
Number of entries: 0
Brick 172.30.1.125:/export/data10
Number of entries: 0
Brick 172.30.1.125:/export/data11
Number of entries: 0
gluster volume heal gvol1 info split-brain
Heal operation on volume gvol1 has been successful
Brick 172.30.1.125:/export/data00
Number of entries: 0
Brick 172.30.1.125:/export/data01
Number of entries: 0
Brick 172.30.1.125:/export/data10
Number of entries: 0
Brick 172.30.1.125:/export/data11
Number of entries: 0
Log detail:
[2012-07-12 09:13:11.666417] I
[afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
gvol1-client-0
[2012-07-12 09:13:11.666998] W
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
operation failed: No such file or directory
[2012-07-12 09:13:11.667132] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:13:11.667141] E
[afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
gvol1-client-0 - No such file or directory
[2012-07-12 09:13:11.667589] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:13:11.668581] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:13:11.669322] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:13:11.669775] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:13:11.670408] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:23:11.770994] I
[afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
gvol1-client-0
[2012-07-12 09:23:11.771358] W
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
operation failed: No such file or directory
[2012-07-12 09:23:11.771416] E
[afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
gvol1-client-0 - No such file or directory
[2012-07-12 09:23:11.771898] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:23:11.772059] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:23:11.773074] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:23:11.773686] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:23:11.774094] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:23:11.774652] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:33:11.874474] I
[afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
gvol1-client-0
[2012-07-12 09:33:11.874919] W
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
operation failed: No such file or directory
[2012-07-12 09:33:11.874978] E
[afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
d9b0c350-33ba-4090-ab08-f91f30dd661f: Failed to remove index on
gvol1-client-0 - No such file or directory
[2012-07-12 09:33:11.875505] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:33:11.875676] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:33:11.876613] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:33:11.877244] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:33:11.877646] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:33:11.878191] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:43:11.971727] I
[afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
gvol1-client-0
[2012-07-12 09:43:11.972004] W
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
operation failed: No such file or directory
[2012-07-12 09:43:11.972066] E
[afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
gvol1-client-0 - No such file or directory
[2012-07-12 09:43:11.972635] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:43:11.972924] I
[afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
background data self-heal triggered. path:
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
pending operations
[2012-07-12 09:43:11.973885] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:43:11.974451] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:43:11.974965] I
[afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
active sinks for performing self-heal on file
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
[2012-07-12 09:43:11.975772] I
[afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
0-gvol1-replicate-1: background data self-heal completed on
<gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
--
Best Regards
Homer Li
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Hi Pranith;
Thanks for your reply.
I checked md5sum too, it is different.
Here is my output:
# getfattr -d -m . -e hex
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
getfattr: Removing leading '/' from absolute path names
# file: export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
trusted.afr.gvol1-client-2=0x000003630000000000000000
trusted.afr.gvol1-client-3=0x000000010000000000000000
trusted.gfid=0xd9b0c35033ba4090ab08f91f30dd661f
trusted.glusterfs.quota.4111d3b4-7e06-483f-aae8-fbefe9e55843.contri=0x00000001c0d15000
# getfattr -d -m . -e hex
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
getfattr: Removing leading '/' from absolute path names
# file: export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
trusted.afr.gvol1-client-2=0x000000010000000000000000
trusted.afr.gvol1-client-3=0x000000010000000000000000
trusted.gfid=0xd9b0c35033ba4090ab08f91f30dd661f
trusted.glusterfs.quota.4111d3b4-7e06-483f-aae8-fbefe9e55843.contri=0x00000001c0d15000
#stat /export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
File:
`/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f'
Size: 7530086400 Blocks: 14706856 IO Block: 4096 regular file
Device: 6900h/26880d Inode: 81395728 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2012-06-21 09:58:59.242136421 +0800
Modify: 2012-07-10 13:42:04.381141510 +0800
Change: 2012-07-12 16:23:15.884163991 +0800
#stat /export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
File:
`/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f'
Size: 7530086400 Blocks: 14706856 IO Block: 4096 regular file
Device: 6910h/26896d Inode: 17956874 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2012-06-21 09:58:59.242136421 +0800
Modify: 2012-07-10 13:42:04.381141510 +0800
Change: 2012-07-12 16:23:15.885163872 +0800
#md5sum /export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
7fba61af476bf379c50f7429c89449ee
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
#md5sum /export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
9b08b7145c171afff863c4ae5884fa01
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
2012/7/12 Pranith Kumar Karampuri <pkarampu at
redhat.com>:> Homer,
> Could you give the output of
> getfattr -d -m . -e hex
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
> getfattr -d -m . -e hex
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
> and also 'stat' of these files.
>
> Pranith.
> ----- Original Message -----
> From: "Homer Li" <01jay.ly at gmail.com>
> To: "gluster-users" <gluster-users at gluster.org>
> Sent: Thursday, July 12, 2012 7:54:24 AM
> Subject: [Gluster-users] One file incessant self heal
>
> Hello ;
> I found many self-heal triggered log in every 10 minutes.
> Only one file , it 's gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f.
> Heal-failed and split-brain have not display anything.
> Does any problem in this file ?
>
>
> GlusterFS config:
>
> OS: 2.6.32-220.17.1.el6.x86_64 Scientific Linux release 6.2 (Carbon)
> # rpm -qa | grep glusterfs
> glusterfs-3.3.0-2.el6.x86_64
> glusterfs-devel-3.3.0-2.el6.x86_64
> glusterfs-fuse-3.3.0-2.el6.x86_64
> glusterfs-geo-replication-3.3.0-2.el6.x86_64
> glusterfs-rdma-3.3.0-2.el6.x86_64
> glusterfs-server-3.3.0-2.el6.x86_64
> glusterfs-debuginfo-3.3.0-2.el6.x86_64
>
> gluster> volume info
>
> Volume Name: gvol1
> Type: Distributed-Replicate
> Volume ID: a7d8ffdf-7296-404b-aeab-824ee853ec59
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: 172.30.1.125:/export/data00
> Brick2: 172.30.1.125:/export/data01
> Brick3: 172.30.1.125:/export/data10
> Brick4: 172.30.1.125:/export/data11
> Options Reconfigured:
> features.limit-usage: /source:500GB
> features.quota: on
> performance.cache-refresh-timeout: 30
> performance.io-thread-count: 32
> nfs.disable: off
> cluster.min-free-disk: 5%
> performance.cache-size: 128MB
>
>
> gluster volume heal gvol1 info
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 1
> /fs126/Graphite-monitor_vdb.qcow2
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 1
> /fs126/Graphite-monitor_vdb.qcow2
>
> # gluster volume heal gvol1 info heal-failed
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 0
>
> gluster volume heal gvol1 info split-brain
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 0
>
>
>
> Log detail:
> [2012-07-12 09:13:11.666417] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:13:11.666998] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:13:11.667132] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:13:11.667141] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:13:11.667589] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:13:11.668581] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.669322] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.669775] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.670408] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.770994] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:23:11.771358] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:23:11.771416] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:23:11.771898] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:23:11.772059] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:23:11.773074] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.773686] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.774094] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.774652] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.874474] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:33:11.874919] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:33:11.874978] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> d9b0c350-33ba-4090-ab08-f91f30dd661f: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:33:11.875505] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:33:11.875676] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:33:11.876613] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.877244] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.877646] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.878191] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.971727] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:43:11.972004] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:43:11.972066] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:43:11.972635] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:43:11.972924] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:43:11.973885] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.974451] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.974965] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.975772] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
>
> --
> Best Regards
> Homer Li
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
--
Best Regards
Homer Li