Strahil Nikolov
2019-Nov-13 20:23 UTC
[Gluster-users] Gluster v6.6 replica 2 arbiter 1 - gfid on arbiter is different
Hello Community, I had a power issue last night and all the systems went down. When they powered up the arbiter was offline (network issue) which was resolved and heal passed for almost all files but 2 per volume. The strange thing is that the arbiter's copy is older and the 'trusted.gfid' is different on the arbiter. In which cases this can happen ? I guess the heal is not passing ,as the 'trusted.gfid' is different while 'trusted.gfid2path.<long_string>' is the same. Here is the output of getfattr and I need to realise if this is a possible situation or it is a bug that needs reporting: Brick1:[root at ovirt1 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b b2.meta getfattr: Removing leading '/' from absolute path names # file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.data_fast-client-2=0x000000020000000200000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x2408e73df137431e99c663783b8bb356 trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746 1 trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12 trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000 Brick2:[root at ovirt2 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b b2.meta getfattr: Removing leading '/' from absolute path names # file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.data_fast-client-2=0x000000020000000200000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x2408e73df137431e99c663783b8bb356 trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746 1 trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12 trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000 Arbiter:[root at ovirt3 ~]# getfattr -d -m . -e hex /gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b b2.meta getfattr: Removing leading '/' from absolute path names # file: gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x48c9c4ebe75c41f7ac45ea4943e9a65a trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746 1 trusted.glusterfs.mdata=0x010000000000000000000000005dcb3a520000000011c7ae2c000000005dcb3a520000000011690f7b000000005dcb3a520000000011690f7b trusted.glusterfs.shard.block-size=0x0000000004000000 trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000 Also, as per 'Troubleshooting Split-Brains - Gluster Docs' there should be another 2 copies of the changelog (local + remote bricks) but in my case they are missing. Why is it like that ? "Each file in a brick maintains the changelog of itself and that of the files present in all the other bricks in its replica set as seen by that brick." | | | | Troubleshooting Split-Brains - Gluster Docs Project documentation for Gluster Filesystem | | | Thanks in advance for your response. Best Regards,Strahil Nikolov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191113/3516aafe/attachment.html>
Strahil Nikolov
2019-Nov-14 00:50 UTC
[Gluster-users] Gluster v6.6 replica 2 arbiter 1 - gfid on arbiter is different
It seems that whenever I reboot a gluster node , I got this problem - so
it's not an arbiter issue.Obviously there is something wrong with v6.6 ,as I
never had such issues with v6.5 .
Any ideas where should I start this up ?
Best Regards,Strahil Nikolov
? ?????, 13 ??????? 2019 ?., 22:23:38 ?. ???????+2, Strahil Nikolov
<hunter86_bg at yahoo.com> ??????:
Hello Community,
I had a power issue last night and all the systems went down. When they powered
up the arbiter was offline (network issue) which was resolved and heal passed
for almost all files but 2 per volume.
The strange thing is that the arbiter's copy is older and the
'trusted.gfid' is different on the arbiter. In which cases this can
happen ?
I guess the heal is not passing ,as the 'trusted.gfid' is different
while 'trusted.gfid2path.<long_string>' is the same.
Here is the output of getfattr and I need to realise if this is a possible
situation or it is a bug that needs reporting:
Brick1:[root at ovirt1 ~]# getfattr -d -m . -e hex
/gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.data_fast-client-2=0x000000020000000200000000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2408e73df137431e99c663783b8bb356
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000
Brick2:[root at ovirt2 ~]# getfattr -d -m . -e hex
/gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.data_fast-client-2=0x000000020000000200000000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x2408e73df137431e99c663783b8bb356
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcc1eae0000000023c985d9000000005dcc1eae000000001f4c18bf000000005dcc1eae000000001ed22d12
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000
Arbiter:[root at ovirt3 ~]# getfattr -d -m . -e hex
/gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673b
b2.meta
getfattr: Removing leading '/' from absolute path names
# file:
gluster_bricks/data_fast/data_fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/25107759-f56b-4d94-872e-0f9b145886d1/571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48c9c4ebe75c41f7ac45ea4943e9a65a
trusted.gfid2path.27d407743ed6191b=0x35333430653763322d666333302d343461382d613633392d3762613138636138653033362f35373161356562342d656239322d343739632d383330352d3739656161303637336262322e6d65746
1
trusted.glusterfs.mdata=0x010000000000000000000000005dcb3a520000000011c7ae2c000000005dcb3a520000000011690f7b000000005dcb3a520000000011690f7b
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-size=0x00000000000001b4000000000000000000000000000000010000000000000000
Also, as per 'Troubleshooting Split-Brains - Gluster Docs' there should
be another 2 copies of the changelog (local + remote bricks) but in my case they
are missing. Why is it like that ?
"Each file in a brick maintains the changelog of itself and that of the
files present in all the other bricks in its replica set as seen by that
brick."
|
|
| |
Troubleshooting Split-Brains - Gluster Docs
Project documentation for Gluster Filesystem
|
|
|
Thanks in advance for your response.
Best Regards,Strahil Nikolov
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20191114/b13fc44a/attachment.html>