Ravishankar N
2018-Sep-28 11:57 UTC
[Gluster-users] sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt
+ gluster-users. Adding Karthik to see if he has some cycles to look into this. -Ravi On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:> > Hi, glusterfs expert > > When I test with glusterfs version 3.12.3 I find it quite often that > sometimes, there are entry remains in gluster volume heal > <vol-name>info output for long time, *it does not disappear until you > visit it from the mount point, is this normal*? > > [root at sn-0:/root] > > # gluster v heal services info > > Brick sn-0.local:/mnt/bricks/services/brick > > Status: Connected > > Number of entries: 0 > > Brick sn-1.local:/mnt/bricks/services/brick > > Status: Connected > > Number of entries: 0 > > Brick sn-2.local:/mnt/bricks/services/brick > > /fstest_88402c989256d6e39e50208c90c1e85d ?????????//this entry remains > in the output until you touch /mnt/services/ > fstest_88402c989256d6e39e50208c90c1e85d > > Status: Connected > > Number of entries: 1 > > [root at sn-0:/root] > > # ssh sn-2.local > > Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts. > > USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR > LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL > (fsclish) AND/OR LIMITED BASH SHELL. > > Read /opt/nokia/share/security/readme_root.txt for more details. > > [root at sn-2:/root] > > # cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/ > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # ls > > 9138e315-efd6-46e0-8a3a-db535078c781 > xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87 > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # getfattr -m . -d -e hex > /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ > > getfattr: Removing leading '/' from absolute path names > > # file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ > > trusted.afr.services-client-1=0x000000000000000100000001 > > trusted.gfid=0x9138e315efd646e08a3adb535078c781 > > trusted.glusterfs.dht=0x000000010000000000000000ffffffff > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # getfattr -m . -d -e hex > /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ > > getfattr: Removing leading '/' from absolute path names > > # file: > mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ > > trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5 > > ?fstest_88402c989256d6e39e50208c90c1e85d? is only seen from sn-2 mount > point and sn-2 service brick, there is no such entry if you ls > /mnt/services on sn-0 or sn-1. > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # cd /mnt/services/ > > [root at sn-2:/mnt/services] > > # ls > > backup db fstest_88402c989256d6e39e50208c90c1e85d? LCM NE3SAgent? > _nokrcpautoremoteuser? PM9? RCP_Backup SS_AlLightProcessor? SymptomDataUpl > > commoncollector EventCorrelationEngine? hypertracer ?????????Log? > netserv??? ODS??????????????????? ptp rcpha?????? SWM > > [root at sn-2:/mnt/services] >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180928/2bcffd8e/attachment.html>
Karthik Subrahmanya
2018-Sep-28 16:32 UTC
[Gluster-users] sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt
Hey, Please provide the glustershd log from all the nodes and client logs on the node from where you did the lookup on the file to resolve this issue. Regards, Karthik On Fri, Sep 28, 2018 at 5:27 PM Ravishankar N <ravishankar at redhat.com> wrote:> + gluster-users. > > Adding Karthik to see if he has some cycles to look into this. > > -Ravi > > On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote: > > Hi, glusterfs expert > > When I test with glusterfs version 3.12.3 I find it quite often that > sometimes, there are entry remains in gluster volume heal <vol-name>info > output for long time, *it does not disappear until you visit it from the > mount point, is this normal*? > > > > > > [root at sn-0:/root] > > # gluster v heal services info > > Brick sn-0.local:/mnt/bricks/services/brick > > Status: Connected > > Number of entries: 0 > > > > Brick sn-1.local:/mnt/bricks/services/brick > > Status: Connected > > Number of entries: 0 > > > > Brick sn-2.local:/mnt/bricks/services/brick > > /fstest_88402c989256d6e39e50208c90c1e85d //this entry remains in > the output until you touch /mnt/services/ > fstest_88402c989256d6e39e50208c90c1e85d > > Status: Connected > > Number of entries: 1 > > > > [root at sn-0:/root] > > # ssh sn-2.local > > Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts. > > > > USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR > LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) > AND/OR LIMITED BASH SHELL. > > > > Read /opt/nokia/share/security/readme_root.txt for more details. > > > > [root at sn-2:/root] > > # cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/ > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # ls > > 9138e315-efd6-46e0-8a3a-db535078c781 > xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87 > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # getfattr -m . -d -e hex > /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ > > getfattr: Removing leading '/' from absolute path names > > # file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ > > trusted.afr.services-client-1=0x000000000000000100000001 > > trusted.gfid=0x9138e315efd646e08a3adb535078c781 > > trusted.glusterfs.dht=0x000000010000000000000000ffffffff > > > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # getfattr -m . -d -e hex > /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ > > getfattr: Removing leading '/' from absolute path names > > # file: > mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ > > trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5 > > > > > > > > ?fstest_88402c989256d6e39e50208c90c1e85d? is only seen from sn-2 mount > point and sn-2 service brick, there is no such entry if you ls > /mnt/services on sn-0 or sn-1. > > [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop] > > # cd /mnt/services/ > > [root at sn-2:/mnt/services] > > # ls > > backup db > fstest_88402c989256d6e39e50208c90c1e85d LCM NE3SAgent > _nokrcpautoremoteuser PM9 RCP_Backup SS_AlLightProcessor SymptomDataUpl > > commoncollector EventCorrelationEngine hypertracer > Log netserv ODS ptp rcpha SWM > > [root at sn-2:/mnt/services] > > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180928/c7e955be/attachment.html>
Zhou, Cynthia (NSB - CN/Hangzhou)
2018-Oct-10 06:32 UTC
[Gluster-users] sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt
Hi, I meet this issue again, from my env on sn-1 mstate volume brick dir there is additional fstest_49e9d4767e95a3eda0a599437ef6ce8f, but on other sn nodes(sn-0, sn-2) there is no such directory. [root at sn-1:/mnt/bricks/mstate/brick] # stat fstest_49e9d4767e95a3eda0a599437ef6ce8f File: fstest_49e9d4767e95a3eda0a599437ef6ce8f Size: 4096 Blocks: 16 IO Block: 4096 directory Device: fd61h/64865d Inode: 45450 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-10-10 06:51:09.045660991 +0300 Modify: 2018-10-10 06:51:09.045660991 +0300 Change: 2018-10-10 07:06:05.060278140 +0300 Birth: - [root at sn-1:/mnt/bricks/mstate/brick] # gluster v heal mstate info Brick sn-0.local:/mnt/bricks/mstate/brick Status: Connected Number of entries: 0 Brick sn-2.local:/mnt/bricks/mstate/brick Status: Connected Number of entries: 0 Brick sn-1.local:/mnt/bricks/mstate/brick /fstest_49e9d4767e95a3eda0a599437ef6ce8f Status: Connected Number of entries: 1 [root at sn-1:/mnt/bricks/mstate/brick] # cd .glusterfs/indices/xattrop/ [root at sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop] # ls 0c40b296-a4d6-46c9-b7a5-037105de84aa xattrop-90daa317-04db-4704-9f34-6f571eb3691c [root at sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop] # cd ../../.. [root at sn-1:/mnt/bricks/mstate/brick] # getfattr -m . -d -e hex fstest_49e9d4767e95a3eda0a599437ef6ce8f # file: fstest_49e9d4767e95a3eda0a599437ef6ce8f trusted.afr.mstate-client-1=0x000000000000000100000001 trusted.gfid=0x0c40b296a4d646c9b7a5037105de84aa trusted.glusterfs.dht=0x000000010000000000000000ffffffff [root at sn-1:/mnt/bricks/mstate/brick] # Enclosed is the glustershd log on sn-1,from which I see following log related to fstest_49e9d4767e95a3eda0a599437ef6ce8f [2018-10-10 06:22:12.976487] I [MSGID: 108026] [afr-self-heald.c:341:afr_shd_selfheal] 0-mstate-replicate-0: entry: path /fstest_49e9d4767e95a3eda0a599437ef6ce8f, gfid: 0c40b296-a4d6-46c9-b7a5-037105de84aa //this is added in the entrance of function afr_shd_selfheal. From: Karthik Subrahmanya <ksubrahm at redhat.com> Sent: Saturday, September 29, 2018 12:33 AM To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com> Cc: gluster-users <gluster-users at gluster.org>; Ravishankar N <ravishankar at redhat.com> Subject: Re: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt Hey, Please provide the glustershd log from all the nodes and client logs on the node from where you did the lookup on the file to resolve this issue. Regards, Karthik On Fri, Sep 28, 2018 at 5:27 PM Ravishankar N <ravishankar at redhat.com<mailto:ravishankar at redhat.com>> wrote: + gluster-users. Adding Karthik to see if he has some cycles to look into this. -Ravi On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote: Hi, glusterfs expert When I test with glusterfs version 3.12.3 I find it quite often that sometimes, there are entry remains in gluster volume heal <vol-name>info output for long time, it does not disappear until you visit it from the mount point, is this normal? [root at sn-0:/root] # gluster v heal services info Brick sn-0.local:/mnt/bricks/services/brick Status: Connected Number of entries: 0 Brick sn-1.local:/mnt/bricks/services/brick Status: Connected Number of entries: 0 Brick sn-2.local:/mnt/bricks/services/brick /fstest_88402c989256d6e39e50208c90c1e85d //this entry remains in the output until you touch /mnt/services/ fstest_88402c989256d6e39e50208c90c1e85d Status: Connected Number of entries: 1 [root at sn-0:/root] # ssh sn-2.local Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts. USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL. Read /opt/nokia/share/security/readme_root.txt for more details. [root at sn-2:/root] # cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/ [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>] # ls 9138e315-efd6-46e0-8a3a-db535078c781 xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87 [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>] # getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ getfattr: Removing leading '/' from absolute path names # file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/ trusted.afr.services-client-1=0x000000000000000100000001 trusted.gfid=0x9138e315efd646e08a3adb535078c781 trusted.glusterfs.dht=0x000000010000000000000000ffffffff [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>] # getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ getfattr: Removing leading '/' from absolute path names # file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/ trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5 ?fstest_88402c989256d6e39e50208c90c1e85d? is only seen from sn-2 mount point and sn-2 service brick, there is no such entry if you ls /mnt/services on sn-0 or sn-1. [root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:root at sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>] # cd /mnt/services/ [root at sn-2:/mnt/services] # ls backup db fstest_88402c989256d6e39e50208c90c1e85d LCM NE3SAgent _nokrcpautoremoteuser PM9 RCP_Backup SS_AlLightProcessor SymptomDataUpl commoncollector EventCorrelationEngine hypertracer Log netserv ODS ptp rcpha SWM [root at sn-2:/mnt/services] -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181010/09a51bcf/attachment-0001.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glustershd-sn1.zip Type: application/x-zip-compressed Size: 274338 bytes Desc: glustershd-sn1.zip URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181010/09a51bcf/attachment-0001.bin>