Hi Amudhan, I have gone through the logs provided by you but I have not seen anything significant which may lead to this issue. I also doubt the logs you have provided in attachments and in mail. In your first mail you were talking about the folder with path " /Packages/Others/MPEG/C/main folder/ sub folder/file " While as in logs, I can see tat the path is "media/disk33/brick33/mainfolder" getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33/mainfolder trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0xa7b38ad363424f60830ba7090e88101f Please confirm if the folder you mentioned is same on both the places. Next, could you also provide the output of the mount command? We would like to check the option you have used to mount the volume. So please give the output of "mount" command. In addition to this I would also like following information/logs - 1 - statedump of mount and brick processes. Please follow this link to know how can we collect state dump - https://github.com/gluster/glusterfs/blob/master/doc/debugging/statedump.md 2 - gluster v info of the volume you are using. 3 - gluster v heal <volname> info - Just to check if the mainfolder is under healing or not. 4 - See if unmount and mount solves this issue? But before doing this get all the information mentioned above. ----- Ashish ----- Original Message ----- From: "Amudhan P" <AmudhanP at realimage.com> To: "Ashish Pandey" <aspandey at redhat.com>, "Pranith Kumar Karampuri" <pkarampu at redhat.com> Cc: "Mohamed Pakkeer" <Pakkeer.Mohideen at realimage.com> Sent: Saturday, October 15, 2016 2:35:23 PM Subject: RE: files not listing from mounpoint Hi Ashish, Attached attachment. Regards Amudhan P From: Ashish Pandey [mailto:aspandey at redhat.com] Sent: Saturday, October 15, 2016 2:19 PM To: Pranith Kumar Karampuri Cc: Amudhan P; Mohamed Pakkeer Subject: Re: files not listing from mounpoint Pranith, I can not find the attachment. Ashish ----- Original Message ----- From: "Pranith Kumar Karampuri" < pkarampu at redhat.com > To: "Amudhan P" < AmudhanP at realimage.com > Cc: "Mohamed Pakkeer" < Pakkeer.Mohideen at realimage.com >, "Ashish Pandey" < aspandey at redhat.com > Sent: Saturday, October 15, 2016 10:36:26 AM Subject: Re: files not listing from mounpoint hi Ashish, Could you help Amudhan? On Thu, Oct 13, 2016 at 12:19 PM, Amudhan P < AmudhanP at realimage.com > wrote: Hi Pranith, Waiting for your reply. Hope you have received attachments which I have sent with my last email. Regards Amudhan P From: Amudhan P Sent: Wednesday, October 05, 2016 4:52 PM To: 'Pranith Kumar Karampuri' Cc: Mohamed Pakkeer Subject: RE: files not listing from mounpoint Hi Pranith, I have attached txt file containing xattr output from all nodes. Regards Amudhan P From: Pranith Kumar Karampuri [ mailto:pkarampu at redhat.com ] Sent: Monday, October 03, 2016 10:12 PM To: Amudhan P Cc: Mohamed Pakkeer Subject: Re: files not listing from mounpoint hi Amudhan, Sorry for the delay in responding. I need the xattr output from all nodes. Please expect some delays in response. There is gluster summit in Berlin so we are all in travels. On Fri, Sep 30, 2016 at 8:27 PM, Amudhan P < AmudhanP at realimage.com > wrote: Hi Pranith, Please find data taken from getfattr of file and bricks from one of the node. Let me know if you require more data. (main folder) root at node:~# getfattr -m. -e hex -d media/disk33/brick33/mainfolder/ getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33/mainfolder trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0xa7b38ad363424f60830ba7090e88101f (subfolder) root at node:~# getfattr -m. -e hex -d media/disk33/brick33/mainfolder/subfolder/ getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33/mainfolder/subfolder trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0x31a58962983b4f9ea1318b1c37f04370 (file) root at node:~# getfattr -m. -e hex -d media/disk33/brick33/mainfolder/subfolder/file1 getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33/mainfolder/subfolder/file1 trusted.bit-rot.version=0x0200000000000000574438b60001924f trusted.ec.config=0x0000080a02000200 trusted.ec.size=0x00000000000027b9 trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0xbd8c028e07144f6592c3fc410f16d55b (file residing brick) root at node:~# getfattr -m. -e hex -d /media/disk33/brick33 getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000009097b3fb92f68491 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 (all brick getfattr of same node) # file: media/disk1/brick1 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000212f6842238e38d8 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk2/brick2 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000097b425c09a12f656 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk3/brick3 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000025ed0970284bda06 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk4/brick4 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000d097b3e8d2f6847e trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk5/brick5 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000009ed09785a12f681b trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk6/brick6 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000002d097b352f684bcb trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk7/brick7 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000a5ed094aa84bd9e0 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk8/brick8 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000ded09772e12f6808 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk9/brick9 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000bb425e99bda12f2f trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk10/brick10 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000038e38e283b425ebe trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk11/brick11 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000003b425ebf3da12f55 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk12/brick12 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000e38e38a0e5ed0936 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk13/brick13 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000006aaaaa8b6d097b21 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk14/brick14 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000425ed08444bda11a trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk15/brick15 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000b425ecd4b684bd6a trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk16/brick16 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000006d097b226f684bb8 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk17/brick17 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000497b42494bda12df trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk18/brick18 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000ad097b0faf684ba5 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk19/brick19 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000005097b40e52f684a4 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk20/brick20 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000052f684a55555553b trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk21/brick21 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000005555553c57b425d2 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk22/brick22 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000057b425d35a12f669 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk23/brick23 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000fb425e86fda12f1c trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk24/brick24 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000009555552997b425bf trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk25/brick25 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000005ed09798612f682e trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk26/brick26 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000e5ed0937e84bd9cd trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk27/brick27 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000897b42368bda12cc trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk28/brick28 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x000000010000000065ed095d684bd9f3 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk29/brick29 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000f8e38deffb425e85 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk30/brick30 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000fda12f1dffffffff trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk31/brick31 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x0000000100000000025ed09704bda12d trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk32/brick32 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007425ece77684bd7d trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk33/brick33 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000009097b3fb92f68491 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk34/brick34 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000000bda12f30e38e389 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk35/brick35 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000007b425eac7da12f42 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 getfattr: Removing leading '/' from absolute path names # file: media/disk36/brick36 trusted.ec.version=0x0000000000000006000000000000007f trusted.gfid=0x00000000000000000000000000000001 trusted.glusterfs.dht=0x00000001000000000e38e38a1097b420 trusted.glusterfs.dht.commithash=0x3239353631353231343000 trusted.glusterfs.volume-id=0x2b575b5cdf2e449cabb9c56cec27e609 regards Amudhan P From: Pranith Kumar Karampuri [mailto: pkarampu at redhat.com ] Sent: Friday, September 30, 2016 5:15 PM To: Amudhan P Cc: Mohamed Pakkeer Subject: Re: files not listing from mounpoint hi Amudhan, For the directory where you are seeing this problem, could you get the output of "getfattr -d -m. -e hex <brick>' from all the bricks? hey Mohammed Pakkeer!! How are you man :-). Long time no see? How is everything with gluster? Will it be possible for you to blog about your setup? I want to gather people who are using EC and gather blogs about what your usecase is, how gluster helped what is the size of the volume etc. On Fri, Sep 30, 2016 at 4:56 PM, Amudhan P < AmudhanP at realimage.com > wrote: Hi Pranith, We are using glusterfs 3.7.8 for our cluster. Cluster is configured with distributed disperse volume. Cluster Config: EC 108 x (8 + 2) = 1080. We are having problem in listing data from a specific folder from the mount point. When I use ?ls? cmd in 'main folder' it is showing empty folder from mount point, but I can see subfolders and files inside the 'main folder' in bricks. Folder path: /Packages/Others/MPEG/C/main folder/ sub folder/file How can we fix this issue?. Adding getfattr output of file, subfolder & main folder. command used for output "getfattr -m. -e hex -d" # file: file (one of the file) trusted.bit-rot.version=0x0200000000000000574438af0003a07c trusted.ec.config=0x0000080a02000200 trusted.ec.size=0x00000000000027b9 trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0xbd8c028e07144f6592c3fc410f16d55b # file: sub folder/ trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0x31a58962983b4f9ea1318b1c37f04370 # file: main folder trusted.ec.version=0x00000000000000010000000000000002 trusted.gfid=0xa7b38ad363424f60830ba7090e88101f Log output generated in client when issuing 'ls' command. [2016-09-14 08:08:45.084197] W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 0-glsvol-disperse-13: Mismatching xdata in answers of 'LOOKUP' [2016-09-14 08:08:45.084353] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-glsvol-disperse-13: Operation failed on some subvolumes (up=3FF, mask=3FF, remaining=0, good=3FD, bad=2) [2016-09-14 08:08:45.085902] I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 0-glsvol-disperse-13: /Packages/Others: name heal successful on 3FF [2016-09-14 08:08:45.096858] W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 0-glsvol-disperse-13: Mismatching xdata in answers of 'LOOKUP' [2016-09-14 08:08:45.096971] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-glsvol-disperse-13: Operation failed on some subvolumes (up=3FF, mask=3FF, remaining=0, good=3FD, bad=2) [2016-09-14 08:08:45.098737] I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 0-glsvol-disperse-13: /Packages/Others/MPEG: name heal successful on 3FF [2016-09-14 08:08:45.109555] W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 0-glsvol-disperse-13: Mismatching xdata in answers of 'LOOKUP' [2016-09-14 08:08:45.109628] W [MSGID: 122056] [ec-combine.c:866:ec_combine_check] 0-glsvol-disperse-13: Mismatching xdata in answers of 'LOOKUP' [2016-09-14 08:08:45.109776] W [MSGID: 122053] [ec-common.c:116:ec_check_status] 0-glsvol-disperse-13: Operation failed on some subvolumes (up=3FF, mask=3FF, remaining=0, good=3FD, bad=2) [2016-09-14 08:08:45.111792] I [MSGID: 122058] [ec-heal.c:2338:ec_heal_do] 0-glsvol-disperse-13: /Packages/Others/MPEG/C: name heal successful on 3FF please let me know if you need any further data. Regards Amudhan P -- Pranith -- Pranith -- Pranith -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161018/72766ce8/attachment.html>