Strahil Nikolov
2022-May-27 03:59 UTC
[Gluster-users] transport endpoint not connected on just 2 files
Check the file attributes on all bricks: getfattr -d -e hex -m. /data/brick/gw-runqueues/<path to file> Best Regards,Strahil Nikolov On Thu, May 26, 2022 at 16:05, Kingsley Tart<gluster at gluster.dogwind.com> wrote: Hi, I've got a strange issue where on all clients I've tested on (tested on 4) I have "transport endpoint is not connected" on two files in a directory, whereas other files can be read fine. Any ideas? On one of the servers (all same version): # gluster --version glusterfs 9.1 On one of the clients (same thing with all of them) - problem with files "gw3" and "gw11": [root at gw6 btl]# cd /mnt/runqueues/runners/ [root at gw6 runners]# ls -la ls: cannot access gw11: Transport endpoint is not connected ls: cannot access gw3: Transport endpoint is not connected total 8 drwxr-xr-x? 2 root root 4096 May 26 09:48 . drwxr-xr-x 13 root root 4096 Apr 12? 2021 .. -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw1 -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw10 -?????????? ? ?? ? ?? ? ? ?? ? ? ? ? ? ? gw11 -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw2 -?????????? ? ?? ? ?? ? ? ?? ? ? ? ? ? ? gw3 -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw4 -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw6 -rw-r--r--? 1 root root? ? 0 May 26 09:49 gw7 [root at gw6 runners]# cat * cat: gw11: Transport endpoint is not connected cat: gw3: Transport endpoint is not connected [root at gw6 runners]# Querying on a server shows those two problematic files: # gluster volume heal gw-runqueues info Brick gluster9a:/data/brick/gw-runqueues /runners /runners/gw11 /runners/gw3 Status: Connected Number of entries: 3 Brick gluster9b:/data/brick/gw-runqueues /runners /runners/gw11 /runners/gw3 Status: Connected Number of entries: 3 Brick gluster9c:/data/brick/gw-runqueues Status: Connected Number of entries: 0 However several hours later there's no obvious change. The servers have hardly any load and the volume is tiny. From a client: # find /mnt/runqueues | wc -l 35 glfsheal-gw-runqueues.log from server gluster9a: https://pastebin.com/7mPszBBM glfsheal-gw-runqueues.log from server gluster9b: https://pastebin.com/rxXs5Tcv Any pointers would be much appreciated! Cheers, Kingsley. ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220527/adcb00d9/attachment.html>
Kingsley Tart
2022-May-27 08:45 UTC
[Gluster-users] transport endpoint not connected on just 2 files
Hi, thanks. OK that's interesting. Picking one of the files, on bricks A and B I see this (and all of the values are identical between bricks A and B): trusted.afr.dirty=0x000000000000000000000000 trusted.afr.gw-runqueues-client-2=0x000000010000000200000000 trusted.gfid=0xa40bb83ff3784ae09c997d272296a7a9 trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733 trusted.glusterfs.mdata=0x01000000000000000000000000628ec57700000000007168bb00000000628ec576000000000000000000000000628ec5760000000000000000 and on brick C I see this: trusted.gfid=0xd73992aee03e4021824b1baced973df3 trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733 trusted.glusterfs.mdata=0x01000000000000000000000000628ec5230000000030136ca000000000628ec523000000000000000000000000628ec5230000000000000000 So brick C is missing the trusted.afr attributes and the trusted.gfid and mdata differ. What do I need to do to fix this? Cheers, Kingsley. On Fri, 2022-05-27 at 03:59 +0000, Strahil Nikolov wrote:> Check the file attributes on all bricks: > > getfattr -d -e hex -m. /data/brick/gw-runqueues/<path to file> > > > Best Regards, > Strahil Nikolov > > > On Thu, May 26, 2022 at 16:05, Kingsley Tart > > <gluster at gluster.dogwind.com> wrote: > > Hi, > > > > I've got a strange issue where on all clients I've tested on > > (tested on > > 4) I have "transport endpoint is not connected" on two files in a > > directory, whereas other files can be read fine. > > > > Any ideas? > > > > On one of the servers (all same version): > > > > # gluster --version > > glusterfs 9.1 > > > > On one of the clients (same thing with all of them) - problem with > > files "gw3" and "gw11": > > > > [root at gw6 btl]# cd /mnt/runqueues/runners/ > > [root at gw6 runners]# ls -la > > ls: cannot access gw11: Transport endpoint is not connected > > ls: cannot access gw3: Transport endpoint is not connected > > total 8 > > drwxr-xr-x 2 root root 4096 May 26 09:48 . > > drwxr-xr-x 13 root root 4096 Apr 12 2021 .. > > -rw-r--r-- 1 root root 0 May 26 09:49 gw1 > > -rw-r--r-- 1 root root 0 May 26 09:49 gw10 > > -????????? ? ? ? ? ? gw11 > > -rw-r--r-- 1 root root 0 May 26 09:49 gw2 > > -????????? ? ? ? ? ? gw3 > > -rw-r--r-- 1 root root 0 May 26 09:49 gw4 > > -rw-r--r-- 1 root root 0 May 26 09:49 gw6 > > -rw-r--r-- 1 root root 0 May 26 09:49 gw7 > > [root at gw6 runners]# cat * > > cat: gw11: Transport endpoint is not connected > > cat: gw3: Transport endpoint is not connected > > [root at gw6 runners]# > > > > > > Querying on a server shows those two problematic files: > > > > # gluster volume heal gw-runqueues info > > Brick gluster9a:/data/brick/gw-runqueues > > /runners > > /runners/gw11 > > /runners/gw3 > > Status: Connected > > Number of entries: 3 > > > > Brick gluster9b:/data/brick/gw-runqueues > > /runners > > /runners/gw11 > > /runners/gw3 > > Status: Connected > > Number of entries: 3 > > > > Brick gluster9c:/data/brick/gw-runqueues > > Status: Connected > > Number of entries: 0 > > > > > > However several hours later there's no obvious change. The servers > > have > > hardly any load and the volume is tiny. From a client: > > > > # find /mnt/runqueues | wc -l > > 35 > > > > > > glfsheal-gw-runqueues.log from server gluster9a: > > https://pastebin.com/7mPszBBM > > > > glfsheal-gw-runqueues.log from server gluster9b: > > https://pastebin.com/rxXs5Tcv > > > > > > Any pointers would be much appreciated! > > > > Cheers, > > Kingsley. > > > > ________ > > > > > > > > Community Meeting Calendar: > > > > Schedule - > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > Bridge: https://meet.google.com/cpu-eiue-hvk > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220527/06ceb0a1/attachment.html>