thr3ads.net - Gluster users - [Gluster-users] Problem with files showing zero links [Jan 2016]

If this information is useful, please help other people find it:
Share via:

Joe Julian

2016-Jan-28 18:28 UTC

[Gluster-users] Problem with files showing zero links

Response inline

On 01/28/2016 07:28 AM, Ronny Adsetts wrote:> Hi all,
>
> Have an issue I'm having trouble explaining or getting to the bottom
of. I have a two node, two brick replicated 75G volume containing ~4661 files:
>
> gotham:~# gluster volume status software
> Status of volume: software
> Gluster process                                         Port    Online  Pid
>
------------------------------------------------------------------------------
> Brick gotham.stor.graysofwestminster.co.uk:/data/gluste
> rfs/software/brick1/brick                               49152   Y      
27296
> Brick metropolis.stor.graysofwestminster.co.uk:/data/gl
> usterfs/software/brick1/brick                           49152   Y      
30335
> NFS Server on localhost                                 2049    Y      
27309
> Self-heal Daemon on localhost                           N/A     Y      
27316
> NFS Server on metropolis.stor.graysofwestminster.co.uk  2049    Y      
30348
> Self-heal Daemon on metropolis.stor.graysofwestminster.
> co.uk                                                   N/A     Y      
30355
>
> Task Status of Volume software
>
------------------------------------------------------------------------------
> There are no active volume tasks
>
>
> gotham:~# gluster volume heal software info
> Brick
gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
> Number of entries: 0
>
> Brick
metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
> Number of entries: 0
>
>
> I have the volume mounted on both nodes at /stor/software. On one of the
nodes, there are lots of files showing with a link count of 0 which is leading
to the Windows software using the data having validation issues:
>
> metropolis:~# ls -ltr /stor/software/win_patches/ | head
> total 21585884
> -rwxr--r-- 0 ainet Domain Admins     27816 Jan  4  2006 nullpatch.exe
> -rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29  2006
W2KA_SP4_128_x86_ENU.exe
> -rwxrwxr-- 0 ronny Domain Admins    432568 Oct 29  2006 MSXML26SP3.exe
> -rwxrwxr-- 0 ronny Domain Admins   1070592 Oct 29  2006 msxml3sp7.msi
> -rwxrwxr-- 0 ronny Domain Admins    711160 Oct 29  2006
Windows2000-KB842773.EXE
> -rwxrwxr-- 0 ronny Domain Admins    760824 Oct 29  2006
Windows2000-KB842933.EXE
> -rwxrwxr-- 0 ronny Domain Admins   2585872 Oct 29  2006
WindowsInstaller-KB893803v2.exe
> -rwxrwxr-- 0 ronny Domain Admins    106240 Oct 29  2006
Windows-KB870669.exe
> -rwxrwxr-- 0 ronny Domain Admins   1260024 Oct 29  2006
Windows2000-KB908506.EXE
>
>
>  From the other node:
>
> gotham:~# ls -ltr /stor/software/win_patches/ | head
> total 21585884
> -rwxr--r-- 1 1066 1016     27816 Jan  4  2006 nullpatch.exe
> -rwxrwxr-- 1 1045 1016 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
> -rwxrwxr-- 1 1045 1016    432568 Oct 29  2006 MSXML26SP3.exe
> -rwxrwxr-- 1 1045 1016   1070592 Oct 29  2006 msxml3sp7.msi
> -rwxrwxr-- 1 1045 1016    711160 Oct 29  2006 Windows2000-KB842773.EXE
> -rwxrwxr-- 1 1045 1016    760824 Oct 29  2006 Windows2000-KB842933.EXE
> -rwxrwxr-- 1 1045 1016   2585872 Oct 29  2006
WindowsInstaller-KB893803v2.exe
> -rwxrwxr-- 1 1045 1016    106240 Oct 29  2006 Windows-KB870669.exe
> -rwxrwxr-- 1 1045 1016   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>
>
> I'm seeing the following and lots more similar in the logs:
>
> [2016-01-28 15:13:15.099478] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote
operation failed: No such file or directory. Path:
/win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f)
client-1 is brick2, metropolis. That error says the file is missing, but 
it also shows the gfid. Perhaps the gfid hardlink is missing (the 
.glusterfs tree).

Check the metadata for the file on both bricks:
    getfattr -m . -d -e hex 
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

Ensure the gfid is the same on both. Check the number of links on both, 
there should be 2. With a gifd of 489f648-9fde-4bcc-b906-6ef88ffcf90f, 
that file should be hard linked to 
.glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on both
bricks.> File does exist on both bricks:
>
> metropolis:~# ls -l
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> -rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
>
> gotham:~# ls -l
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> -rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
>
> Any pointers to get to the bottom of this would be greatly appreciated.
Look for log entries in the brick logs that corrospond to the timestamps 
on the client-side and see if there's some correlation.
>
> Thanks.
>
> Ronny
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160128/4e3f01e2/attachment.html>

Ronny Adsetts

2016-Jan-28 22:29 UTC

head link

[Gluster-users] Problem with files showing zero links

Joe Julian wrote on 28/01/2016 18:28:> Response inline
Ditto. And thanks BTW. :-).
> On 01/28/2016 07:28 AM, Ronny Adsetts wrote:
>> Hi all,
>>
>> Have an issue I'm having trouble explaining or getting to the
bottom of. I have a two node, two brick replicated 75G volume containing ~4661
files:
>>
>> gotham:~# gluster volume status software
>> Status of volume: software
>> Gluster process                                         Port    Online 
Pid
>>
------------------------------------------------------------------------------
>> Brick gotham.stor.graysofwestminster.co.uk:/data/gluste
>> rfs/software/brick1/brick                               49152   Y      
27296
>> Brick metropolis.stor.graysofwestminster.co.uk:/data/gl
>> usterfs/software/brick1/brick                           49152   Y      
30335
>> NFS Server on localhost                                 2049    Y      
27309
>> Self-heal Daemon on localhost                           N/A     Y      
27316
>> NFS Server on metropolis.stor.graysofwestminster.co.uk  2049    Y      
30348
>> Self-heal Daemon on metropolis.stor.graysofwestminster.
>> co.uk                                                   N/A     Y      
30355
>>
>> Task Status of Volume software
>>
------------------------------------------------------------------------------
>> There are no active volume tasks
>>
>>
>> gotham:~# gluster volume heal software info
>> Brick
gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
>> Number of entries: 0
>>
>> Brick
metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
>> Number of entries: 0
>>
>>
>> I have the volume mounted on both nodes at /stor/software. On one of
the nodes, there are lots of files showing with a link count of 0 which is
leading to the Windows software using the data having validation issues:
>>
>> metropolis:~# ls -ltr /stor/software/win_patches/ | head
>> total 21585884
>> -rwxr--r-- 0 ainet Domain Admins     27816 Jan  4  2006 nullpatch.exe
>> -rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29  2006
W2KA_SP4_128_x86_ENU.exe
>> -rwxrwxr-- 0 ronny Domain Admins    432568 Oct 29  2006 MSXML26SP3.exe
>> -rwxrwxr-- 0 ronny Domain Admins   1070592 Oct 29  2006 msxml3sp7.msi
>> -rwxrwxr-- 0 ronny Domain Admins    711160 Oct 29  2006
Windows2000-KB842773.EXE
>> -rwxrwxr-- 0 ronny Domain Admins    760824 Oct 29  2006
Windows2000-KB842933.EXE
>> -rwxrwxr-- 0 ronny Domain Admins   2585872 Oct 29  2006
WindowsInstaller-KB893803v2.exe
>> -rwxrwxr-- 0 ronny Domain Admins    106240 Oct 29  2006
Windows-KB870669.exe
>> -rwxrwxr-- 0 ronny Domain Admins   1260024 Oct 29  2006
Windows2000-KB908506.EXE
>>
>>
>> From the other node:
>>
>> gotham:~# ls -ltr /stor/software/win_patches/ | head
>> total 21585884
>> -rwxr--r-- 1 1066 1016     27816 Jan  4  2006 nullpatch.exe
>> -rwxrwxr-- 1 1045 1016 135477136 Oct 29  2006 W2KA_SP4_128_x86_ENU.exe
>> -rwxrwxr-- 1 1045 1016    432568 Oct 29  2006 MSXML26SP3.exe
>> -rwxrwxr-- 1 1045 1016   1070592 Oct 29  2006 msxml3sp7.msi
>> -rwxrwxr-- 1 1045 1016    711160 Oct 29  2006 Windows2000-KB842773.EXE
>> -rwxrwxr-- 1 1045 1016    760824 Oct 29  2006 Windows2000-KB842933.EXE
>> -rwxrwxr-- 1 1045 1016   2585872 Oct 29  2006
WindowsInstaller-KB893803v2.exe
>> -rwxrwxr-- 1 1045 1016    106240 Oct 29  2006 Windows-KB870669.exe
>> -rwxrwxr-- 1 1045 1016   1260024 Oct 29  2006 Windows2000-KB908506.EXE
>>
>>
>> I'm seeing the following and lots more similar in the logs:
>>
>> [2016-01-28 15:13:15.099478] W
[client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote
operation failed: No such file or directory. Path:
/win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f)
> 
> client-1 is brick2, metropolis. That error says the file is missing, but it
also shows the gfid. Perhaps the gfid hardlink is missing (the .glusterfs tree).
> 
> Check the metadata for the file on both bricks:
>    getfattr -m . -d -e hex
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
> 
> Ensure the gfid is the same on both. Check the number of links on both,
there should be 2. With a gifd of 489f648-9fde-4bcc-b906-6ef88ffcf90f, that file
should be hard linked to .glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on
both bricks.
"Good" brick:

gotham:~# getfattr -m . -d -e hex
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
trusted.afr.software-client-0=0x000000000000000000000000
trusted.afr.software-client-1=0x000000000000000000000000
trusted.gfid=0x3489f6489fde4bccb9066ef88ffcf90f

gotham:~# ls -l
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

gotham:~# ls -i
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
114563677
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

gotham:~# find /data/glusterfs/software -inum 114563677
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
/data/glusterfs/software/brick1/brick/.glusterfs/34/89/3489f648-9fde-4bcc-b906-6ef88ffcf90f


"Bad" brick:

metropolis:~# getfattr -m . -d -e hex
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
getfattr: Removing leading '/' from absolute path names
# file: data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
trusted.afr.software-client-0=0x000000000000000000000000
trusted.afr.software-client-1=0x000000000000000000000000
trusted.gfid=0x3489f6489fde4bccb9066ef88ffcf90f

metropolis:~# ls -l
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

metropolis:~# ls -i
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
132120540
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi

metropolis:~# find /data/glusterfs/software -inum 132120540
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi


So we're missing the file in .glusterfs.


The magic question is I guess do you happen to know how to put the node back
together?

Thanks.

Ronny

-- 
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160128/c1fef89c/attachment.sig>

Ronny Adsetts

2016-Jan-29 10:46 UTC

head link

[Gluster-users] Problem with files showing zero links

Follow up for the archives. I fixed this by 'touch'ing all the files
with zero links:

Testing:

# ls -al /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe
-rwxr--r-- 0 ainet Domain Admins 55915216 Oct 15  2013
IE11-Windows6.1-x64-en-us.exe

# touch -a /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe

# ls -al /stor/software/win_patches/IE11-Windows6.1-x64-en-us.exe
-rwxr--r-- 1 ainet Domain Admins 55915216 Oct 15  2013
IE11-Windows6.1-x64-en-us.exe

and then:

# find /stor/software -links 0 -print0 | xargs -0 touch -a

No idea how it came about in the first place. Suspect it happened during either
an operating system upgrade or upgrading glusterfs packages to those from
gluster.org.

Anyway, thanks to Joe for the pointers.

Ronny
-- 
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160129/2b395804/attachment.sig>

Gluster users - Jan 2016 - Problem with files showing zero links

[Gluster-users] Problem with files showing zero links

[Gluster-users] Problem with files showing zero links

[Gluster-users] Problem with files showing zero links