thr3ads.net - Gluster users - [Gluster-users] Hundreds of duplicate files [Feb 2015]

If this information is useful, please help other people find it:
Share via:

Olav Peeters

2015-Feb-18 18:52 UTC

[Gluster-users] Hundreds of duplicate files

Hi all,
I'm have this problem after upgrading from 3.5.3 to 3.6.2.
At the moment I am still waiting for a heal to finish (on a 31TB volume 
with 42 bricks, replicated over three nodes).

Tom,
how did you remove the duplicates?
with 42 bricks I will not be able to do this manually..
Did a:
find $brick_root -type f -size 0 -perm 1000 -exec /bin/rm {} \;
work for you?

Should this type of thing ideally not be checked and mended by a heal?

Does anyone have an idea yet how this happens in the first place? Can it 
be connected to upgrading?

Cheers,
Olav

On 01/01/15 03:07, tbenzvi at 3vgeomatics.com wrote:> No, the files can be read on a newly mounted client! I went ahead and 
> deleted all of the link files associated with these duplicates, and 
> then remounted the volume. The problem is fixed!
> Thanks again for the help, Joe and Vijay.
> Tom
>
>     --------- Original Message ---------
>     Subject: Re: [Gluster-users] Hundreds of duplicate files
>     From: "Vijay Bellur" <vbellur at redhat.com>
>     Date: 12/28/14 3:23 am
>     To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
>
>     On 12/28/2014 01:20 PM, tbenzvi at 3vgeomatics.com wrote:
>     > Hi Vijay,
>     > Yes the files are still readable from the .glusterfs path.
>     > There is no explicit error. However, trying to read a text file in
>     > python simply gives me null characters:
>     >
>     > >>> open('ott_mf_itab').readlines()
>     >
>    
['\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
>     >
>     > And reading binary files does the same
>     >
>
>     Is this behavior seen with a freshly mounted client too?
>
>     -Vijay
>
>     > --------- Original Message ---------
>     > Subject: Re: [Gluster-users] Hundreds of duplicate files
>     > From: "Vijay Bellur" <vbellur at redhat.com>
>     > Date: 12/27/14 9:57 pm
>     > To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
>     >
>     > On 12/28/2014 10:13 AM, tbenzvi at 3vgeomatics.com wrote:
>     > > Thanks Joe, I've read your blog post as well as your post
>     > regarding the
>     > > .glusterfs directory.
>     > > I found some unneeded duplicate files which were not being
read
>     > > properly. I then deleted the link file from the brick. This
always
>     > > removes the duplicate file from the listing, but the file
does not
>     > > always become readable. If I also delete the associated file
>     in the
>     > > .glusterfs directory on that brick, then some more files
become
>     > > readable. However this solution still doesn't work for
all files.
>     > > I know the file on the brick is not corrupt as it can be read
>     > directly
>     > > from the brick directory.
>     >
>     > For files that are not readable from the client, can you check
>     if the
>     > file is readable from the .glusterfs/ path?
>     >
>     > What is the specific error that is seen while trying to read one
>     such
>     > file from the client?
>     >
>     > Thanks,
>     > Vijay
>     >
>     >
>     >
>     > _______________________________________________
>     > Gluster-users mailing list
>     > Gluster-users at gluster.org
>     > http://www.gluster.org/mailman/listinfo/gluster-users
>     >
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150218/5c4532a8/attachment.html>

tbenzvi at 3vgeomatics.com

2015-Feb-18 19:43 UTC

head link

[Gluster-users] Hundreds of duplicate files

Hi Olav,

I have a hunch that our problem was caused by improper unmounting of the gluster
volume, and have since found that the proper order should be: kill all jobs
using volume -> unmount volume on clients -> gluster volume stop ->
stop gluster service (if necessary)
 
In my case, I wrote a Python script to find duplicate files on the mounted
volume, then delete the corresponding link files on the bricks (making sure to
also delete files in the .glusterfs directory)
 
However, your find command was also suggested to me and I think it's a
simpler solution. I believe removing all link files (even ones that are not
causing duplicates) is fine since the next file access gluster will do a lookup
on all bricks and recreate any link files if necessary. Hopefully a gluster
expert can chime in on this point as I'm not completely sure.
 
Keep in mind your setup is somewhat different than mine as I have only 5 bricks
with no replication.
 
Regards,
Tom
 
--------- Original Message --------- Subject: Re: [Gluster-users] Hundreds of
duplicate files
From: "Olav Peeters" <opeeters at gmail.com>
Date: 2/18/15 10:52 am
To: gluster-users at gluster.org, tbenzvi at 3vgeomatics.com

 Hi all,
 I'm have this problem after upgrading from 3.5.3 to 3.6.2.
 At the moment I am still waiting for a heal to finish (on a 31TB volume with 42
bricks, replicated over three nodes).
 
 Tom,
 how did you remove the duplicates?
 with 42 bricks I will not be able to do this manually..
 Did a:
 find $brick_root -type f -size 0 -perm 1000 -exec /bin/rm {} \;
 work for you?
 
 Should this type of thing ideally not be checked and mended by a heal?
 
 Does anyone have an idea yet how this happens in the first place? Can it be
connected to upgrading?
 
 Cheers,
 Olav
   On 01/01/15 03:07, tbenzvi at 3vgeomatics.com wrote:
 No, the files can be read on a newly mounted client! I went ahead and deleted
all of the link files associated with these duplicates, and then remounted the
volume. The problem is fixed!
Thanks again for the help, Joe and Vijay.
 
Tom
 
--------- Original Message --------- Subject: Re: [Gluster-users] Hundreds of
duplicate files
 From: "Vijay Bellur" <vbellur at redhat.com>
 Date: 12/28/14 3:23 am
 To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
 
 On 12/28/2014 01:20 PM, tbenzvi at 3vgeomatics.com wrote:
 > Hi Vijay,
 > Yes the files are still readable from the .glusterfs path.
 > There is no explicit error. However, trying to read a text file in
 > python simply gives me null characters:
 >
 > >>> open('ott_mf_itab').readlines()
 >
['\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
 >
 > And reading binary files does the same
 >
 
 Is this behavior seen with a freshly mounted client too?
 
 -Vijay
 
 > --------- Original Message ---------
 > Subject: Re: [Gluster-users] Hundreds of duplicate files
 > From: "Vijay Bellur" <vbellur at redhat.com>
 > Date: 12/27/14 9:57 pm
 > To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
 >
 > On 12/28/2014 10:13 AM, tbenzvi at 3vgeomatics.com wrote:
 > > Thanks Joe, I've read your blog post as well as your post
 > regarding the
 > > .glusterfs directory.
 > > I found some unneeded duplicate files which were not being read
 > > properly. I then deleted the link file from the brick. This always
 > > removes the duplicate file from the listing, but the file does not
 > > always become readable. If I also delete the associated file in the
 > > .glusterfs directory on that brick, then some more files become
 > > readable. However this solution still doesn't work for all files.
 > > I know the file on the brick is not corrupt as it can be read
 > directly
 > > from the brick directory.
 >
 > For files that are not readable from the client, can you check if the
 > file is readable from the .glusterfs/ path?
 >
 > What is the specific error that is seen while trying to read one such
 > file from the client?
 >
 > Thanks,
 > Vijay
 >
 >
 >
 > _______________________________________________
 > Gluster-users mailing list
 > Gluster-users at gluster.org
 > http://www.gluster.org/mailman/listinfo/gluster-users
 >
 

 _______________________________________________
 Gluster-users mailing list
 Gluster-users at gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150218/a552f25a/attachment.html>

Joe Julian

2015-Feb-18 19:50 UTC

head link

[Gluster-users] Hundreds of duplicate files

On 02/18/2015 10:52 AM, Olav Peeters wrote:> Hi all,
> I'm have this problem after upgrading from 3.5.3 to 3.6.2.
> At the moment I am still waiting for a heal to finish (on a 31TB 
> volume with 42 bricks, replicated over three nodes).
>
> Tom,
> how did you remove the duplicates?
> with 42 bricks I will not be able to do this manually..
> Did a:
> find $brick_root -type f -size 0 -perm 1000 -exec /bin/rm {} \;
> work for you?
>
> Should this type of thing ideally not be checked and mended by a heal?
This problem should never happen. If it does, yes, I believe this should 
be recognized by gluster and fixed. The problem is, nobody's been able 
to reproduce this in a lab and those that have found it have only found 
it in production. Since those in production want it fixed quickly, 
there's generally no ability to go back and forth with the developers to 
adequately identify the problem and file a proper bug report.

If you have the time to figure out what happened, please file a bug 
report and provide any feedback the developers need to be able to 
reproduce this problem.

https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>
> Does anyone have an idea yet how this happens in the first place? Can 
> it be connected to upgrading?
You have given me one clue, above, that may be relevant. "I am waiting 
for a heal to finish". One guess would be that if you have volumes in an 
unhealed state and upgrade while unhealed, perhaps that may be a 
possible issue.
>
> Cheers,
> Olav
> On 01/01/15 03:07, tbenzvi at 3vgeomatics.com wrote:
>> No, the files can be read on a newly mounted client! I went ahead and 
>> deleted all of the link files associated with these duplicates, and 
>> then remounted the volume. The problem is fixed!
>> Thanks again for the help, Joe and Vijay.
>> Tom
>>
>>     --------- Original Message ---------
>>     Subject: Re: [Gluster-users] Hundreds of duplicate files
>>     From: "Vijay Bellur" <vbellur at redhat.com>
>>     Date: 12/28/14 3:23 am
>>     To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
>>
>>     On 12/28/2014 01:20 PM, tbenzvi at 3vgeomatics.com wrote:
>>     > Hi Vijay,
>>     > Yes the files are still readable from the .glusterfs path.
>>     > There is no explicit error. However, trying to read a text
file in
>>     > python simply gives me null characters:
>>     >
>>     > >>> open('ott_mf_itab').readlines()
>>     >
>>    
['\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
>>     >
>>     > And reading binary files does the same
>>     >
>>
>>     Is this behavior seen with a freshly mounted client too?
>>
>>     -Vijay
>>
>>     > --------- Original Message ---------
>>     > Subject: Re: [Gluster-users] Hundreds of duplicate files
>>     > From: "Vijay Bellur" <vbellur at redhat.com>
>>     > Date: 12/27/14 9:57 pm
>>     > To: tbenzvi at 3vgeomatics.com, gluster-users at gluster.org
>>     >
>>     > On 12/28/2014 10:13 AM, tbenzvi at 3vgeomatics.com wrote:
>>     > > Thanks Joe, I've read your blog post as well as your
post
>>     > regarding the
>>     > > .glusterfs directory.
>>     > > I found some unneeded duplicate files which were not
being read
>>     > > properly. I then deleted the link file from the brick.
This
>>     always
>>     > > removes the duplicate file from the listing, but the file
>>     does not
>>     > > always become readable. If I also delete the associated
file
>>     in the
>>     > > .glusterfs directory on that brick, then some more files
become
>>     > > readable. However this solution still doesn't work
for all files.
>>     > > I know the file on the brick is not corrupt as it can be
read
>>     > directly
>>     > > from the brick directory.
>>     >
>>     > For files that are not readable from the client, can you check
>>     if the
>>     > file is readable from the .glusterfs/ path?
>>     >
>>     > What is the specific error that is seen while trying to read
>>     one such
>>     > file from the client?
>>     >
>>     > Thanks,
>>     > Vijay
>>     >
>>     >
>>     >
>>     > _______________________________________________
>>     > Gluster-users mailing list
>>     > Gluster-users at gluster.org
>>     > http://www.gluster.org/mailman/listinfo/gluster-users
>>     >
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20150218/a5747a8e/attachment.html>

Gluster users - Feb 2015 - Hundreds of duplicate files

[Gluster-users] Hundreds of duplicate files

[Gluster-users] Hundreds of duplicate files

[Gluster-users] Hundreds of duplicate files