Frederic Harmignies
2017-Nov-15 14:27 UTC
[Gluster-users] Missing files on one of the bricks
Hello, we have 2x files that are missing from one of the bricks. No idea how to fix this. Details: # gluster volume info Volume Name: data01 Type: Replicate Volume ID: 39b4479c-31f0-4696-9435-5454e4f8d310 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 192.168.186.11:/mnt/AIDATA/data Brick2: 192.168.186.12:/mnt/AIDATA/data Options Reconfigured: performance.cache-refresh-timeout: 30 client.event-threads: 16 server.event-threads: 32 performance.readdir-ahead: off performance.io-thread-count: 32 performance.cache-size: 32GB transport.address-family: inet nfs.disable: on features.trash: off features.trash-max-filesize: 500MB # gluster volume heal data01 info Brick 192.168.186.11:/mnt/AIDATA/data Status: Connected Number of entries: 0 Brick 192.168.186.12:/mnt/AIDATA/data <gfid:7e8513f4-d4e2-4e66-b0ba-2dbe4c803c54> <gfid:9612ecd2-106d-42f2-95eb-fef495c1d8ab> Status: Connected Number of entries: 2 # gluster volume heal data01 info split-brain Brick 192.168.186.11:/mnt/AIDATA/data Status: Connected Number of entries in split-brain: 0 Brick 192.168.186.12:/mnt/AIDATA/data Status: Connected Number of entries in split-brain: 0 Both files is missing from the folder on Brick1, the gfid files are also missing in the .gluster folder on that same Brick1. Brick2 has both the files and the gfid file in .gluster We already tried: #gluster heal volume full Running a stat and ls -l on both files from a mounted client to try and trigger a heal Would a re-balance fix this? Any guidance would be greatly appreciated! Thank you in advance! -- *Frederic Harmignies* *High Performance Computer Administrator* www.elementai.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171115/e0e6598d/attachment.html>
Nithya Balachandran
2017-Nov-16 10:42 UTC
[Gluster-users] Missing files on one of the bricks
On 15 November 2017 at 19:57, Frederic Harmignies < frederic.harmignies at elementai.com> wrote:> Hello, we have 2x files that are missing from one of the bricks. No idea > how to fix this. > > Details: > > # gluster volume info > > Volume Name: data01 > Type: Replicate > Volume ID: 39b4479c-31f0-4696-9435-5454e4f8d310 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: 192.168.186.11:/mnt/AIDATA/data > Brick2: 192.168.186.12:/mnt/AIDATA/data > Options Reconfigured: > performance.cache-refresh-timeout: 30 > client.event-threads: 16 > server.event-threads: 32 > performance.readdir-ahead: off > performance.io-thread-count: 32 > performance.cache-size: 32GB > transport.address-family: inet > nfs.disable: on > features.trash: off > features.trash-max-filesize: 500MB > > # gluster volume heal data01 info > Brick 192.168.186.11:/mnt/AIDATA/data > Status: Connected > Number of entries: 0 > > Brick 192.168.186.12:/mnt/AIDATA/data > <gfid:7e8513f4-d4e2-4e66-b0ba-2dbe4c803c54> > <gfid:9612ecd2-106d-42f2-95eb-fef495c1d8ab> > Status: Connected > Number of entries: 2 > > # gluster volume heal data01 info split-brain > Brick 192.168.186.11:/mnt/AIDATA/data > Status: Connected > Number of entries in split-brain: 0 > > Brick 192.168.186.12:/mnt/AIDATA/data > Status: Connected > Number of entries in split-brain: 0 > > > Both files is missing from the folder on Brick1, the gfid files are also > missing in the .gluster folder on that same Brick1. > Brick2 has both the files and the gfid file in .gluster > > We already tried: > > #gluster heal volume full > Running a stat and ls -l on both files from a mounted client to try and > trigger a heal > > Would a re-balance fix this? Any guidance would be greatly appreciated! >A rebalance would not help here as this is a replicate volume. Ravi, any idea what could be going wrong here? Regards, Nithya> > Thank you in advance! > > -- > > *Frederic Harmignies* > *High Performance Computer Administrator* > > www.elementai.com > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171116/ced87954/attachment.html>
On 11/16/2017 04:12 PM, Nithya Balachandran wrote:> > > On 15 November 2017 at 19:57, Frederic Harmignies > <frederic.harmignies at elementai.com > <mailto:frederic.harmignies at elementai.com>> wrote: > > Hello, we have 2x files that are missing from one of the bricks. > No idea how to fix this. > > Details: > > # gluster volume info > Volume Name: data01 > Type: Replicate > Volume ID: 39b4479c-31f0-4696-9435-5454e4f8d310 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: 192.168.186.11:/mnt/AIDATA/data > Brick2: 192.168.186.12:/mnt/AIDATA/data > Options Reconfigured: > performance.cache-refresh-timeout: 30 > client.event-threads: 16 > server.event-threads: 32 > performance.readdir-ahead: off > performance.io-thread-count: 32 > performance.cache-size: 32GB > transport.address-family: inet > nfs.disable: on > features.trash: off > features.trash-max-filesize: 500MB > > # gluster volume heal data01 info > Brick 192.168.186.11:/mnt/AIDATA/data > Status: Connected > Number of entries: 0 > > Brick 192.168.186.12:/mnt/AIDATA/data > <gfid:7e8513f4-d4e2-4e66-b0ba-2dbe4c803c54> > <gfid:9612ecd2-106d-42f2-95eb-fef495c1d8ab> > Status: Connected > Number of entries: 2 > > # gluster volume heal data01 info split-brain > Brick 192.168.186.11:/mnt/AIDATA/data > Status: Connected > Number of entries in split-brain: 0 > > Brick 192.168.186.12:/mnt/AIDATA/data > Status: Connected > Number of entries in split-brain: 0 > > > Both files is missing from the folder on Brick1, the gfid files > are also missing in the .gluster folder on that same Brick1. > Brick2 has both the files and the gfid file in .gluster > > We already tried: > > ?#gluster heal volume full > Running a stat and ls -l on both files from a mounted client to > try and trigger a heal > > Would a re-balance fix this? Any guidance would be greatly > appreciated! > > > A rebalance would not help here as this is a replicate volume. Ravi, > any idea what could be going wrong here?No, explicit lookup should have healed the file on the missing brick. Unless lookup did not hit afr and is served from caching translators. Frederic, what version of gluster are you running? Can you launch 'gluster heal volume' and see glustershd logs for possible warnings? Use DEBUG client-log-level if you have to.? Also, instead of stat, try a getfattr on the file from the mount. -Ravi> > Regards, > Nithya > > > Thank you in advance! > > -- > * > * > *Frederic Harmignies* > /High Performance Computer Administrator/ > > www.elementai.com <http://www.elementai.com/> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://lists.gluster.org/mailman/listinfo/gluster-users > <http://lists.gluster.org/mailman/listinfo/gluster-users> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171116/43d4f701/attachment.html>