Lindsay Mathieson
2016-Apr-17 19:32 UTC
[Gluster-users] How accurate is heal statistics heal-count?
On 18/04/2016 12:23 AM, Krutika Dhananjay wrote:> I just checked the code. All `statistics heal-count` does is to count > the number of indices (in other words the number of files to be > healed) present per brick in the .glusterfs/indices/xattrop directory > - which is where we hold empty files representing inodes that need > heal - and prints them. I even tested it on my local setup, in terms > of creating a few thousand files and writing data into them while a > brick is down and then executing the command after bringing the brick > back up periodically. It worked for me. > > `heal info` also reads individual indices in the same directory, but > it takes locks to confirm they need heal by examining the pending xattrs. > > Could you describe how you ran into this issue?Just ran the cluster, as per my later email "Continual heals happening on cluster". I've: - shutdown the VM's - waited until all heals completed - restarted the VM'a - heal info immediately starts showing an every changing list of shards being healed on all three nodes. - heal statistics heal-count shows nothing Something I've just noticed. I have 3 nodes, vnb, vng, vna. The heal info on vng & vna shows shards being healed. but heal info on *vnb* shows nothing. when I looked at indices directory on all three nodes there: - is a xattrop dir which contains one 0 length file which never changes, different name on all three nodes - is a dirty dir which contains an ever changing list of 0 byte files -- Lindsay Mathieson
Kevin Lemonnier
2016-Apr-17 19:49 UTC
[Gluster-users] How accurate is heal statistics heal-count?
> - heal info immediately starts showing an every changing list of shards > being healed on all three nodes. > - heal statistics heal-count shows nothing >Interestingly, I have the exact oposite on my cluster : heal info shows nothing but I constantly have between 3 and ~50 in statistics heal-count. I assumed it was just syncing the writes of the different VM's, that it was expected. -- Kevin Lemonnier PGP Fingerprint : 89A5 2283 04A0 E6E9 0111 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160417/df622e9e/attachment.sig>
Lindsay Mathieson
2016-Apr-17 20:22 UTC
[Gluster-users] How accurate is heal statistics heal-count?
On 18/04/2016 5:32 AM, Lindsay Mathieson wrote:> - shutdown the VM's > - waited until all heals completed > - restarted the VM'a > - heal info immediately starts showing an every changing list of > shards being healed on all three nodes. > - heal statistics heal-count shows nothingI missed a step, I also restarted all gluster services before restarting the VM's. Wanted a clean slate. -- Lindsay Mathieson
Joe Julian
2016-Apr-17 20:55 UTC
[Gluster-users] How accurate is heal statistics heal-count?
All the steps necessary to marking something as needing healed are performed for every write (or metadata change). If you happen to check in the middle of that transaction, you'll see a heal pending. On 04/17/2016 12:32 PM, Lindsay Mathieson wrote:> On 18/04/2016 12:23 AM, Krutika Dhananjay wrote: >> I just checked the code. All `statistics heal-count` does is to count >> the number of indices (in other words the number of files to be >> healed) present per brick in the .glusterfs/indices/xattrop directory >> - which is where we hold empty files representing inodes that need >> heal - and prints them. I even tested it on my local setup, in terms >> of creating a few thousand files and writing data into them while a >> brick is down and then executing the command after bringing the brick >> back up periodically. It worked for me. >> >> `heal info` also reads individual indices in the same directory, but >> it takes locks to confirm they need heal by examining the pending >> xattrs. >> >> Could you describe how you ran into this issue? > > Just ran the cluster, as per my later email "Continual heals happening > on cluster". I've: > > - shutdown the VM's > - waited until all heals completed > - restarted the VM'a > - heal info immediately starts showing an every changing list of > shards being healed on all three nodes. > - heal statistics heal-count shows nothing > > Something I've just noticed. I have 3 nodes, vnb, vng, vna. The heal > info on vng & vna shows shards being healed. but heal info on *vnb* > shows nothing. > > when I looked at indices directory on all three nodes there: > - is a xattrop dir which contains one 0 length file which never > changes, different name on all three nodes > - is a dirty dir which contains an ever changing list of 0 byte files >