Gluster users - Apr 2016 - "gluster volume heal datastore2 info" blocking, heal not completing

Have this happening at the moment. Issue the command and it just hangs. 
More disturbingly, io on the hosted VM's *stops* until I kill the heal 
info cmd.


Also the heal status seems to be stuck as well, on the last 6 shards on 
two nodes. Shard size is 4MB:

    gluster volume heal datastore2 statistics heal-count
    Gathering count of entries to be healed on volume datastore2 has
    been successful

    Brick vnb.proxmox.softlog:/tank/vmdata/datastore2
    Number of entries: 6

    Brick vng.proxmox.softlog:/tank/vmdata/datastore2
    Number of entries: 6

    Brick vna.proxmox.softlog:/tank/vmdata/datastore2
    Number of entries: 0



Has been like that for over an hour.

I killed and restarted the glusterd, glusterfsd processes on each node. 
No of shards needing healed increased upto a couple of 100 each time, 
then healed back down to 6 (as above).

I can leave it like this for a while if anyone wants to suggest tests or 
logging.

nb. Heal count just dropped down to 5 shards, so there is some progress, 
albeit very slow.

Current settings:

    Volume Name: datastore2
    Type: Replicate
    Volume ID: 7d93a1c6-ac39-4d94-b136-e8379643bddd
    Status: Started
    Number of Bricks: 1 x 3 = 3
    Transport-type: tcp
    Bricks:
    Brick1: vnb.proxmox.softlog:/tank/vmdata/datastore2
    Brick2: vng.proxmox.softlog:/tank/vmdata/datastore2
    Brick3: vna.proxmox.softlog:/tank/vmdata/datastore2
    Options Reconfigured:
    network.remote-dio: enable
    cluster.eager-lock: enable
    performance.io-cache: off
    performance.read-ahead: off
    performance.quick-read: off
    performance.stat-prefetch: on
    performance.strict-write-ordering: on
    performance.write-behind: off
    nfs.enable-ino32: off
    nfs.addr-namelookup: off
    nfs.disable: on
    cluster.server-quorum-type: server
    cluster.quorum-type: auto
    features.shard: on
    cluster.data-self-heal: off
    performance.readdir-ahead: off


Thanks,

-- 
Lindsay Mathieson

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.gluster.org/pipermail/gluster-users/attachments/20160413/83857586/attachment.html>

nb. iowait on vna was very high (20%) from other non gluster processes, 
but its now settled down to around 10%

-- 
Lindsay Mathieson