Gluster users - Jan 2020 - healing of big files

Hi there. I'm trying to deploy replicated 2+1 cluster with gluster 5.5.1 
(debian 10) as vm image storage for libvirt, but have an issue with 
healing disconnected node. Example:

1) node4 + node5 - storages with one brick, node2 - arbiter.
2) Creating volume with defaults, and making a file (5Gb) in mounted fs, 
running 'dd bs=1024k count=5000 if=/dev/zero of=/mnt/zero'
3) Now restarting node5, and executing 'echo >>/mnt/zero' on
node2. So
file /mnt/zero become little changed and need to heal.

# gluster volume heal vmpool info

Brick node5:/data/glusterfs/vmpool/brick1/brick
Status: enpoint not connected
Number of entries: -

Brick node4:/data/glusterfs/vmpool/brick1/brick
/zero
Status: Connected
Number of entries: 1

Brick node2:/data/glusterfs/vmpool/brick1/brick
/zero
Status: Connected
Number of entries: 1


4) After node5 has reconnected, healing procedure has begun, but looks 
that about to entire file have transferred from synced node to unsynced, 
instead just changed blocks. (ifconfig on node5 showed '        RX 
packets 269790  bytes 5293724127 (4.9 GiB)) - so little less  than 5Gb 
file'   I tried 'difff' mode (gluster volume set vmpool 
cluster.data-self-heal-algorithm diff) but with no effect. Any idea, 
whats wrong?

-- 
vlad f halilov
swemel

Am Di., 14. Jan. 2020 um 10:38 Uhr schrieb vlad f halilov <vfh at
swemel.ru>:
> Hi there. I'm trying to deploy replicated 2+1 cluster with gluster
5.5.1
> (debian 10) as vm image storage for libvirt, but have an issue with
> healing disconnected node. Example:
>
...
>
> 4) After node5 has reconnected, healing procedure has begun, but looks
> that about to entire file have transferred from synced node to unsynced,
> instead just changed blocks. (ifconfig on node5 showed '        RX
> packets 269790  bytes 5293724127 (4.9 GiB)) - so little less  than 5Gb
> file'   I tried 'difff' mode (gluster volume set vmpool
> cluster.data-self-heal-algorithm diff) but with no effect. Any idea,
> whats wrong?
>
Actually i think diff should be working the way you expect it to, but diff
is very cpu-intensive as it needs to calculate and check checksums.

I would advise you to look at the example configuration for volumes with
vm-disks, it should be "/var/lib/glusterd/groups/virt". The main thing
is
sharding is activated ("split big files internally in many small
ones") and
set heal-algorithm to full.

This way heal is very fast, as it only resyncs the changed shards without
the need of checksums. I use it the same way on multiple hosts, each with
about 20-30 vms.

And Check the value of "features.shard-block-size", don't know
what the
default is atm - i use 64MB and it works quite well.

Regards.

Mika
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20200114/d3a43121/attachment.html>