Charles
2021-Sep-13 08:51 UTC
rsync dest high disk write request avg waiting time (w_await) and probable low speed
Is it likely that rsync is the cause of the symptoms?
If so, how and can the rsync command be changed to prevent the problem?
While running this command ...
rsync --filter=protect _Changed and deleted files/ --exclude=/_Changed
and deleted files/ --backup --backup-dir=_Changed and deleted
files/2021/Sep/13 at 01:20 --compress --delete-during --delete-excluded
--devices --group --links --owner --numeric-ids --perms --timeout=600
--protect-args --recursive --specials --times --verbose
/home/vbox/VirtualBox VMs/aems3.iciti.av/
vbox2_to_backup.sar:/srv/backup/vms.vbox2.iciti.av
... Zabbix reported reported high disk write request avg waiting time
(w_await) values of ~40 ms
The VirtualBox VM was shut down during the sync so the source files were
static. There were seven files to be synced, six trivially small and
one 38 GB. The sync took ~0.5 hours and summarised
sent 1,009,799,482 bytes received 2,428,622 bytes 636,021.43 bytes/sec
total size is 39,736,701,202 speedup is 39.26
Investigation showed the high w_await values were associated with rsync
rather than the HDD
1. Moved the destination to a different HDD. Same
2. Re-scheduled to a different time. Same
3. Wrote to the HDD locally
dd if=/dev/zero of=/srv/backup/vms.vbox2.iciti.av/trash
That wrote at ~130 MB/s and gave w_await 56 ms
4. Wrote to the HDD from the source computer
scp -p '/home/vbox/VirtualBox VMs/aems3.iciti.av/aems3.iciti.av.vdi' \
storage4.iciti.av:/srv/backup/vms.vbox2.iciti.av/trash
...
100%
37GB 76.5MB/s 08:15
...
That wrote at ~76 MB/s and gave w_await 62 ms
5. Experimented with --bwlimit
Progressively reducing from 50M to 5M made no significant difference to
the time taken to sync (stayed around 20 to 30 m) or w_await (stayed
around 40 ms).
Only reducing from 5M to 3M made a significant difference to the time
taken to sync (60 m) while w_await stayed around 40 ms.