rsync - Feb 2023 - syncing huge files/devices: bmapfs

Hi,

recently I had to sync really huge files (VM images), represented
either as block devices (on one or both sides) or as regular files.

It *seemed* that Rsync didn't work well with block devices (the
--copy-devices didn't 'work for me, maybe I'm stupid or it's
broken in
the Rsync that ships with Debian bullseye). And additionally I ran into
performance and resource issues using Rsync for huge files. 

While this all can be my fault, I started a small project "bmapfs": a
fuse driver that maps the huge file (or device) as a directory of files
representing blocks of the backing file (or device).

Using Rsync now on these smaller files I circumvented the above
mentioned issues (be they real or just a result of my stupidity).

Maybe you find bmapfs useful for other scenarios too. I'd be happy if
you provide me feedback of *any* kind.

https://gitea.schlittermann.de/hs12/bmapfs

    Best regards from Dresden/Germany
    Viele Gr??e aus Dresden
    Heiko Schlittermann
--
 SCHLITTERMANN.de ---------------------------- internet & unix support -
 Heiko Schlittermann, Dipl.-Ing. (TU) - {fon,fax}: +49.351.802998{1,3} -
 gnupg encrypted messages are welcome --------------- key ID: F69376CE -
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20230210/5c033df7/signature.sig>

nice,

do you have performance comparison vs. plain fail on the same hardware
setup ?

thank you

roland

Am 10.02.23 um 07:56 schrieb Heiko Schlittermann via
rsync:
> fuse driver that maps the huge file (or device) as a directory of files
> representing blocks of the backing file (or device).
>
> Using Rsync now on these smaller files I circumvented the above
> mentioned issues (be they real or just a result of my stupidity).
>
> Maybe you find bmapfs useful for other scenarios too. I'd be happy if