Libguestfs - Jun 2022 - Libguestfs Rsync-in with delta transfer enabled

Hi,

I am exploring libguestfs rsync-in to copy the incremental changes to
qcow2. I am following rsync example
<https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/> to
use
the rsync-in. As per the example, I am using rsync daemon and rysnc-in is
working for me. But, If the changes are in the same file then the complete
file is getting transferred.

For example, if my file is of 100Mi then the first qcow2 is of approx
105Mi. After the first qcow2, I am appending data of 20Mi and then creating
the second qcow2 using the first qcow2 as a backing file. Here then the
second qcow2 should have been around approx 22Mi but it's showing 125Mi in
qemu-img info.
>From the above observation, I think the delta transfer algorithm is not
getting used while doing rsync-in.

Just out of curiosity, I tried adding arg --no-whole-file in this function
<https://github.com/libguestfs/libguestfs/blob/master/daemon/rsync.c#L40>
but
it's still not working.

Can anyone please point me to how to enable the delta transfer algorithm?
Is it already supported ? or Do I need to make changes in the code? If yes,
then please let me know the places where I should add the changes.

Thanks,
Ajay
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://listman.redhat.com/archives/libguestfs/attachments/20220618/ba55189b/attachment.htm>

On Sat, Jun 18, 2022 at 06:18:58PM +0530, Ajay Nemade
wrote:
> Hi,
>
> I am exploring libguestfs rsync-in to copy the incremental changes
> to qcow2. I am following rsync example?to use the rsync-in. As per
> the example, I am using rsync daemon and rysnc-in is working for
> me. But, If the changes are in the same file then the complete file
> is getting transferred.
>
> For example, if my file is of 100Mi then the first qcow2 is of
> approx 105Mi.  After the first qcow2, I am appending data of 20Mi
> and then creating the second qcow2 using the first qcow2 as a
> backing file. Here then the second qcow2 should have been around
> approx 22Mi but it's showing?125Mi?in qemu-img info.
>
> From the above observation, I think the delta transfer algorithm is
> not getting used while doing rsync-in.
>
> Just out of?curiosity, I tried adding arg --no-whole-file in this
> function?but it's still not working.
>
> Can anyone please point me to how to enable the delta transfer
> algorithm? Is it already supported ? or Do I need to make changes in
> the code? If yes, then please let?me know the places where I should
> add the changes.
I'm not sure I know the correct answer.

I guess you have already tried modifying this code:

https://github.com/libguestfs/libguestfs/blob/60e8ea0312d94860403c956f0f12d491743d3fdd/daemon/rsync.c#L49

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

On 06/18/22 14:48, Ajay Nemade wrote:
> Hi,
> 
> I am exploring libguestfs rsync-in to copy the incremental changes to
> qcow2. I am following rsync example
>
<https://rwmj.wordpress.com/2013/04/22/using-rsync-with-libguestfs/%20>?to
> use the rsync-in. As per the example, I am using rsync daemon and
> rysnc-in is working for me. But, If the changes are in the same file
> then the complete file is getting transferred.
> 
> For example, if my file is of 100Mi then the first qcow2 is of approx
> 105Mi. After the first qcow2, I am appending data of 20Mi and then
> creating the second qcow2 using the first qcow2 as a backing file. Here
> then the second qcow2 should have been around approx 22Mi but it's
> showing?125Mi?in qemu-img info.
> 
> From the above observation, I think the delta transfer algorithm is not
> getting used while doing rsync-in.
> 
> Just out of?curiosity, I tried adding arg --no-whole-file in this
> function
>
<https://github.com/libguestfs/libguestfs/blob/master/daemon/rsync.c#L40>?but
> it's still not working.
> 
> Can anyone please point me to how to enable the delta transfer
> algorithm? Is it already supported ? or Do I need to make changes in the
> code? If yes, then please let?me know the places where I should add the
> changes.
Please try to reproduce the symptom without libguestfs; that is, by
running rsync in the guest manually. If you can reproduce the symptom
like that, then please (a) paste the full output of "qemu-img info"
from
the second (overlay) qcow2 file, (b) add "--stats" to the rsync
command
in the guest, and paste the output from that too.

(The behavior you are reporting is not impossible in general IMO, just
unexpected when only appending data.)

Thanks
Laszlo