Just note this patch has nothing to do with memory consumption vs
performance. It just avoids allocating memory that was left unused anyway.
On Sat, Oct 2, 2021, 12:51 devzero--- via rsync <rsync at lists.samba.org>
wrote:
> >In the exchange I argued that proper use of ram as a buffer would have
> cut down backup time to minutes instead of days.
>
> could you give an example where rsync is slowing things down so much due
> to ram constraints or inefficient ram use?
>
> please mind that disk bandwith and file copy bandwith is not the same.
> random i/o and seek time is the culprit.
>
> why should rsync use ram for buffering data it copies, if the linux kernel
> / vm subsystem already does this?
>
> roland
>
> *Gesendet:* Samstag, 02. Oktober 2021 um 12:07 Uhr
> *Von:* "Rupert Gallagher via rsync" <rsync at
lists.samba.org>
> *An:* makovick at gmail.com, rsync at lists.samba.org
> *Betreff:* Re: [PATCH] Reduce memory usage
> If you look at my previous exchange in the list, I raised the need for
> more ram usage via a tool option. In the exchange I argued that proper use
> of ram as a buffer would have cut down backup time to minutes instead of
> days. At the time, my proposal was dismissed by someone saying that rsync
> uses as much ram as it needs. I still feel the need to free rsync from this
> mindless constraint, while also welcoming ram usage optimisations such as
> yours in this patch. How hard can it be to allow rsync to use 1GB of ram
> instead of 100MB? The benefit would be huge. In my case, where a supermicro
> server uses a shared bus to transfer data from two disks, the overhead
> caused by frequent small buffer IO is so high that backup time is still
> huge. And I am using server hardware! PC and laptops are even worse.
>
> RG
>
>
>
>
> -------- Original Message --------
> On Sep 26, 2021, 13:54, Jind?ich Makovi?ka via rsync <
> rsync at lists.samba.org> wrote:
> Hi,
>
> When using rsync to back up the file system on my laptop, containing a
> pretty much default linux desktop, I was wondering how rsync uses over
> 100MB of RAM it allocates.
>
> It turned out that most of the memory is used for the arrays of
> file_struct pointers, most of which end up unused - much more than the
> actual file_struct entries. In my case, the peak usage was 135MB of
> pointers, and just 1.5MB of the file_struct entries themselves.
>
> The problem seems to be that the default file_list allocation parameters
> predate the incremental recursion, which allocates a huge number of small
> file lists, while AFAICS originally rsync allocated just one large list.
>
> Applying the attached patch, which reduces the default allocation to 32
> pointers, and preallocates 32K pointers only for the main file lists in
> send_file_list and recv_file_list, reduces the peak memory usage in my case
> from 142MB to 12MB.
>
> Regards,
> --
> Jind?ich Makovi?ka
> -- Please use reply-all for most replies to avoid omitting the mailing
> list. To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync Before posting, read:
> http://www.catb.org/~esr/faqs/smart-questions.html
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20211002/89cc6317/attachment.htm>