rsync - Aug 2020 - Rsync 3.2.3 released

Rupert Gallagher via rsync <rsync at lists.samba.org> wrote:
> On 7 Aug 2020, 23:44, Wayne Davison < wayne at opencoder.net> wrote:
>
> >> Also, I have 12GB of cache in ecc ram that rsync is not using.
>
> >It uses whatever memory it needs plus whatever filesystem caching
> >your OS provides.
>
> Hmmm... bad day today...
>
> No, it is not using all available resources. It is doing frantic
> I/O instead of buffering. The result is a appalling low transfer
> rate.
This is an OS configuration/tuning problem, not an rsync problem.

Application-level code (such as rsync) does not directly deal with
NUMA features such as external RAM banks.  Support for that sort of
thing would normally be provided by the OS memory management and
filesystem code -- but (depending on the hardware details) it might
need configuration or even a specialized driver.

Beach ball play is fun, but I see this ball bounched betwen rsync, openbsd and
supermicro, and I am not enjoying it. I see a large cache delivered by the OS on
server hardware and a program unable to use it.

-------- Original Message --------
On 8 Aug 2020, 20:14, < pluto at agora.rdrop.com> wrote:
Rupert Gallagher via rsync <rsync at lists.samba.org>
wrote:
> On 7 Aug 2020, 23:44, Wayne Davison < wayne at opencoder.net> wrote:
>
> >> Also, I have 12GB of cache in ecc ram that rsync is not using.
>
> >It uses whatever memory it needs plus whatever filesystem caching
> >your OS provides.
>
> Hmmm... bad day today...
>
> No, it is not using all available resources. It is doing frantic
> I/O instead of buffering. The result is a appalling low transfer
> rate.
This is an OS configuration/tuning problem, not an rsync problem.
Application-level code (such as rsync) does not directly deal with
NUMA features such as external RAM banks. Support for that sort of
thing would normally be provided by the OS memory management and
filesystem code -- but (depending on the hardware details) it might
need configuration or even a specialized driver.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20200809/49463a7c/attachment.htm>

Rupert Gallagher <ruga at protonmail.com> wrote:
> ... I see this ball bounched betwen rsync, openbsd and supermicro
> ... I see a large cache delivered by the OS on server hardware and
> a program unable to use it.
It sounds as if the problem may be OpenBSD "delivering" the cache
rather than "utilizing" it.  What does the documentation -- or the
OpenBSD support community -- say about configuring OpenBSD to use
that sort of cache for filesystem optimization?