rsync - Sep 2021 - high memory usage transferring about 27.000.000 files

We are trying to back up a large number of small files and have issues 
with high memory usage.
On both ends debian 11.0 is running with rsync version 3.2.3  protocol 
version 31.
Ideally we want to use the options `-a --delete --backup -e ssh` to run 
this transfer and pull a backup from the target machine.
The memory usage continually rises until using all free memory on the 
initiator side, about 4.5GB split across two processes. On the sender 
side one process uses about half of that.
The folder to transfer contains about 11TB of data split over 
~27.000.000 files in a structure with ~19.000.000 directories.
We first tried to make sure that rsync is using it's incremental 
recursion mode. After reading the manual we explicitly used the 
`--delete-during` option as well as trying to run without deletion or 
backup mode with no noticeable improvements.
The command line options on the sender side, as far as i understand, 
indicate that incremental recursion mode is requested 
(`-vlogDtpre.iLsfxCIvu`).

We also tried --omit-dir-times to reduce memory usage without success.

We desperately try not to split up the transfer for subfolders as we 
would have to do so dynamically because the folder names change.

Does anyone have experience with such transfers and can help us with 
reducing the memory usage?
Or is there an issue and rsync shouldn't use that much memory?