Direct I/O (assuming you mean O_DIRECT on open) can be a bit fiddly, but I
doubt it's out of reach. The main difficulty is allocating a buffer with
appropriate alignment.
I put together a library to facilitate O_DIRECT I/O a while back:
http://stromberg.dnsalias.org/~dstromberg/odirect/
odirect is callable from C, C++ and CPython. I hope to add support for
calling it via Python "ctypes" at some point in the future.
On Sun, Sep 4, 2011 at 6:09 AM, Sandon Van Ness <sandon at van-ness.com>
wrote:
> As far as I can tell rsync doesn't support this. How hard would it be
to
> implement this? Is it trivial enough to just change the calls in the code
> with sed? I think this can significantly reduce CPU usage and increase I/O
> speed when dealing with fast storage solutions. It can make a huge
> difference when say transferring 30TB of data.
>
> Here are some tests I did. So far the only thing I know of that can use
> direct I/O is dd which showed a huge improvement. CPU usage went from very
> high down to under 15% and the speed almost doubled.
>
> Is there anything else I can do to tune rsync to give me better speed
> and/or less CPU usage as I seem to be hitting a CPU bottleneck in the case
> of rsync.
>
> In my tests I saw the highest CPU usage with rsync and lowest throughput:
>
> *cp (no direct I/O):*
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23356 root 20 0 10740 676 584 D 74 0.0 0:16.48 cp
> 23344 root 20 0 0 0 0 D 9 0.0 0:04.80 flush-8:64
> 639 root 20 0 0 0 0 R 9 0.0 0:12.28 kswapd0
>
>
> root at dekabutsu: 05:54 AM :/data2# time cp -avf /data/winxp.img /data2/
> `/data/winxp.img' -> `/data2/winxp.img'
>
> real 1m10.767s
> user 0m0.692s
> sys 0m47.401s
>
> *Rsync (no direct I/O):*
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23397 root 20 0 11572 616 256 R 97 0.0 0:08.48 rsync
> 23395 root 20 0 11864 1292 848 R 84 0.0 0:07.35 rsync
>
>
> root at dekabutsu: 05:56 AM :/data2# time rsync --progress -av
> /data/winxp.img /data2/
> building file list ...
> 1 file to consider
> winxp.img
> 37589497344 100% 223.43MB/s 0:02:40 (xfer#1, to-check=0/1)
>
> sent 37594086010 bytes received 42 bytes 234231065.74 bytes/sec
> total size is 37589497344 speedup is 1.00
>
> real 2m40.552s
> user 3m36.020s
> sys 1m8.181s
>
> *dd (direct I/O)=*
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23452 root 20 0 14960 1664 580 D 13 0.0 0:00.76 dd
>
>
> root at dekabutsu: 06:00 AM :/data2# time dd bs=1M iflag=direct
oflag=direct
> if=/data/winxp.img of=/data2/winxp.img
> 35848+1 records in
> 35848+1 records out
> 37589497344 bytes (38 GB) copied, 46.8669 s, 802 MB/s
>
> real 0m46.869s
> user 0m0.031s
> sys 0m6.043s
>
>
> *dd (no direct I/O):*
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 23455 root 20 0 14960 1664 576 R 65 0.0 0:06.59 dd
> 23344 root 20 0 0 0 0 S 20 0.0 0:17.88 flush-8:64
>
>
> root at dekabutsu: 06:01 AM :/data2# time dd bs=1M if=/data/winxp.img
> of=/data2/winxp.img
> 35848+1 records in
> 35848+1 records out
> 37589497344 bytes (38 GB) copied, 83.5282 s, 450 MB/s
>
> real 1m23.530s
> user 0m0.048s
> sys 0m54.304s
>
> (so pretty close to the speed of cp).
>
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/**
> mailman/listinfo/rsync
<https://lists.samba.org/mailman/listinfo/rsync>
> Before posting, read:
http://www.catb.org/~esr/faqs/**smart-questions.html<http://www.catb.org/%7Eesr/faqs/smart-questions.html>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20110904/0e4ebffb/attachment.html>