Hi, Since a --direct-io feature was requested a few times the past decade with little response and the actual patch is quite trivial, I patched both v3.0.9 and master branch and included the patches here. If this functionality is acceptable I don't mind spending the additional effort to update the documentation, etc. Beware that the underlying filesystem needs Direct I/O support, therefore for this to work on NFS, one needs the kernel CONFIG_NFS_DIRECTIO enabled (probably default these days anyway). This is the reason why --direct-io is not enabled by default and should be used with care. I don't know what the behavior is if the local or remote side do not support O_DIRECT. In my tests it provides a much more stable copy process (on NFS) with improved performance, but I didn't do any prolonged repeated tests yet (I only used it during a large copy that was ongoing so I prefer not to juggle with numbers here). Please test yourself and report back. Kind regards, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- dagit linux solutions, info at dagit.net, http://dagit.net/ [Any errors in spelling, tact or fact are transmission errors] -------------- next part -------------- A non-text attachment was scrubbed... Name: rsync-master-direct-io.patch Type: text/x-patch Size: 2975 bytes Desc: URL: <http://lists.samba.org/pipermail/rsync/attachments/20130214/6c3cf8b6/attachment.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: rsync-3.0.9-direct-io.patch Type: text/x-patch Size: 2141 bytes Desc: URL: <http://lists.samba.org/pipermail/rsync/attachments/20130214/6c3cf8b6/attachment-0001.bin>
On 2/14/2013 9:50 AM, Dag Wieers wrote:> Hi, > > Since a --direct-io feature was requested a few times the past decade > with little response and the actual patch is quite trivial, I patched > both v3.0.9 and master branch and included the patches here. > > If this functionality is acceptable I don't mind spending the additional > effort to update the documentation, etc. > > Beware that the underlying filesystem needs Direct I/O support, > therefore for this to work on NFS, one needs the kernel > CONFIG_NFS_DIRECTIO enabled (probably default these days anyway). This > is the reason why --direct-io is not enabled by default and should be > used with care. I don't know what the behavior is if the local or remote > side do not support O_DIRECT. > > In my tests it provides a much more stable copy process (on NFS) with > improved performance, but I didn't do any prolonged repeated tests yet > (I only used it during a large copy that was ongoing so I prefer not to > juggle with numbers here). Please test yourself and report back. > > Kind regards, > >When I drop the 3.0.9 diff into my otherwise working spec file for 3.0.9 on opensuse build service, it patches and builds with no error, but "make test" fails: + cd rsync-3.0.9 + /usr/bin/make test gcc -std=gnu99 -I. -I. -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -DHAVE_CONFIG_H -Wall -W -c tls.c -o tls.o gcc -std=gnu99 -fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -DHAVE_CONFIG_H -Wall -W -o tls tls.o syscall.o lib/compat.o lib/snprintf.o lib/permstring.o lib/sysxattrs.o -lacl -lpopt syscall.o: In function `do_open': syscall.c:(.text+0x29b): undefined reference to `direct_io' collect2: ld returned 1 exit status make: *** [tls] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.isF55i (%check) What should I look for? This is opensuse 11.4 x86_64 linux 2.6.37 reiserfs (reiser3) -- blw
On Thu, 14 Feb 2013, Brian K. White wrote:> On 2/14/2013 9:50 AM, Dag Wieers wrote: > >> Since a --direct-io feature was requested a few times the past decade >> with little response and the actual patch is quite trivial, I patched >> both v3.0.9 and master branch and included the patches here. > > When I drop the 3.0.9 diff into my otherwise working spec file for 3.0.9 on > opensuse build service, it patches and builds with no error, but "make test" > fails:Attached is an updated patch which takes care of the test cases. Kind regards, -- -- dag wieers, dag at wieers.com, http://dag.wieers.com/ -- dagit linux solutions, info at dagit.net, http://dagit.net/ [Any errors in spelling, tact or fact are transmission errors] -------------- next part -------------- A non-text attachment was scrubbed... Name: rsync-3.0.9-direct-io.patch Type: text/x-patch Size: 2945 bytes Desc: URL: <http://lists.samba.org/pipermail/rsync/attachments/20130214/f1440ba7/attachment.bin>
On 02/14/2013 02:21 PM, Dag Wieers wrote:> On Thu, 14 Feb 2013, Brian K. White wrote: > >> On 2/14/2013 9:50 AM, Dag Wieers wrote: >> >>> Since a --direct-io feature was requested a few times the past decade >>> with little response and the actual patch is quite trivial, I patched >>> both v3.0.9 and master branch and included the patches here. >> >> When I drop the 3.0.9 diff into my otherwise working spec file for >> 3.0.9 on opensuse build service, it patches and builds with no error, >> but "make test" fails: > > Attached is an updated patch which takes care of the test cases. > > Kind regards,Awesome I will have to give this a try. I seem to recall that rsync was CPU bottlenecked on my machine around 200 megabytes/sec where as a DD device -> device copy with direct I/O I could get around 800 megabytes/sec (just a regular cp/dd was like 300-400 megabytes/sec). Also nice that you are not trashing your disk cache as well. I have been waiting for this for a while.
On 2/14/2013 5:21 PM, Dag Wieers wrote:> On Thu, 14 Feb 2013, Brian K. White wrote: > >> On 2/14/2013 9:50 AM, Dag Wieers wrote: >> >>> Since a --direct-io feature was requested a few times the past decade >>> with little response and the actual patch is quite trivial, I patched >>> both v3.0.9 and master branch and included the patches here. >> >> When I drop the 3.0.9 diff into my otherwise working spec file for >> 3.0.9 on opensuse build service, it patches and builds with no error, >> but "make test" fails: > > Attached is an updated patch which takes care of the test cases. > > Kind regards,Success! Thank you very much! -- bkw
I wondered about that as well -- could speed things up by 30% over going through the slow linux buffers. One thing that the 'dd' people found out though was that if you do direct I/O, memory and your I/O really do have to line up -- it may be that only 512 byte alignment is necessary (or 4096 on some newer disks)...but ideally, you look at the stat's claim for write size since the last param in stat isn't the allocation size, but the smallest optimal write size -- i.e. the "stripe size" if you have a RAID...as there, you want to write whole strips at once, otherwise you get into a read/modify/write cycle that slows down your disk I/O with 200% overhead -- *ouch!*... Dag Wieers wrote:> Hi, > > Since a --direct-io feature was requested a few times the past decade > with little response and the actual patch is quite trivial, I patched > both v3.0.9 and master branch and included the patches here. >
Dag Wieers wrote:> On Thu, 14 Feb 2013, Brian K. White wrote: > >> On 2/14/2013 9:50 AM, Dag Wieers wrote: >> >>> Since a --direct-io feature was requested a few times the past decade >>> with little response and the actual patch is quite trivial, I patched >>> both v3.0.9 and master branch and included the patches here. >> >> When I drop the 3.0.9 diff into my otherwise working spec file for >> 3.0.9 on opensuse build service, it patches and builds with no error, >> but "make test" fails: > > Attached is an updated patch which takes care of the test cases. > > Kind regards, >--- Tried this patch, but it failed to work: cmd: cd /Media && { sudo rsync --archive --hard-links --acls -xattrs --whole-file --drop-cache \ --one-file-system --drop-cache --8-bit-output \ --human-readable --progress --inplace --del . /backups/Media/. } ---- first it deleted a bunch of stuff (~30 files) deleting Library / rsync: delete_file: unlink(MediaBack) failed: Operation not permitted (1) .recycle/ .recycle/SDT27D6.tmp 0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=1006/1019) .recycle/Library/ .recycle/Library/[Commie] Kotoura - 04 [FC6C5497].mkv 32.77K 0% 0.00kB/s 0:00:00 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:Broken pipe (32) Then a bunch more deletes...(~17) Then: rsync: write failed on "/backups/Media/./.recycle/Library/[Commie] Kotoura - 04 [FC6C5497].mkv": Invalid argument (22) rsync: connection unexpectedly closed (57 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] ???