rsync - Dec 2002 - Suggestion: rsync and direct IO

We have some cases when copying indexes that we'd like rsync to avoid
the system cache when copying a chunk of data from some other machine. 
I am probably going to modify our own copy of rsync to do this, but if
we could write directly to the disk using direct-io (or from for that
matter) we could avoid polluting the operating system's buffer cache
before we're ready to use the new data.  Obviously this feature isn't
tied to Direct-IO, so I'd call the new feature something like
-avoid-system-cache or something.  I will probably make the feature an
optional one because I can imagine having a situation where sometimes
I'd want it to hit the system filecache and sometimes I wouldn't.

	Sample use:  Here at Altavista we copy large indexes around (multiple
gigs of data).   Obviously the new version of the index isn't good until
the whole thing is ready.  The old version of the index is using the
operating system's buffer cache to increase performance.  Thus you don't
want to make the old version of the index stale until you're done with
it.  Also, you don't need the new version of the index in the system
filecache until you're ready to use it.  Here's where avoiding writing
to the system cache when copying the new file to the system is a good
thing.

	Sample non-use:  You've got an FTP server where people can use rsync to
make mirrors of your site.  You expect lots of people to use the mirror
and get the same files (as well as normal users downloading the files)
so you want rsyncd to put the files in the cache for other people
accessing your mirror.  (Although if you think about it, the
infrequently used files are going to go through the cache as well as the
frequent ones and you'd want to use the cache for the ones already in
memory and not for the infrequent ones -- bleah).



	On another topic while I'm thinking about it -- does any one know of a
reason that you can't set socket options in rsync like you can in rsyncd
-- if you have a push going in the opposite direction to the server it'd
be nice to have a larger outgoing buffer size (useful for cross country
pushes).  We have a local version that has been modified to set the
buffer size and it performs 3 times faster than the unmodified version.

From: David Sisson [mailto:David.Sisson@av.com] asks:
> [...]
> If we could write directly to the disk using direct-io (or from for that
> matter) we could avoid polluting the operating system's buffer cache
> before we're ready to use the new data.  Obviously this feature
isn't
> tied to Direct-IO, so I'd call the new feature something like
> -avoid-system-cache or something.  I will probably make the feature an
> optional one because I can imagine having a situation where sometimes
> I'd want it to hit the system filecache and sometimes I wouldn't.
I think such a feature would be highly desireable. Here at Stratus our disk
cache manager has a feature the caller can set called the "transient file
attribute" that does exactly this; it discards the data block from the disk
cache as soon as the application is done with it (the default is to hold a
referenced data block in the disk cache for several minutes).  We encourage
our customers to enable this bit whenever they are referencing data in a
"single use" mode, such as when they are scanning files to prepare
reports.
As you say, it prevents such activities from interfering with the online
transaction processing system's data.

But is there an OS-independent way to set this attribute?  I haven't
memorized the POSIX standard, but I don't recall seeing a way to set or get
such an attribute.

My general feeling is that rsync should, by default, avoid the system cache,
given that it will reference any given block only once. It is just chance
that data that it brings in would be useful to another app.

PG
--
Paul Green, Senior Technical Consultant, Stratus Computer, Inc.
Voice: +1 978-461-7557; FAX: +1 978-461-3610; Video on request.
Speaking from Stratus not for Stratus

On Wed, Dec 04, 2002 at 03:41:33PM -0800, David Sisson
wrote:
> 	We have some cases when copying indexes that we'd like rsync to avoid
> the system cache when copying a chunk of data from some other machine. 
> I am probably going to modify our own copy of rsync to do this, but if
> we could write directly to the disk using direct-io (or from for that
> matter) we could avoid polluting the operating system's buffer cache
> before we're ready to use the new data.  Obviously this feature
isn't
> tied to Direct-IO, so I'd call the new feature something like
> -avoid-system-cache or something.  I will probably make the feature an
> optional one because I can imagine having a situation where sometimes
> I'd want it to hit the system filecache and sometimes I wouldn't.
Yeah, that sounds like a very good idea.
> 	On another topic while I'm thinking about it -- does any one know of a
> reason that you can't set socket options in rsync like you can in
rsyncd
> -- if you have a push going in the opposite direction to the server
it'd
> be nice to have a larger outgoing buffer size (useful for cross country
> pushes).  We have a local version that has been modified to set the
> buffer size and it performs 3 times faster than the unmodified version.
Probably just an oversight, although I would have thought the buffer size
would be up to the transport program (rsh or ssh).



On Thu, Dec 05, 2002 at 10:01:08AM -0800, Jos Backus
wrote:
> On Thu, Dec 05, 2002 at 07:39:53AM -0500, Green, Paul wrote:
> > But is there an OS-independent way to set this attribute?  I
haven't
> > memorized the POSIX standard, but I don't recall seeing a way to
set or get
> > such an attribute.
>
> Fyi: Solaris has directio(3C), FreeBSD has open(..., O_DIRECT).
What about Linux?  There's an O_DIRECT but the comment in asm/fcntl.h says
it is currently ignored.

- Dave Dykstra