rsync - Sep 2006 - post-xfer exec -- more juice!

Hello,

I'm attempting to use the pre-xfer exec and post-xfer exec utilities to
replace a complicated script which tails the rsync log and does A Bunch
Of Stuff at the beginning and completion of rsync transfers.
(It will also allow us to throw away a nasty hack which forces rsync to
log the fact that is exiting - trying to provide a catch for /all/
possible exit conditions (ie, strings in the log file) seemed
overwhelming.)

However, my efforts are hampered by not knowing the name of the
transferred file in the post-xfer script - the fine print says
"RSYNC_REQUEST: (pre-xfer only)"

This pair of config items is _almost_ perfect for managing all the
wonderful things which we need to have done to freshly arrived files.
As well, we have a situation where the state of the filesystem must be
guaranteed unchanging whilst an rsync is in process (sending data), so
we set a flag at the start of the transfer and remove it at the end. 
pre-exec is perfect for doing this, and so would be post-exec -- _if_ we
knew which rsync had just completed!!

Am I missing some trivial way to give this information to a post-exec
script?
Are there some logistical restrictions which would prevent me from
(easily) modifying the rsync source to send this information to the
post-exec script?


cheers,
Mike

On Fri, Sep 08, 2006 at 12:42:16PM +0800, Mike Daws
wrote:
> Am I missing some trivial way to give this information to a post-exec
> script?
The easiest thing to do with the current system is to use some kind of
caching mechanism in the pre-xfer exec to note the RSYNC_REQUEST string
for use in the  post-xfer exec.  Unfortunately, the best you can do with
the information currently at hand is to cache based on a combination of
fairly unique items, such as RSYNC_HOST_ADDR + RSYNC_USER_NAME +
RSYNC_MODULE_NAME.  (You can't use something like parent PID, because
the pre-xfer exec isn't the child of the transfer PID.)
> Are there some logistical restrictions which would prevent me from
> (easily) modifying the rsync source to send this information to the
> post-exec script?
The xfer forks all happen prior to the process receiving the request
info because it has already run chroot and changed the user permissions
when starting the transfer.  A special pipe is used to communicate the
request info to the post-xfer process.  There are several things that
could be done to rsync to effect the change you want:

1. Use a similar pipe to communicate the info to the pre-xfer process.

2. Fork a single pre/post xfer process that uses the single pipe's info
   to setup the environment and then handles the pre- & post- xfer
   commands (using another sub-process for the pre-exec command if it
   needed to run both).

3. Change the code to not chroot & set user permissions until after the
   start of the transfer begins, making the code in this first part of
   the transfer critical to get right to avoid security bugs.

I personally think that option 2 is a good choice and could be put into
the current code without too much difficulty.

..wayne..

The way I have accomplished this is to modify the rsync request slightly and
cache the modulename and request into a database.  In my case, this provides
me with a unique key in the database so I know the requests match up.  On
the client side I have added the top-level path into the request (e.g. rsync
-optionshere /usr/bin username@backupserver::modulename/bin/).  This is the
only way I can get the path into the request but it does work.  On the
pre-xfer side I insert the modulename and path as well as other pertinent
information.  On the post-xfer side I fetch the record from the database
matching the modulename and pull in the request value to get the path.  

A bit of a hack but it has worked for us to track all open connections to
the rsync server and create a database of exit codes by module (user
surrogate in our world) and path.

Rob