rsync - Jan 2004 - Resolving problems in the generator->receiver pipes

When I was working on the the hard-link change, I noticed that many
of the hard-link verbose messages were getting lost.  These messages
get output very near the end of the transfer, and it turns out that
the reason for the loss was that there are two pipes flowing from
the generator and the receiver, and it was possible for the "we're
all done" message to get received down the redo pipe without all of
the messages getting sent down the error pipe.  It's also a long-
known bug that the redo pipe can clog (but only if a really large
number of redo items build up).

In looking at this code, I figured out that it would be much better
to just dump the redo pipe and use the error pipe as a message pipe.
Since it already uses the same transfer protocol as the multiplexed
IO between other components, it turned out to be a very simple thing
to add a REDO and a DONE message to the existing code.  This gets
rid of the redo pipe and all of its problems.

At the same time I though it would be good to separate the existing
message-sending code out of the log-file code.  This is because one
other non-logging message is already supported (the "data" stream in
multiplexed IO) so the addition of two extra messages makes it a
better fit for the io.c code than the log.c code (i.e. when the log
code needs to send a message to another program, it calls the code
in io.c to do it instead of doing it itself).

Finally, the flushing code was tweaked to allow it to signal that
the writing out of the receiver->generator pipe should be completely
flushed (from the in-memory list of messages).  This allows the
final flushing to complete more efficiently than calling io_flush()
in a loop.

In my exuberance for having finally solved the redo-hang problem in
a much better way than my previous suggested fixes for it, I went
ahead and checked in my changes.  However, feedback is still
gratefully accepted.  The changes can be found here:

    http://www.blorf.net/redo.patch

..wayne..

On Thu, Jan 15, 2004 at 12:11:39PM -0800, Wayne Davison
wrote:
> When I was working on the the hard-link change, I noticed that many
> of the hard-link verbose messages were getting lost.  These messages
> get output very near the end of the transfer, and it turns out that
> the reason for the loss was that there are two pipes flowing from
> the generator and the receiver, and it was possible for the "we're
> all done" message to get received down the redo pipe without all of
> the messages getting sent down the error pipe.  It's also a long-
> known bug that the redo pipe can clog (but only if a really large
> number of redo items build up).
> 
> In looking at this code, I figured out that it would be much better
> to just dump the redo pipe and use the error pipe as a message pipe.
> Since it already uses the same transfer protocol as the multiplexed
> IO between other components, it turned out to be a very simple thing
> to add a REDO and a DONE message to the existing code.  This gets
> rid of the redo pipe and all of its problems.
> 
> At the same time I though it would be good to separate the existing
> message-sending code out of the log-file code.  This is because one
> other non-logging message is already supported (the "data" stream
in
> multiplexed IO) so the addition of two extra messages makes it a
> better fit for the io.c code than the log.c code (i.e. when the log
> code needs to send a message to another program, it calls the code
> in io.c to do it instead of doing it itself).
> 
> Finally, the flushing code was tweaked to allow it to signal that
> the writing out of the receiver->generator pipe should be completely
> flushed (from the in-memory list of messages).  This allows the
> final flushing to complete more efficiently than calling io_flush()
> in a loop.
> 
> In my exuberance for having finally solved the redo-hang problem in
> a much better way than my previous suggested fixes for it, I went
> ahead and checked in my changes.  However, feedback is still
> gratefully accepted.  The changes can be found here:
> 
>     http://www.blorf.net/redo.patch
What i looked at from the cvs digest looked like decent
code.  It was just lacking the overview you provided here.

This sounds pretty good.  Just needs some thorough testing.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt