thr3ads.net - rsync - [patch] rsync over existing I/O connections (new feature) [Mar 2003]

If this information is useful, please help other people find it:
Share via:

Tony Clayton

2003-Mar-08 09:40 UTC

[patch] rsync over existing I/O connections (new feature)

Hello,

I wrote this patch that allows you to run rsync over an existing I/O
connection, instead of creating a new rsh or socket connection.

For example, I have a client and server that talk over ssh via a simple
custom protocol.  I want to use the existing ssh connection to transfer files,
but I want the flexibility of rsync to do it.  

With this patch, I can do the following in my client (assuming input fd of 0
and output fd of 1):

- send the server a START_RSYNC request on stdout (my protocol)
- receive a START_RSYNC_ACK reply on fd stdin (my protocol)
- clear the close-on-exec flags for the input and output (ssh) descriptors if
necessary using fcntl (probably not required if using stdin/stdout)
- fork/exec: rsync --partial --quiet --fd-in 0 --fd-out 1 :. myfile
- return to blocking I/O for stdin/stdout using fcntl (rsync uses non-blocking)
- continue on with my protocol 

And on the server, I do the following:

- receive START_RSYNC request from client (my protocol)
- send START_RSYNC_REPLY from client (my protocol)
- fork/exec: rsync -p --server --sender --quiet . /path/to/file/on/server
- return to blocking I/O for stdin/stdout using fcntl (rsync uses non-blocking)
- continue on with my protocol 


I find this extremely useful. Hopefully it's useful enough to others to get
added to the rsync codebase in one form or another so I don't have to keep
patching rsync.

This patch was made against rsync 2.5.4.

cheers,

Tony

rsync-2.5.4-force_fd.apc.patch
-----

diff -urN rsync-2.5.4-2/main.c rsync-2.5.4-2apc/main.c
--- rsync-2.5.4-2/main.c	Mon Sep 30 10:09:17 2002
+++ rsync-2.5.4-2apc/main.c	Thu Mar  6 15:29:27 2003
@@ -665,6 +665,8 @@
 	extern int whole_file;
 	extern int write_batch;
 	extern int read_batch;
+	extern int force_f_in;
+	extern int force_f_out;
 	int rc;
 
 	/* Don't clobber argv[] so that ps(1) can still show the right
@@ -770,9 +772,14 @@
 		list_only = 1;
 	}
 	
-	pid =
do_cmd(shell_cmd,shell_machine,shell_user,shell_path,&f_in,&f_out);
+	if (force_f_in != -1) {
+	  ret = client_run(force_f_in, force_f_out, -1, argc, argv);
+	}
+	else {
+	  pid =
do_cmd(shell_cmd,shell_machine,shell_user,shell_path,&f_in,&f_out);
 	
-	ret = client_run(f_in, f_out, pid, argc, argv);
+	  ret = client_run(f_in, f_out, pid, argc, argv);
+	}
 
 	fflush(stdout);
 	fflush(stderr);
diff -urN rsync-2.5.4-2/options.c rsync-2.5.4-2apc/options.c
--- rsync-2.5.4-2/options.c	Mon Sep 30 10:09:17 2002
+++ rsync-2.5.4-2apc/options.c	Thu Mar  6 15:34:59 2003
@@ -106,6 +106,9 @@
 int write_batch = 0;
 int read_batch = 0;
 
+int force_f_in = -1;
+int force_f_out = -1;
+
 char *backup_suffix = BACKUP_SUFFIX;
 char *tmpdir = NULL;
 char *compare_dest = NULL;
@@ -261,6 +264,7 @@
   rprintf(F,"     --bwlimit=KBPS          limit I/O bandwidth, KBytes per
second\n");
   rprintf(F,"     --write-batch=PREFIX    write batch fileset starting
with PREFIX\n");
   rprintf(F,"     --read-batch=PREFIX     read batch fileset starting with
PREFIX\n");
+  rprintf(F,"     --fd-in=FD --fd-out=FD  use these open file descriptors
for input and output\n");
   rprintf(F," -h, --help                  show this help screen\n");
 #ifdef INET6
   rprintf(F," -4                          prefer IPv4\n");
@@ -356,6 +360,8 @@
   {"hard-links",      'H', POPT_ARG_NONE,  
&preserve_hard_links , 0, 0, 0 },
   {"read-batch",       0,  POPT_ARG_STRING, &batch_prefix,
OPT_READ_BATCH, 0, 0 },
   {"write-batch",      0,  POPT_ARG_STRING, &batch_prefix,
OPT_WRITE_BATCH, 0, 0 },
+  {"fd-in",            0,  POPT_ARG_INT,    &force_f_in , 0, 0, 0
},
+  {"fd-out",           0,  POPT_ARG_INT,    &force_f_out , 0, 0,
0 },
 #ifdef INET6
   {0,		      '4', POPT_ARG_VAL,    &default_af_hint,   AF_INET , 0,
0 },
   {0,		      '6', POPT_ARG_VAL,    &default_af_hint,   AF_INET6 ,
0, 0 },
@@ -586,6 +592,20 @@
 	    return 0;
 	}
 
+	if ((force_f_in == -1) && (force_f_out != -1)) {
+	    snprintf(err_buf,sizeof(err_buf),
+		"fd-out can not be used without fd-in\n");
+	    rprintf(FERROR,"ERROR: fd-out can not be used without fd-in\n");
+	    return 0;
+	}
+	if ((force_f_out == -1) && (force_f_in != -1)) {
+	    snprintf(err_buf,sizeof(err_buf),
+		"fd-in can not be used without fd-out\n");
+	    rprintf(FERROR,"ERROR: fd-in can not be used without fd-out\n");
+	    return 0;
+	}
+
+
         *argv = poptGetArgs(pc);
         if (*argv)
                 *argc = count_args(*argv);

jw schultz

2003-Mar-08 10:39 UTC

head link

[patch] rsync over existing I/O connections (new feature)

On Fri, Mar 07, 2003 at 05:40:17PM -0500, Tony Clayton
wrote:> Hello,
> 
> I wrote this patch that allows you to run rsync over an existing I/O
> connection, instead of creating a new rsh or socket connection.
> 
> For example, I have a client and server that talk over ssh via a simple
> custom protocol.  I want to use the existing ssh connection to transfer
files,
> but I want the flexibility of rsync to do it.  
> 
> With this patch, I can do the following in my client (assuming input fd of
0
> and output fd of 1):
> 
> - send the server a START_RSYNC request on stdout (my protocol)
> - receive a START_RSYNC_ACK reply on fd stdin (my protocol)
> - clear the close-on-exec flags for the input and output (ssh) descriptors
if
> necessary using fcntl (probably not required if using stdin/stdout)
> - fork/exec: rsync --partial --quiet --fd-in 0 --fd-out 1 :. myfile
> - return to blocking I/O for stdin/stdout using fcntl (rsync uses
non-blocking)
> - continue on with my protocol 
> 
> And on the server, I do the following:
> 
> - receive START_RSYNC request from client (my protocol)
> - send START_RSYNC_REPLY from client (my protocol)
> - fork/exec: rsync -p --server --sender --quiet . /path/to/file/on/server
> - return to blocking I/O for stdin/stdout using fcntl (rsync uses
non-blocking)
> - continue on with my protocol 
> 
> 
> I find this extremely useful. Hopefully it's useful enough to others to
get
> added to the rsync codebase in one form or another so I don't have to
keep
> patching rsync.
> 
> This patch was made against rsync 2.5.4.
Interesting idea.  In terms of invoking rsync as part of a
larger communications tool it makes some sense.

The logic you put in really already exists under the
--server option.  I think the only part missing in current
code is what to do with the log messages.  Connection to
arbitrary file descriptors should not be necessary.
am_server is currently only tested for true and set to
either 1 or 0. 
#define AM_SERVER 1
#define AM_CLIENT 2
and a --client option that sets am_server to AM_CLIENT, then
make a few changes in log.c so that instead of testing for
non-zero am_server is tested for AM_SERVER or AM_CLIENT
to direct the log messages where they should go.

That should allow you do do something like
	rsync -p --client . myfile


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Possibly Parallel Threads

Search for more reasonably related threads

rsync - Mar 2003 - [patch] rsync over existing I/O connections (new feature)

[patch] rsync over existing I/O connections (new feature)

[patch] rsync over existing I/O connections (new feature)

Possibly Parallel Threads