rsync - Nov 2004 - Flag file management techniques using rsync

G'day,

I am just getting into rsync and have been very impressed with performance
and reliability.  There is one thing I haven't worked out how to do yet and
I haven't found much with several Google sessions or by way of consulting
the examples.

I am looking for a way to check for (or duplicate) single flag files to and
from a remote host (without any dummy spit errors when it is not present)
and ideally avoiding the need for a specialised sub directory for each flag
file.  For example to see if the remote host is otherwise occupied (i.e.. in
the middle of building the import data files) I was expecting this might
work for a "pull" action...

	rsync -vptgo rmtusr@rmthost.net:/somepath/file1.flg file1.flg
      while [ -f file1.flg ]
		do
		sleep 30
		rsync -vptgo --delete rmtusr@rmthost.net:/somepath/file1.flg file1.flg
	done
	# Now safe to transfer the data
	rsync -rvptgo rmtusr@rmthost.net:/somepath/export import/

... then I want to drop a flag file on the remote host before a "push"
rsync
starts and then remove it on completion. eg.

	touch file2.flg
	rsync -vptgo file2.flg rmtusr@rmthost.net:/somepath/file2.flg
	rsync -rvptgo export/ rmtusr@rmthost.net:/somepath/import/
	rm -f file2.flg
	rsync -vptgo --delete file2.flg rmtusr@rmthost.net:/somepath/file2.flg

In my initial attempts the --delete options didn't seem to work to well on
single files.  I had thought of using discrete subdirectories for each flag
file so I can use the -r option with the --delete but I was hoping to avoid
this so a general solution would be much easier to deploy (and scale).

Is there a better way to achieve these goals using rsync or perhaps are
there better options using ssh alone?  Regards, Frank.

On Sun, Nov 07, 2004 at 10:54:56PM +1100, Frank Hamersley
wrote:
> I am looking for a way to check for (or duplicate) single flag files
> to and from a remote host (without any dummy spit errors when it is
> not present) and ideally avoiding the need for a specialised sub
> directory for each flag file.
For such a simple operation, you should be just using ssh (or rsh) to
run simple shell commands, such as "touch" and "rm". 
Certainly rsync
could be used, and the mailing list has just recently explained twice
now how to delete an individual file using rsync (for those times that
a remote shell is not involved), but since you're using a remote shell,
you should use it to run the necessary shell commands:

    ssh -l rmtusr rmthost.net touch /somepath/file1.flg
    ssh -l rmtusr rmthost.net rm /somepath/file1.flg
    ssh -l rmtusr rmthost.net test -e /somepath/file1.flg

You can use the last command to cache the file1.flg locally if you need
to test it more than once (or just use the result if you don't).  Keep
in mind that on failure you'll probably want to distinguish between the
file not being there (error return 1) and the remote connection not
working (error return 255).

Also, if multiple processes are all competing for the same lock file,
you'll want to replace the touch with a script/program on the remote
host that does an atomic locking operation that only lets one process
acquire the lock (sleeping or returning failure, as appropriate).

..wayne..

G'day,

I am just getting into rsync and have been very impressed with performance
and reliability.  There is one thing I haven't worked out how to do yet and
I haven't found much with several Google sessions or by way of consulting
the examples.

I am looking for a way to check for (or duplicate) single flag files to and
from a remote host (without any dummy spit errors when it is not present)
and ideally avoiding the need for a specialised sub directory for each flag
file.  For example to see if the remote host is otherwise occupied (i.e.. in
the middle of building the import data files) I was expecting this might
work for a "pull" action...

	rsync -vptgo rmtusr@rmthost.net:/somepath/file1.flg file1.flg
      while [ -f file1.flg ]
		do
		sleep 30
		rsync -vptgo --delete rmtusr@rmthost.net:/somepath/file1.flg file1.flg
	done
	# Now safe to transfer the data
	rsync -rvptgo rmtusr@rmthost.net:/somepath/export import/

... then I want to drop a flag file on the remote host before a "push"
rsync
starts and then remove it on completion. eg.

	touch file2.flg
	rsync -vptgo file2.flg rmtusr@rmthost.net:/somepath/file2.flg
	rsync -rvptgo export/ rmtusr@rmthost.net:/somepath/import/
	rm -f file2.flg
	rsync -vptgo --delete file2.flg rmtusr@rmthost.net:/somepath/file2.flg

In my initial attempts the --delete options didn't seem to work to well on
single files.  I had thought of using discrete subdirectories for each flag
file so I can use the -r option with the --delete but I was hoping to avoid
this so a general solution would be much easier to deploy (and scale).

Is there a better way to achieve these goals using rsync or perhaps are
there better options using ssh alone?

Regards, Frank.