rsync - Aug 2008 - mirroring a huge file using rsync while it's being written to.

Hi All,

In the application that I am building, I've to "incrementally"
copy several
files of large sizes (around 1 GB) from several machines (some 300 in
number) spread across various geographic locations. We've been using scp
until now and are exploring rsync for the same. In my application, these
files are being written to (still being updated) while rsync will be reading
from them. To clarify my point further, we invoke rsync to read a file from
a remote server where it is being continuously written to by another daemon.
So we want to rsync repeatedly every few minutes to get the latest appended
information to the remote file.

I should like to know:

(1) While reading from a file, can rsync be made to wait for some duration
after it has read the last byte in the file, to re-read the next
installments of additions done to it by the write daemon on the remote
server?

(2) When we know that the only additions/changes to the remote file will
happen at its tail, i.e, all the "writes" to the file are basically
appending to the file.

(3) While rsync is updating the local copy of the remote file (at the local
end), and is merely appending to it all the time, can I read from it
"live",
ie, read it while it is being appended to. I am invoking rsync with the
--inplace option. Will the data be inconsistent while I am reading from it?


thanks
-y
-------------- next part --------------
HTML attachment scrubbed and removed

Hi All,


In the application that I am building, I've to "incrementally"
copy several
files of large sizes (around 1 GB) from several machines (some 300 in
number) spread across various geographic locations. We've been using scp
until now and are exploring rsync for the same. In my application, these
files are being written to (still being updated) while rsync will be reading
from them. To clarify my point further, we invoke rsync to read a file from
a remote server where it is being continuously written to by another daemon.
So we want to rsync repeatedly every few minutes to get the latest appended
information to the remote file.

I should like to know:

(1) While reading from a file, can rsync be made to wait for some duration
after it has read the last byte in the file, to re-read the next
installments of additions done to it by the write daemon on the remote
server?

(2) When we know that the only additions/changes to the remote file will
happen at its tail, i.e, all the "writes" to the file are basically
appending to the file.

(3) While rsync is updating the local copy of the remote file (at the local
end), and is merely appending to it all the time, can I read from it
"live",
ie, read it while it is being appended to. I am invoking rsync with the
--inplace option. Will the data be inconsistent while I am reading from it?


thanks
-y
-------------- next part --------------
HTML attachment scrubbed and removed

On Tue, 2008-08-05 at 17:42 +0530, Yogesh Kumar wrote:
> In the application that I am building, I've to
"incrementally" copy
> several files of large sizes (around 1 GB) from several machines (some
> 300 in number) spread across various geographic locations. We've been
> using scp until now and are exploring rsync for the same. In my
> application, these files are being written to (still being updated)
> while rsync will be reading from them. To clarify my point further, we
> invoke rsync to read a file from a remote server where it is being
> continuously written to by another daemon. So we want to rsync
> repeatedly every few minutes to get the latest appended information to
> the remote file.
> 
> I should like to know:
> 
> (1) While reading from a file, can rsync be made to wait for some
> duration after it has read the last byte in the file, to re-read the
> next installments of additions done to it by the write daemon on the
> remote server?
No.  You'll have to run rsync in a loop or from a cron job scheduled to
execute every few minutes.
> (2) When we know that the only additions/changes to the remote file
> will happen at its tail, i.e, all the "writes" to the file are
> basically appending to the file. 
Then use --append to avoid the work of rechecking the beginning of the
file.
> (3) While rsync is updating the local copy of the remote file (at the
> local end), and is merely appending to it all the time, can I read
> from it "live", ie, read it while it is being appended to. I am
> invoking rsync with the --inplace option. Will the data be
> inconsistent while I am reading from it?
Yes, you can read the destination file while rsync is appending to it.
The file will be incomplete but all the data in it so far will be
correct (as with wget).
      * 
Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url :
http://lists.samba.org/archive/rsync/attachments/20080805/7b55f4ae/attachment.bin