rsync - May 2004 - File sync/delete question

Hi,

I need to solve a specific problem and I think that I need an extension to 
rsync for this to work :)
I have an FTP server in the DMZ and a samba server in my intranet. The ftp 
server has an "incoming" directory, where people can put uploads and
it is
also running rsync in daemon mode.
The samba server syncs the incoming directory of the ftp server every 
minute to make access to the uploaded files easier (I don't want the 
internal people to deal with ftp/sftp to get uploaded files. In fact, 
there is also an "outgoing" directory, that is also on the samba
server
and mirrored to the ftp server, so incoming should work the same way).

My problem is, that any file that is deleted on the samba server is synced 
again from the ftp server a minute later. I use
rsync -rvt $FTPSRV /ftp_mirror/incoming
for syncing. The --delete option just doesn't fit here, because I want to 
delete on the SENDING side, not on mine. Also, still-running uploads would 
not be completely rsync'd if the file would be deleted while being 
uploaded.

Imagine this happenings:
10:00: file "a.txt" gets uploaded to ftp server
10:01: file "a.txt" is rsynced to samba server
10:02: file "a.txt" gets deleted from samba server (after rsync run)
10:03: file would again be synced :(

Now I think that I need a new option for rsync:
"Transfer all files, and remotely delete those that must have been 
transferred before and are not here anymore".

E.g. rsync -rvt --delete-remote-before `cat last-sync-date` ...
last-sync-date is a file that contains the unix timestamp (or any other 
format) of the last syncronization. It could probably also be created by 
rsync after successful completion, or created on return code 0 by my 
script.

Again my scenario from above:
10:00: file "a.txt" gets uploaded to ftp server
10:01: file "a.txt" is rsynced to samba server
10:02: file "a.txt" gets deleted from samba server (after rsync run)
10:03: last rsync run was at 10:02, but file "a.txt" on server was
        modified at 10:00, so it must already have been transferred
        --> delete it on server

Is this feasible? Is anyone willing to implement this, or shall I try on 
my own (haha, of course you would like me to do, but I'm asking because I 
don't have too much time ;-)

kind regards
Markus Gaugusch
-- 
__________________    /"\ 
Markus Gaugusch       \ /    ASCII Ribbon Campaign
markus(at)gaugusch.at  X     Against HTML Mail
                       / \

On Fri 21 May 2004, Markus Gaugusch wrote:
> 
> Imagine this happenings:
> 10:00: file "a.txt" gets uploaded to ftp server
> 10:01: file "a.txt" is rsynced to samba server
> 10:02: file "a.txt" gets deleted from samba server (after rsync
run)
> 10:03: file would again be synced :(
> 
> Now I think that I need a new option for rsync:
> "Transfer all files, and remotely delete those that must have been 
> transferred before and are not here anymore".
It may be that unison will do what you want.


Paul Slootman

I think this would be an excellent addition to rsync and could find many
applications for it myself.  However like Markus I haven't the time or
the programing knowledge to make it myself.  

just wanted to get my two cents worth in,

Matt

>>> Markus Gaugusch <markus@gaugusch.at> 05/21/04 04:10PM
>>>
Hi,

I need to solve a specific problem and I think that I need an extension
to 
rsync for this to work :)
I have an FTP server in the DMZ and a samba server in my intranet. The
ftp 
server has an "incoming" directory, where people can put uploads and
it
is 
also running rsync in daemon mode.
The samba server syncs the incoming directory of the ftp server every 
minute to make access to the uploaded files easier (I don't want the 
internal people to deal with ftp/sftp to get uploaded files. In fact, 
there is also an "outgoing" directory, that is also on the samba
server

and mirrored to the ftp server, so incoming should work the same way).

My problem is, that any file that is deleted on the samba server is
synced 
again from the ftp server a minute later. I use
rsync -rvt $FTPSRV /ftp_mirror/incoming
for syncing. The --delete option just doesn't fit here, because I want
to 
delete on the SENDING side, not on mine. Also, still-running uploads
would 
not be completely rsync'd if the file would be deleted while being 
uploaded.

Imagine this happenings:
10:00: file "a.txt" gets uploaded to ftp server
10:01: file "a.txt" is rsynced to samba server
10:02: file "a.txt" gets deleted from samba server (after rsync run)
10:03: file would again be synced :(

Now I think that I need a new option for rsync:
"Transfer all files, and remotely delete those that must have been 
transferred before and are not here anymore".

E.g. rsync -rvt --delete-remote-before `cat last-sync-date` ...
last-sync-date is a file that contains the unix timestamp (or any other

format) of the last syncronization. It could probably also be created
by 
rsync after successful completion, or created on return code 0 by my 
script.

Again my scenario from above:
10:00: file "a.txt" gets uploaded to ftp server
10:01: file "a.txt" is rsynced to samba server
10:02: file "a.txt" gets deleted from samba server (after rsync run)
10:03: last rsync run was at 10:02, but file "a.txt" on server was
        modified at 10:00, so it must already have been transferred
        --> delete it on server

Is this feasible? Is anyone willing to implement this, or shall I try
on 
my own (haha, of course you would like me to do, but I'm asking because
I 
don't have too much time ;-)

kind regards
Markus Gaugusch
-- 
__________________    /"\ 
Markus Gaugusch       \ /    ASCII Ribbon Campaign
markus(at)gaugusch.at  X     Against HTML Mail
                       / \