rsync - Apr 2015 - Options for a "I'm done" flag file

As part of my backup system, I use Rsync to keep a copy of each server on one
central backup server. This backup server then uses StoreBackup to keep multiple
iterations of each clone directory.
So that the StoreBackup archives don't keep adding "redundant" and
misleading backups, I update a flag file with the current date/time before doing
the Rsync update, and test to see if this file is newer than the one in the
latest StoreBackup backup. If it isn't, then I skip the StoreBackup for that
server.

The end result is that if a system is down or out of communication (one or two
are at sites that can be offline for days), then the list of backups in
StoreBackup will reflect that. Eg, if the system did a sync on the 1st, but not
on 2nd - 5th, then there will be no backups for 2nd-5th, and when looking later
I won't be "fooled" into thinking that I have a backup from (say)
the 4th.

Where this breaks down is if the sync fails part way through. The flag file has
already been synced, so I have multiple backups which aren't actually
complete.
I actually have this at the moment. Just put a small system on a customer site,
it has a database that creates 1GB journal files (not that it handles anything
like that volume of data), and at the moment their connectivity is a bit flakey.

My first thought was "do the flag file last", but a quick search
confirms what I thought - that there isn't an option for this.

So, does anyone have any suggestions how I might reasonably easily get the
ability for my script to see if the previous sync completed ?

rsync -av /src/ /dst/ && touch /dst/done

That should do it as the touch only happens if rsync exits with a code of
0.  If you need to consider other non zero exit code, it is still doable,
just a bit more shell code.

There are surely other options as well, but this is probably the most
simple.
On Apr 28, 2015 3:47 AM, "Simon Hobson" <linux at
thehobsons.co.uk> wrote:
> As part of my backup system, I use Rsync to keep a copy of each server on
> one central backup server. This backup server then uses StoreBackup to keep
> multiple iterations of each clone directory.
> So that the StoreBackup archives don't keep adding
"redundant" and
> misleading backups, I update a flag file with the current date/time before
> doing the Rsync update, and test to see if this file is newer than the one
> in the latest StoreBackup backup. If it isn't, then I skip the
StoreBackup
> for that server.
>
> The end result is that if a system is down or out of communication (one or
> two are at sites that can be offline for days), then the list of backups in
> StoreBackup will reflect that. Eg, if the system did a sync on the 1st, but
> not on 2nd - 5th, then there will be no backups for 2nd-5th, and when
> looking later I won't be "fooled" into thinking that I have a
backup from
> (say) the 4th.
>
> Where this breaks down is if the sync fails part way through. The flag
> file has already been synced, so I have multiple backups which aren't
> actually complete.
> I actually have this at the moment. Just put a small system on a customer
> site, it has a database that creates 1GB journal files (not that it handles
> anything like that volume of data), and at the moment their connectivity is
> a bit flakey.
>
> My first thought was "do the flag file last", but a quick search
confirms
> what I thought - that there isn't an option for this.
>
> So, does anyone have any suggestions how I might reasonably easily get the
> ability for my script to see if the previous sync completed ?
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20150428/923cbda0/attachment.html>

Michael Johnson - MJ <mj at revmj.com> wrote:
> rsync -av /src/ /dst/ && touch /dst/done
Aaaaahhhh, knew I'd miss some detail.
All the syncs are pushed to the backup server.

But that does give me an idea. I guess I could do that on the source, then sync
the flag file over.
rsync -avH ${other_gubbins} / user at remote.machine:/dest/ &&
  touch /etc/donefile &&
  rsync -av ${some_other_gubbins} /etc/donefile user at remote.machine:/dest/

That (or some variation of it) could work.

For a push job.

Run the rsync for the files,
if the exit code is 0, create the flag file and then rsync just that 
file on its own.




On 28/04/2015 10:38 p.m., Simon Hobson wrote:
> As part of my backup system, I use Rsync to keep a copy of each server on
one central backup server. This backup server then uses StoreBackup to keep
multiple iterations of each clone directory.
> So that the StoreBackup archives don't keep adding
"redundant" and misleading backups, I update a flag file with the
current date/time before doing the Rsync update, and test to see if this file is
newer than the one in the latest StoreBackup backup. If it isn't, then I
skip the StoreBackup for that server.
>
> The end result is that if a system is down or out of communication (one or
two are at sites that can be offline for days), then the list of backups in
StoreBackup will reflect that. Eg, if the system did a sync on the 1st, but not
on 2nd - 5th, then there will be no backups for 2nd-5th, and when looking later
I won't be "fooled" into thinking that I have a backup from (say)
the 4th.
>
> Where this breaks down is if the sync fails part way through. The flag file
has already been synced, so I have multiple backups which aren't actually
complete.
> I actually have this at the moment. Just put a small system on a customer
site, it has a database that creates 1GB journal files (not that it handles
anything like that volume of data), and at the moment their connectivity is a
bit flakey.
>
> My first thought was "do the flag file last", but a quick search
confirms what I thought - that there isn't an option for this.
>
> So, does anyone have any suggestions how I might reasonably easily get the
ability for my script to see if the previous sync completed ?
>
>