rsync - Jun 2016 - rsync Digest, Vol 162, Issue 18

Folks,

Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
  > The scripts I use analyze the rsync log after it completes and then 
sftp's a summary to the root of the just completed rsync.
  > If no summary is found or the summary is that it failed, the folder 
rotation for that set is skipped and that folder is re-used on the 
subsequent rsync.
  > The key here is that the folder rotation script runs separately from 
the rsync script(s).

That is what we found to be important some years ago - do most of the 
management outside of the scripts that do rsync.

In particular, our scripts prepare a target backup directory with a name 
of the form:
	slash.20151122.seq.1379.current
   - the name of the area being backed-up is obvious, as is the date.
We use Tower of Hanoi management (see the update at 
https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), so 
embed a sequence number in the backup directory name, and the suffix 
.current to indicate a current backup target.

we can keep on trying the rsync backups until we get success, and the 
scripts then remove the .current suffix.  This works because the rsync 
includes the --delete option, used since the backup target can be a 
recycled hard-linked directory: also recommended.

  > For each entity I want to rsync, I create a named folder to identify 
it and the rsync'd data is held in sub-folders:
  > daily.[1-7] and monthly.[1-3]
  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
  > Then the rotation script checks daily.0/rsync.summary - and if it 
worked, it removes daily.7 and renames the daily folders.
  > On the first of the month, the rotation script removes monthly.3, 
renames the other 2 and makes a complete hard-link copy of daily.1 to 
monthly.1
  > It's been running now for about 4 years and, in my environment, the 
10 copies take about 4 times the space of a single copy.
  > (we do complete copies of linux servers - starting from /)
  > If there's a good spot to post the scripts, I'd be glad to put
them up.

I'd recommend Tower of Hanoi, as referenced above, for managing sets of 
backups - far simpler than dealing with days and dates, etc.  For 
example, the above mentions special actions on the first of the month - 
what if that is missed because of some failure - more special case 
scripting? Tower of Hanoi is self-healing....



Regards

Rob.

Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
CSIRO Information Management and Technology
T +61 3 9545 2368 (Docklands and Clayton)
T +61 3 9669 8102 (Docklands only)
Mob +61 428 108 333
Robert.Bell at csiro.au<mailto:Robert.Bell at csiro.au> | www.csiro.au
https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage

Also: Bureau of Meteorology Scientific Computing Services

Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia


PLEASE NOTE
The information contained in this email may be confidential or
privileged.  Any unauthorised use or disclosure is prohibited.  If
you have received this email in error, please delete it immediately
and notify the sender by return email. Thank you.  To the extent
permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been
maintained or that the communication is free of errors, virus,
interception or interference.

Please consider the environment before printing this email.

On 22/06/2016 10:00 pm, rsync-request at lists.samba.org
wrote:
> Send rsync mailing list submissions to
> 	rsync at lists.samba.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://lists.samba.org/mailman/listinfo/rsync
> or, via email, send a message with subject or body 'help' to
> 	rsync-request at lists.samba.org
>
> You can reach the person managing the list at
> 	rsync-owner at lists.samba.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rsync digest..."
>
>
>
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> ---------------------------------------
>
>
>
> Today's Topics:
>
>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>
>
>
> _______________________________________
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> rsync mailing list
> rsync at lists.samba.org
> https://lists.samba.org/mailman/listinfo/rsync
>

I agree that the Tower of Hanoi distribution is a good method.
It ensures a better depth. And with 10 copies, it would really, really 
be good depth.
With accounting systems though, it's good to have a backup of the data 
just -before- they close the fiscal period...
(the call we get is -- OMG we forgot to post that $250K before we closed 
-- admin is going to kill us -- can we go back?)
We've only had a couple of connectivity issues with 2 or 3 clients over 
4 years that caused it to miss one of the monthly rotations.
(the auto-retry off business hours is what has increased the resiliency 
of the process at sites with less than perfect internet service)

-- 
Larry Irwin
Email: lrirwin at alum.wustl.edu

On 06/22/2016 09:20 PM, Bell, Robert (CSIRO IM&T, Docklands)
wrote:
> Folks,
>
> Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail):
>  > The scripts I use analyze the rsync log after it completes and then 
> sftp's a summary to the root of the just completed rsync.
>  > If no summary is found or the summary is that it failed, the folder 
> rotation for that set is skipped and that folder is re-used on the 
> subsequent rsync.
>  > The key here is that the folder rotation script runs separately 
> from the rsync script(s).
>
> That is what we found to be important some years ago - do most of the 
> management outside of the scripts that do rsync.
>
> In particular, our scripts prepare a target backup directory with a 
> name of the form:
>     slash.20151122.seq.1379.current
>   - the name of the area being backed-up is obvious, as is the date.
> We use Tower of Hanoi management (see the update at 
> https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), 
> so embed a sequence number in the backup directory name, and the 
> suffix .current to indicate a current backup target.
>
> we can keep on trying the rsync backups until we get success, and the 
> scripts then remove the .current suffix.  This works because the rsync 
> includes the --delete option, used since the backup target can be a 
> recycled hard-linked directory: also recommended.
>
>  > For each entity I want to rsync, I create a named folder to 
> identify it and the rsync'd data is held in sub-folders:
>  > daily.[1-7] and monthly.[1-3]
>  > When I rsync, I rsync into daily.0 using daily.1 as the link-dest.
>  > Then the rotation script checks daily.0/rsync.summary - and if it 
> worked, it removes daily.7 and renames the daily folders.
>  > On the first of the month, the rotation script removes monthly.3, 
> renames the other 2 and makes a complete hard-link copy of daily.1 to 
> monthly.1
>  > It's been running now for about 4 years and, in my environment,
the
> 10 copies take about 4 times the space of a single copy.
>  > (we do complete copies of linux servers - starting from /)
>  > If there's a good spot to post the scripts, I'd be glad to
put them
> up.
>
> I'd recommend Tower of Hanoi, as referenced above, for managing sets 
> of backups - far simpler than dealing with days and dates, etc.  For 
> example, the above mentions special actions on the first of the month 
> - what if that is missed because of some failure - more special case 
> scripting? Tower of Hanoi is self-healing....
>
>
>
> Regards
>
> Rob.
>
> Dr Robert C. Bell
> HPC National Partnerships | Scientific Computing
> CSIRO Information Management and Technology
> T +61 3 9545 2368 (Docklands and Clayton)
> T +61 3 9669 8102 (Docklands only)
> Mob +61 428 108 333
> Robert.Bell at csiro.au<mailto:Robert.Bell at csiro.au> |
www.csiro.au
> https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage
>
> Also: Bureau of Meteorology Scientific Computing Services
>
> Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, 
> Australia
> Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia
> Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia
> Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia
> Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia
>
>
> PLEASE NOTE
> The information contained in this email may be confidential or
> privileged.  Any unauthorised use or disclosure is prohibited.  If
> you have received this email in error, please delete it immediately
> and notify the sender by return email. Thank you.  To the extent
> permitted by law, CSIRO does not represent, warrant and/or
> guarantee that the integrity of this communication has been
> maintained or that the communication is free of errors, virus,
> interception or interference.
>
> Please consider the environment before printing this email.
>
> On 22/06/2016 10:00 pm, rsync-request at lists.samba.org wrote:
>> Send rsync mailing list submissions to
>>     rsync at lists.samba.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>     https://lists.samba.org/mailman/listinfo/rsync
>> or, via email, send a message with subject or body 'help' to
>>     rsync-request at lists.samba.org
>>
>> You can reach the person managing the list at
>>     rsync-owner at lists.samba.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of rsync digest..."
>>
>>
>>
>> To unsubscribe or change options: 
>> https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> ---------------------------------------
>>
>>
>>
>> Today's Topics:
>>
>>    1. Re: rsync script for snapshot backups (Dennis Steinkamp)
>>    2. Re: rsync script for snapshot backups (Petros Angelatos)
>>
>>
>>
>> _______________________________________
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>>
>> rsync mailing list
>> rsync at lists.samba.org
>> https://lists.samba.org/mailman/listinfo/rsync
>>
>