Louis-David Mitterrand
2002-Nov-28 15:53 UTC
rsync as a bakcup tool and the case of rotated logs
Hello, I use rsync (among many other things) as an incremental backup tool. Every night I save /{home,etc,var} from several servers. On one of them we keep 52 weeks of system logs, so each time syslog is rotated all old syslog.x.gz (where x is between 1 and 213) change names. This impacts our backup transfers because the incremental snapshot now has to include all syslog.x.gz files even though only one of them has been added (and one deleted). Now I am a very happy and grateful user of rsync. I was just wondering if there is any way to: 1) have rsync "understand" that file names might have changed, maybe by comparing files through their md5 signature instead of by their name, that way rsync would see that /backup/syslog.198.gz is the same as /var/log/syslog.197.gz and not retransfer it, 2) create hard-links to identical files in --backup-dir=/backup/incremental-2002-11-27 when is detects that /server/sylog.138.gz is the actually the same as /backup/current/syslog.137.gz, Thanks in advance for your insight, -- PHEDRE: Elle s'en est punie, et fuyant mon courroux, A cherch? dans les flots un supplice trop doux. (Ph?dre, J-B Racine, acte 5, sc?ne 7)
> 1) have rsync "understand" that file names might have changed, maybe by > comparing files through their md5 signature instead of by their name, > that way rsync would see that /backup/syslog.198.gz is the same as > /var/log/syslog.197.gz and not retransfer it,The best choice is to rename the syslog files with a date, and don't repeatedly rename them, eg: syslog.YYYYMMDD.gz (eg: syslog.20021128.gz). Pruning old ones isn't too hard: simply reverse sort the names and remove everything after the first 213. It also makes it easier to find a particular log file.> 2) create hard-links to identical files in > --backup-dir=/backup/incremental-2002-11-27 when is detects that > /server/sylog.138.gz is the actually the same as > /backup/current/syslog.137.gz,BackupPC is one package that does this; see http://backuppc.sourceforge.net. (disclaimer: I'm the author). I'm in the process of adding rsync support. Craig
Hans-Juergen Beie
2002-Nov-29 18:51 UTC
rsync as a bakcup tool and the case of rotated logs
Louis-David Mitterrand wrote: > Hello, > > I use rsync (among many other things) as an incremental backup tool. > Every night I save /{home,etc,var} from several servers. On one of them > we keep 52 weeks of system logs, so each time syslog is rotated all old > syslog.x.gz (where x is between 1 and 213) change names. This impacts > our backup transfers because the incremental snapshot now has to include > all syslog.x.gz files even though only one of them has been added (and > one deleted). > > Now I am a very happy and grateful user of rsync. I was just wondering > if there is any way to: > > 1) have rsync "understand" that file names might have changed, maybe by > comparing files through their md5 signature instead of by their name, > that way rsync would see that /backup/syslog.198.gz is the same as > /var/log/syslog.197.gz and not retransfer it, As Craig already proposed, rename or copy it. If you are using logrotate you can do that in a postrotate job. > 2) create hard-links to identical files in > --backup-dir=/backup/incremental-2002-11-27 when is detects that > /server/sylog.138.gz is the actually the same as > /backup/current/syslog.137.gz, rsback is a frontend to rsync (disclaimer: I'm the author ;-) which allows to make rotating backup archives by hard-linking. See http://www.pollux.franken.de/hjb/rsback hjb :-? -- Hans-Juergen Beie Phone: +49 911 396628 / +49 173 3546274 Fax: +49 911 396663 mailto:hjb@pollux.franken.de