rsync - Nov 2008 - How to delete files older than X on backup during sync?

Hi list,

I have been reading MAN pages and listarchives, but not found the
answer to my question though I am sure it must be possible to acheive
my wish.

I wish to use rsync to create a backup BUT only keep the files for a
limited period of time, EG two weeks.
I have not yet been able to figure out how to do this inside rsync
(while the backup is being performed) and my understanding of the
"--delete" flag is not for this wish.

My plan is - unless I find out that it indeed is possible with rsync -
to make a cronjob removing files older than X in the folder containing
the backups?

BR,

Thomas von Eyben

You can do something like

"rsync OPTIONS --rsync-path="find /root-dir/ -mtime +14 -delete ;
rsync"
SRC DEST"

although, to be safe you'll probably want something like:

"rsync OPTIONS --rsync-path="find /root-dir/ -type f -mtime +14
-delete
; rsync" --prune-empty-dirs SRC DEST"

I'm not sure whether or not find would delete a directory that's not
empty, but this the safe bet.

Thomas von Eyben wrote:
> Hi list,
> 
> I have been reading MAN pages and listarchives, but not found the
> answer to my question though I am sure it must be possible to acheive
> my wish.
> 
> I wish to use rsync to create a backup BUT only keep the files for a
> limited period of time, EG two weeks.
> I have not yet been able to figure out how to do this inside rsync
> (while the backup is being performed) and my understanding of the
> "--delete" flag is not for this wish.
> 
> My plan is - unless I find out that it indeed is possible with rsync -
> to make a cronjob removing files older than X in the folder containing
> the backups?
> 
> BR,
> 
> Thomas von Eyben

On Sun, 2008-11-09 at 03:18 -0500, Matthew Monaco wrote:
> Thomas von Eyben wrote:
> > I wish to use rsync to create a backup BUT only keep the files for a
> > limited period of time, EG two weeks.
> > I have not yet been able to figure out how to do this inside rsync
> > (while the backup is being performed) and my understanding of the
> > "--delete" flag is not for this wish.
> > 
> > My plan is - unless I find out that it indeed is possible with rsync -
> > to make a cronjob removing files older than X in the folder containing
> > the backups?
> 
> You can do something like
> 
> "rsync OPTIONS --rsync-path="find /root-dir/ -mtime +14 -delete ;
rsync"
> SRC DEST"
If you count the two-week period from the mtime of the file, then
backups of really old files will be deleted as soon as they are saved.
I think it would make more sense to count from the time the backup file
is saved.  To do that, use a separate --backup-dir each day (e.g.,
--backup-dir=../backups/$(date +%Y-%m-%d) ) and simply delete each
backup dir as a whole when it becomes two weeks old.

The only disadvantage is that with multiple backup dirs, it becomes less
trivial to find backups of a particular path, but it would be easy to
write a script to help with this.

Matt

On Sun, Nov 9, 2008 at 4:34 PM, Matt McCutchen <matt@mattmccutchen.net>
wrote:
> On Sun, 2008-11-09 at 03:18 -0500, Matthew Monaco wrote:
>> Thomas von Eyben wrote:
>> > I wish to use rsync to create a backup BUT only keep the files for
a
>> > limited period of time, EG two weeks.
>> > I have not yet been able to figure out how to do this inside rsync
>> > (while the backup is being performed) and my understanding of the
>> > "--delete" flag is not for this wish.
>> >
>> > My plan is - unless I find out that it indeed is possible with
rsync -
>> > to make a cronjob removing files older than X in the folder
containing
>> > the backups?
>>
>> You can do something like
>>
>> "rsync OPTIONS --rsync-path="find /root-dir/ -mtime +14
-delete ; rsync"
>> SRC DEST"
>
> If you count the two-week period from the mtime of the file, then
> backups of really old files will be deleted as soon as they are saved.
> I think it would make more sense to count from the time the backup file
> is saved.  To do that, use a separate --backup-dir each day (e.g.,
> --backup-dir=../backups/$(date +%Y-%m-%d) ) and simply delete each
> backup dir as a whole when it becomes two weeks old.
>
> The only disadvantage is that with multiple backup dirs, it becomes less
> trivial to find backups of a particular path, but it would be easy to
> write a script to help with this.
>
> Matt
>
>
Actually, I think I need to be a little more specific (now I think
about it): I only want to keep the files that are deleted on the
source for the additional period of two weeks.

So I want to keep the back up identical to the source, just not
deleting files in the backup that are deleted in the source (should be
fairly easy with rsync ;-) BUT I  only want to keep these
deleted-on-source files for an additional period of two weeks?

- TvE

Matthew, please remember to CC rsync@lists.samba.org .

On Sun, 2008-11-09 at 15:46 -0500, Matthew Monaco wrote:
> You could use touch to update all of the files in your backup-dir.  Then
> use my original suggestion to delete files that are older than two weeks.
I assume you mean, touch only the destination files corresponding to
source files that still exist (and you'd get that list from itemize
output)?  That would work, but it loses the mtimes of the files, which
will make the rsync quick check fail and may make the backup less
useful.  Using the atime instead would work nicely provided that the
atime field isn't being used for anything else and the destination
filesystem is mounted noatime.

Matt
> Matt McCutchen wrote:
> > On Sun, 2008-11-09 at 19:03 +0100, Thomas von Eyben wrote:
> >> Actually, I think I need to be a little more specific (now I think
> >> about it): I only want to keep the files that are deleted on the
> >> source for the additional period of two weeks.
> >>
> >> So I want to keep the back up identical to the source, just not
> >> deleting files in the backup that are deleted in the source
(should be
> >> fairly easy with rsync ;-) BUT I  only want to keep these
> >> deleted-on-source files for an additional period of two weeks?

You could find these files using find. Then, copy these to another
directory and sycn the directory which has only the new files.
If you were cleaver you could hard link the files to save space before the
sync. But maybe someone on this list will know how to do this with just
rsync.

This is just one possiblity. Maybe somone has already written a tool to do
the first step.

If you were to use LBackup and run a backup once per day and set the
rotations to 20. Then after 20 days, the old files would be removed from
the backup.

>LBackup is not intended for large scale deployment, unless used in
>combination with some sort of file system management tool.
If you are backing up up client machines, chances are that their
configuration will be changing in the near future. In addition, there are
may be different operating systems or different operating system versions
on these machines.

You may certainly use LBackup to backup these machines. However, another
tool may be better suited to the job.

As workstations, these machines will most likely get moved around,
re-imaged swapped between people and upgraded on a very regular basis. If
this is the case then the maintenance of the backups for these
workstations without any kind of automated system will result in great
deal of work on your part. 

As such, if you decide to use LBackup for the backup of client machines, I
would strongly recommend some sort of centralized file system management
system already be place and tested on these machines. This will help you
deal with the modified versions of rsync on all of these machines. In
addition, some machines may require special configuration such as privacy
restrictions. 

LBackup is not currently designed to deal with client backups at this
scale without assistance from some other scripts or tools. As such, I
would strongly recommend that if you are going to use LBackup for the
backup of 200 client machines, that you have some sort of file management
system or other automated deployment system in place on these machines
before you start. Otherwise, you will be asking for a wold of pain, with
regards the maintenance of such a solution. In addition, LBackup is not
yet very good at dealing with clients, which are constantly changing
between the states available and ready for backup and not available for
backup. Think about, a laptop which moves around the world and may be
behind a firewall. Remote backup of such a laptop is something which is
under development but it not anywhere near ready. 

The BackupPC project is aimed at the backup of workstations. As such, I
would suggest you have a look at this project and see if that will meet
your client backup requirements. You could even use rsync to centralize
the data.

Once the data has been pulled together by BackupPC or some other tools,
you could then effectively use  a tool like LBackup to move the data to a
remote location.

I hope this clarifies the statement on the web page. 

If you decided to write some sort of system to assist with the management
and deployment of LBackup to workstations and/or roaming clients then
please let me know, as I would be most interested.

Thanks.


PS : Sorry if this is not the right place for the discussion of this topic.