Hello,
I make my production backups with Rsync.
Here is an example of my backup tree on the destination server :
/backups
/2013-04-03
/2013-04-02
/2013-04-01
/2013-03-31
/2013-03-30
/2013-03-29
At the end of the backup process, I upload a logfile in the backup
directory and delete oldest backups.
For this, I use an include/exclude file, for example this inclexcl.txt :
+ /2013-04-03
+ /2013-04-03/logfile.log
- /2013-04-03/*
- /2013-04-02
- /2013-04-01
I also use this empty directory where my logfile is :
/tmp
/path
/2013-04-03
/logfile.log
And I run this rsync command :
rsync -a --delete-after --exclude-from=inclexcl.txt /tmp/path/
myserver::backups/
Perfect, it works, my logfile is uploaded and oldest backups are
deleted (in this example all backups of March).
However, what I can see in the daemon log is that Rsync is browsing
from the top down all files of the backup directories to delete, and
delete them one by one.
As each of my backup directories contains hundreds of thousands of
files, deletion take a very long time and eats a lot of IOs.
Is there a way to tell Rsync to directly delete backup directories at
the top of the hierarchy without browsing them ?
Thank you very much for your support !
Best regards,
Ben
On 06.04.2013 16:57, AZ 9901 wrote:> > Is there a way to tell Rsync to directly delete backup directories at > the top of the hierarchy without browsing them ?No. Deleting a directory-tree has to be done "file by file" and "directory by directory" there is no magic "remove that mountain of files"(*). *: Altough Snapshots can work like that. Use something that supports snapshots, put a day into a snapshot and then you can remove the whole snapshot. Depending on the circumstances (what as to happend to do the work) it may or may not be faster than removing a directory tree "file by file" but at least the work is done "invisibly" by whatever is doing the snapshoting. -- Matthias
Le 6 avr. 2013 ? 17:15, Matthias Schniedermeyer a ?crit :> On 06.04.2013 16:57, AZ 9901 wrote: >> >> Is there a way to tell Rsync to directly delete backup directories at >> the top of the hierarchy without browsing them ? > > No. > > Deleting a directory-tree has to be done "file by file" and "directory > by directory" there is no magic "remove that mountain of files"(*). >Matthias, thank you for your answer ! I made some tests, I created a directory with 300k files in it. I deleted it with Rsync, then with rm command. I did this test several times. On my server, it took about 20 seconds with rm, about 40 seconds with Rsync. In the Rsync deamon logfile, I can see a "del." line for each file of the directory, beginning from the top down. Does Rsync compares each file against the incl/excl list to know if it can be deleted ? If so, is there a way to tell Rsync the following ? "delete this top directory and all its files, but stop lacking time with comparisons against the incl/excl list for each element of this top directory" Certainly that it could speed up deletion with Rsync :-) Thank you very much ! Ben
Le 6 avr. 2013 ? 16:57, AZ 9901 a ?crit :> I make my production backups with Rsync. > > > Here is an example of my backup tree on the destination server : > /backups > /2013-04-03 > /2013-04-02 > /2013-04-01 > /2013-03-31 > /2013-03-30 > /2013-03-29 > > At the end of the backup process, I upload a logfile in the backup > directory and delete oldest backups. > > For this, I use an include/exclude file, for example this inclexcl.txt : > + /2013-04-03 > + /2013-04-03/logfile.log > - /2013-04-03/* > - /2013-04-02 > - /2013-04-01 > > I also use this empty directory where my logfile is : > /tmp > /path > /2013-04-03 > /logfile.log > > And I run this rsync command : > rsync -a --delete-after --exclude-from=inclexcl.txt /tmp/path/ > myserver::backups/ > > Perfect, it works, my logfile is uploaded and oldest backups are > deleted (in this example all backups of March). > > > However, what I can see in the daemon log is that Rsync is browsing > from the top down all files of the backup directories to delete, and > delete them one by one. > As each of my backup directories contains hundreds of thousands of > files, deletion take a very long time and eats a lot of IOs. > > Is there a way to tell Rsync to directly delete backup directories at > the top of the hierarchy without browsing them ?Le 7 avr. 2013 ? 11:25, AZ 9901 a ?crit :> I made some tests, I created a directory with 300k files in it. > I deleted it with Rsync, then with rm command. > I did this test several times. > On my server, it took about 20 seconds with rm, about 40 seconds with Rsync. > > In the Rsync deamon logfile, I can see a "del." line for each file of the directory, beginning from the top down. > Does Rsync compares each file against the incl/excl list to know if it can be deleted ? > > If so, is there a way to tell Rsync the following ? > "delete this top directory and all its files, but stop lacking time with comparisons against the incl/excl list for each element of this top directory"Hello ! Wayne, do you have any idea about this ? Thank you very much ! Best regards, Ben