rsync - Mar 2007 - rsync backup - order by size

Hey all,

 

I have setup a system using rsync to backup our data daily to an offsite
backup server. We are having issues with large (1GB-20GB) files taking
longer than a day to complete (up to a week), meaning that smaller files are
not being backed up every day. The smaller files are the more critical ones,
and the larger files aren't usually critical.

 

If I could get the smaller files to be backed up first, and then the larger
ones afterwards, then the large file transfers can be killed at the end of
the day and resumed the next day after the smaller files have been backed up
again.

 

Therefore, is there any way that I can get rsync to backup the files in
order of size?

 

Thanks,
Ty

 

 

-------------- next part --------------
HTML attachment scrubbed and removed

On 3/28/07, Ty Miller <tyronmiller@gmail.com>
wrote:
> Therefore, is there any way that I can get rsync to backup the files in
> order of [increasing] size?
Rsync has no option to transfer files in order of increasing size, but
there are two ways to accomplish something similar to that.  See
comments #1 and #2 at:

https://bugzilla.samba.org/show_bug.cgi?id=4082

Matt

Hey guys,

Just thought I'd let you know that this worked perfectly. 

If the rsync guys would like a copy of my backup script for their examples
page, or anything like that, then I am happy to sanitize it and pass it on. 

It performs the following functions;

The backup script implements a 31 day rotating incremental backup. The
incrementals will go into subdirectories named after the day of the month
(1,2,3,...,31) and the current full backup goes into a directory called
"0".
(This was named "0" because the Windows version of rsync/cygwin has
limitations on the filename length.)

A log file is kept, which is parsed at the end of the backup to determine
whether any errors have occurred. If so, then an email is sent to the
administrator containing the backup errors.

The script uses the --partial option so that if a transfer is interrupted
then partially transferred files will be kept so they can be continued on
the next transfer. I also added a check for a "connection reset" error
so
that the backup is automatically kicked off immediately if the connection
drops out and alerts the administrator.

Large files were causing smaller files to not be backed up daily cause the
large files were taking longer than a day to complete. It now performs two
passes of backups for small and large files. The small files get backed up,
and then the large files start getting transferred. The next time the backup
gets kicked off it kills the large file backup if it is still running,
alerts the administrator, and restarts the process again so that the small
files get backed up, then the large files backup starts off where it left
off.

The large files were also causing issues with network performance, so a
bandwidth limit has been implemented for both small and large file
transfers.

Thanks again for your help,
Ty


-----Original Message-----
From: hashproduct@gmail.com [mailto:hashproduct@gmail.com] On Behalf Of Matt
McCutchen
Sent: Thursday, 29 March 2007 12:18 PM
To: Ty Miller
Cc: rsync@lists.samba.org
Subject: Re: rsync backup - order by size

On 3/28/07, Ty Miller <tyronmiller@gmail.com>
wrote:
> If I changed this to include the --min-size and --max-size so that there
are
> the following two rsync passes;
>
> - Empty the daily incremental backup dir
> - Backup small files to the "Current" dir, moving modified or
deleted
files
> to the daily incremental dir
> - Backup large files to the "Current" dir, moving modified or
deleted
files
> to the daily incremental dir
>
> How will this affect the reality of what happens? For example;
>
> - Will the first pass cause the Current dir to only contain small files,
> causing it to move the large files to the daily incremental dir? [...]
It's good that you bring this up, but I don't think it will be a
problem.  The --min-size and --max-size options tell rsync to skip
transferring regular files whose sizes are out of range, but those
files are still present in the file list.  Thus, the receiver will
know not to delete any of the files from the "Current" dir that exist
on the sender, even files that belong to the other pass.  The two
passes should give you the same result as the one pass currently does.

Matt