All I am use rsync to backup a large filesystem over the internet. I do an incremental backup of the 2 file systems once a day to keep an offsite backup. However, this backup takes about 60 minutes. Are there some tweaks I can do to speed up the backup? Here is the command I run via cron every day: rsync --exclude-from=/home/backups/rsync/attachments-exclude.lst --delete -ave ssh bisil@mysourceserver:/var/attachments/secure/ /backups/var/varattachments This size of the source directory is about 7 GB. even though I back up every day, it takes about 60 minutes. There are about 10-50 MB of files that are added into the source every day. Can you advise? Regards Srinivas ===========================Srinivas Kotapally, Analyst -------------- next part -------------- HTML attachment scrubbed and removed
On Sat, 2006-05-06 21:50:58 -0400, Srinivas Kotapally <ksrinivas@bisil.com> wrote:> I am use rsync to backup a large filesystem over the internet. I do an > incremental backup of the 2 file systems once a day to keep an offsite > backup. However, this backup takes about 60 minutes. Are there some tweaks > I can do to speed up the backup? Here is the command I run via cron every > day: > > rsync --exclude-from=/home/backups/rsync/attachments-exclude.lst --delete > -ave ssh bisil@mysourceserver:/var/attachments/secure/ > /backups/var/varattachments > > This size of the source directory is about 7 GB. even though I back up every7GB? That doesn't even qualify as "large" :-)> day, it takes about 60 minutes. There are about 10-50 MB of files that are > added into the source every day.Well, you guess it: there's no way to give you some hints based on your description. "I copy a dataset of 7GB with rsync. I don not know about the filesystems used to keep the data, I don't know the mount options used, I don't know how files change (in data as well as in metadata) and it takes about 60 min over a link with unknown speed and latency." Please give us some more hints. One hour over eg. a mobile phone connection with 9600 Baud (like 1kb/sec) for a dataset which changes completely would be quite well-working, while one hour with fast disks over a well-connected site (eg. gigabit internet upstream) with non-changing data would be catastrophic... MfG, JBG -- Jan-Benedict Glaw jbglaw@lug-owl.de . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O f?r einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://lists.samba.org/archive/rsync/attachments/20060507/cc0a5b38/attachment.bin
On Sun, 2006-05-07 09:53:47 -0400, Srinivas Kotapally <ksrinivas@bisil.com> wrote:> The filesystem that I backup is one where files keep getting added or > deleted no other files are modified at all. These files are usually word/ext2? ext3? xfs? jfs? reiserfs? ...> PDF documents. These files are only added not changed. The number of files > added are about 20-200 each day. I did not understand the question aboutEach in the 10mb range?> mount options... the disks are already mounted. The issue over here is notWhat are the -o xxxxxx options added to the mount call? These are eg. noted in /etc/fstab (if it's a regularly mounted filesystem on a unix-like box). MfG, JBG -- Jan-Benedict Glaw jbglaw@lug-owl.de . +49-172-7608481 _ O _ "Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O f?r einen Freien Staat voll Freier B?rger" | im Internet! | im Irak! O O O ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://lists.samba.org/archive/rsync/attachments/20060507/84e3f6df/attachment.bin
On Sat, May 06, 2006 at 09:50:58PM -0400, Srinivas Kotapally wrote:> I am use rsync to backup a large filesystem over the internet. I do an > incremental backup of the 2 file systems once a day to keep an offsite > backup. However, this backup takes about 60 minutes.The first thing I'd recommend is to switch from using --delete to using --delete-during, as this will require less disk I/O on the receiver. If that doesn't speed things up enough, you need to do some testing to figure out what factor(s) are limiting your speed. For instance, run "top" or some other CPU-monitoring tool during the rsync run to see if your CPU is being maxxed out (e.g. a slower CPU can become saturated with the task of running rsync's file-difference algorithm at the same time that ssh is encoding (and compressing?) the data being sent. You also want to figure out how near capacity your disk I/O. If you don't have a good disk-I/O-monitoring tool, you can at least get an estimation of how much time is spent in the filesystem scan by doing a large "find" operation and using that to estimate the amount of time that rsync must use just getting the list of filenames on the sending side. Finally, use the stats that rsync outputs about how much data is transmitted to see if bandwidth might be your limiting factor. Once you determine what is the bottleneck for the backup, you can try to figure out if there is a way to speed it up. ..wayne..