hi all, I use rsync to copy/backup ALL my stuff to another disk. When I run this seems like my machine (4 GIG ram centos 5.1) now begins to swap out more programs. Is there a way to reduce that swapping? I am running with echo 1 > /proc/sys/vm/swappiness I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year then rsync -a /home /mnt/backup/mon.day.year This is approximately 102G of data. Thanks for any suggestions. Jerry
Jerry Geis wrote:> hi all, > > I use rsync to copy/backup ALL my stuff to another disk. > > When I run this seems like my machine (4 GIG ram centos 5.1) > now begins to swap out more programs. Is there a way to reduce > that swapping? I am running with echo 1 > /proc/sys/vm/swappiness > > I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year > then rsync -a /home /mnt/backup/mon.day.year > > This is approximately 102G of data.It's the number of files in the run that matters more than the amount of date. Rsync loads the entire directory listing into RAM before starting to copy so there is a certain amount of per-file overhead. It should help if you could break the run up, perhaps doing a few directories separately, then make another pass that excludes those directories. -- Les Mikesell lesmikesell at gmail.com
On Jan 30, 2008 11:24 AM, Jerry Geis <geisj at pagestation.com> wrote:> hi all, > > I use rsync to copy/backup ALL my stuff to another disk. > > When I run this seems like my machine (4 GIG ram centos 5.1) > now begins to swap out more programs. Is there a way to reduce > that swapping? I am running with echo 1 > /proc/sys/vm/swappiness > > I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year > then rsync -a /home /mnt/backup/mon.day.yearIMHO, rsync is overkill here. I would: mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year cd /home;find . | cpio -vdump /mnt/backup/mon.day.year -- Marcelo "?No ser? acaso que ?sta vida moderna est? teniendo m?s de moderna que de vida?" (Mafalda)
on 1/30/2008 5:24 AM Jerry Geis spake the following:> hi all, > > I use rsync to copy/backup ALL my stuff to another disk. > > When I run this seems like my machine (4 GIG ram centos 5.1) > now begins to swap out more programs. Is there a way to reduce > that swapping? I am running with echo 1 > /proc/sys/vm/swappiness > > I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year > then rsync -a /home /mnt/backup/mon.day.year > > This is approximately 102G of data. > > Thanks for any suggestions. > > JerryRsync's main benefit is on backups of changed files. dumping to a new destination every time makes rsync less efficient than just about every other option. Now if you made the new directory, and hardlinked the old stuff to the new directory, then rsync would shine. -- MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos/attachments/20080130/173a8613/attachment-0002.sig>
Yohoo!>Rsync's main benefit is on backups of changed files. dumping to a new >destination every time makes rsync less efficient than just about every >other >option. >Now if you made the new directory, and hardlinked the old stuff to the new >directory, then rsync would shine.That's what rsnapshot is designed for. It uses rsync to sync the files to a backup destination and hardlinks any existing file, so you can go back to any level you like. Christian
Scott Silva wrote:> on 1/30/2008 5:24 AM Jerry Geis spake the following: >> hi all, >> >> I use rsync to copy/backup ALL my stuff to another disk. >> >> When I run this seems like my machine (4 GIG ram centos 5.1) >> now begins to swap out more programs. Is there a way to reduce >> that swapping? I am running with echo 1 > /proc/sys/vm/swappiness >> >> I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year >> then rsync -a /home /mnt/backup/mon.day.year >> >> This is approximately 102G of data. >> >> Thanks for any suggestions. >> >> Jerry > Rsync's main benefit is on backups of changed files. dumping to a new > destination every time makes rsync less efficient than just about > every other option. > Now if you made the new directory, and hardlinked the old stuff to the > new directory, then rsync would shine. >I did the rsync hard link for a while. After 30+ hardlinks to each file built up, filesystem operations slowed down - not in a killer way, but I did notice it. I think it's better to just use --backup and write the previous version to a new dir with --backup-dir=`date +%F` or some such scheme. You don't see the backups represented as a whole directory structure, but it's less messy. -- Toby Bluhm Alltech Medical Systems America, Inc. 30825 Aurora Road Suite 100 Solon Ohio 44139 440-424-2240
Scott Silva wrote:> on 1/30/2008 5:24 AM Jerry Geis spake the following: >> hi all, >> >> I use rsync to copy/backup ALL my stuff to another disk. >> >> When I run this seems like my machine (4 GIG ram centos 5.1) >> now begins to swap out more programs. Is there a way to reduce >> that swapping? I am running with echo 1 > /proc/sys/vm/swappiness >> >> I simply mount /dev/sdc1 /mnt/backup; mkdir /mnt/backup/month.day.year >> then rsync -a /home /mnt/backup/mon.day.year >> >> This is approximately 102G of data. >> >> Thanks for any suggestions. >> >> Jerry > Rsync's main benefit is on backups of changed files. dumping to a new > destination every time makes rsync less efficient than just about > every other option. > Now if you made the new directory, and hardlinked the old stuff to the > new directory, then rsync would shine.Yes. That's my experience, too. I do something like this twice weekly to 2 external USB drives. One drive contains backups from the most recent 2 Sunday mornings, the other drive contains two backups from Wednesday mornings. After mounting the backup drive, doing a space-available check and a couple other housekeeping chores, I do this. # (I've purposely left out a bunch of stuff to avoid being blamed when someone's system craters. ) $UD #-- Previously identified as the USB drive mount point # DT=`date +%F` #-- Today's date, for naming the backup directory: yyyy-mm-dd # # Shuffle directories so that oldest backup directory is renamed to today's date. # OD=`ls -1 $UD | head -1` #echo "Renaming $UD/$OD to $UD/$DT" mv $UD/$OD/ $UD/$DT touch $UD/$DT # Now, data that is only 2 weeks old will be overwritten with current data. for SD in <list of directories to back up> ; do rsync -ar --exclude '*.iso' --delete-excluded /$SD/* $UD/$DT/$SD done # I've used the basic outline above since Sept 13, 2006 and have found that the backup takes slightly more than half as long as when I used "find" and "cpio". As with anything else, YMMV.