Im brand new to rsync and i been doing alot of reading about it and it looks like the tool I have long been wanting. Bascially i got 1 server, 2 hard drives. partions are / /boot /home /usr /var /backup (2nd drive) Bascially what i am looking at doing is mirroring the first drive to the 2nd drive, and once that intial one is moved over, have a cron run nightly (or when specified) that will update any modified files from the cron run. I am looking to have this as a perl script (or whatever) so i can easily port it to all my servers across my network. So that if hard drive 1 fails, i will just need to pop in hard drive two and be up and running within minutes rather then hours (my last outage was pretty major on my /boot partion and if had rysnc setup in the manor i am speaking of, downtime would have went from 8 hours to less then 30 minutes....) Does anyone have such examples of what i can do to accomplish this? or if even better, someone is willing to create the script in question for some a quick $50 or something like that I have absolutly no problem in. I am rather busy with daily operations and if i can get someone to make me the script, that would be very favorable ------------------------------------- Thanks! Gary Peltola
>>>>> "Gary" == Gary Peltola <clist@www-hosting.net> writes:Gary> Bascially what i am looking at doing is mirroring the first Gary> drive to the 2nd drive, and once that intial one is moved Gary> over, have a cron run nightly (or when specified) that will Gary> update any modified files from the cron run. I am looking to Gary> have this as a perl script (or whatever) so i can easily Gary> port it to all my servers across my network. So that if hard Gary> drive 1 fails, i will just need to pop in hard drive two and Gary> be up and running within minutes rather then hours (my last Gary> outage was pretty major on my /boot partion and if had rysnc Gary> setup in the manor i am speaking of, downtime would have Gary> went from 8 hours to less then 30 minutes....) Gary> Does anyone have such examples of what i can do to Gary> accomplish this? Below is a couple of scripts that do this... but wait there's still more. :-) The first script does this: * Backup filesystems on /dev/hda to /dev/hdb, which gets mounted under /backup. * /dev/hdb is bigger than /dev/hda, so (backwards) rsync incrementals are done to /backup/incremental, which uses the left over space on /dev/hdb. * Filesystems on /dev/hdb are backed up to tape using "dump". This is cool, because the system can be up and running (and busy) while this is happening, since the filesystems on /dev/hdb aren't active. The 2nd script: * Cleans up the incrementals, so there's always a certain percentage of free space. When it needs space it removes one or more whole days worth of incrementals. Note that in this script there is *some* sanity checking to try and ensure that sensible stuff is removed. However, you should always *BEWARE* of scripts that run as root and automatically recursively remove directories. Just to be sure, I've changed the line that is supposed to remove stuff so that it looks like: echo "Not currently executing this command: rm -rf" "${this}" so it now has to be modified so that the "rm -rf ..." is actually activated... :-) Remember, no warranty and all that... peace & happiness, martin --------8<---------8<-------- CUT HERE --------8<---------8<-------- #!/bin/sh # Uncomment these for debugging. #set -x DEBUG=echo # Backup disk /dev/hda to /dev/hdb and then backup /dev/hdb to tape: # 1. Using rsync, backup each mounted filesystem /X to # /backup/current/X, which is mounted especially for the task. Old # version of files are stored under # /backup/incremental/DATETIME/X/... # 2. Each mounted filesystem /X has its corresponding # /backup/current/X backed up using dump. The /incremental # filesystem is also dumped at the end. # Note that the incrementals aren't currently cleaned up. Once we get # an idea of the size of incrementals, we can started removing any # directories older than (say) 6 months. incr=/backup/incremental yip="${incr}/`date +'%Y/%m/%d/%H%M%S'`" tape=/dev/ntape filesystems=`mount -t ext2 | awk '{print $3}'` do_rsync () { $DEBUG /usr/bin/rsync -axH --delete \ --backup --backup-dir="${yip}$1" \ "$1/" "/backup/current$1" } do_mt () { ${DEBUG} /bin/mt -f "${tape}" "$1" } do_label () { tmp="/tmp/bkup$$" date > "${tmp}" n=1 for i in $* ; do echo "$n $i" >> "${tmp}" n=`expr $n + 1` done $DEBUG /bin/dd if="${tmp}" of="${tape}" bs=512 conv=sync #$DEBUG /bin/tar cvf "${tape}" "${tmp}" do_mt eof /bin/rm -f "${tmp}" } do_dump () { ${DEBUG} /sbin/dump -0 -b 32 -a -f "${tape}" "$1" } ###################################################################### echo "Check backup partitions starting: `date`." fscks="${incr}" disks="" tapes="" for i in ${filesystems} ; do case "$i" in /var/spool/squid) : ;; /) fscks="${fscks} /backup/current" disks="${disks} /" tapes="${tapes} /backup/current" ;; *) fscks="${fscks} /backup/current${i}" disks="${disks} ${i}" tapes="${tapes} /backup/current${i}" esac done for i in ${fscks} ; do ${DEBUG} /sbin/fsck -a $i if [ $? -gt 1 ] ; then echo "fsck of backup filesystem $i failed. Quitting!" fi done echo "Check backup partitions finished: `date`." ###################################################################### echo "Disk backup starting: `date`." ${DEBUG} mount ${incr} for i in ${disks} ; do b="/backup/current${i}" if ${DEBUG} mount "${b}" ; then echo " ${i}" do_rsync "${i}" ${DEBUG} umount "${b}" else echo "*** Failed to mount backup volume \"${b}\"" fi done ${DEBUG} umount ${incr} echo "Disk backup finished: `date`." ###################################################################### echo "Tape backup starting: `date`." do_mt rewind # Eek! Tape drive doesn't do hardware compression. #do_mt datcompression do_label ${tapes} for b in ${tapes} ; do if ${DEBUG} mount "${b}" ; then echo " ${b}" do_dump "${b}" ${DEBUG} umount "${b}" else echo "*** Failed to mount backup volume \"${b}\"" fi done do_mt offline echo "Tape backup finished: `date`." # Last line. --------8<---------8<-------- CUT HERE --------8<---------8<-------- #!/bin/sh # Cleanup the incremental backups on /backup/incremental to create # free space. base=/backup/incremental threshold=90 mount $base dirs=`echo /backup/incremental/*/*/*` percent_used=`df /backup/incremental/ | tail -1 | sed -e 's/^.*[^0-9]\([0-9][0-9]*\).*$/\1/'` while [ \( "${percent_used}" -gt ${threshold} \) -a -n "$dirs" ] ; do this=`echo $dirs | sed -e 's/ .*$//'` dirs=`echo $dirs | sed -e 's/^[^ ]* *//'` case "$this" in ${base}/*/*/*) echo "Removing old backup directory ${this}." echo "Not currently executing this command: rm -rf "${this}" ;; *) echo "ERROR!!! Bad name for backup directory: ${this}." dirs="" esac percent_used=`df /backup/incremental/ | tail -1 | sed -e 's/^.*[^0-9]\([0-9][0-9]*\).*$/\1/'` done if [ "${percent_used}" -gt ${threshold} ] ; then echo "WARNING!!!" echo "${base} has ${percent_used}% used, which is greater than limit of ${threshold}%" echo "but there are no backups to remove!!!" fi umount $base --------8<---------8<-------- CUT HERE --------8<---------8<--------
Check out rdiff-backup. I have been very, very impressed with how easy it is to use. It uses the same algorithm as rsync to create incremental diffs. http://www.stanford.edu/~bescoto/rdiff-backup/ (also packaged for Debian, in the 'rdiff-backup' package)
What about software RAID 1 (mirroring)? If the backup is not on another physical machine, seems like mirroring might be as easy as rsync, w/o any scary scripted rm -rf'ing or --delete. Plus, it's always up-to-date (at least between disk sync intervals, someone else can remind me what those are under ext2fs, ~30 sec, IIRC), whereas a scripted rsync would only be good up to the last go-round on the cron job, nightly in your example. Don't get me wrong, I love rsync, but seems like not quite the right tool for this job, IMHO. -Dan Young -Parkrose School District -Portland, OR Gary Peltola wrote:>Im brand new to rsync and i been doing alot of reading about it and it looks like the tool I have long been wanting. > >Bascially i got 1 server, 2 hard drives. partions are > >/ >/boot >/home >/usr >/var >/backup (2nd drive) > >Bascially what i am looking at doing is mirroring the first drive to the 2nd drive, and once that intial one is moved over, have a >cron run nightly (or when specified) that will update any modified files from the cron run. I am looking to have this as a perl >script (or whatever) so i can easily port it to all my servers across my network. So that if hard drive 1 fails, i will just need to >pop in hard drive two and be up and running within minutes rather then hours (my last outage was pretty major on my /boot partion >and if had rysnc setup in the manor i am speaking of, downtime would have went from 8 hours to less then 30 minutes....) > >Does anyone have such examples of what i can do to accomplish this? or if even better, someone is willing to create the script in >question for some a quick $50 or something like that I have absolutly no problem in. I am rather busy with daily operations and if i >can get someone to make me the script, that would be very favorable > > >------------------------------------- >Thanks! >Gary Peltola >
> What about software RAID 1 (mirroring)? If the backup is not on another > physical machine, seems like mirroring might be as easy as rsync, w/o > any scary scripted rm -rf'ing or --delete. Plus, it's always up-to-date > (at least between disk sync intervals, someone else can remind me what > those are under ext2fs, ~30 sec, IIRC), whereas a scripted rsync would > only be good up to the last go-round on the cron job, nightly in your > example. > > Don't get me wrong, I love rsync, but seems like not quite the right > tool for this job, IMHO. >We already got RAID 1, and honestly its worthless with file system errors / corruptions since it will in fact mirror errors and all. The only true way to have a reliable backup of your file system is an exact mirror