What do you guys recommend for backing up a small CentOS server in a business environment. It will have (3) 300gb drives in a raid 5 array but I don't anticipate more than about 25gb of data that needs to be backed up each night. I want a lot of backups with a rotation scheme that included daily, weekly, and monthly copies. I want the daily copies of the data kept until the next week, and the weekly copy being kept for four weeks, and the monthly copies being kept for a year. The vendor is recommending a RD1000 Removable Disk device. This looks like it has great specs. Each cartridge holds 160gb (non-compressed) and the drive costs about $420 but seems that with each removable cartridge costing $128, we may be limited to how many cartridges we could have, thus perhaps not retaining backup instances as long as I like. I asked about a HP DAT160 tape drive. Each tape holds 160gb (non-compressed) and the drive costs about $730, and each tape only costs about $24, so it would be economical to have lots of backup instances saved for a long period of time. I have been using tape and the backup rotation scheme mentioned above for over 20 years. The vendor is telling me they don't recommend tape drives anymore and all of their customers are using removable hard drive for local backups. Am I missing something? My instincts tell me the tape drive is the right solution for a system with a small amount of data, where the system is used only from 8am - 5pm (so backup speed is not critical) and where we want to save backup instances for a long time before overwriting them. Any input would be welcomed.
On 03/11/2012 08:12 PM, Scott Walker wrote:> What do you guys recommend for backing up a small CentOS server in a > business environment. It will have (3) 300gb drives in a raid 5 array but I > don't anticipate more than about 25gb of data that needs to be backed up > each night. > I want a lot of backups with a rotation scheme that included daily, weekly, > and monthly copies. I want the daily copies of the data kept until the next > week, and the weekly copy being kept for four weeks, and the monthly copies > being kept for a year. > > The vendor is recommending a RD1000 Removable Disk device. This looks like > it has great specs. Each cartridge holds 160gb (non-compressed) and the > drive costs about $420 but seems that with each removable cartridge costing > $128, we may be limited to how many cartridges we could have, thus perhaps > not retaining backup instances as long as I like. > > I asked about a HP DAT160 tape drive. Each tape holds 160gb > (non-compressed) and the drive costs about $730, and each tape only costs > about $24, so it would be economical to have lots of backup instances saved > for a long period of time. > > I have been using tape and the backup rotation scheme mentioned above for > over 20 years. The vendor is telling me they don't recommend tape drives > anymore and all of their customers are using removable hard drive for local > backups. Am I missing something? My instincts tell me the tape drive is > the right solution for a system with a small amount of data, where the > system is used only from 8am - 5pm (so backup speed is not critical) and > where we want to save backup instances for a long time before overwriting > them. > > Any input would be welcomed. > >What do you consider to be a "long time" to keep backups on hand? Tape, and tape drives, have a bad reputation. They are difficult and time consuming to verify. I run my backups nightly to a hard drive using rsync. I use a directory named by the day of the week. I cycle through the seven daily directories until the 1st of the month when I run a complete backup to an monthly directory. Then for the next seven days I wipe the daily directories and start the cycle over again. A couple of minor variations to this plan should work for you. I don't know what your network configuration looks like so this may not apply to you. Here's a peek at the logic I use. # BUILD DATE STAMP Date=`date +%Y%m%d` echo "Date= \"$Date\"" # Rev. 5.6 start Day=`date +%a` echo "Day= \"$Day\"" DayNum=`date +%d` # Rev. 7.0 # IF THIS IS A SUNDAY USE THE CALANDAR DATE if [ "$Day" == "Sun" ];then Day="$Date" else # IF THIS IS THE 1ST OF THE MONTH USE THE CALANDAR DATE if [ "$DayNum" == "01" ];then Day="$Date" fi fi # USE THE DAY OF THE WEEK, EXCEPT FOR SUNDAY AND THE 1ST OF THE MONTH WHICH IS HANDLED ABOVE, AS THE DIRECTORY NAME Date="$Day" # Rev. 5.6 end # REMOVE PREVIOUS $Date DIRECTORY IF THIS IS THE FIRST USE THIS MONTH # Rev. 7.0 ENTIRE CASE STATEMENT ADDED case $DayNum in 02) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 03) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 04) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 05) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 06) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; 07) echo "Removing /home/homebu/$Date directory" rm -rf /home/homebu/$Date ;; *) echo "Old $Date directory not deleted" ;; esac # TRANSER FILES -- _ ?v? /(_)\ ^ ^ Mark LaPierre Registerd Linux user No #267004
On Sun, Mar 11, 2012 at 8:12 PM, Scott Walker <Scott_Walker at ramsystemscorp.com> wrote:> What do you guys recommend for backing up a small CentOS server in a > business environment. ?It will have (3) 300gb drives in a raid 5 array but I > don't anticipate more than about 25gb of data that needs to be backed up > each night. > I want a lot of backups with a rotation scheme that included daily, weekly, > and monthly copies. ?I want the daily copies of the data kept until the next > week, and the weekly copy being kept for four weeks, and the monthly copies > being kept for a year. > > The vendor is recommending a RD1000 Removable Disk device. ?This looks like > it has great specs. ?Each cartridge holds 160gb (non-compressed) and the > drive costs about $420 but seems that with each removable cartridge costing > $128, we may be limited to how many cartridges we could have, thus perhaps > not retaining backup instances as long as I like. > > I asked about a HP DAT160 tape drive. ?Each tape holds 160gb > (non-compressed) and the drive costs about $730, and each tape only costs > about $24, so it would be economical to have lots of backup instances saved > for a long period of time. > > I have been using tape and the backup rotation scheme mentioned above for > over 20 years. ?The vendor is telling me they don't recommend tape drives > anymore and all of their customers are using removable hard drive for local > backups. ?Am I missing something? ?My instincts tell me the tape drive is > the right solution for a system with a small amount of data, where the > system is used only from 8am - 5pm (so backup speed is not critical) and > where we want to save backup instances for a long time before overwriting > them. > > Any input would be welcomed.The cost of disks is so low, it's very hard to justify tape. Don't forget you also need to have someone swapping the tapes every day or week, or spend more for a robot. For the amount you would spend on those tapes, you can get many TBs of disk space. In general it works very well to spend your money on disks and backup to multiple locations. With disk, you get so many benefits, such as random-access recovery, and most disk-based systems support some level of data deduplication. If you use something like rsync backups with hard links, there's also never a need for a full backup after the first one. I'm sure you will be able to come up with a few arguments against using disk, and in some situations tape is better, but almost never for some little server somewhere. Once you start talking about long-term archives and stuff like that, then yes, tapes are good. Disks also need a different type of maintenance, such as running a full read/refresh of the data every so often. In the SAN world they call this "scrubbing", though don't confuse it with the 'scrub' command that securely wipes all data from the disk... Some common disk-to-disk backup tools: - BackupPC - rdiff-backup - dirvish - Duplicity - Duplicati An overview of using rsync for backups: http://www.mikerubel.org/computers/rsync_snapshots/ ? Brian Mathis
Scott Walker <Scott_Walker at ...> writes:> > What do you guys recommend for backing up a small CentOS server in a > business environment. It will have (3) 300gb drives in a raid 5 array but I > don't anticipate more than about 25gb of data that needs to be backed up > each night.. . . I stumbled on http://storebackup.org/ the other day. It looks pretty good for disc-disc backup. Myself, I use a daily backup with rsync creating hard links to the backup directory - both remote (offsite) and local. http://bhepple.freeshell.org/oddmuse/wiki.cgi/backup-copy and supporting stuff at: http://bhepple.freeshell.org/oddmuse/wiki.cgi/UnixScripts
On Sun, Mar 11, 2012 at 7:12 PM, Scott Walker <Scott_Walker at ramsystemscorp.com> wrote:> What do you guys recommend for backing up a small CentOS server in a > business environment. ?It will have (3) 300gb drives in a raid 5 array but I > don't anticipate more than about 25gb of data that needs to be backed up > each night. > I want a lot of backups with a rotation scheme that included daily, weekly, > and monthly copies. ?I want the daily copies of the data kept until the next > week, and the weekly copy being kept for four weeks, and the monthly copies > being kept for a year.I'd look at backuppc first - running on a different machine which can be fairly low-powered or one that does something else in the daytime. It pools multiple copies of identical content with hardlinks so you can keep much more history online than you would expect, and it provides a nice web interface for browsing backups and doing restores. And it basically takes care of itself.> I have been using tape and the backup rotation scheme mentioned above for > over 20 years. ?The vendor is telling me they don't recommend tape drives > anymore and all of their customers are using removable hard drive for local > backups. ?Am I missing something? ?My instincts tell me the tape drive is > the right solution for a system with a small amount of data, where the > system is used only from 8am - 5pm (so backup speed is not critical) and > where we want to save backup instances for a long time before overwriting > them. > > Any input would be welcomed.Tapes are really inconvenient compared to online backups and disks are so inexpensive these days that tape only makes sense for archiving. -- Les Mikesell lesmikesell at gmail.com
There are three reasons backups are made.
1: Protect from hardware failure
2: Protection from user deletion and/or corruption
of files whether accidental or otherwise. (Yes,
there are people who will intentionally damage
a filesystem.)
3: Fire or other natural disasters.
From your description below, you seem reasonably well
protected against hardware failure. I would not run
RAID5 with three disks though. Instead, I would run
RAID1 with the third disk as a hot-standby. For the
amount of data you are managing, 300GB is plenty of
space for 25GB and CentOS.
Protection from filesystem damage requires a rolling
archive such as can be created the dump program. Think
about dump in a tower-of-hanoi level sequence, or similar.
This archive should be maintained on a separate device.
Running a crontab scheduled dump every night will preserve
all files as they exist at the close of business each day.
Protection of data from natural disasters requires that
off-site backups be made.
The old school way to address all above was tape rotation.
As disks have become more affordable various forms of RAID
have taken #1 off the list of reasons to do tape rotation.
These days other choices are available.
An archive tree still needs to be maintained. My view is
that the tree should be maintained on a local disk. Creating
an off-site backup is simply taking a snapshot of the dump
tree and transporting it to a safe place.
The off-site issue is something you need to address with your
client. I always make the client assume the responsibility
for setting the frequency of off-site backups and of actually
doing them.
As to the backup device, I would suggest a USB hard-drive.
I happen to have a 64GB USB flash drive I use to transport
data. Such a device or smaller, if rotated weekly could be
the backup device. My recommendation would be to attach a
USB hard-drive to the system and maintain the dump tree
on it. I would use the USB thumb drive for a transport device.
25GB of data isn't very much.
I don't know if you and perl have made friends. Personally,
I do most of my system specific scripting in Perl rather
than shell code, but that's just me.
I have a perl program I use to do exactly what I have described
above for a client who has about 100GB of data to protect.
This program builds a hierarchy of weekly folders. On the
first work-week of each quarter, it will write a level 0
dump file. All the daily files are incremental from that
dump. The drive the dump-tree is on has more than sufficient
space to keep a year's worth of backups. Once a quarter
the client takes a snapshot of the most recently completed
quarter's tree.
If you (or any others on the list) would like the program
I will send if. If several ask, I will just post it. It
is only about 350 lines with some comments.
Ray
On 03/11/2012 05:12 PM, Scott Walker wrote:> What do you guys recommend for backing up a small CentOS server in a
> business environment. It will have (3) 300gb drives in a raid 5 array but
I
> don't anticipate more than about 25gb of data that needs to be backed
up
> each night.
> I want a lot of backups with a rotation scheme that included daily, weekly,
> and monthly copies. I want the daily copies of the data kept until the
next
> week, and the weekly copy being kept for four weeks, and the monthly copies
> being kept for a year.
>
> The vendor is recommending a RD1000 Removable Disk device. This looks like
> it has great specs. Each cartridge holds 160gb (non-compressed) and the
> drive costs about $420 but seems that with each removable cartridge costing
> $128, we may be limited to how many cartridges we could have, thus perhaps
> not retaining backup instances as long as I like.
>
> I asked about a HP DAT160 tape drive. Each tape holds 160gb
> (non-compressed) and the drive costs about $730, and each tape only costs
> about $24, so it would be economical to have lots of backup instances saved
> for a long period of time.
>
> I have been using tape and the backup rotation scheme mentioned above for
> over 20 years. The vendor is telling me they don't recommend tape
drives
> anymore and all of their customers are using removable hard drive for local
> backups. Am I missing something? My instincts tell me the tape drive is
> the right solution for a system with a small amount of data, where the
> system is used only from 8am - 5pm (so backup speed is not critical) and
> where we want to save backup instances for a long time before overwriting
> them.
>
> Any input would be welcomed.
>
>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
On 3/11/2012 6:12 PM, Scott Walker wrote:> What do you guys recommend for backing up a small CentOS server in a > business environment. It will have (3) 300gb drives in a raid 5 array but I > don't anticipate more than about 25gb of data that needs to be backed up > each night. > I want a lot of backups with a rotation scheme that included daily, weekly, > and monthly copies. I want the daily copies of the data kept until the next > week, and the weekly copy being kept for four weeks, and the monthly copies > being kept for a year. > > The vendor is recommending a RD1000 Removable Disk device. This looks like > it has great specs. Each cartridge holds 160gb (non-compressed) and the > drive costs about $420 but seems that with each removable cartridge costing > $128, we may be limited to how many cartridges we could have, thus perhaps > not retaining backup instances as long as I like. > > I asked about a HP DAT160 tape drive. Each tape holds 160gb > (non-compressed) and the drive costs about $730, and each tape only costs > about $24, so it would be economical to have lots of backup instances saved > for a long period of time. > > I have been using tape and the backup rotation scheme mentioned above for > over 20 years. The vendor is telling me they don't recommend tape drives > anymore and all of their customers are using removable hard drive for local > backups. Am I missing something? My instincts tell me the tape drive is > the right solution for a system with a small amount of data, where the > system is used only from 8am - 5pm (so backup speed is not critical) and > where we want to save backup instances for a long time before overwriting > them. > > Any input would be welcomed.I believe in tape... it's just not a viable option with the large disk sizes we have today unless you have a lot of money for a fast, multi-drive solution. I can backup a bit over 500GB daily in 3 hours to external disk. Using a single tape drive that would (and did) take far too long. So today I use TB size drives dropped into an external docking station. The docking station plugs into the server using eSATA. Then it's a relatively simple script run by cron to handle the daily backup. I'm happy to share the script if you're interested but it has long lines that don't do well in email. I'll send it offline if you'd like to use it as an example. Buying multiple drives allows us to do media rotation just like we did with tape. The big difference with disks is that I just do full backups each time. In our situation there is time for that and it saves a lot of grief when trying to restore something in particular. None of this running back thru the incrementals to get at what you want. Of course, with incremental backups the typical daily time would be much much shorter. For backing up multiple production servers I have a backup server and a private GB network to each system. Each server runs a backup script at night (via cron) to backup to the backup server. Then we backup the backup server to the external disk during the day. At least one external disk is off site at any given time. I'm aware of the fancy tools to do the job for you but I like the simplicity of our home grown solution. And the only thing I absolutely need for a restore is tar. No databases, no extra applications, just tar. The catch is that I'm not sure it would scale up to a huge number of servers gracefully. It's nice to know what works for the other guy but you gotta look for what will work for you. Good luck. //Steve
Over the years I have run into several situations where for one reason or another a backup utility such as dump or tar couldn't read a particular backup. For that reason, I like to periodically do a backup using another backup format. So I might use backuppc for my main backup system, but once a month do a full backup using dump onto a completely separate media. I have been sucessfully using 8GB dual layer DVDs for some of my backups/archiving and now that the price of Blu ray has come down I am about to experiment with that. I have been writing dump format files to the DVD's and then writing an SHA256 checksum for each dump file so it's very easy to verify the integrity of the dump. I am also about to try daily emcrypted backups to http://rsync.net along with periodic archival to blu-ray disk for one of my backup needs. I have noticed that two of the recently mentioned backup packages, duplicity and storebackup appear to support some kind of block level deduplication where you can backup a large file, database or possibly even a disk partition, incrementally over the network. I am interested in trying that for backup up of mysql databases. Nataraj
On Wed, March 14, 2012 00:26, Nataraj wrote:> I think the reality is that nothing lasts forever. > Optical media is probably much more likely to > survive ICBM's, but then you may not have a drive > to read them... > > Nataraj >About five years ago I was asked to recover data from a 2400' reel mag-tape of unknown provenance in an unknown format. The possessor was a university professor and the tape dated from the late 1970s. It contained data from social science research projects he had conducted at that time. I eventually managed to read the tape at 1600 bpi in raw block format and from the headers determined that the encoding was EBCDIC and that the tape had been created on a CDC machine. However, the entire tape after the headers was blank. Not corrupt, not zeroed, just blank. Apparently the operator had mounted, initialized and labelled an evidently new tape but never actually put any data on it. So the professor had been carefully storing a scratch tape for decades. Always restore from your backups on a regular basis just to ensure there is something there to restore from. -- *** E-Mail is NOT a SECURE channel *** James B. Byrne mailto:ByrneJB at Harte-Lyne.ca Harte & Lyne Limited http://www.harte-lyne.ca 9 Brockley Drive vox: +1 905 561 1241 Hamilton, Ontario fax: +1 905 561 0757 Canada L8E 3C3