We are running backup softwares for incrementals/differentials and full backups with variouse softwares currently using dirvish scripts + amanda .. what is everyones views on other opensourced backup software? is there anything better or other options we have missed? We are looking at backula as an option? any thoughts? Thanks - Nathan - http://www.linuxcare.ca
Nathan wrote:> We are running backup softwares for incrementals/differentials and full backups > with variouse softwares currently using dirvish scripts + amanda .. what is > everyones views on other opensourced backup software? is there anything better > or other options we have missed? We are looking at backula as an option? any > thoughts?I wanted to recommend you to take a look at Bacula, than I saw it is already on your list. Bacula is very nice peace of software. It can do almost everything that very expensive products from Veritas and Legato do. The only thing it is missing is a nice GUI. You'll need to configure it using your favorite text editor. It can be a bit complex to configure, however it does come with good default configuration files that you can use to build on. However, you should read documentation and really understand how Bacula works before deploying. It is much more powerful and therefore also much more complex system than Amanda. It is based on three separate components. The center piece is Director. This is basically backup server. Other two pieces are Storage Daemon that handles the storage (you can have as many as you like, it can run on same machine as Director, or on dedicated machine, or even on the client) and File Daemon (this is basically the client). The storage format isn't standard (so you can't simply take the tapes and run tar or restore commands on them, like you could with Amanda). It also needs SQL database (MySQL, PostgreSQL and SQLite are supported at the moment). There's couple of utilities distributed with Bacula that allow you to read and extract data directly from tapes (in case you loose your backup server or SQL database), and/or recreate SQL database from content of the tapes. Bacula also has some nice security features. You can use SSL to encrypt communication between all three components (Director, Storage Daemon and File Daemon), so you don't have to worry of somebody sniffing your /etc/shadow while your network backup runs. It also supports encryption of the backup on the client machine. That means that the data is encrypted. So even if somebody steals your tapes, he can't do anything with them. He would need decryption key, which is normally stored on the client machine (this allows you to centrally backup several departments or clients that don't trust each other, or you). Also, think offsite tapes stored in "trusted" location somewhere far away. There are some very interesting features planned for the future. All in all, very interesting project. Something worth including into repository such as centosplus or Dag's. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature URL: <http://lists.centos.org/pipermail/centos/attachments/20061022/0e61b04c/attachment-0002.sig>
On Sun, 2006-10-22 at 11:41 -0700, Nathan wrote:> We are running backup softwares for incrementals/differentials and full backups > with variouse softwares currently using dirvish scripts + amanda .. what is > everyones views on other opensourced backup software? is there anything better > or other options we have missed? We are looking at backula as an option? any > thoughts? > >I use backuppc ... which is great if you have a PC to designate as a backup machine. I do not backup to tape, but to a separate machine. It is very easy to recover specific files with the web interface. I don't have any idea if it is good as tape backup front though. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://lists.centos.org/pipermail/centos/attachments/20061022/83bb1dd2/attachment-0002.sig>
On Sun, 2006-10-22 at 13:41, Nathan wrote:> We are running backup softwares for incrementals/differentials and full backups > with variouse softwares currently using dirvish scripts + amanda .. what is > everyones views on other opensourced backup software? is there anything better > or other options we have missed? We are looking at backula as an option? any > thoughts?If you are going strictly to disk or with an occasional archive to tape, backuppc is great because its compression and duplicate linking will hold about 10x what you'd expect online and it has a nice web interface for browsing and restores. In normal operation it is 'full-auto' - even more so than amanda since you don't need to swap tapes. -- Les Mikesell lesmikesell at gmail.com
> We are running backup softwares for incrementals/differentials and full backups > with variouse softwares currently using dirvish scripts + amanda .. what is > everyones views on other opensourced backup software? is there anything better > or other options we have missed? We are looking at backula as an option? any > thoughts?I recommend Arkeia. <http://www.arkeia.com> Barry
On Sun, 22 Oct 2006 at 2:21pm, Aleksandar Milivojevic wrote> Nathan wrote: >> We are running backup softwares for incrementals/differentials and full >> backups with variouse softwares currently using dirvish scripts + amanda .. >> what is everyones views on other opensourced backup software? is there >> anything better or other options we have missed? We are looking at backula >> as an option? any thoughts? > > I wanted to recommend you to take a look at Bacula, than I saw it is already > on your list. Bacula is very nice peace of software. It can do almost > everything that very expensive products from Veritas and Legato do. The only > thing it is missing is a nice GUI. You'll need to configure it using your > favorite text editor. It can be a bit complex to configure, however it does > come with good default configuration files that you can use to build on. > However, you should read documentation and really understand how Bacula works > before deploying. It is much more powerful and therefore also much more > complex system than Amanda.I'd take a bit of an issue with your last statement, there. Amanda's best feature is its *very* powerful scheduling algorithm. Essentially, amanda tries to ensure that, within the parameters you set for time between full backups, each night's backup run backs up about the same amount of data. This is very handy when you are, as I am, backing up ~10TB of data to small (even LTO3 is small when you're talking that much data) tapes. AFAICT, all scheduling with bacula is user driven. Thus you end up with the traditional *nix incrementals on weekdays, fulls on the weekend shuffle where your fulls can take a *long* time. And that doesn't work so well, e.g., when you're dealing with grad students who work on weekends. If I'm wrong about bacula's lack of any sort of scheduler, I'd be happy to hear about it. But, for me, amanda is a rather powerful and complete solution -- moreso than bacula. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Barry Brimer wrote:>> We are running backup softwares for incrementals/differentials and >> full backups with variouse softwares currently using dirvish scripts >> + amanda .. what is everyones views on other opensourced backup >> software? is there anything better or other options we have missed? >> We are looking at backula as an option? any thoughts? > > I recommend Arkeia. <http://www.arkeia.com> > > BarryUnless they've changed things since v4.x, I CAN'T reccomend Arkeia. They store the backup database on a local drive, and if you should loose that db, you'll have to manually scan all your tapes before you can retrieve any data. Yes, you can manually backup the db somewhere else, but you can't easily put it on tape. Sure, you could simply dump the contents of each tape to another system with Arkeia's CLI, but in my case, it was 24 hours per AIT3 tape! Since this was in a library, I had 9 tapes to get through. The best backup arrangement I've used is rysnc to another system. I have it worked out in a very nice way to automate offsite storage to my colo. I never recommend Arkeia to anyone... Sorry to be such a stick-in-the-mud about it, but Arkeia let me down in a very severe manner, and I don't wish anyone else to suffer through the same problems I had. HTH Mark
Barry Brimer wrote:>> Unless they've changed things since v4.x, I CAN'T reccomend Arkeia. >> They store the backup database on a local drive, and if you should >> loose that db, you'll have to manually scan all your tapes before >> you can retrieve any data. Yes, you can manually backup the db >> somewhere else, but you can't easily put it on tape. Sure, you could >> simply dump the contents of each tape to another system with >> Arkeia's CLI, but in my case, it was 24 hours per AIT3 tape! Since >> this was in a library, I had 9 tapes to get through. > > Are you familiar with Arkeia's DR module? > > BarryNo, I think this came out in version 5, which by that time, I didn't want to have anything to do with Arkeia. Mark
On Sun, 22 Oct 2006 11:41:51 -0700, you wrote:>We are running backup softwares for incrementals/differentials and full backups >with variouse softwares currently using dirvish scripts + amanda .. what is >everyones views on other opensourced backup software? is there anything better >or other options we have missed? We are looking at backula as an option? any >thoughts?I am looking for the answer to the same question. I have got amanda going (but not used in anger - just doing my first Centos install for my home server). Just today got amanda to write and restore some data. So my requirements: * cheap/free * multiple backups per tape * fully automated backup each day * easy recovery (happy with either a Kdat type GUI or amanda type) - needs to know which tape to recover the latest (or chosen) version. * ideal for a home network (mixed Linux Windows) The only thing I don't like about amanda (so far) is that it needs a new tape each backup, mainly because I'd like it to be a completely automatic backup, only requiring the tape to be changed when full (or maybe each month). I typically don't backup much data each day (because it's a home network), so I'd like to be able to store multiple backups on each tape. I have 20GB Travan tape drive, so that's enough for a complete full backup and several incremental's. I want something cheap (ideally free). Looking at Kdat (but that doesn't seem to recognise my tape drive and not sure it's automated) and tar (but that isn't going to keep track of what's where if I need to do a recovery) I've had a (brief look) at the Bacula website and it looks like that will do what I would like - anything I've missed? -- Peter Crighton
Matt Hyclak wrote:> On Tue, Oct 24, 2006 at 04:01:45PM +0100, Peter Crighton enlightened > us: >>> We are running backup softwares for incrementals/differentials and >>> full backups with variouse softwares currently using dirvish >>> scripts + amanda .. what is everyones views on other opensourced >>> backup software? is there anything better or other options we have >>> missed? We are looking at backula as an option? any thoughts? >> >> I am looking for the answer to the same question. I have got amanda >> going (but not used in anger - just doing my first Centos install for >> my home server). Just today got amanda to write and restore some >> data. >> >> So my requirements: >> >> * cheap/free >> * multiple backups per tape >> * fully automated backup each day >> * easy recovery (happy with either a Kdat type GUI or amanda type) - >> needs to know which tape to recover the latest (or chosen) version. >> * ideal for a home network (mixed Linux Windows) >> >> >> >> The only thing I don't like about amanda (so far) is that it needs a >> new tape each backup, mainly because I'd like it to be a completely >> automatic backup, only requiring the tape to be changed when full >> (or maybe each month). >> >> I typically don't backup much data each day (because it's a home >> network), so I'd like to be able to store multiple backups on each >> tape. I have 20GB Travan tape drive, so that's enough for a complete >> full backup and several incremental's. >> > > If you have enough holding disk, just leave the tape out until you > hit about 20GB worth of data. I do this here at work on a weekly > basis - holding disk is a pair of RAID 1 disks, then once a week I > pop a tape in and it flushes the entire week's worth of data. > > MattIf you have 20GB of data, using tapes is OK. In my case, I have about 3TB of data that needs to be backed up, and taken offsite. So, the only real option is rsync going out to disks. We started out with using one of the recipes from the Linux Server Hacks book, #38, #41 & #42 to essentially build up a poor man's SAN. Using CentOS installed on systems with 3Ware cards, I have 2 onsite 4 TB NAS. The first one is for network use, the second is for hourly, daily and weekly snapshots of the main NAS. There's a third 4TB NAS that's located offsite in a colo facility that's fed with dual T1s. We can have anywhere from 2-5 GB of data change every day. We're a company of about 50 employees, and we do legal work - so nothing can be thrown away. This system runs 7 days a week, and it's fully automated with email alerts, etc. The big benefit is restores. We've had our graphics dept accidently delete 250GB of data, and it was trivial to scp the missing data back to the main NAS. It all happened at network speeds, over a GB switch. All the NASes have dual NICS in them, and the second NICS are connected to their own private GB switch - hence the poor man's SAN. When hourly snapshots run, all the data that changes has a seperate GB network to move the data, leaving the office network alone. No user can tell that backups are happening throughout the day. Maybe this is something I should write up in more detail. The entire system runs on just a couple of shell scripts, rsync, and Perl program to mail out logs.... HTH Mark Mark
>>>> Maybe this is something I should write up in more detail. The >>>> entire system runs on just a couple of shell scripts, rsync, and >>>> Perl program to mail out logs.... >>>> >>>> HTH >>>> Mark >>>> >>>> >>> If you do by chance write that up, I would be interested in seeing >>> it. Dustin >> >> Agreed! Hints about where you put the snapshot files, how you are >> mounting the NAS, the commands you use to generate the snapshots, >> etc. would be most helpful. I know that each of these pieces are >> documented sparerately, but gathering this stuff you do into a nice >> little recipe can be very helpful to other folks.My goal is to have the rough draft posted on my blog later today. Mark
On Fri, 2006-03-11 at 10:28 -0800, Bbt Lists wrote:> Mark Schoonover wrote: > >>>>> Maybe this is something I should write up in more detail. The > >>>>> entire system runs on just a couple of shell scripts, rsync, and > >>>>> Perl program to mail out logs.... > >>>>> > >>>>> HTH > >>>>> Mark > >>>>> > >>>>> > >>>>> > >>>> If you do by chance write that up, I would be interested inseeing> >>>> it. Dustin > >>>> > >>> Agreed! Hints about where you put the snapshot files, how you are > >>> mounting the NAS, the commands you use to generate the snapshots, > >>> etc. would be most helpful. I know that each of these pieces are > >>> documented sparerately, but gathering this stuff you do into anice> >>> little recipe can be very helpful to other folks. > >>> > > > > My goal is to have the rough draft posted on my blog later today. > > > > Mark > > > > > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > http://lists.centos.org/mailman/listinfo/centos > > > Please send me the link either on list or off list if others do notwant it.> > =-)On list if at all possible. Jeff
Jeff Stacey wrote:> On Fri, 2006-03-11 at 10:28 -0800, Bbt Lists wrote: >> Mark Schoonover wrote: >>>>>>> Maybe this is something I should write up in more detail. The >>>>>>> entire system runs on just a couple of shell scripts, rsync, and >>>>>>> Perl program to mail out logs.... >>>>>>> >>>>>>> HTH >>>>>>> MarkA day late, but ran into some problems with blogspot. The article is posted at: http://marks-tech-pages.blogspot.com - titled Linux Based Poor Man's SAN. Let me know what you think! Thanks! Mark Schoonover IS Manager American Geotechnical V: 858-450-4040 - F: 714-685-3909 - C: 858-472-3816 "Stop the Earth!! I want off!!"
-----Original Message----- From: Les Mikesell [mailto:lesmikesell at gmail.com] Sent: Monday, November 06, 2006 8:59 AM To: CentOS mailing list Subject: RE: [CentOS] Best backup software for linux On Fri, 2006-11-03 at 07:51 -0800, Mark Schoonover wrote:> >>>> Maybe this is something I should write up in more detail. The > >>>> entire system runs on just a couple of shell scripts, rsync, and > >>>> Perl program to mail out logs.... > >>>> > >>> If you do by chance write that up, I would be interested in seeing > >>> it. Dustin> My goal is to have the rough draft posted on my blog later today.Do you have a comparison to backuppc (http://backuppc.sourceforge.net/)? No, but you can read my blog, then look at backuppc and see what the differences are. Let us know. Mark
Mark Schoonover wrote:> From: Les Mikesell [mailto:lesmikesell at gmail.com] > > On Fri, 2006-11-03 at 07:51 -0800, Mark Schoonover wrote: > > > > > > Maybe this is something I should write up in more detail. > > > > > > The entire system runs on just a couple of shell scripts, > > > > > > rsync, and Perl program to mail out logs.... > > > > > > > > > > > If you do by chance write that up, I would be interested in > > > > > seeing it. Dustin > > > My goal is to have the rough draft posted on my blog later today. > > Do you have a comparison to backuppc > (http://backuppc.sourceforge.net/)? > > No, but you can read my blog, then look at backuppc and see what the > differences are. Let us know.Sounds interesting. Where is your blog? -- Bowie
Les Mikesell wrote:> On Mon, 2006-11-06 at 18:42 +0000, Peter Crighton wrote: > >> You wrote "Hardlinks are key to this backup strategy. Using cp -al >> creates hardlinks to files, and this simple command is what does all >> the heavy lifting for daily and weekly backups. Wikipedia has a very >> good explanation on how hardlinks work. In a nutshell, when there's a >> hardlink pointing to a file from the hourly directory, to a file in >> the current directory, and that current file gets deleted, all the >> links that point to that now deleted current file gets the file data >> 'pushed' back towards all the links. I'll have to think how to >> explain this better." >> >> Do you mean that the hourly files are written when created, the >> hardlink for the daily doesn't actually copy the file (simply makes a >> link), but if the file is set to be deleted from it's location >> (because it's gone from the server) then it is actually moved so that >> it still exists in the daily backup but is removed from the hourly? >> -- > > Think of all directory entries as links. The real entries that > map disk space to files are inodes and links are names pointing > to the inodes. There can be any number - including 0 - of links > to an inode. The space is not released for re-use until the > link count goes to 0 and no process has the file open. So hardlinks > are just multiple names pointing to the same data, and the data > doesn't go away until the last name is removed.You did much better explaining what's going on with hardlinks than I did. I'm going to have to rewrite that part of the blog a few times before it reads better. I can picture it all in my head, but describing how it works is another.>Note that this only works as a backup if the original filename is removed.If it>is overwritten or truncated instead, all links now point to the changedversion. This is true if you're doing it with only filesystem tools, but this system is using rsync. What's happening is the cp -al occurs first making hardlinks that point to an hourly directory into the current directory, then rsync is run to update current. Because rsync will create a new temp file when any file changes, the original is deleted with it's data 'pushed' to any hardlinks pointing at the original file. Rsync then renames the temp file the original file name that has changed, therefore assuring that any hardlinks will always have the previous copy of any changed files. With rsync running in --delete mode, any files from the source server that get deleted, will get deleted out of current in the backup server, causing this cascade of hardlinks to get updated with the deleted files data. That's how this system can create incremental backups of only changed data, but with hardlinks, it looks like full backups are made each and every time. Really saves disk space, that's for sure! Hope this clears things up... Mark
Les Mikesell wrote:> On Mon, 2006-11-06 at 14:54 -0800, Mark Schoonover wrote: > >>> Note that this only works as a backup if the original filename is >>> removed. If it is overwritten or truncated instead, all links now >>> point to the changed version. >> >> This is true if you're doing it with only filesystem tools, but this >> system is using rsync. What's happening is the cp -al occurs first >> making hardlinks that point to an hourly directory into the current >> directory, then rsync is run to update current. Because rsync will >> create a new temp file when any file changes, the original is >> deleted with it's data 'pushed' to any hardlinks pointing at the >> original file. Rsync then renames the temp file the original file >> name that has changed, therefore assuring that any hardlinks will >> always have the previous copy of any changed files. With rsync >> running in --delete mode, any files from the source server that get >> deleted, will get deleted out of current in the backup server, >> causing this cascade of hardlinks to get updated with the deleted >> files data. That's how this system can create incremental backups of >> only changed data, but with hardlinks, it looks like full backups >> are made each and every time. Really saves disk space, that's for >> sure! >> >> Hope this clears things up... > > Backuppc is even more extreme in the space savings. It first > compresses the files, then detects duplicates using an efficient > hashing scheme and links all duplicates to one pooled copy whether > they came from > the same source or not. It includes a custom rsync on the server > side that understands the compressed storage format but works with > stock versions on the remote side so you don't need any special client > software. And it has a nice web interface for browsing the backup > archive and doing restores.Sounds like a good product! Since I had plenty of terabytes of storage, having compression on the files wasn't a requirement. One of my requirements was not having a gui to configure things, or do restores. Thanks for pointing some info out on backuppc, I've never heard of it. Mark