Hello all, I am down to my last hurdle of my project, backups. I am thinking of three different ways to go and wanted to ask for input on what you think is the better choice. Not asking for 'how to' but more of 'what is best in your experience' The scenario... centos server acting as a virtual host. Virtual machines are webservers and dns servers. All on one machine, all running centos 6. Virtual machines are kvm, sitting in lvm storage. What I want to do.. auto backups of the virtual machines to be stored on the virtual host's extra drives for later download to my home computer. Many backup solutions and programs seem centered on a network of computers with file sharing. I do not have this and don't think I want to go that way on my host. My three thoughts, not sure which one to pursue... (involves certain folders, /home/ (which includes maildir), /var/www/, /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the comp pretty quick and then restore, or maybe just do one big backup of each server, then work on the folders as a solution) 1- Amanda. I do not know much about it or how it would deal with mysql databases, but it look promising. I do not have a NFS in place on any of the installs. 2- rsnyc - some kind of rsync going from the host to each machine, putting it on the host's backup drives. Adding a mysql hotcopy of some kind on the VMs, along with bin files, saved to a special folder that will then be part of the rsync. Once a week full of both rsync and mysqlcopy, then incremental daily. 3- Use kpartx ? and access the lvm the VM is on to rsync internally on the host, ditto above with the mysql copy/bin setup. Number 3 seems like it is the securest way, but obviously not much info out there on it. Number 2 seems like the 'old way' and will require some real work to get it right number 1 looks good, but do not really know anything about it. Which way would you go, or do you have a different way you like better?
On Sat, Mar 24, 2012 at 8:19 AM, Bob Hoffman <bob at bobhoffman.com> wrote:> The scenario... > centos server acting as a virtual host. Virtual machines are webservers > and dns servers. All on one machine, all running centos 6. > Virtual machines are kvm, sitting in lvm storage. > > What I want to do.. > auto backups of the virtual machines to be stored on the virtual host's > extra drives for later download to my home computer.Hi Bob, You want to backup the whole VM? You can set a script to periodically shutdown the VM and then just rsync the VM files to the other partition. -- http://linux3.arinet.org
Snapshot the lvm and rsync it to your house. Then wipe out the snap. Seems easiest. Though it won't be the best for things like running databases. Sent from my iPhone On Mar 23, 2012, at 8:20 PM, Bob Hoffman <bob at bobhoffman.com> wrote:> Hello all, > I am down to my last hurdle of my project, backups. > I am thinking of three different ways to go and wanted to ask for input > on what you think is the better choice. > Not asking for 'how to' but more of 'what is best in your experience' > > The scenario... > centos server acting as a virtual host. Virtual machines are webservers > and dns servers. All on one machine, all running centos 6. > Virtual machines are kvm, sitting in lvm storage. > > What I want to do.. > auto backups of the virtual machines to be stored on the virtual host's > extra drives for later download to my home computer. > > Many backup solutions and programs seem centered on a network of > computers with file sharing. I do not have this and don't think I want > to go that way on my host. > > > My three thoughts, not sure which one to pursue... > (involves certain folders, /home/ (which includes maildir), /var/www/, > /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the > comp pretty quick and then restore, or maybe just do one big backup of > each server, then work on the folders as a solution) > > 1- Amanda. I do not know much about it or how it would deal with mysql > databases, but it look promising. I do not have a NFS in place on any of > the installs. > > 2- rsnyc - some kind of rsync going from the host to each machine, > putting it on the host's backup drives. Adding a mysql hotcopy of some > kind on the VMs, along with bin files, saved to a special folder that > will then be part of the rsync. Once a week full of both rsync and > mysqlcopy, then incremental daily. > > 3- Use kpartx ? and access the lvm the VM is on to rsync internally on > the host, ditto above with the mysql copy/bin setup. > > Number 3 seems like it is the securest way, but obviously not much info > out there on it. > Number 2 seems like the 'old way' and will require some real work to get > it right > number 1 looks good, but do not really know anything about it. > > Which way would you go, or do you have a different way you like better? > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos
>> On Fri, 23 Mar 2012 20:19:41 -0400, >> Bob Hoffman <bob at bobhoffman.com> said:B> I am down to my last hurdle of my project, backups. Not asking for 'how B> to' but more of 'what is best in your experience'. Some questions: * What's the hardest stuff for you to recreate? I'd have that on both DVD and something network-accessible. * What's your biggest PITA problem (for me, it would be bare-metal restore) vs. your most likely one (I'd assume loss of a MySQL table or a VM)? You mentioned being able to rebuild the host quickly, so if the bare-metal thing isn't a big problem, concentrate on the VMs instead. * What are your priorities? If it's speed of the restore, and you have the IO/network bandwidth and room, then do like another poster said and rsync the VM files after shutting them down. If it's more like history where you want to go back in time to lots of versions, something finer-grained would be in order. B> The scenario... centos server acting as a virtual host. Virtual B> machines are webservers and dns servers. All on one machine, all running B> centos 6. Virtual machines are kvm, sitting in lvm storage. What I B> want to do.. auto backups of the virtual machines to be stored on the B> virtual host's extra drives for later download to my home computer. Your VMs sound like they start out identical, and then you add stuff to specialize each one. If so, I'd keep these backups: a. one generic bare-bones VM that can be installed with as few commands as possible. b. each change-set you use to specialize for basic DNS, web, etc. c. smaller groups of individual files like DB schemas, web content, mailboxes, etc. This way, any given restore breaks down to (a) plus (one or more b) plus (whatever's appropriate from c). When you get to the individual file backups within a VM, something like this might be all you need: # cd / # find . -depth -type f -newer /etc/BKUP -print | pax -x cpio -wd | gzip -c > /path/to/$(date '+%Y/%m%d/%H%M').pax.gz # touch /etc/BKUP B> 1- Amanda. I do not know much about it or how it would deal with mysql B> databases, but it look promising. I set it up once, but it wasn't a close enough match to what we needed for me to craft an entire backup strategy around it. It's not a trivial thing to install or run, so you'll be spending time finding out how Amanda wants to do things and matching that to your goals. B> 2- rsync - some kind of rsync going from the host to each machine, B> putting it on the host's backup drives. That's what I use at work, but we're closer to the "networked fileservers with remote shares" setup. I use the find/pax/touch setup above to handle hourly backups for 800-1000 users, and they're happy little campers when they find out the spreadsheet they created at 6am and mangled around noon isn't completely gone. -- Karl Vogel I don't speak for the USAF or my company Texted a friend meaning to ask if she was busy. Me: "Are you busty?" Her: "They're manageable, what's up?" --Jimmy Fallon, #textingdisaster tweets, 13 Jan 2012
On 03/23/2012 05:19 PM, Bob Hoffman wrote:> Hello all, > I am down to my last hurdle of my project, backups. > I am thinking of three different ways to go and wanted to ask for input > on what you think is the better choice. > Not asking for 'how to' but more of 'what is best in your experience' > > The scenario... > centos server acting as a virtual host. Virtual machines are webservers > and dns servers. All on one machine, all running centos 6. > Virtual machines are kvm, sitting in lvm storage. > > What I want to do.. > auto backups of the virtual machines to be stored on the virtual host's > extra drives for later download to my home computer. > > Many backup solutions and programs seem centered on a network of > computers with file sharing. I do not have this and don't think I want > to go that way on my host. > > > My three thoughts, not sure which one to pursue... > (involves certain folders, /home/ (which includes maildir), /var/www/, > /mysqlhotcopys and bin files, and maybe a few more. I can rebuild the > comp pretty quick and then restore, or maybe just do one big backup of > each server, then work on the folders as a solution) > > 1- Amanda. I do not know much about it or how it would deal with mysql > databases, but it look promising. I do not have a NFS in place on any of > the installs. > > 2- rsnyc - some kind of rsync going from the host to each machine, > putting it on the host's backup drives. Adding a mysql hotcopy of some > kind on the VMs, along with bin files, saved to a special folder that > will then be part of the rsync. Once a week full of both rsync and > mysqlcopy, then incremental daily.I think backuppc, syncing over rsync directly to the VM makes sense here.> 3- Use kpartx ? and access the lvm the VM is on to rsync internally on > the host, ditto above with the mysql copy/bin setup.You can't safely mount the VM's partitions without shutting down the VM. You might get away with it with an LVM snapshot taken inside the host, but that assumes that it's safe to access the LVM datastructure from outside the VM while the VM is running and it may not be.> Number 3 seems like it is the securest way, but obviously not much info > out there on it. > Number 2 seems like the 'old way' and will require some real work to get > it right > number 1 looks good, but do not really know anything about it.For the database you have many options. I would suggest at least weekly do this: 1) shutdown the mysql database server 2) take a snapshot of the partition with the mysql database server 3) restart your mysql server (the whole shutdown/snapshot/restart take less than 15 seconds) 4) Do a full backup of that snapshot, using backuppc, tar, dump or whatever you choose In order to do this, you must of course setup the database on an LV partition (MY LV partition is simply an ext4 or ext 3 partition, mounted on /var/lib/mysql). Make sure there is adequate space on your snapshot LV partition for any data that might get written to the database before the snapshot gets backuped and deleted. If your database is small you can do daily backups of the entire database like this. Otherwise you can use tools like mysqldump and mysqlhotcopy, but you should make sure you have a pretty good understanding of these tools and that you are backing things up properly with them. Some people run replication between two database servers and then shutdown the secondary server and back it up. There are also backup tools like duplicity and storebackup which I have recently learned about that are supposed to be able to do incremental backups of binary files, where you could actually do a restore of the database from your full backup and then somehow apply an update to the database files from incrementals. I have not yet tested this method. There again, I would still shutdown/snapshot/restart the mysql server to ensure database integrity during the backup. Whatever you choose, I suggest you perform test restores and if you choose database level backups, make sure you understand what you are doing and that you are saving the backup in a way that you are able to recover it. If you data is important consider doing backing up in two different formats to 2 different media types. I think simply doing at least a weekly full backup of the entire /var/lib/mysql directory (via snapshot or server shutdown) is the least prone to user error. Be aware that things like changes in database format, etc can make old backups difficult or impossible to restore. Many people, myself included, ran into some problems when mysql changed the way it handled character set encodings and it was not so straightforward to restore mysqldump backups made with an older version of mysql, using the latest version of mysql. Many options had to be specified and some people still had to write scripts to manually go through and make some conversions. I ran into these problems while upgrading a mediawiki database that was running under a very old version of mysql for a client. So if having long term access to old backups is important, you must check for compatibility between newer/older versions of mysql and either do conversion/refresh of your data, or make sure you still have functional copies of the tools necessary to restore older versions. Nataraj> Which way would you go, or do you have a different way you like better? > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos
Hi Bob, I just want to to throw in another alternative to make choice harder ... :-)> The scenario... > centos server acting as a virtual host. Virtual machines are webservers > and dns servers. All on one machine, all running centos 6. > Virtual machines are kvm, sitting in lvm storage.My basic setup is quite similar to yours. CentOS 5 machine working as host, several virtual CentOS 5/6 boxes on that host, but additionally there are some Mac OS X and Ubuntu boxes around that also need to be backed up, some of them in remote locations. I have a NAS drive set up in my small datacenter that works as an iSCSI host and serves a LUN for backups. The virtual host machine runs Bacula dir and sd, with the backup volumes on the LUN. All Clients run bacula fd and connect to the server, the ones connecting from the internet use SSL encryption and certificate authentication. I run full backups once a week, and daily incrementals. For the last half year or so, the solution has proven rock-solid, not a single failure. I had to restore several files during that period, and there wasn't any problem at all with that as well. Database backup is done using pre-scripts that perform a database dump and then back up that dump, which is a bit of a downside as other solutions often have database backup plugins, but nothing that could not be solved. After initial setup, configuration is done on the Bacular directory server alone, which saves some amount of maintenance. Backup strategies can be defined in a very flexible way, too. There also is a feature that allows you to define a base system (e.g. a freshly installed CentOS box) and tell the server just to back up differences from that base system, saving large amounts of storage if you have many similar machines to be backed up. Best regards, Peter.