Mr. Teo En Ming (Zhang Enming)
2009-Nov-13 13:56 UTC
[Xen-users] Data Backup and Restore Scripts for your Xen Servers
There are 2 file attachments in this email. One is for harddisk cloning, the other is for data restoration. If there are any bugs with the scripts, please let me know. Thank you. -- Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical Engineering) Alma Maters: (1) Singapore Polytechnic (2) National University of Singapore My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com My Secondary Blog: http://enmingteo.wordpress.com My Youtube videos: http://www.youtube.com/user/enmingteo Email: space.time.universe@gmail.com Mobile Phone (Starhub Prepaid): +65-8369-2618 Street: Bedok Reservoir Road Country: Singapore _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mr. Teo En Ming (Zhang Enming)
2009-Nov-13 19:34 UTC
[Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
On Fri, Nov 13, 2009 at 9:56 PM, Mr. Teo En Ming (Zhang Enming) < space.time.universe@gmail.com> wrote:> There are 2 file attachments in this email. One is for harddisk cloning, > the other is for data restoration. If there are any bugs with the scripts, > please let me know. > > Thank you. > > -- > Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical > Engineering) > Alma Maters: > (1) Singapore Polytechnic > (2) National University of Singapore > My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com > My Secondary Blog: http://enmingteo.wordpress.com > My Youtube videos: http://www.youtube.com/user/enmingteo > Email: space.time.universe@gmail.com > Mobile Phone (Starhub Prepaid): +65-8369-2618 > Street: Bedok Reservoir Road > Country: Singapore >Attached updates to backup and restore scripts. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mr. Teo En Ming (Zhang Enming)
2009-Nov-15 01:02 UTC
[Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
On Sat, Nov 14, 2009 at 3:34 AM, Mr. Teo En Ming (Zhang Enming) < space.time.universe@gmail.com> wrote:> > > On Fri, Nov 13, 2009 at 9:56 PM, Mr. Teo En Ming (Zhang Enming) > <space.time.universe@gmail.com> wrote: >> >> There are 2 file attachments in this email. One is for harddisk cloning, >> the other is for data restoration. If there are any bugs with thescripts,>> please let me know. >> >> Thank you. >> >> -- >> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical >> Engineering) >> Alma Maters: >> (1) Singapore Polytechnic >> (2) National University of Singapore >> My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com >> My Secondary Blog: http://enmingteo.wordpress.com >> My Youtube videos: http://www.youtube.com/user/enmingteo >> Email: space.time.universe@gmail.com >> Mobile Phone (Starhub Prepaid): +65-8369-2618 >> Street: Bedok Reservoir Road >> Country: Singapore > > Attached updates to backup and restore scripts. > > >Hi All, I have tested my self-written harddisk backup/cloning script by performing a backup of my Western Digital 500 GB SATA harddisk to my 160 GB Hitachi harddisk. The backup operation is successful except for 3 virtual machines. The backup of these 3 virtual machines fail because partimage does not support ext4 filesystems. I have performed a backup of these 3 VMs *again* using the following modified script to handle ext4 filesystems by using fsarchiver: <SCRIPT> #!/bin/sh ############################################################################################################### # Declare Variables ############################################################################################################### HARDDISK=/dev/sda DEST=/media/hitachi/redo FSA_COMPLEVEL=3 PROCESSOR_CORES=2 GZIP_COMPLEVEL=fast PARTIMAGE_COMPLEVEL=1 ############################################################################################################### # Create Storage Directory for Backup Images ############################################################################################################### if [ ! -d $DEST ] then mkdir -p $DEST fi # VM 19: openSUSE 11.2 x86_64 Linux PV domU ############################################################################################################### # Add partition mappings in /dev/mapper kpartx -av /dev/virtualmachines/opensuse # Backup MBR dd if=/dev/virtualmachines/opensuse of=$DEST/virtualmachines-opensuse.mbr bs=512 count=1 # Backup Partition Geometry sfdisk -d /dev/virtualmachines/opensuse > $DEST/virtualmachines-opensuse.sfdisk # Backup Filesystems # partimage does not support ext4 filesystems #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save /dev/mapper/virtualmachines-opensuse2 \ #$DEST/virtualmachines-opensuse2.img fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES $DEST/virtualmachines-opensuse2.fsa \ /dev/mapper/virtualmachines-opensuse2 # Delete partition mappings in /dev/mapper kpartx -dv /dev/virtualmachines/opensuse # VM 16: Slackware64 13.0 amd64 HVM domU ############################################################################################################### # Add partition mappings in /dev/mapper kpartx -av /dev/virtualmachines/slackware64 # Backup MBR dd if=/dev/virtualmachines/slackware64 of=$DEST/virtualmachines-slackware64.mbr bs=512 count=1 # Backup Partition Geometry sfdisk -d /dev/virtualmachines/slackware64 > $DEST/virtualmachines-slackware64.sfdisk # Backup Filesystems # partimage does not support ext4 filesystems #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save /dev/mapper/virtualmachines-slackware64p1 \ #$DEST/virtualmachines-slackware64p1.img #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save /dev/mapper/virtualmachines-slackware64p2 \ #$DEST/virtualmachines-slackware64p2.img fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES $DEST/virtualmachines-slackware64p1.fsa \ /dev/mapper/virtualmachines-slackware64p1 fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES $DEST/virtualmachines-slackware64p2.fsa \ /dev/mapper/virtualmachines-slackware64p2 # Delete partition mappings in /dev/mapper kpartx -dv /dev/virtualmachines/slackware64 # VM 17: Ubuntu 9.10 Karmic Koala Linux HVM domU ############################################################################################################### # Add partition mappings in /dev/mapper kpartx -av /dev/virtualmachines/ubuntu910 # Backup MBR dd if=/dev/virtualmachines/ubuntu910 of=$DEST/virtualmachines-ubuntu910.mbr bs=512 count=1 # Backup Partition Geometry sfdisk -d /dev/virtualmachines/ubuntu910 > $DEST/virtualmachines-ubuntu910.sfdisk # Backup Filesystems # partimage does not support ext4 filesystems #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save /dev/mapper/virtualmachines-ubuntu910p1 \ #$DEST/virtualmachines-ubuntu910p1.img fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES $DEST/virtualmachines-ubuntu910p1.fsa \ /dev/mapper/virtualmachines-ubuntu910p1 # Delete partition mappings in /dev/mapper kpartx -dv /dev/virtualmachines/ubuntu910 # EOF </SCRIPT> I have attached the latest harddisk backup and restore scripts in this email at the time of this writing. -- Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical Engineering) Alma Maters: (1) Singapore Polytechnic (2) National University of Singapore My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com My Secondary Blog: http://enmingteo.wordpress.com My Youtube videos: http://www.youtube.com/user/enmingteo Email: space.time.universe@gmail.com Mobile Phone (Starhub Prepaid): +65-8369-2618 Street: Bedok Reservoir Road Country: Singapore _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mr. Teo En Ming (Zhang Enming)
2009-Nov-15 01:07 UTC
[Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
On Sun, Nov 15, 2009 at 9:02 AM, Mr. Teo En Ming (Zhang Enming) < space.time.universe@gmail.com> wrote:> > > On Sat, Nov 14, 2009 at 3:34 AM, Mr. Teo En Ming (Zhang Enming) < > space.time.universe@gmail.com> wrote: > > > > > > On Fri, Nov 13, 2009 at 9:56 PM, Mr. Teo En Ming (Zhang Enming) > > <space.time.universe@gmail.com> wrote: > >> > >> There are 2 file attachments in this email. One is for harddisk cloning, > >> the other is for data restoration. If there are any bugs with the > scripts, > >> please let me know. > >> > >> Thank you. > >> > >> -- > >> Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical > >> Engineering) > >> Alma Maters: > >> (1) Singapore Polytechnic > >> (2) National University of Singapore > >> My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com > >> My Secondary Blog: http://enmingteo.wordpress.com > >> My Youtube videos: http://www.youtube.com/user/enmingteo > >> Email: space.time.universe@gmail.com > >> Mobile Phone (Starhub Prepaid): +65-8369-2618 > >> Street: Bedok Reservoir Road > >> Country: Singapore > > > > Attached updates to backup and restore scripts. > > > > > > > > Hi All, > > I have tested my self-written harddisk backup/cloning script by performing > a backup of my Western Digital 500 GB SATA harddisk to my 160 GB Hitachi > harddisk. > > The backup operation is successful except for 3 virtual machines. The > backup of these 3 virtual machines fail because partimage does not support > ext4 filesystems. I have performed a backup of these 3 VMs *again* using the > following modified script to handle ext4 filesystems by using fsarchiver: > > <SCRIPT> > #!/bin/sh > > > ############################################################################################################### > # Declare Variables > > ############################################################################################################### > > HARDDISK=/dev/sda > DEST=/media/hitachi/redo > FSA_COMPLEVEL=3 > PROCESSOR_CORES=2 > GZIP_COMPLEVEL=fast > PARTIMAGE_COMPLEVEL=1 > > > ############################################################################################################### > # Create Storage Directory for Backup Images > > ############################################################################################################### > > if [ ! -d $DEST ] > then > mkdir -p $DEST > fi > > # VM 19: openSUSE 11.2 x86_64 Linux PV domU > > ############################################################################################################### > > # Add partition mappings in /dev/mapper > kpartx -av /dev/virtualmachines/opensuse > > # Backup MBR > dd if=/dev/virtualmachines/opensuse of=$DEST/virtualmachines-opensuse.mbr > bs=512 count=1 > > # Backup Partition Geometry > sfdisk -d /dev/virtualmachines/opensuse > > $DEST/virtualmachines-opensuse.sfdisk > > # Backup Filesystems > # partimage does not support ext4 filesystems > #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save > /dev/mapper/virtualmachines-opensuse2 \ > #$DEST/virtualmachines-opensuse2.img > > fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES > $DEST/virtualmachines-opensuse2.fsa \ > /dev/mapper/virtualmachines-opensuse2 > > # Delete partition mappings in /dev/mapper > kpartx -dv /dev/virtualmachines/opensuse > > # VM 16: Slackware64 13.0 amd64 HVM domU > > ############################################################################################################### > > # Add partition mappings in /dev/mapper > kpartx -av /dev/virtualmachines/slackware64 > > # Backup MBR > dd if=/dev/virtualmachines/slackware64 > of=$DEST/virtualmachines-slackware64.mbr bs=512 count=1 > > # Backup Partition Geometry > sfdisk -d /dev/virtualmachines/slackware64 > > $DEST/virtualmachines-slackware64.sfdisk > > # Backup Filesystems > # partimage does not support ext4 filesystems > #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save > /dev/mapper/virtualmachines-slackware64p1 \ > #$DEST/virtualmachines-slackware64p1.img > > #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save > /dev/mapper/virtualmachines-slackware64p2 \ > #$DEST/virtualmachines-slackware64p2.img > > fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES > $DEST/virtualmachines-slackware64p1.fsa \ > /dev/mapper/virtualmachines-slackware64p1 > > fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES > $DEST/virtualmachines-slackware64p2.fsa \ > /dev/mapper/virtualmachines-slackware64p2 > > # Delete partition mappings in /dev/mapper > kpartx -dv /dev/virtualmachines/slackware64 > > # VM 17: Ubuntu 9.10 Karmic Koala Linux HVM domU > > ############################################################################################################### > > # Add partition mappings in /dev/mapper > kpartx -av /dev/virtualmachines/ubuntu910 > > # Backup MBR > dd if=/dev/virtualmachines/ubuntu910 of=$DEST/virtualmachines-ubuntu910.mbr > bs=512 count=1 > > # Backup Partition Geometry > sfdisk -d /dev/virtualmachines/ubuntu910 > > $DEST/virtualmachines-ubuntu910.sfdisk > > # Backup Filesystems > # partimage does not support ext4 filesystems > #partimage -d -M -b -z$PARTIMAGE_COMPLEVEL save > /dev/mapper/virtualmachines-ubuntu910p1 \ > #$DEST/virtualmachines-ubuntu910p1.img > > fsarchiver savefs -v -z $FSA_COMPLEVEL -j $PROCESSOR_CORES > $DEST/virtualmachines-ubuntu910p1.fsa \ > /dev/mapper/virtualmachines-ubuntu910p1 > > # Delete partition mappings in /dev/mapper > kpartx -dv /dev/virtualmachines/ubuntu910 > > # EOF > </SCRIPT> > > I have attached the latest harddisk backup and restore scripts in this > email at the time of this writing. > > > -- > Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical > Engineering) > Alma Maters: > (1) Singapore Polytechnic > (2) National University of Singapore > My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com > My Secondary Blog: http://enmingteo.wordpress.com > My Youtube videos: http://www.youtube.com/user/enmingteo > Email: space.time.universe@gmail.com > Mobile Phone (Starhub Prepaid): +65-8369-2618 > Street: Bedok Reservoir Road > Country: Singapore > >Forgot to mention that the duration of harddisk cloning has been shortened from 7 hours to 4.5 hours. The filesize of the backup image set has been reduced from 300 GB to 126 GB, which is a substantial reduction of more than 50%. -- Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical Engineering) Alma Maters: (1) Singapore Polytechnic (2) National University of Singapore My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com My Secondary Blog: http://enmingteo.wordpress.com My Youtube videos: http://www.youtube.com/user/enmingteo Email: space.time.universe@gmail.com Mobile Phone (Starhub Prepaid): +65-8369-2618 Street: Bedok Reservoir Road Country: Singapore _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Steven Anderson
2009-Nov-15 12:50 UTC
[Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
I was trying to understand why one wouldnt use LV snapshots take a snapshot backup snapshot dd snapshot to another LV for cloning hikenboot _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra
2009-Nov-16 01:04 UTC
Re: [Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
Steven Anderson wrote:> I was trying to understand why one wouldnt use LV snapshotsbecause it gives you a backup of a ''dirty'' filesystem. that is, a snapshot when it''s mounted, and without any guarantees about all processes having flushed important data to disk. recovering from such a backup is to the system just like recovering from a ''pull the plug'' event. In most cases an fsck is enough and the (little) lost data is within reason for a catastrophic failure. In other cases (usually when a busy database is involved), it''s just not OK, you could recover data that is totally inconsistent, maybe needing a complex analysis to bring it back to a reasonable state. IOW, it''s better to use a ''normal'' backup system, and be sure to get cooperation from any DB system. -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
We are running xen on more 50 servers and recently updated xen from 3.1.3 to 3.4.1. Gitco yum repository has been used for updating. The problem is that more than 50% of servers failed to boot after updating xen. We followed exactly same procedure to update xen version on every server but some servers were successfully updated and others failed (about half and half). All xen and system log files did not show any error message. Physical Server Model: IBM x3650 Operating System: CentOS 5.3 Raid Controller: IBM ServeRAID-MR10i SAS/SATA Controller Steps we followed for updating xen 1. edit yum repository (add gitco repository) 2. remove current virtualization package (yum groupremove) 3. install updated xen version (yum groupinstall) 4. edit menu.list 5. reboot It is hard to distinguish that it is caused from hardware, Linux Kernel, or Xen. Would you guys give me any suggestions that we should check to find out which caused this problem? Thank you so much in advance. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Nov 16, 2009 at 03:40:02PM +0900, Dongkyu Lee wrote:> We are running xen on more 50 servers and recently updated xen from > 3.1.3 to 3.4.1. > Gitco yum repository has been used for updating. The problem is that > more than 50% of servers failed to boot after updating xen. > We followed exactly same procedure to update xen version on every server > but some servers were successfully > updated and others failed (about half and half). All xen and system log > files did not show any error message. > > Physical Server Model: IBM x3650 > Operating System: CentOS 5.3 > Raid Controller: IBM ServeRAID-MR10i SAS/SATA Controller > > Steps we followed for updating xen > 1. edit yum repository (add gitco repository) > 2. remove current virtualization package (yum groupremove) > 3. install updated xen version (yum groupinstall) > 4. edit menu.list > 5. reboot > > It is hard to distinguish that it is caused from hardware, Linux Kernel, > or Xen. > Would you guys give me any suggestions that we should check to find out > which caused this problem? >Set up a serial console and check/log what''s happening.. -- Pasi> Thank you so much in advance. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mr. Teo En Ming (Zhang Enming)
2009-Nov-18 19:45 UTC
Re: [Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
On Mon, Nov 16, 2009 at 9:04 AM, Javier Guerra <javier@guerrag.com> wrote:> Steven Anderson wrote: >> I was trying to understand why one wouldnt use LV snapshots > > because it gives you a backup of a ''dirty'' filesystem. that is, a snapshot when it''s mounted, and without any guarantees about all processes having flushed important data to disk. > > recovering from such a backup is to the system just like recovering from a ''pull the plug'' event. In most cases an fsck is enough and the (little) lost data is within reason for a catastrophic failure. In other cases (usually when a busy database is involved), it''s just not OK, you could recover data that is totally inconsistent, maybe needing a complex analysis to bring it back to a reasonable state. > > IOW, it''s better to use a ''normal'' backup system, and be sure to get cooperation from any DB system. > > -- > Javier > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >Hi All, I have tested both the backup and restore scripts by doing an actual backup and restore operation on my computer. For your convenience, I have attached the final backup and restore scripts in this email. -- Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical Engineering) Alma Maters: (1) Singapore Polytechnic (2) National University of Singapore My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com My Secondary Blog: http://enmingteo.wordpress.com My Youtube videos: http://www.youtube.com/user/enmingteo Email: space.time.universe@gmail.com Mobile Phone (Starhub Prepaid): +65-8369-2618 Street: Bedok Reservoir Road Country: Singapore _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mr. Teo En Ming (Zhang Enming)
2009-Nov-20 16:25 UTC
Re: [Xen-users] Re: Data Backup and Restore Scripts for your Xen Servers
On Thu, Nov 19, 2009 at 3:45 AM, Mr. Teo En Ming (Zhang Enming) < space.time.universe@gmail.com> wrote:> On Mon, Nov 16, 2009 at 9:04 AM, Javier Guerra <javier@guerrag.com> wrote: > > Steven Anderson wrote: > >> I was trying to understand why one wouldnt use LV snapshots > > > > because it gives you a backup of a ''dirty'' filesystem. that is, a > snapshot when it''s mounted, and without any guarantees about all processes > having flushed important data to disk. > > > > recovering from such a backup is to the system just like recovering from > a ''pull the plug'' event. In most cases an fsck is enough and the (little) > lost data is within reason for a catastrophic failure. In other cases > (usually when a busy database is involved), it''s just not OK, you could > recover data that is totally inconsistent, maybe needing a complex analysis > to bring it back to a reasonable state. > > > > IOW, it''s better to use a ''normal'' backup system, and be sure to get > cooperation from any DB system. > > > > -- > > Javier > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > Hi All, > > I have tested both the backup and restore scripts by doing an actual > backup and restore operation on my computer. > > For your convenience, I have attached the final backup and restore > scripts in this email. > >Hi All, I have performed an extensive testing of the restore of the data backup image set using my backup and restore scripts. Two virtual machines, namely, openSUSE 11.2 PV domU, and Slackware64 13.0 HVM domU, did not restore properly. Please refer to my latest updated restore script for the fixes to these problems. Attached latest updated restore script. -- Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical Engineering) Alma Maters: (1) Singapore Polytechnic (2) National University of Singapore My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com My Secondary Blog: http://enmingteo.wordpress.com My Youtube videos: http://www.youtube.com/user/enmingteo Email: space.time.universe@gmail.com Mobile Phone (Starhub Prepaid): +65-8369-2618 Street: Bedok Reservoir Road Country: Singapore _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users