Tom Hibbert
2005-Jan-13 03:08 UTC
[Xen-devel] Debian Sarge Root Raid + LVM + XEN install guide (LONG)
Hello fellow xenophiles and happy new year!
I''ve documented the install procedure for a prototype server here since
I found no similar document
Anywhere on the net. It''s a Sarge-based Domain0 on linux root raid from
scratch, using LVM to store
the data for the domU mail server and its mailstore. I humbly submit my
notes in the hope that they are useful to some weary traveller.
Have fun!
Debian Sarge XEN dom0 with Linux Root Raid and LVM
Hardware: P4 3.2ghz LG775
Asus P5GD1-VM
1gb DDR400 DRAM
2x80gb Seagate SATA disks
Reasons for using software raid (over Intel ICH raid or more expensive
SCSI raid)
1. Speed
Bonnie++ shows Linux Software Raid is MUCH faster than ICH5
(at least under Linux)
2. Reliability
I have observed that frequent disk access with small files
has destroyed ICH5 raid arrays in the past (at least under
Linux)
3. Recovery
I had a bad experience with the death of an Adaptec 3200S
controller not long ago. The array
was nonrecoverable because a replacement card could not be
sourced in time. Additionally the
firmware revision for the 3200s was unknown. (Recovery from
controller death if even possible requires the same firmware
revision as the original card, since that was not known
we would have had to guess which takes time and time is money when you
have a dead server)
4. Price
Reduce cost of hardware to the client because we arent using
expensive raid controllers
5. Prevalence
It is much easier to source standard disks than it is to
source SCSI disks (in the case
of using SCSI raid controllers). It is also much easier to
source a standard SATA controller than it is to source a RAID
controller
Reasons for using XEN
1. Recovery
Putting all network services inside XEN virtual machines that
can be backed up makes disaster recovery a non-brainer
2. Better utilisation of hardware
Stacking virtual machines allows more efficient use of
hardware (cost effectiveness)
3. It''s just cooler :)
Methodology
1. Setting up the hardware - setting SATA to compatible mode
2. Boot off Feather Linux USB key
3. Partition primary drive
4. Install base system
5. Chroot into base system
6. Install C/C++ development packages
7. Install XEN packages
8. Configure/build/install XEN Dom0 kernel
9. Install GRUB
10. Reboot to base system and set SATA to enhanced mode
11. Migrate system into RAID1 and test
12. Configure/build/install XEN DomU kernel
13. Configure LVM
14. Create DomU environment
* 15. Install services into DomU
16. Configure XEN to boot DomU automatically
* 17. Testing
* 18. Deployment
* Not covered by this document
1. Setting up the hardware
-----------------------
Standard stuff here. Set the mode for SATA to Compatible so that
Feather''s kernel was able to access the hard disks.
2. Boot off Feather Linux USB key
------------------------------
Feather is fantastic because it allows one to setup a Debian system
without having to boot from the now heavily outdated Woody install CD.
It supports more hardware and also allows easy installation to a system
without a CDRom drive in a build network without an ''evil''
segment (PXE
boot). It also makes a convenient rescue platform.
http://featherlinux.berlios.de
3. Partition primary drive
-----------------------
Feather Linux does not properly support the ICHx and it doesnt have the
administration tools for making raid arrays. Therefore the setup method
we will use is to build the base system on a single disk and then
migrate it into RAID1. Trust me, this is much easier than it sounds!
I partitioned the primary drive as follows
Device Boot Start End Blocks Id System
/dev/hda1 1 3 24066 fd Linux raid
autodetect
/dev/hda2 4 501 4000185 fd Linux raid
autodetect
/dev/hda3 502 9605 73127880 fd Linux raid
autodetect
/dev/hda4 9606 9729 996030 fd Linux raid
autodetect
using hda2 for root and hda1 for boot with swap on hda4. hda3 is not
used yet.
Format and mount up the drive to /target:
# mkdir /target
# mkfs.ext3 /dev/hda1
# mkfs.ext3 /dev/hda2
# mount /dev/hda2 /target
# mkdir /target/boot
# mount /dev/hda1 /target/boot
4. Install the base system
----------------------
Set up Feather with APT and debootstrap:
# dpkg-get
# apt-get install debootstrap
Install the base system
# debootstrap sarge /target
Perform basic configuration
# vi /target/etc/fstab
/dev/sda2 / ext3 defaults 0 1
/dev/sda1 /boot ext3 defaults 0 2
proc /proc proc defaults 0 0
You may be asking why am I putting sda here? The reason is because once
I set the ICH6 to use Enhanced Mode and reboot into the fresh 2.6.9 xen0
kernel with SATA support compiled the drives appear as SCSI devices. hda
will be enumerated as /dev/sda.
5. Chroot into base system
-----------------------
# umount /dev/hda1
# cd /target
# chroot .
# su -
# mount /dev/hda1 /boot
Unmounting and remounting boot is important for configuring GRUB later.
Some more configuration needs to be done at this point:
# rm /etc/resolv.conf
# rm /etc/hostname
# echo xen0-test > /etc/hostname
# echo nameserver 210.55.13.3 > /etc/resolv.conf
6. Install C/C++ packages
----------------------
# apt-setup
# apt-get update
# dselect update
# tasksel
(Select C/C++ development packages)
7. Install XEN packages
--------------------
Until Adam''s packages get released I am using some homebrew packages
descended from Brian''s original
work.
# mkdir xen
# cd xen
# apt-get install wget
# wget -r http://cryptocracy.hn.org/xen/
# cd cryptocracy.hn.org/xen
# dpkg -i *.deb
# apt-get -f install
8. Configure/build/install XEN dom0 kernel
---------------------------------------
Since this is the first time configuring XEN on this hardware I am
building the kernel from scratch.
When we get more of these servers I will install a prebuilt debianised
kernel on them.
# cd /usr/src/
# tar -jxvf ./kernel-source-2.6.9_2.6.9-3_all.deb
# cd kernel-source-2.6.9
# export ARCH=xen
# cp ~/xen/cryptocracy.hn.org/xen/config.xen0 .config
# make menuconfig
(Make changes as appropriate for this hardware)
# make
# make modules_install
# cp vmlinuz /boot/vmlinuz-2.6.9-dom0
9. Configure GRUB
--------------
# apt-get install grub
# grub-install
# update-grub
Now edit the grub menu.lst file and modify the kernel definition so it
looks like this:
title Xen 2.0.1 / Xenolinux 2.6.9
root (hd0,0)
kernel /xen.gz dom0_mem=131072
module /269-xen0 root=/dev/sda2 ro console=tty0
10. Reboot to base system and revert SATA configuration to Enhanced mode
--------------------------------------------------------------------
# reboot
Set the relevant option in the BIOS and we''re good to go.
11. Migrate to RAID1 and test
-------------------------
We''ve just built a complete Dom0 base system on the first disk. In
order
to migrate this into RAID1,
we will create a RAID array using the second disk only, duplicate the
data onto the second drive, reboot into it and then readd the first
drive to the array. Sounds complex, but it isnt. This is another
advantage of Linux RAID over conventional RAID: it is easy to migrate
from a single disk to a RAID configuration.
First we need to partition the second disk exactly like the first:
# sfdisk -d /dev/sda > ~/partitions.sda
Having this data backed up is an incredibly good idea. I experienced a
catastrophic faliure on
one server once by enabling DMA with a buggy OSB4 driver. The partition
table was destroyed. Using
the partition data backed up in the manner above i was able to restore
the partition to find
that my data (an important IMAP store) was still intact.
Duplicating the partition table (or restoring from backup) is simple:
# sfdisk /dev/sdb < ~/partitions.sda
That''s it. The two drives are now identically partitioned.
Now we need to initialise the RAID on the second disk without destroying
the data on the first.
# apt-get install mdadm raidtools2
Begin by creating the raidtab. My one looks like this:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sda1
failed-disk 0
device /dev/sdb1
raid-disk 1
... repeated for each partition. Marking the partitions on sda - our
source drive - as failed BEFORE
creating the raid array is very important as it prevents them from being
overwritten by mkraid.
Create the RAID disks now.
# for i in ''seq 0 3''; do mkraid /dev/md$i; done
Format and mount the root and boot partitions and initialise swap:
# mkfs.ext3 /dev/md0
# mkfs.ext3 /dev/md1
# mkswap /dev/md2
# mkdir /target
# mount /dev/md1 /target
# mkdir /target/boot
# mount /dev/md0 /target/boot
Copy the contents of our base system into the RAID we''ve just created:
# ls -1 / | grep -v proc | while read line ; do cp -afx /$line /target;
done
# cp -afx /boot/* /target/boot
Modify the target''s fstab and grub configuration as follows:
/target/etc/fstab now looks like this:
/dev/md1 / ext3 defaults 0 1
/dev/md0 /boot ext3 defaults 0 2
proc /proc proc defaults 0 0
/dev/md2 none swap sw 0 0
And change the kernel definition in /target/boot/menu.lst slightly:
module /269-xen0 root=/dev/md1 ro console=tty0
Umount /target/boot:
# umount /target/boot
Chroot into the target:
# cd /target
# chroot .
# su -
Remount boot and install grub:
# mount -a
# grub-install
# update-grub
# exit
# logout
We''re now ready to reboot into our new RAID!
# reboot
Most modern boards these days (at least the ASUS ones which is all I
use) have an option to select
the boot device. On the P4 and P5 series mainboards this is accessed
through F8. As your system is
booting hit F8 and choose the second drive. If your system does not
support this you can change the
boot order in the bios or if you prefer you can edit the GRUB options by
pressing ''e'' at the prompt.
Once the system has rebooted you should now be inside your RAID setup.
It''s time to import the first
drive into the array.
First edit the raidtab and mark sda as usable:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
persistent-superblock 1
chunk-size 8
device /dev/sda1
raid-disk 0
device /dev/sdb1
raid-disk 1
... etc. Now add the partitions on sda as members using raidhotadd:
# raidhotadd /dev/md0 /dev/sda1
Rinse and repeat for each partition, or use a tricky bash one liner :)
The mirror is now syncing each partition in sequence. You can check the
status of this process
by periodically cating /proc/mdstat.
Once each partition is synced your mirror is complete and you can
reboot, remove and shuffle drives
about to your hearts content, or at least until you''re satisfied that
the root raid is working
correctly.
12. Configure/build/install XEN domU kernel
There''s no point in building the domU kernel until you''re
ready to use
it. If I was using a prebuilt
kernel package I would have included the domU kernel so this step would
be avoided.
# cd /usr/src/kernel-source-2.6.9
# make clean
# export ARCH=xen
# cp ~/xen/cryptocracy.hn.org/xen/config.xenU .config
# make menuconfig
(Make changes as appropriate)
# make
# make modules_install
# cp vmlinuz /boot/vmlinuz-2.6.9-domU
13. Configure LVM
I use LVM (or devmapper) to store the domU VBDs, including their data.
This allows for easy resizing of
partitions/images as required by services.
# apt-get install lvm10 lvm2
Initialise the partition as a physical volume:
# pvcreate /dev/md3
Create a volume group for xen:
# vgcreate xen /dev/md3
14. Create domU environment
-----------------------
Create logical volumes for the service domU and its mailstore:
# lvcreate -L4096M -n mail xen
# lvcreate -L65000M -n store xen
Format and mount the domU VBD:
# mount.ext3 /dev/xen/mail
# mount /dev/xen/mail /target
Install the base system on the domU:
# export ARCH=i386
# apt-get install debootstrap
# debootstrap /target
Configure the target:
# cd /target
# chroot .
# su -
# rm /etc/hostname
# rm /etc/resolv.conf
# echo mail > /etc/hostname
# echo nameserver 210.55.13.3 > /etc/resolv.conf
# apt-setup
Edit /etc/fstab:
/dev/hda1 / ext3 errors=remount-ro 0 1
/dev/hdb1 /store reiserfs defaults 0 2
proc /proc proc defaults 0 0
Edit /etc/network/interfaces:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
# exit
# logout
Create the config file for the new domain
# cp /etc/xen/xmexample1 /etc/xen/mail
Edit the file and change the name and disk parameters:
name = mail
disk = [ ''phy:xen/mail,hda1,w'',
''phy:xen/store,hdb1,w'']
Unmount the target and format the store partition:
# umount /target
# apt-get install reiserfsprogs
# mkfs.reiserfs /dev/xen/store
Fire up your new xenU domain!
# /etc/init.d/xend start
# xm create -f /etc/xen/mail
# xm console mail
Have a play and to return to the xen0 hit ctrl-].
16. Configure xen to start up the domain automatically
--------------------------------------------------
# ln -s /etc/init.d/xend /etc/rc2.d/S20xen
# ln -s /etc/init.d/xendomains /etc/rc2.d/S21xendomains
# mv /etc/xen/main /etc/xen/auto
That''s it! :) Enjoy your fresh new server.
-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It''s fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
Ian Pratt
2005-Jan-13 09:10 UTC
RE: [Xen-devel] Debian Sarge Root Raid + LVM + XEN install guide (LONG)
> I''ve documented the install procedure for a prototype server > here since > I found no similar document > Anywhere on the net. It''s a Sarge-based Domain0 on linux root > raid from > scratch, using LVM to store > the data for the domU mail server and its mailstore. I humbly > submit my > notes in the hope that they are useful to some weary traveller.Thanks for the comprehensive document -- we''ll definitely find a home for it on the web site. Out of interest, any reason not to use debbootstrap for setting up the domU partitions rather than copying over evrything from dom0? (I normally use the latter technique too for my RH/FC installs, but it always feels slightly distasteful. I was hoping that we could come up with a procedure around debbootstrap and dpkg that was easy yet enabled individual tailoring of each install). Best, Ian ------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It''s fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Martin Maney
2005-Jan-13 13:15 UTC
Re: [Xen-devel] Debian Sarge Root Raid + LVM + XEN install guide (LONG)
On Thu, Jan 13, 2005 at 09:10:30AM -0000, Ian Pratt wrote:> always feels slightly distasteful. I was hoping that we could come up > with a procedure around debbootstrap and dpkg that was easy yet enabled > individual tailoring of each install).The drawback to debootstrap, IME, is just that it skips some basic setup that isn''t needed for a chroot, but has to be filled in for a standalone system image. Networking setup, apt sources, and other things I''m unable to recall in the pre-coffee state. I have some notes I made a few months ago when I was messing about with setting up xen domains, but I know they''re incomplete. More recently I''ve been too busy to get the project box set back up for further work towards a script to (mostly?) automate those things. As well, I keep thinking that someone must have done this before, if only I could find it. :-) -- And in the end, reality always tends to hit theory hard in the face when you least expect it. - Linus ------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It''s fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel