Xen devel - Sep 2004 - Custom build the Xen0 kernel && miscellany

Hi everyone,

I''m new to the list. I''m a systems admin at a multimedia
company in New
Zealand. I''m currently constructing a small cluster of dual Xeon
machines
which will host web sites in Xen VMs.

I would like to stick with the ''best practices'' of the Xen
project in terms of
managing the kernel source trees and builds, but I find that the default 
build contains a bit more than I would like, and not everything I would need. 
For example, I our cluster machines have RageXL graphics adapters, and 
therefore I would like to compile in framebuffer support so I can run X. Not 
critical, no, but I prefer it especially during my investigations.

I am using the Xen 2.0 bk builds, and the host OS is Debian Sarge. Both seem 
to be close to stable releases and so hopefully my timing is going to be 
fairly good!

Can I tell the default build system to use a modified .config file for the 
kernel build? From the looks of the makefile it appears that much more than 
patching and compiling is going on with the kernel, so if I do a make 
menuconfig and make my changes, am I able to install the kernel in the normal 
way?

The support I''m adding is for SMP, hypterthreading, big memory, and ATI
framebuffer support. I''d also like to pull out various bits and pieces
--
mostly drivers for hardware I don''t have, as this would speed up the
build
process.

Thank you for your time. I''ve been watching Xen with some interest for
a while
now and I''m thrilled that my employer has given me the go ahead to put 
together this cluster. I would like to feed the results back to the Xen 
project somewhere, especially where it pertains to performance of virtual web 
and database servers. It seems that the wiki has gone west (been dead for a 
while now), so perhaps it would be appropriate for me to submit stuff back to 
this list for archival purposes.

(Speaking of the email archive, could it be modified for searching? I''m
sure
my questions have been asked before)

One final question (and this is bound to be unpopular). I would like to have 
some idea as to how many virtual web servers I might be able to run on dual 
Xeon 2.8GHz machines with 4-12GB RAM. Our sites are typically LAMP, with 
medium sized databases. On average, our current web servers are 900MHz PIII 
machines that don''t often get pushed very hard. Perhaps just some
anecdotal
accounts would satisfy my curiosity. I realize I''m not being too
practical
here -- my apologies! ;o).

Thanks all for your time. 

Regards,
Paul Dorman
Systems Administrator
CWA New Media Ltd.


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

The Makefile pulls your config files from install/boot/config-*.
It is fine to modify the configurations as neceesary. Note that SMP
isn''t available for guest OSes -- Xen currently only supports
uniprocessor guests, even though it itself is SMP capable.

 -- Keir
> Hi everyone,
> 
> I''m new to the list. I''m a systems admin at a multimedia
company in New
> Zealand. I''m currently constructing a small cluster of dual Xeon
machines
> which will host web sites in Xen VMs.
> 
> I would like to stick with the ''best practices'' of the
Xen project in terms of
> managing the kernel source trees and builds, but I find that the default 
> build contains a bit more than I would like, and not everything I would
need.
> For example, I our cluster machines have RageXL graphics adapters, and 
> therefore I would like to compile in framebuffer support so I can run X.
Not
> critical, no, but I prefer it especially during my investigations.
> 
> I am using the Xen 2.0 bk builds, and the host OS is Debian Sarge. Both
seem
> to be close to stable releases and so hopefully my timing is going to be 
> fairly good!
> 
> Can I tell the default build system to use a modified .config file for the 
> kernel build? From the looks of the makefile it appears that much more than
> patching and compiling is going on with the kernel, so if I do a make 
> menuconfig and make my changes, am I able to install the kernel in the
normal
> way?
> 
> The support I''m adding is for SMP, hypterthreading, big memory,
and ATI
> framebuffer support. I''d also like to pull out various bits and
pieces --
> mostly drivers for hardware I don''t have, as this would speed up
the build
> process.
> 
> Thank you for your time. I''ve been watching Xen with some interest
for a while
> now and I''m thrilled that my employer has given me the go ahead to
put
> together this cluster. I would like to feed the results back to the Xen 
> project somewhere, especially where it pertains to performance of virtual
web
> and database servers. It seems that the wiki has gone west (been dead for a
> while now), so perhaps it would be appropriate for me to submit stuff back
to
> this list for archival purposes.
> 
> (Speaking of the email archive, could it be modified for searching?
I''m sure
> my questions have been asked before)
> 
> One final question (and this is bound to be unpopular). I would like to
have
> some idea as to how many virtual web servers I might be able to run on dual
> Xeon 2.8GHz machines with 4-12GB RAM. Our sites are typically LAMP, with 
> medium sized databases. On average, our current web servers are 900MHz PIII
> machines that don''t often get pushed very hard. Perhaps just some
anecdotal
> accounts would satisfy my curiosity. I realize I''m not being too
practical
> here -- my apologies! ;o).
> 
> Thanks all for your time. 
> 
> Regards,
> Paul Dorman
> Systems Administrator
> CWA New Media Ltd.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> I''m new to the list.
Welcome on board :-)  That''s another country to add to the list!
> Can I tell the default build system to use a modified .config file for the
> kernel build? From the looks of the makefile it appears that much more than
> patching and compiling is going on with the kernel, so if I do a make
> menuconfig and make my changes, am I able to install the kernel in the
> normal way?
As Keir said, you should put your XenLinux config files into install/boot/ and 
they will be used when you rebuild.  You could even stick your old Linux 
configs into there (with the appropriate names, to match the Xen build 
system''s expectations) and they should work (though the Linux build
system
might have to confirm a few Xen-specific options interactively).
> The support I''m adding is for SMP, hypterthreading, big memory,
and ATI
> framebuffer support. I''d also like to pull out various bits and
pieces --
> mostly drivers for hardware I don''t have, as this would speed up
the build
> process.
UP guests (that includes dom0) only at the moment.  You''ll use both
CPUs by
running (multiple) uniprocessor domains on each CLU. SMP guests will be a 
feature of a future version (perhaps a 3.0 release???).

This also means guests won''t see the hyperthreading but instead you can
run a
domain on each hyperthread - you may want to see if this is better or worse 
for performance than disabling HT - it''ll depend somewhat in your
workload.
> Thank you for your time. I''ve been watching Xen with some interest
for a
> while now and I''m thrilled that my employer has given me the go
ahead to
> put together this cluster. I would like to feed the results back to the Xen
> project somewhere, especially where it pertains to performance of virtual
> web and database servers. It seems that the wiki has gone west (been dead
> for a while now), so perhaps it would be appropriate for me to submit stuff
> back to this list for archival purposes.
I update the Wiki every so often but it doesn''t see that much activity
at the
moment (maybe more after the 2.0 release...).  Contributions to the user 
guide are useful, too ;-)  You may like to look at it, although it''s
not
fully up to date at the moment.
> One final question (and this is bound to be unpopular). I would like to
> have some idea as to how many virtual web servers I might be able to run on
> dual Xeon 2.8GHz machines with 4-12GB RAM. Our sites are typically LAMP,
> with medium sized databases. On average, our current web servers are 900MHz
> PIII machines that don''t often get pushed very hard. Perhaps just
some
> anecdotal accounts would satisfy my curiosity. I realize I''m not
being too
> practical here -- my apologies! ;o).
Not sure about actual numbers, I''m afraid.  Since Xen has a low
overhead you
should get a large proportion of the available power for running virtual 
machines...  You''ll also get some "statistical multiplexing"
benefit, since
slack time from idle domains can be used by active ones, rather than wasted 
(as in the case of a dedicated machine).  The scheduler should ensure fair 
sharing when multiple domains are busy.

Cheers,
Mark

-----
September 19th - Talk Like A Pirate Day
http://www.talklikeapirate.com


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Personally I found it a bit tough to manage configs with the curent
kernel building phase of the xen sources. As a result I wrote the
beginings of a kernel builder for xenlinux images and modules.

If youi hit http://www.terrabox.com/debian/binary you shouyld see a
package called xenlinuxbuilder. It''s a simplified method for specificly
building xenlinux kernels based on the installed version of
xen-kernel-packages. To build a specific configuration of a kernel I do:
debian/rules clean menuconfig tarball PRIV=[yes|no]
LIN_VER=[2.4.27|2.6.8] EXTRA=[whatever extraversion label]

note: you MUST specify clean prior to tarball curently IF you apply any
3rd party patches to the source. It doens''t handle detecting extra
patches, yet. My goal is to create a mak-kpkg like system specificly for
debian initially, later for general use. It will save your individual
configs into configs/ for reuse after you have fully configured the
individual kernel.

 It will produce a single tarball for /boot and /lib/modules for
privileged kernels, and separate /boot and /lib/modules tarballs for
unpriv kernels. This has made it a bit easier for myself to generate and
maintain separate and easilly built kernels for my various xenlinux
images.

If you are managing more than just 1 or 2 kernel versions then this may
be a smoother method for compiling and packaging since *eventually* it
will produce .deb packages that will verify which xen version it is that
you are running on the underlying xen kernel.

I have also packaged up the xen distro as pre-compiled binaries for xen
on Debian Sarge. I''m close to having a nightly export run that will bk
update, debian patch, compile, and package them automagically. :) I''m
willing to share my custom makefile for doing this if you want to
maintain a local repository of xen packages.

I stil have a good bit of work to do before I can send the debian/* tree
upstream for inclusion into xen proper if the gang will accept them. :)

I also need ideas on how best to tie the debs into the netbsd vhost disk
image so that I can build netbsd server images dynamicly. Right now i''m
taking a look at the  xenoboot software to see if I can''t package it up
for debian to keep from reproducing a lot of the work that has already
been done.


as far as number of images that I run per machine, a friend of mine runs
~20 images on a 4GB dual p4 machine at his offices. I run 6 athlon xp
2000+ machines, each with 1GB ram that run from 2 to 10 images each. So
far not a single beta client has complained of any kind of CPU or
performance issues. So far it appears that memory is more of an issue
here than anything else due to the amazingly low overhead of xen when
using NAS or SAN based storage. :)

As far as selling vhosts using xenlinux kernels, I wouldn''t make it an
official product just yet. It''s stable for local disks, and in general,
however I have run into various issues that have poped up related to
iSCSI and other momentary glitches as I do semi-daily updates of the
cluster. I would let the xen team get to an official 2.0 release before
truly pushing vhosts out as a paid-service (unless you get a few clients
to sign up for them with the understanding that they aren''t guaranteed
stable yet).

If you go lok at http://xen.terrabox.com there is a page for companies
to list themselves on that offer xen based virtual hosts that I started.

Hope this helps from the business perspective for you. :)

Brian

On Mon, 2004-09-20 at 07:16, Mark A. Williamson wrote:
> > I''m new to the list.
> 
> Welcome on board :-)  That''s another country to add to the list!
> 
> > Can I tell the default build system to use a modified .config file for
the
> > kernel build? From the looks of the makefile it appears that much more
than
> > patching and compiling is going on with the kernel, so if I do a make
> > menuconfig and make my changes, am I able to install the kernel in the
> > normal way?
> 
> As Keir said, you should put your XenLinux config files into install/boot/
and
> they will be used when you rebuild.  You could even stick your old Linux 
> configs into there (with the appropriate names, to match the Xen build 
> system''s expectations) and they should work (though the Linux
build system
> might have to confirm a few Xen-specific options interactively).
> 
> > The support I''m adding is for SMP, hypterthreading, big
memory, and ATI
> > framebuffer support. I''d also like to pull out various bits
and pieces --
> > mostly drivers for hardware I don''t have, as this would speed
up the build
> > process.
> 
> UP guests (that includes dom0) only at the moment.  You''ll use
both CPUs by
> running (multiple) uniprocessor domains on each CLU. SMP guests will be a 
> feature of a future version (perhaps a 3.0 release???).
> 
> This also means guests won''t see the hyperthreading but instead
you can run a
> domain on each hyperthread - you may want to see if this is better or worse
> for performance than disabling HT - it''ll depend somewhat in your
workload.
> 
> > Thank you for your time. I''ve been watching Xen with some
interest for a
> > while now and I''m thrilled that my employer has given me the
go ahead to
> > put together this cluster. I would like to feed the results back to
the Xen
> > project somewhere, especially where it pertains to performance of
virtual
> > web and database servers. It seems that the wiki has gone west (been
dead
> > for a while now), so perhaps it would be appropriate for me to submit
stuff
> > back to this list for archival purposes.
> 
> I update the Wiki every so often but it doesn''t see that much
activity at the
> moment (maybe more after the 2.0 release...).  Contributions to the user 
> guide are useful, too ;-)  You may like to look at it, although
it''s not
> fully up to date at the moment.
> 
> > One final question (and this is bound to be unpopular). I would like
to
> > have some idea as to how many virtual web servers I might be able to
run on
> > dual Xeon 2.8GHz machines with 4-12GB RAM. Our sites are typically
LAMP,
> > with medium sized databases. On average, our current web servers are
900MHz
> > PIII machines that don''t often get pushed very hard. Perhaps
just some
> > anecdotal accounts would satisfy my curiosity. I realize I''m
not being too
> > practical here -- my apologies! ;o).
> 
> Not sure about actual numbers, I''m afraid.  Since Xen has a low
overhead you
> should get a large proportion of the available power for running virtual 
> machines...  You''ll also get some "statistical
multiplexing" benefit, since
> slack time from idle domains can be used by active ones, rather than wasted
> (as in the case of a dedicated machine).  The scheduler should ensure fair 
> sharing when multiple domains are busy.
> 
> Cheers,
> Mark
> 
> -----
> September 19th - Talk Like A Pirate Day
> http://www.talklikeapirate.com
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
> Project Admins to receive an Apple iPod Mini FREE for your judgement on
> who ports your project to Linux PPC the best. Sponsored by IBM.
> Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Thanks for your reply Brian,

looks like you are doing some great work with Debianizing Xen (or Xenifying 
Debian!). I thank you from the bottom of my heart :o)

Actually, I am hoping to replace our current LAMP server infrastructure with a 
Xen cluster of around 8 machines. My ambition is to get to the point where I 
can run them stably for 24hrs at a time with no corruption to the actual 
images, at which point our developers can start working on the VMs. Hopefully 
the releases Sarge and Xen 2.0 will coincide with this!

I''d be very interested in helping with any Debian related stuff you are
doing
(so please keep on doing it!), though it will take time for me to get up to 
speed with everything.

Could I ask the list members if they have had any issues with the stability of 
the VM images themselves? That is, if I tell our developers they can start 
working on VMs, will their work be safe (even if VM uptimes aren''t that
great)? Naturally I''ll have backup systems in place, so I''m
kind of wondering
where the potential dangers lie. Initially by the way the servers will only 
be hosting a few sites, perhaps four per server, so the work load (and disk 
IO, and network IO..) will be pretty low.

Thanks for your contributions and time!

Regards,
Paul

On Tuesday 21 September 2004 03:44 am, Brian Wolfe
wrote:
> Personally I found it a bit tough to manage configs with the curent
> kernel building phase of the xen sources. As a result I wrote the
> beginings of a kernel builder for xenlinux images and modules.
>
> If youi hit http://www.terrabox.com/debian/binary you shouyld see a
> package called xenlinuxbuilder. It''s a simplified method for
specificly
> building xenlinux kernels based on the installed version of
> xen-kernel-packages. To build a specific configuration of a kernel I do:
> debian/rules clean menuconfig tarball PRIV=[yes|no]
> LIN_VER=[2.4.27|2.6.8] EXTRA=[whatever extraversion label]
>
> note: you MUST specify clean prior to tarball curently IF you apply any
> 3rd party patches to the source. It doens''t handle detecting extra
> patches, yet. My goal is to create a mak-kpkg like system specificly for
> debian initially, later for general use. It will save your individual
> configs into configs/ for reuse after you have fully configured the
> individual kernel.
>
>  It will produce a single tarball for /boot and /lib/modules for
> privileged kernels, and separate /boot and /lib/modules tarballs for
> unpriv kernels. This has made it a bit easier for myself to generate and
> maintain separate and easilly built kernels for my various xenlinux
> images.
>
> If you are managing more than just 1 or 2 kernel versions then this may
> be a smoother method for compiling and packaging since *eventually* it
> will produce .deb packages that will verify which xen version it is that
> you are running on the underlying xen kernel.
>
> I have also packaged up the xen distro as pre-compiled binaries for xen
> on Debian Sarge. I''m close to having a nightly export run that
will bk
> update, debian patch, compile, and package them automagically. :)
I''m
> willing to share my custom makefile for doing this if you want to
> maintain a local repository of xen packages.
>
> I stil have a good bit of work to do before I can send the debian/* tree
> upstream for inclusion into xen proper if the gang will accept them. :)
>
> I also need ideas on how best to tie the debs into the netbsd vhost disk
> image so that I can build netbsd server images dynamicly. Right now
i''m
> taking a look at the  xenoboot software to see if I can''t package
it up
> for debian to keep from reproducing a lot of the work that has already
> been done.
>
>
> as far as number of images that I run per machine, a friend of mine runs
> ~20 images on a 4GB dual p4 machine at his offices. I run 6 athlon xp
> 2000+ machines, each with 1GB ram that run from 2 to 10 images each. So
> far not a single beta client has complained of any kind of CPU or
> performance issues. So far it appears that memory is more of an issue
> here than anything else due to the amazingly low overhead of xen when
> using NAS or SAN based storage. :)
>
> As far as selling vhosts using xenlinux kernels, I wouldn''t make
it an
> official product just yet. It''s stable for local disks, and in
general,
> however I have run into various issues that have poped up related to
> iSCSI and other momentary glitches as I do semi-daily updates of the
> cluster. I would let the xen team get to an official 2.0 release before
> truly pushing vhosts out as a paid-service (unless you get a few clients
> to sign up for them with the understanding that they aren''t
guaranteed
> stable yet).
>
> If you go lok at http://xen.terrabox.com there is a page for companies
> to list themselves on that offer xen based virtual hosts that I started.
>
> Hope this helps from the business perspective for you. :)
>
> Brian

-------------------------------------------------------
This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170
Project Admins to receive an Apple iPod Mini FREE for your judgement on
who ports your project to Linux PPC the best. Sponsored by IBM.
Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel