Xen devel - Jan 2006 - Kernel with very low memory requirement (~12MB) for domU

Hello,

We are currently working on a master/bachelor level project, producing a 
xen + knoppix bootable CD for router simulation. Our task is to provide 
students as ourselves with a portable router lab that will run on any 
inexpesive PC, enabling students to exercise Cisco CCNA/CCNP level 
studies without access to physical equipment.

The goal is run 10x domU with nothing but the quagga/zebra daemons, on a 
P3/450MHz computer with 256MB ram.

So far, the CD and it''s file systems is put together. We can boot it
and
run 6 routers in it''s own domU before we run out of memory. And
that''s
where we are now. We need to reduce the memory requirements for each 
domU, and possibly the dom0 later on.

Removing everything we''re not dependent of (in make menuconfig) has 
reduced the vmlinuz kernel to just over 800KB. Still, it is not possible 
to start a domU with less than memory=17 in the config file. Although, I 
do think this is possible. Some tests we did last year (Xen 2.0.6 
installed in regular desktop system) showed that it was possible to boot 
a domU with 2.6.11-xenU kernel with just memory=12 (I hope I remember 
this correct).

I''ve spent a lot of time reading Xen documentation, wiki and lists. 
Sometimes I find "evidence" for my theories, like this post:

http://article.gmane.org/gmane.comp.emulators.xen.devel/11459

...where this config is present:

#-------configuration xmsec1-------------------------
kernel = "/boot/vmlinuz-2.6.11-xenU"
ramdisk="/boot/U1_ramdisk.img"
#security reference identifier
ssidref= 0x00010001
memory = 10
name = "xmsec1"
cpu = -1   # leave to Xen to pick
# Number of network interfaces. Default is 1.
nics=1
dhcp="dhcp"
#-----------------------------------------------------

As you can see, Reiner uses 10M for the each domU. Further down his 
post, it seems he is even running on less than that. I have no idea on 
what he is up to using ramdrives and such. We use an ext2 filesystem in 
a file, with cow_scripts from the xen 3 demo live cd.

To sum it up. We need to shrink the memory needed to start a domU down 
to somewhere around 12MB. And we do not have much time left to 
accomplish it. We''d seriuosly appreciate any feedback or tips on how to
get going.

Thanks in advance,

An by the way, xen is a stunningly cool product.

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Rickard Borgmäster wrote:
> To sum it up. We need to shrink the memory needed to start a domU down
> to somewhere around 12MB.
Any chance you could make your work so far available?

I''d be interested in, well, just playing with it :-).

But it might even help you reach your goal if some of the devs can
actually download it and see for themselves what''s using up memory.

Just a suggestion.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jan 31, 2006 at 03:09:18PM +0100, Rickard Borgmäster
wrote:
> To sum it up. We need to shrink the memory needed to start a domU down 
> to somewhere around 12MB. And we do not have much time left to 
> accomplish it. We''d seriuosly appreciate any feedback or tips on
how to
> get going.
I am using Xen in a similar network testing environment and I can boot my
regular domU kernel with memory=10 without a problem.

I''ve attached my kernel and xen config for you to tryout. Note though
that I
am using reiserfs and not ext3.

I am using this kernel config with the 2.6.16-rc1 kernel at
http://xenbits.xensource.com/linux-2.6-xen.hg

Good luck!

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Molle Bestefich wrote:
 >
> Any chance you could make your work so far available?
> 
> I''d be interested in, well, just playing with it :-).
> 
> But it might even help you reach your goal if some of the devs can
> actually download it and see for themselves what''s using up
memory.
That could be arranged. I''ll have to talk to our project supervisor for
access to the webspace. In fact, the "old" solution is already up at:
http://www.isk.kth.se/proj/crab/

That solution is made with UML, and our task is to replace it with Xen 
for better performance and isolation. Once our project is done, we will 
publish it on some website. But I guess it shouldn''t be impossible to 
post some kind of preview aswell. If it helps anyone to help us that''s
a
good idea.

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
> I am using Xen in a similar network testing environment and I can boot my
> regular domU kernel with memory=10 without a problem.
> 
> I''ve attached my kernel and xen config for you to tryout. Note
though that I
> am using reiserfs and not ext3.
Fantastic! So now there is further more evidence that it works :-) I''ll
have a look at your config tomorrow and see if I can build a 2.6.16-rc1 
kernel tomorrow. Have only been using 2.6.12.6 so far...

Btw, are you running Xen-2 or Xen-3?

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jan 31, 2006 at 06:25:08PM +0100, Rickard Borgmäster
wrote:
> CJ van den Berg wrote:
> >I am using Xen in a similar network testing environment and I can boot
my
> >regular domU kernel with memory=10 without a problem.
> >
> >I''ve attached my kernel and xen config for you to tryout. Note
though that
> >I
> >am using reiserfs and not ext3.
> 
> Fantastic! So now there is further more evidence that it works :-)
I''ll
> have a look at your config tomorrow and see if I can build a 2.6.16-rc1 
> kernel tomorrow. Have only been using 2.6.12.6 so far...
> 
> Btw, are you running Xen-2 or Xen-3?
I am running Xen 3.0-testing on Debian sid using Ralph Passgang''s
packages [1]. I use his kernel for dom0 and several custom kernels in domUs.
I am running Debian sid in both dom0 and all domUs.

1. http://lists.xensource.com/archives/html/xen-users/2006-01/msg00641.html

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
>
> I am running Xen 3.0-testing on Debian sid using Ralph Passgang''s
> packages [1]. I use his kernel for dom0 and several custom kernels in
domUs.
> I am running Debian sid in both dom0 and all domUs.
> 
> 1. http://lists.xensource.com/archives/html/xen-users/2006-01/msg00641.html
Looks promising :-)

Although... kernel 2.6.16-rc1... how did you do that? "clone hg 
http://xenbits.xensource.com/linux-2.6-xen.hg" results in 2.6.15 
sources, doesn''t it? Have you manually patched it up to 2.6.16-rc1? And
then applied some patches (again) for xen?

I''m just learning the custom compilation steps involved with xen so 
please have understanding ;-)

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Rickard Borgmäster wrote:
 >
> Although... kernel 2.6.16-rc1... how did you do that? "clone hg 
> http://xenbits.xensource.com/linux-2.6-xen.hg" results in 2.6.15 
> sources, doesn''t it? Have you manually patched it up to
2.6.16-rc1? And
> then applied some patches (again) for xen?
After the hg clone finished, I noticed that 2.6.16-rc1 is the current 
kernel version. I''m compiling it right now. Do we''ll see how
it works.
Noticed though that your kernel is quite "big", and could be stripped 
rather much. Don''t know how much it will affect speed and memory usage,
but it will at least reduce compile time :-)

Anyways, writing this, the compile just finished. Altough... it won''t 
start. xenconsole: Could not read tty from store: No such file or 
directory. I killed and restarted xenconsoled. And now something else is 
crashing...

# xm crate -c crab-domU.cfg name=test1
Using config file "crab-domU.cfg".
Started domain test1
#

xm list only returns Domain-0 :-(

Will try a "normal" 2.6.16-rc1 kernel compile now.

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Feb 01, 2006 at 10:58:56AM +0100, Rickard Borgmäster
wrote:
> After the hg clone finished, I noticed that 2.6.16-rc1 is the current 
> kernel version. I''m compiling it right now. Do we''ll see
how it works.
> Noticed though that your kernel is quite "big", and could be
stripped
> rather much. Don''t know how much it will affect speed and memory
usage,
> but it will at least reduce compile time :-)
"big" is relative I guess. :-) That''s about as small as it
gets for my
purposes (which is testing netfilter modules).
> Anyways, writing this, the compile just finished. Altough... it
won''t
> start. xenconsole: Could not read tty from store: No such file or 
> directory. I killed and restarted xenconsoled. And now something else is 
> crashing...
I''ve never had that error so I''m afraid I can''t
comment.
> # xm crate -c crab-domU.cfg name=test1
> Using config file "crab-domU.cfg".
> Started domain test1
> #
> 
> xm list only returns Domain-0 :-(
I''ve had plenty of strange things like this happen when the memory
parameter was border line too small. If it works once though it seems to
work always.

If you want you can post your crab-domU.cfg here. That might shed more
light on the problem. I can try booting my local build of that kernel
config with it too.
> Will try a "normal" 2.6.16-rc1 kernel compile now.
I would imagine that would be a *lot* fatter.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
> 
> "big" is relative I guess. :-) That''s about as small as
it gets for my
> purposes (which is testing netfilter modules).
Well, I had a hard time convincing myself that someone would need 802.11 
drivers in a domU kernel... ;-)
>># xm crate -c crab-domU.cfg name=test1
>>Using config file "crab-domU.cfg".
>>Started domain test1
>>#
>>
>>xm list only returns Domain-0 :-(
> 
> 
> I''ve had plenty of strange things like this happen when the memory
> parameter was border line too small. If it works once though it seems to
> work always.
Of course I''ve tried to adjust the memory parameter so that
shouldn''t be
  the limiting factor. I do of course want to see that the new kernel 
will start at all, before trying to get it to boot at ~12MB. However, 
even after setting memory to 54 it won''t start.
> If you want you can post your crab-domU.cfg here. That might shed more
> light on the problem. I can try booting my local build of that kernel
> config with it too.
Well... it isn''t very sophisticated but here goes:

kernel = "/boot/vmlinuz-2.6.16-rc1-vdberg-xenU"
memory = 54
disk = [''cow:/crab/.xen/crab-domU.cfg 30,hda1,w'']
root = "/dev/hda1 ro"
>>Will try a "normal" 2.6.16-rc1 kernel compile now.
> 
> I would imagine that would be a *lot* fatter.
I never did that. Instead, I started over and configured myself a new 
2.6.16-rc1 kernel. But... that didn''t boot either. Now, I think I have 
to try compile a *stock* 2.6.16-rc1 kernel to see if that will boot at 
all. Something is wrong.

Oh... just got a thought.. Do the dom0 kernel have to be 2.6.16-rc1 
aswell...?

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 03, 2006 at 10:07:15AM +0100, Rickard Borgmäster
wrote:
> CJ van den Berg wrote:
> >"big" is relative I guess. :-) That''s about as small
as it gets for my
> >purposes (which is testing netfilter modules).
> 
> Well, I had a hard time convincing myself that someone would need 802.11 
> drivers in a domU kernel... ;-)
Oops. ;-) But they are compiled as modules anyway...
> >>Will try a "normal" 2.6.16-rc1 kernel compile now.
> >
> >I would imagine that would be a *lot* fatter.
> 
> I never did that. Instead, I started over and configured myself a new 
> 2.6.16-rc1 kernel. But... that didn''t boot either. Now, I think I
have
> to try compile a *stock* 2.6.16-rc1 kernel to see if that will boot at 
> all. Something is wrong.
What hardware platform and compiler version are you using? The configuration
I sent you was Pentium 4 optimised. If you''re trying to run in on a
Pentium
II as your previous mail suggested I doubt it''ll work as-is. I
don''t see
anything else that should effect it though.
> Oh... just got a thought.. Do the dom0 kernel have to be 2.6.16-rc1 
> aswell...?
No. In fact dom0 doesn''t even have to be running Linux.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
> 
> What hardware platform and compiler version are you using? The
configuration
> I sent you was Pentium 4 optimised. If you''re trying to run in on
a Pentium
> II as your previous mail suggested I doubt it''ll work as-is. I
don''t see
> anything else that should effect it though.
I''m running a Pentium4 Dell-somethingish desktop computer for the 
testing pusposes. So no need to change the code optimization for that...

I''m getting quite frustrated about this now... :-(

Here is what I do;

hg clone http://xenbits.xensource.com/linux-2.6-xen.hg
cd linux-2.6-xen.hg
cp ~/config-vandenberg .config
make
cp vmlinuz /boot/vmlinuz-2.6.16-rc1-xenU
cp System.map System.map-2.6.16-rc1-xenU
xm create -c /crab/.xen/crab-domU.cfg

And that just produces:


Using config file "crab-domU.cfg".
Started domain testcrab
xenconsole: Could not read tty from store: No such file or directory


My config looks like this:

kernel = "/boot/vmlinuz-2.6.16-rc1-xenU"
memory = 17
disk = [''cow:/crab/.xen/crab-domU.cfg 30, hda1,w'']
root = "/dev/hda1 ro"

But I''ve also tried this:

kernel = "/boot/vmlinuz-2.6.16-rc1-xenU"
memory = 64
disk = [''file:/crab/.xen/crab-domU.cfg, hda1,w'']
root = "/dev/hda1 ro"

None of the above will start. The latter example doesn''t give the 
"xenstore" error, but just crashes instantly.

Am I missing something? I''m running Ralph Passgangs debian packages
just
like you...

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 03, 2006 at 02:56:34PM +0100, Rickard Borgmäster
wrote:
> Here is what I do;
> 
> hg clone http://xenbits.xensource.com/linux-2.6-xen.hg
> cd linux-2.6-xen.hg
> cp ~/config-vandenberg .config
> make
> cp vmlinuz /boot/vmlinuz-2.6.16-rc1-xenU
> cp System.map System.map-2.6.16-rc1-xenU
> xm create -c /crab/.xen/crab-domU.cfg
Just to be certain I followed these exact steps, except of course that I
used my own config instead of crab-domU.cfg, including pulling a completely
new clone. It builds and boots just fine, although it won''t run with
memory=10 anymore. It now only works from memory=12 up. If you want I can
send you the built kernel in a private mail.
> My config looks like this:
...
> disk = [''cow:/crab/.xen/crab-domU.cfg 30, hda1,w'']
I don''t use cow, so I''m not sure, but this line looks very
strange to me.
> But I''ve also tried this:
...
> disk = [''file:/crab/.xen/crab-domU.cfg, hda1,w'']
And this definitely must be wrong.
> Am I missing something? I''m running Ralph Passgangs debian
packages just
> like you...
I''m afraid I don''t know what else to suggest. It just works
here...

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
> On Fri, Feb 03, 2006 at 02:56:34PM +0100, Rickard Borgmäster wrote:
> 
>>hg clone http://xenbits.xensource.com/linux-2.6-xen.hg
>>cd linux-2.6-xen.hg
>>cp ~/config-vandenberg .config
>>make
>>cp vmlinuz /boot/vmlinuz-2.6.16-rc1-xenU
>>cp System.map System.map-2.6.16-rc1-xenU
>>xm create -c /crab/.xen/crab-domU.cfg
> > 
> Just to be certain I followed these exact steps, except of course that I
> used my own config instead of crab-domU.cfg, including pulling a completely
> new clone. It builds and boots just fine, although it won''t run
with
> memory=10 anymore. It now only works from memory=12 up. If you want I can
> send you the built kernel in a private mail.
Please do, reply to my private email in the header. If it boots on 12 it 
  is good enough for me. But I could probably strip it further to make 
sure it would fit a smaller memory footprint.
>>disk = [''cow:/crab/.xen/crab-domU.cfg 30, hda1,w'']
>>disk = [''file:/crab/.xen/crab-domU.cfg, hda1,w'']
> 
> And this definitely must be wrong.
s/cfg/img :-)

I wonder if there is something else on the system that produces broken 
kernels. Bad compiler or mismatched libraries.

I guess I do not need xen installed to compile the kernel, right? Maybe 
I should try to ssh to my home machine (which I know is 100% fine) and 
compile on that. It it also faster..

But anyways, please send me the binary kernel and the system.map and I 
will test it.

Thanks!

Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
>>
>>hg clone http://xenbits.xensource.com/linux-2.6-xen.hg
>>cd linux-2.6-xen.hg
>>cp ~/config-vandenberg .config
>>make
>>cp vmlinuz /boot/vmlinuz-2.6.16-rc1-xenU
>>cp System.map System.map-2.6.16-rc1-xenU
>>xm create -c /crab/.xen/crab-domU.cfg
> 
> Just to be certain I followed these exact steps, except of course that I
> used my own config instead of crab-domU.cfg, including pulling a completely
> new clone. It builds and boots just fine, although it won''t run
with
> memory=10 anymore. It now only works from memory=12 up. If you want I can
> send you the built kernel in a private mail.
I received your kernel and the System.map in a private mail. Thank you, 
very kind of you. Unfortunately it produced the same result. Waiting for 
your mail to get trough, I decided to "hg clone" the sources to my
home
machine and try a compile there as well. It is running 2.6.14 Gentoo, 
but I suppose it shouldn''t matter.

Both your kernel, my home-compiled one, and the ones I''ve been
compiling
in the test envirionment... all fail. Something is really strange. How 
come I can only run a kernel compiled out of the 2.6.12.6-sources from 
xen-3.0-testing... there must be something unique to this machine. Maybe 
I should reinstall xen or somethink. But that doesn''t explain why a 
2.6.12.6-kernel works and a 2.6.16-rc1 doesn''t... sigh.

Well. Now I''ll take a weekends rest from this project. We''ll
see what
happens next week.

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

CJ van den Berg wrote:
>>Oh... just got a thought.. Do the dom0 kernel have to be 2.6.16-rc1 
>>aswell...?
> 
> No. In fact dom0 doesn''t even have to be running Linux.
Hello again. So we''re back to work :-)

Deciced to test what other things could be stopping a kernel later than 
2.6.12.6 to run on my system. So I downloaded and compiled xen-unstable 
and xen-3.0-testing.

xen-3.0-testing didn''t produce any better results than the 3.0 release.
But after installing xen-unstable, I at least could boot a 2.6.15 
kernel. And that with memory = 11

:-D

But... xen-unstable isn''t named unstable for nothing. I''m not
sure if
this is a good way to acheive our goals. I don''t feel too comfortable 
putting a snapshot on a CD for our "release product" :-/

Anyways. We''re just about to put the xen-unstable on the CDROM image 
together with a 2.6.12.6 dom0 kernel and the new, stripped, 2.6.15 domU 
kernel. Doing so, the project will at least move forward and we can 
continue testing and probably make a beta. Some problems will just have 
to be solved as they appear ;-)

Thanks for your help so far, it has been good tutorial.

Best regards
Rickard Borgmäster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Feb 06, 2006 at 02:11:54PM +0100, Rickard Borgmäster
wrote:
> CJ van den Berg wrote:
> >>Oh... just got a thought.. Do the dom0 kernel have to be 2.6.16-rc1
> >>aswell...?
> >
> >No. In fact dom0 doesn''t even have to be running Linux.
> 
> Hello again. So we''re back to work :-)
> 
> Deciced to test what other things could be stopping a kernel later than 
> 2.6.12.6 to run on my system. So I downloaded and compiled xen-unstable 
> and xen-3.0-testing.
> 
> xen-3.0-testing didn''t produce any better results than the 3.0
release.
> But after installing xen-unstable, I at least could boot a 2.6.15 
> kernel. And that with memory = 11
Ah, sorry for the confusion. When you send you were using Passgang''s I
assumed you were running 3.0.1 like me.

In general though, new xen versions are backward compatible with old
kernels, but not the other way round. So technically, although it works for
me on 3.0.1, the -unstable kernel can only, AFAIK, be expected to work on
xen-unstable.
> :-D
:-)
> But... xen-unstable isn''t named unstable for nothing. I''m
not sure if
> this is a good way to acheive our goals. I don''t feel too
comfortable
> putting a snapshot on a CD for our "release product" :-/
I would feel the same way. Maybe you should try again with 3.0.1? At least
then if it works one half of the equation would be "stable".
> Anyways. We''re just about to put the xen-unstable on the CDROM
image
> together with a 2.6.12.6 dom0 kernel and the new, stripped, 2.6.15 domU 
> kernel. Doing so, the project will at least move forward and we can 
> continue testing and probably make a beta. Some problems will just have 
> to be solved as they appear ;-)
I''m not sure what the Xen release schedule is, but if you are only
around
beta stage it''s possible Xen will release a stable 2.6.16 before you
have
to ship.
> Thanks for your help so far, it has been good tutorial.
No problem. You''re welcome.

-- 
CJ van den Berg

mailto:cj@vdbonline.com
  xmpp:cj@vdbonline.com
   sip:8918316@sipgate.de

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel