Xen devel - Sep 2005 - What''s the magic to compile a unified dom0 and domU kernel image?

Hi list,

I''m playing around with XEN 2.0.7 (using the .debs) and want to have
only one kernel image for dom0 and domU (like SuSE has) with different
initrds.
Now my problem: First I''ve tried to use my old 2.6.11 config and just
enable all the XEN specific options to have it work as dom0 and domU ->
no go, see output in [1].
Second I''ve tried using the config of the XEN domU kernel and enabling
all the modules I need for my hardware -> no go, output is the same as
in [1]. (I should remark that the original XEN domU config of course
works here, but not after enabling what I need).
Third I''ve extracted the SuSE 9.3 rpm and "borrowed" their
config. After
enabling all the options Debian needs to successfully boot with an
initrd (cramfs and devfs) I got a working kernel that boots just fine in
dom0 AND domU (of course with different hand-optimized initrds)!

Now I could just say wow, how cool this is (actually XEN is quite cool I
think :)) but I want to understand what''s the magic kernel option SuSE
has set (or not set) and I have (not). After hours of research and
comparing the two config files (using the excelent kdiff3) and compiling
the kernel at least 20 times with different options from the SuSE config
I still have no working vmlinuz image from my own config file.
I''ve uploaded the two config-files (the working and the not working
one)
to http://markus.schuster.name/xen/ maybe someone has time to have a
look at it (the kernel has much too much options to try all by hand ;))

Thanks for your help in advance.

Greetings,
Markus


[1]
dom0:~# xm create ttylinux -c
Using config file "/etc/xen/ttylinux".
Started domain ttylinux, console on port 9605
************ REMOTE CONSOLE: CTRL-] TO QUIT ********

************ REMOTE CONSOLE EXITED *****************
dom0:~#


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Markus,

You could use a dom0 kernel for both dom0 and domU.
If compiling your own ''unified'' kernel, make sure you
make ARCH=xen menuconfig
make ARCH=xen install (or whatever)

Marcus.

Markus Schuster wrote:
> Hi list,
> 
> I''m playing around with XEN 2.0.7 (using the .debs) and want to
have
> only one kernel image for dom0 and domU (like SuSE has) with different
> initrds.
> Now my problem: First I''ve tried to use my old 2.6.11 config and
just
> enable all the XEN specific options to have it work as dom0 and domU ->
> no go, see output in [1].
> Second I''ve tried using the config of the XEN domU kernel and
enabling
> all the modules I need for my hardware -> no go, output is the same as
> in [1]. (I should remark that the original XEN domU config of course
> works here, but not after enabling what I need).
> Third I''ve extracted the SuSE 9.3 rpm and "borrowed"
their config. After
> enabling all the options Debian needs to successfully boot with an
> initrd (cramfs and devfs) I got a working kernel that boots just fine in
> dom0 AND domU (of course with different hand-optimized initrds)!
> 
> Now I could just say wow, how cool this is (actually XEN is quite cool I
> think :)) but I want to understand what''s the magic kernel option
SuSE
> has set (or not set) and I have (not). After hours of research and
> comparing the two config files (using the excelent kdiff3) and compiling
> the kernel at least 20 times with different options from the SuSE config
> I still have no working vmlinuz image from my own config file.
> I''ve uploaded the two config-files (the working and the not
working one)
> to http://markus.schuster.name/xen/ maybe someone has time to have a
> look at it (the kernel has much too much options to try all by hand ;))
> 
> Thanks for your help in advance.
> 
> Greetings,
> Markus
> 
> 
> [1]
> dom0:~# xm create ttylinux -c
> Using config file "/etc/xen/ttylinux".
> Started domain ttylinux, console on port 9605
> ************ REMOTE CONSOLE: CTRL-] TO QUIT ********
> 
> ************ REMOTE CONSOLE EXITED *****************
> dom0:~#
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marcus Brown schrieb:
> You could use a dom0 kernel for both dom0 and domU.
Yes, that''s actually what I want to to. But the default XEN dom0 config
includes much too less modules to be of any use for me. And the XEN
specific options (CONFIG_XEN*) are the same in my two configs and the
default XEN dom0 config.
> If compiling your own ''unified'' kernel, make sure you
> make ARCH=xen menuconfig
> make ARCH=xen install (or whatever)
That''s more or less what I did to get a kernel. I did:
# ARCH=XEN make clean
# ARCH=XEN make menuconfig
# ARCH=XEN make vmlinuz
# ARCH=XEN make modules
# ARCH=XEN make modules_install

In general I know how to compile a XEN kernel, but there must be some
difference between my two configs that I don''t realize to be important
for XEN.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Markus,

How about this, then?
(for Xen 2.0.7)

CONFIG_XEN=y
CONFIG_ARCH_XEN=y
CONFIG_NO_IDLE_HZ=y

#
# XEN
#
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PHYSDEV_ACCESS=y
CONFIG_XEN_BLKDEV_BACKEND=y
# CONFIG_XEN_NETDEV_BACKEND is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_XEN_NETDEV_FRONTEND_PIPELINED_TRANSMITTER is not set
CONFIG_XEN_WRITABLE_PAGETABLES=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_X86=y
# CONFIG_X86_64 is not set
CONFIG_HAVE_ARCH_DEV_ALLOC_SKB=y

Which is the start of the config I use for a Bluetooth Proxy Server
(for an ipaq)

I believe PRIVILEGED_GUEST and PHYSDEV_ACCESS to be the
''magical'' options.

Marcus

Markus Schuster wrote:
> Marcus Brown schrieb:
> 
>>You could use a dom0 kernel for both dom0 and domU.
> 
> 
> Yes, that''s actually what I want to to. But the default XEN dom0
config
> includes much too less modules to be of any use for me. And the XEN
> specific options (CONFIG_XEN*) are the same in my two configs and the
> default XEN dom0 config.
> 
> 
>>If compiling your own ''unified'' kernel, make sure you
>>make ARCH=xen menuconfig
>>make ARCH=xen install (or whatever)
> 
> 
> That''s more or less what I did to get a kernel. I did:
> # ARCH=XEN make clean
> # ARCH=XEN make menuconfig
> # ARCH=XEN make vmlinuz
> # ARCH=XEN make modules
> # ARCH=XEN make modules_install
> 
> In general I know how to compile a XEN kernel, but there must be some
> difference between my two configs that I don''t realize to be
important
> for XEN.
> 
> Greetings,
> Markus
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marcus Brown schrieb:
> How about this, then?
> (for Xen 2.0.7)
> [..]
That''s exactly what I have standing on the top of my two config files.
I''ve also set CONFIG_XEN_NETDEV_BACKEND, but that''s set in the
working
and in the non-working config, so I don''t think it is the problem.
> I believe PRIVILEGED_GUEST and PHYSDEV_ACCESS to be the
> ''magical'' options.
I''ve set both options in both of my config files. One works, the other
not...
Maybe you want to have a look at my two config files (URL is in my first
message).

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

The trick for me was to be sure that I included ARCH=xen in EVERY make
line when building the kernel.

make ARCH=xen menuconfig
make ARCH=xen modules
make ARCH=xen -j 6

etc.

You may need to start over again with your original config file AND to
run a make mrproper if you have not been using ARCH=xen everywhere.



On Thu, 2005-09-15 at 12:41 +0200, Markus Schuster
wrote:
> Marcus Brown schrieb:
> > How about this, then?
> > (for Xen 2.0.7)
> > [..]
> 
> That''s exactly what I have standing on the top of my two config
files.
> I''ve also set CONFIG_XEN_NETDEV_BACKEND, but that''s set
in the working
> and in the non-working config, so I don''t think it is the problem.
> 
> > I believe PRIVILEGED_GUEST and PHYSDEV_ACCESS to be the
> > ''magical'' options.
> 
> I''ve set both options in both of my config files. One works, the
other
> not...
> Maybe you want to have a look at my two config files (URL is in my first
> message).
> 
> Greetings,
> Markus
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Markus,

Haven''t had much time to check the whole thing,
but look at this first:
diff config-fail config-ok | grep DEV_LOOP
< CONFIG_BLK_DEV_LOOP=m
> CONFIG_BLK_DEV_LOOP=y
Other things that I''d check (that you haven''t mentioned) are:
- initrd
- location of modules (in your domU''s filesystem I hope!:) )
- does the domU boot with another kernel?
- have you correctly specified all these options in the xen
	domU config?

Marcus.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Marcus Brown schrieb:
> diff config-fail config-ok | grep DEV_LOOP
> < CONFIG_BLK_DEV_LOOP=m
> >CONFIG_BLK_DEV_LOOP=y
Now I''ve compiled loop support statically into the kernel -> no
change,
doesn''t work.

> - initrd
> - location of modules (in your domU''s filesystem I hope!:) )
Well, I don''t think these two should make any difference, as the kernel
doesn''t even start to boot. But to make sure I''ve tried with
and without
initrd (last tests without) and with modules in
/lib/modules/2.6.11-xen-universal in the domU filesystem.

> - does the domU boot with another kernel?
Yes, as I said, it boots with the original XEN domU kernel and it boots
with a kernel built from ''config-ok''. But when using
''config-fail'' WITH
THE SAME kernel source I''m unable get a working kernel image.

> - have you correctly specified all these options in the xen
> 	domU config?
I think so, as I just switch the kernel=... option between the working
version and the version I can''t get to work.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Michael Gregg schrieb:
> The trick for me was to be sure that I included ARCH=xen in EVERY make
> line when building the kernel.
I did so, as I know this from my time playing around with UML. And I
just have so switch the .config in the kernel source to have a working
or a non-working vmlinuz image. So I think the problem must lay in the
config file.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi list,

has anybody any further ideas I could try?

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I am assuming that you have dom0 working. Why can''t you just use the 
xen0 kernel to boot up domU?

Markus Schuster wrote:
>Hi list,
>
>has anybody any further ideas I could try?
>
>Greetings,
>Markus
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-users mailing list
>Xen-users@lists.xensource.com
>http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

David F Barrera schrieb:
> I am assuming that you have dom0 working. Why can''t you just use
the
> xen0 kernel to boot up domU?
Yep, dom0 is working. And it''s the question I''d like to find
an answer
for why I can''t use that kernel image for my domUs.
As I said, I''ve one .config that works and one that doesn''t
work. My
question is why. I want to use XEN in a production environment (as 3.0
gets stable) and I really want to know what could be the problem.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, 2005-09-27 at 21:53 +0200, Markus Schuster
wrote:
> David F Barrera schrieb:
> > I am assuming that you have dom0 working. Why can''t you just
use the
> > xen0 kernel to boot up domU?
> 
> Yep, dom0 is working. And it''s the question I''d like to
find an answer
> for why I can''t use that kernel image for my domUs.
> As I said, I''ve one .config that works and one that
doesn''t work. My
> question is why. I want to use XEN in a production environment (as 3.0
> gets stable) and I really want to know what could be the problem.
> 
> Greetings,
> Markus
> _______________________________________________
The kernel for a dum0 and domU are very different.

Just take a look at the choices you get when choosing compile options,
that will probably tell a large portion of the story.

If you want exact differences pour over the source code, hell just diff
the two tree''s for that matter.


Regards,
Ted






_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ted Kaczmarek schrieb:
> The kernel for a dum0 and domU are very different.
I know, but it is possible to build one that fits dom0 and domU.
> If you want exact differences pour over the source code, hell just diff
> the two tree''s for that matter.
I think my main problem is that I''m no native english speaker, so I may
not be able to describe my problem like I would like to.
I don''t understand what you mean by "two trees" of source
code. I use
ONE kernel source tree and TWO .config files. One is based on the SuSE
XEN kernel image (that works) and one is based on my non-XEN Debian
kernel config (doesn''t work).
And I looked at diffs of that two .config files over and over again and
set a lot of parameters from the SuSE one in the other (formerly non-XEN
Debian) .config file but I can''t find the "magic" option that
stops my
Debian based .config from producing a vmlinuz kernel image that can be
used for dom0 and domU (it works for dom0, but not for domU).
But I wrote that a lot of times now in my older posts, I really don''t
know how to write that more clearer.

Greetings,
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I know, but it is possible to build one that fits dom0 and domU.
See our -xen config files under
linux-*-sparse/arch/xen/configs/xen_defconfig_x86_32

The -xen kernel has pretty much everything compiled as modules, and
should be what most users want. The -xen0 and -xenU kernels are really
just for developers, having a relatively small configuration so they
build fast, and with functions compiled-in rather than as modules so as
to obviate the need to install modules before booting.
The only advantage of using the -xenU kernel over the -xen0 kernel for
guests is that it''s a bit smaller. Its still rather larger than the
-xen
modularized kernel, though)

Ian

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users