With RHEL6 beta 2 now available, I am once again trying
to get a fresh Xen 4.0-testing tree booted "underneath" it.
(This is for my Xen development machines, no political
comments please ;-) I''m sure there are some similarities
between this process and the process under FC13(?);
any help appreciated.
I have cloned xen-4.0-testing.hg and successfully built
a full set of Xen "stuff" with "make world". The dom0
kernel that is installed in /boot is vmlinuz-2.6.31.13
I used the default config (and default responses to
the "make oldconfig" that seems to be run by default)
to build this dom0 kernel.
1) Is there a better config file to start from for
building the dom0 kernel? For example, I see that
CONFIG_SYSFS_DEPRECATED{,V2} are not set, and I
am fairly confident that this kernel will not boot
a RHEL6 environment without it.
2) What do I use to build the initrd? The standard
"mkinitrd initrd.img version" yields LOTS of error
messages and, in any case, I believe that dracut
should now be used instead of mkinitrd?
3) Should I be using a different/newer pvops kernel?
IIRC, there is an issue with booting >=2.6.32
pvops dom0 with pre-xen-unstable hypervisors? Or
is that fixed in latest xen-4.0-testing?
4) Anything else I should watch out for? E.g. recent
tool changes due to udev or xencommon or ???
Most of these final steps are black magic to me and
I''ve been using a recipe on RHEL5uX for them for
so long, I''m not even sure what to look for if
(when) the dom0 boot fails... and trial and error
is not very productive for this kind of thing.
Thanks,
Dan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Thu, Jul 08, 2010 at 03:16:23PM -0700, Dan Magenheimer wrote:> With RHEL6 beta 2 now available, I am once again trying > to get a fresh Xen 4.0-testing tree booted "underneath" it. > (This is for my Xen development machines, no political > comments please ;-) I''m sure there are some similarities > between this process and the process under FC13(?); > any help appreciated. > > I have cloned xen-4.0-testing.hg and successfully built > a full set of Xen "stuff" with "make world". The dom0 > kernel that is installed in /boot is vmlinuz-2.6.31.13 > I used the default config (and default responses to > the "make oldconfig" that seems to be run by default) > to build this dom0 kernel. > > 1) Is there a better config file to start from for > building the dom0 kernel? For example, I see that > CONFIG_SYSFS_DEPRECATED{,V2} are not set, and I > am fairly confident that this kernel will not boot > a RHEL6 environment without it.You could try my .config .. it works on Fedora 13 for me, with xen/stable-2.6.32.x dom0 kernel: http://pasik.reaktio.net/xen/pv_ops-dom0-debug/config-2.6.32.15-pvops-dom0-xen-stable-x86_64> 2) What do I use to build the initrd? The standard > "mkinitrd initrd.img version" yields LOTS of error > messages and, in any case, I believe that dracut > should now be used instead of mkinitrd?Yeah, you should use dracut instead of mkinitrd. The usage is the same.> 3) Should I be using a different/newer pvops kernel? > IIRC, there is an issue with booting >=2.6.32 > pvops dom0 with pre-xen-unstable hypervisors? Or > is that fixed in latest xen-4.0-testing? >Xen 4.0.1-rc3 and newer work OK with the latest xen/stable-2.6.32.x at least for me on F13.> 4) Anything else I should watch out for? E.g. recent > tool changes due to udev or xencommon or ??? >Not sure..> Most of these final steps are black magic to me and > I''ve been using a recipe on RHEL5uX for them for > so long, I''m not even sure what to look for if > (when) the dom0 boot fails... and trial and error > is not very productive for this kind of thing. >I was thinking of posting a tutorial about Xen 4 + F13, and after that try RHEL6 myself.. :) -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer wrote:> With RHEL6 beta 2 now available, I am once again trying > to get a fresh Xen 4.0-testing tree booted "underneath" it. > (This is for my Xen development machines, no political > comments please ;-) I''m sure there are some similarities > between this process and the process under FC13(?); > any help appreciated. > > I have cloned xen-4.0-testing.hg and successfully built > a full set of Xen "stuff" with "make world". The dom0 > kernel that is installed in /boot is vmlinuz-2.6.31.13 > I used the default config (and default responses to > the "make oldconfig" that seems to be run by default) > to build this dom0 kernel. > > 1) Is there a better config file to start from for > building the dom0 kernel? For example, I see that > CONFIG_SYSFS_DEPRECATED{,V2} are not set, and I > am fairly confident that this kernel will not boot > a RHEL6 environment without it.RHEL6 requires this not to be set, i.e., the redhat .config''s have: # CONFIG_SYSFS_DEPRECATED_V2 is not set So, general rule of thumb around here: build RHEL6 kernel on RHEL6(guest). The beauty of a 2.6.32 guest (disk image) is that it can be a PV guest or an HVM guest, using just two different xen guest config files (surely simplifies my testing efforts -- far less guest disk images, kernel copy/installs, etc.), of course only one of them running at any one time, unless you like ''boom''! ;-) Following that rule: (a) install/make a RHEL6 xen guest: PV or HVM (although PV is much faster) on your favorite (working?) dom0/xen host. (b) install devel pkg & build the variant kernel you want in that guest (c) ... and for the issue below, in (2):> 2) What do I use to build the initrd? The standard > "mkinitrd initrd.img version" yields LOTS of error > messages and, in any case, I believe that dracut > should now be used instead of mkinitrd?(c)... continued.... mkinitrd is a script in rhel6 that invokes dracut with the proper params to make an initrd. of course, in mkinitrd fashion, it uses the current host config to determine what goes into the initrd. So a PV guest will be better, since it''ll include the xen-[blk|net]front drivers (et. al.) so you can build a -dom0 with needed boot drivers.> 3) Should I be using a different/newer pvops kernel? > IIRC, there is an issue with booting >=2.6.32 > pvops dom0 with pre-xen-unstable hypervisors? OrNot necessarily. Do this all the time: RHEL6-domU on RHEL5-dom0. ... maybe some intermediate versions of xen dont work w/pv-ops kernel, but it does with rhel5-dom0/xen.> is that fixed in latest xen-4.0-testing?good question! curious minds want to know....> 4) Anything else I should watch out for? E.g. recent > tool changes due to udev or xencommon or ??? > > Most of these final steps are black magic to me andI haven''t followed the how-to-build a dom0 from F12/F13, so don''t know the black magic to makde a dom0 from a 2.6.32 base. I''m sure the magic has to do w/making the dom0 kernel in synch with xen HV base code (HV calls, defines, etc.), but I believe how that is done in the late Fedora trees is quite diff. than how it was done in rhel5 sources (due to pv-ops kernel design).> I''ve been using a recipe on RHEL5uX for them for > so long, I''m not even sure what to look for if > (when) the dom0 boot fails... and trial and error > is not very productive for this kind of thing. > > Thanks, > Dan > >Hope some of the above helps. Good luck!... Don ____________________________________________> Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Jul-09 17:44 UTC
Re: [Xen-devel] Help booting Xen under RHEL6 beta2
> > > 3) Should I be using a different/newer pvops kernel? > > IIRC, there is an issue with booting >=2.6.32 > > pvops dom0 with pre-xen-unstable hypervisors? Or > Not necessarily. Do this all the time: RHEL6-domU on RHEL5-dom0. > ... maybe some intermediate versions of xen dont work w/pv-ops kernel, > but it does with rhel5-dom0/xen. > > > is that fixed in latest xen-4.0-testing? > good question! curious minds want to know....Yes. The xen/evtchn and xen/gntdev are differently created on 2.6.32 and xen-4.0-testing. For details: http://lists.xensource.com/archives/html/xen-devel/2010-06/msg01129.html You can "fix" this by doing: git revert 376d908f52427591cef4acd172db9c3ef28676ec and git revert 9d85e25edbcb8a684db8470119077475ac4d90b5 That will make it possible for you to use the 2.6.32 with Xen 4.0 (and earlier versions).> > > 4) Anything else I should watch out for? E.g. recent > > tool changes due to udev or xencommon or ???There is a bug 1612 for udev. But that is not hit by Fedora 13, but rather other distros. There is a patch that fixes some of the udev versions, but not all. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 9 Jul 2010, Konrad Rzeszutek Wilk wrote:> Yes. The xen/evtchn and xen/gntdev are differently created on 2.6.32 > and xen-4.0-testing.That isn''t quite true. A dom0 kernel based on the current xen/stable-2.6.32.x won''t work with xen-4.0.0 but it will work with xen-4.0-testing (currently at 4.0.1-rc4-pre) as that has the corresponding patches since 4.0.1-rc2. Michael Young _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Jul-09 20:01 UTC
Re: [Xen-devel] Help booting Xen under RHEL6 beta2
On Fri, Jul 09, 2010 at 07:09:50PM +0100, M A Young wrote:> On Fri, 9 Jul 2010, Konrad Rzeszutek Wilk wrote: > > >Yes. The xen/evtchn and xen/gntdev are differently created on 2.6.32 > >and xen-4.0-testing. > > That isn''t quite true. A dom0 kernel based on the current > xen/stable-2.6.32.x won''t work with xen-4.0.0 but it will work with > xen-4.0-testing (currently at 4.0.1-rc4-pre) as that has the > corresponding patches since 4.0.1-rc2.Right. Thanks for correcting me. It should have said xen-4.0.0 as you pointed out. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks all for the replies. I''ve made some good forward progress
but have run into another roadblock, so wanted to see if anyone
has seen any of this before:
Booting RHEL6 distro with xen-4.0-testing (64-bit) and a
2.6.32.16 dom0 kernel (config attached), I usually get:
dracut: Switching root
Welcome to Red Hat Enterprise Linux Server
Press ''I'' to enter interactive startup
Starting udev: init: rcS main process (439) killed by KILL signal
Kernel panic - not syncing: Out of memory and no killable processes
(Xen is definitely booting... but this is a laptop and
I haven''t found a way to capture console output so am
copying text manually. Also, I''ve used this laptop
for a long time with a RHEL5 distro on it with no problems,
so it''s highly likely any differences are due to RHEL6 vs RHEL5.)
Sometimes it gets past udev OK, completes the init scripts
and then gets stuck in a very slow loop repeatedly trying to
bring up the graphical console. It is alive at this point...
I can ssh into it but the disk is being continually pounded
so I can''t do anything useful. I can "telinit 3" and the
graphical console cycling stops, but the disk continues
to scream. I am now able to login on the laptop screen,
but again everything is VERY slow. Eventually the
disk screaming stops...
Trying "service xend start" it eventually completes (with
a sequence of dots), but fails... logs attached.
I tried "xl info" but get "cannot init xl context".
I do have xenfs in my /etc/fstab and "ll /proc/xen"
shows everything normal.
By changing /etc/inittab to boot to runlevel 3, I can
avoid the graphics issue, but at the loss of ability
to VNC with a graphical display (which worked with
RHEL5).
I tried disabling selinux in /etc/selinux/config but
this made no difference. Realizing that there might
be some never-before-tried-under-Xen service running
in this distro (e.g. ksm? libvirtd?), I''ve also
attached the output of "service --status-all".
Interestingly, vmstat seems to imply that the disk
activity is due to swapping as si/so are large when
the disk light is active and small/0 when the disk
light is off. Why? No clue. I tried disabling
ksm and ksmtuned, but no difference. It does appear
with "top" that there are a lot of gnome apps
using a lot of memory, so all the swapping may
be completely unrelated to the Xen issues.
Thanks for any help or suggestions!
Dan
BTW, the 2.6.32.16 "dom0" kernel boots fine without Xen, which
may be a good litmus test to ensure the kernel and initramfs
are built properly.
> -----Original Message-----
> From: Konrad Rzeszutek Wilk
> Sent: Friday, July 09, 2010 2:01 PM
> To: M A Young
> Cc: Don Dutile; Dan Magenheimer; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2
>
> On Fri, Jul 09, 2010 at 07:09:50PM +0100, M A Young wrote:
> > On Fri, 9 Jul 2010, Konrad Rzeszutek Wilk wrote:
> >
> > >Yes. The xen/evtchn and xen/gntdev are differently created on
2.6.32
> > >and xen-4.0-testing.
> >
> > That isn''t quite true. A dom0 kernel based on the current
> > xen/stable-2.6.32.x won''t work with xen-4.0.0 but it will
work with
> > xen-4.0-testing (currently at 4.0.1-rc4-pre) as that has the
> > corresponding patches since 4.0.1-rc2.
>
> Right. Thanks for correcting me. It should have said xen-4.0.0 as you
> pointed out.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Fri, 9 Jul 2010, Dan Magenheimer wrote:> Thanks all for the replies. I''ve made some good forward progress > but have run into another roadblock, so wanted to see if anyone > has seen any of this before: > > Booting RHEL6 distro with xen-4.0-testing (64-bit) and a > 2.6.32.16 dom0 kernel (config attached), I usually get: > > dracut: Switching root > Welcome to Red Hat Enterprise Linux Server > Press ''I'' to enter interactive startup > Starting udev: init: rcS main process (439) killed by KILL signal > Kernel panic - not syncing: Out of memory and no killable processesAre you limiting the memory for dom0? If so I suggest you allow it more, or don''t limit it at all and see if it boots any more successfully. It is also probably worth booting direct to run level 3 (add 3 to the boot line) to so you can eliminate whether or not graphics is an issue. Also I suggest you check whether the hypervisor and xen software are the same version. If they aren''t then xend may not be able to the hypervisor. Michael Young _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Michael -- Thanks for the help!> Also I suggest you check whether the hypervisor and xen software are > the same version. If they aren''t then xend may not be able to the > hypervisor.They are definitely the same version as I cloned only xen-4.0-testing and built/installed both on the test machine itself.> Are you limiting the memory for dom0? If so I suggest you allow it > more, or don''t limit it at all and see if it boots any more > successfully.Yes, I think you are right. I have always booted dom0 with 256M on this box, but I guess more recent gnome tools are much more bloated so that accounts for the swapping and slowness.> It is > also probably worth booting direct to run level 3 (add 3 to the boot > line) to so you can eliminate whether or not graphics is an issue.Yes, I did that. This skips the graphics looping but xend still fails to start (and I can no longer VNC to the box... fortunately it is a laptop at least). Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Slightly more progress. I discovered xenstored is not running... so I did: # modprobe xen-evtchn # xenstored # xenconsoled # /etc/init.d/xend start but xend fails to start with the same messages (ending in "Connection refused").> -----Original Message----- > From: Dan Magenheimer > Sent: Saturday, July 10, 2010 3:42 PM > To: M A Young > Cc: xen-devel@lists.xensource.com; Don Dutile; Konrad Wilk > Subject: RE: [Xen-devel] Help booting Xen under RHEL6 beta2 > > Hi Michael -- > > Thanks for the help! > > > Also I suggest you check whether the hypervisor and xen software are > > the same version. If they aren''t then xend may not be able to the > > hypervisor. > > They are definitely the same version as I cloned only xen-4.0-testing > and built/installed both on the test machine itself. > > > Are you limiting the memory for dom0? If so I suggest you allow it > > more, or don''t limit it at all and see if it boots any more > > successfully. > > Yes, I think you are right. I have always booted dom0 with 256M > on this box, but I guess more recent gnome tools are much more > bloated so that accounts for the swapping and slowness. > > > It is > > also probably worth booting direct to run level 3 (add 3 to the boot > > line) to so you can eliminate whether or not graphics is an issue. > > Yes, I did that. This skips the graphics looping but xend > still fails to start (and I can no longer VNC to the box... > fortunately it is a laptop at least). > > Thanks, > Dan > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Jul-12 10:37 UTC
RE: [Xen-devel] Help booting Xen under RHEL6 beta2
On Sun, 11 Jul 2010, Dan Magenheimer wrote:> Slightly more progress. I discovered xenstored is not running... > so I did: > > # modprobe xen-evtchn > # xenstored > # xenconsoledxencommons should take care of starting xenstored and xenconsoled _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Is xencommons in 4.0-testing? If so, where? Sent from my iPhone On Jul 12, 2010, at 4:37 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Sun, 11 Jul 2010, Dan Magenheimer wrote: >> Slightly more progress. I discovered xenstored is not running... >> so I did: >> >> # modprobe xen-evtchn >> # xenstored >> # xenconsoled > > xencommons should take care of starting xenstored and xenconsoled > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2010-Jul-12 14:06 UTC
Re: [Xen-devel] Help booting Xen under RHEL6 beta2
On Mon, 12 Jul 2010, Dan M @ Oracle wrote:> Is xencommons in 4.0-testing? If so, where?in the sources: tools/hotplug/Linux/init.d/xencommons it should be installed in /etc/init.d _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/07/2010 15:06, "Stefano Stabellini" <Stefano.Stabellini@eu.citrix.com> wrote:> On Mon, 12 Jul 2010, Dan M @ Oracle wrote: >> Is xencommons in 4.0-testing? If so, where? > > in the sources: > > tools/hotplug/Linux/init.d/xencommons > > it should be installed in /etc/init.dIt doesn''t exist in 4.0-testing. In that tree, the xend wrapper script is still starting xenstored and xenconsoled. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2010-07-12 at 15:06 +0100, Stefano Stabellini wrote:> On Mon, 12 Jul 2010, Dan M @ Oracle wrote: > > Is xencommons in 4.0-testing? If so, where? > > in the sources: > > tools/hotplug/Linux/init.d/xencommons > > it should be installed in /etc/init.dThat change is only in xen-unstable, isn''t it? In xen-4.0-testing it looks to me like tools/misc/xend still takes care of it so if you''ve switched to xl in that branch you might still need to start stuff by hand. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
OK, so here''s another interesting piece of information: The xend.log python error trace ends with: xs_handle = xen.lowlevel.xs.xs() Error: (111, ''Connection refused'') So I rebooted (with xend disabled), manually started xenstored, and ensured xenstored is indeed running with ps -ef. Then I tried "xenstore-ls" and all I get is: tool = "" xenstored = "" local = "" I tried this on a EL5-based distro running a 2.6.18.8-xen dom0 (with xend disabled on boot) and xenstore-ls gives me a long list of dom0 related registry entries. So maybe xenstored is dependent on some distro configuration that is missing on RHEL6b2? Thanks, Dan P.S. I checked /dev/xen and it does contain evtchn and gntdev and my kernel config does have CONFIG_XEN_DEV_EVTCHN=y> -----Original Message----- > From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > Sent: Monday, July 12, 2010 8:37 AM > To: Stefano Stabellini > Cc: Dan Magenheimer; xen-devel@lists.xensource.com; Konrad Rzeszutek > Wilk; DonDutile; Young > Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 > > On Mon, 2010-07-12 at 15:06 +0100, Stefano Stabellini wrote: > > On Mon, 12 Jul 2010, Dan M @ Oracle wrote: > > > Is xencommons in 4.0-testing? If so, where? > > > > in the sources: > > > > tools/hotplug/Linux/init.d/xencommons > > > > it should be installed in /etc/init.d > > That change is only in xen-unstable, isn''t it? In xen-4.0-testing it > looks to me like tools/misc/xend still takes care of it so if you''ve > switched to xl in that branch you might still need to start stuff by > hand. > > Ian. > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/07/2010 21:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> So I rebooted (with xend disabled), manually started xenstored, > and ensured xenstored is indeed running with ps -ef. > Then I tried "xenstore-ls" and all I get is: > > tool = "" > xenstored = "" > local = "" > > I tried this on a EL5-based distro running a 2.6.18.8-xen > dom0 (with xend disabled on boot) and xenstore-ls gives > me a long list of dom0 related registry entries.Those dom0 registry entries would be written by xend. If you just run xenstored, you''d expect the database to be pretty empty. Looks like xenstored is running okay. Perhaps xend is trying to connect to xenstored in a different way to xenstore-ls. There are two different ways to connect -- via a Unix domains socket /var/run/xenstored/socket; or via a kernel device /proc/xen/xenbus. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
This recent thread from xen-users with the exact symptoms:
http://www.gossamer-threads.com/lists/xen/users/177223
(see last in thread) implies that this is the result of
a regression since xen-4.0.1-rc3, though it''s vague on
exactly what regressed ("evtchn stuff"?).
I''ll try rebuilding to confirm.
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Monday, July 12, 2010 2:39 PM
> To: Dan Magenheimer; Ian Campbell; Stefano Stabellini
> Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk
> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2
>
> On 12/07/2010 21:19, "Dan Magenheimer"
<dan.magenheimer@oracle.com>
> wrote:
>
> > So I rebooted (with xend disabled), manually started xenstored,
> > and ensured xenstored is indeed running with ps -ef.
> > Then I tried "xenstore-ls" and all I get is:
> >
> > tool = ""
> > xenstored = ""
> > local = ""
> >
> > I tried this on a EL5-based distro running a 2.6.18.8-xen
> > dom0 (with xend disabled on boot) and xenstore-ls gives
> > me a long list of dom0 related registry entries.
>
> Those dom0 registry entries would be written by xend. If you just run
> xenstored, you''d expect the database to be pretty empty. Looks
like
> xenstored is running okay. Perhaps xend is trying to connect to
> xenstored in
> a different way to xenstore-ls. There are two different ways to connect
> --
> via a Unix domains socket /var/run/xenstored/socket; or via a kernel
> device
> /proc/xen/xenbus.
>
> -- Keir
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Xen-4.0-testing:19983 would be the c/s being discussed there. You could try reverting it and see how that goes for you. But if you have /dev/xen/evtchn and /dev/xen/gntdev then (assuming the device nodes aren''t bogus for some reason, and there''s no reason they should be) you should be good to go in this respect. You could double check by ''ls -l /dev/xen'' and compare the device-node minors to the list in /proc/misc. -- Keir On 12/07/2010 21:44, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:> This recent thread from xen-users with the exact symptoms: > http://www.gossamer-threads.com/lists/xen/users/177223 > (see last in thread) implies that this is the result of > a regression since xen-4.0.1-rc3, though it''s vague on > exactly what regressed ("evtchn stuff"?). > > I''ll try rebuilding to confirm. > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >> Sent: Monday, July 12, 2010 2:39 PM >> To: Dan Magenheimer; Ian Campbell; Stefano Stabellini >> Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk >> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 >> >> On 12/07/2010 21:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> >> wrote: >> >>> So I rebooted (with xend disabled), manually started xenstored, >>> and ensured xenstored is indeed running with ps -ef. >>> Then I tried "xenstore-ls" and all I get is: >>> >>> tool = "" >>> xenstored = "" >>> local = "" >>> >>> I tried this on a EL5-based distro running a 2.6.18.8-xen >>> dom0 (with xend disabled on boot) and xenstore-ls gives >>> me a long list of dom0 related registry entries. >> >> Those dom0 registry entries would be written by xend. If you just run >> xenstored, you''d expect the database to be pretty empty. Looks like >> xenstored is running okay. Perhaps xend is trying to connect to >> xenstored in >> a different way to xenstore-ls. There are two different ways to connect >> -- >> via a Unix domains socket /var/run/xenstored/socket; or via a kernel >> device >> /proc/xen/xenbus. >> >> -- Keir >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Xen-4.0-testing:19983 would be the c/s being discussed there.Could you confirm that number? The changeset says "xend: Fix use of set() for Python 2.3" and it was 11 months ago which doesn''t match the "roll back to xen-4.0.1-rc3" suggestion in the xen-users thread, so I''m confused. The device-node minors do match between /dev/xen and /proc/misc. I''m in the process of doing a fresh build which will take awhile. Thanks, Dan> -----Original Message----- > From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > Sent: Monday, July 12, 2010 3:10 PM > To: Dan Magenheimer; Ian Campbell; Stefano Stabellini > Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk > Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 > > Xen-4.0-testing:19983 would be the c/s being discussed there. You could > try > reverting it and see how that goes for you. But if you have > /dev/xen/evtchn > and /dev/xen/gntdev then (assuming the device nodes aren''t bogus for > some > reason, and there''s no reason they should be) you should be good to go > in > this respect. You could double check by ''ls -l /dev/xen'' and compare > the > device-node minors to the list in /proc/misc. > > -- Keir > > On 12/07/2010 21:44, "Dan Magenheimer" <dan.magenheimer@oracle.com> > wrote: > > > This recent thread from xen-users with the exact symptoms: > > http://www.gossamer-threads.com/lists/xen/users/177223 > > (see last in thread) implies that this is the result of > > a regression since xen-4.0.1-rc3, though it''s vague on > > exactly what regressed ("evtchn stuff"?). > > > > I''ll try rebuilding to confirm. > > > >> -----Original Message----- > >> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] > >> Sent: Monday, July 12, 2010 2:39 PM > >> To: Dan Magenheimer; Ian Campbell; Stefano Stabellini > >> Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk > >> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 > >> > >> On 12/07/2010 21:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> > >> wrote: > >> > >>> So I rebooted (with xend disabled), manually started xenstored, > >>> and ensured xenstored is indeed running with ps -ef. > >>> Then I tried "xenstore-ls" and all I get is: > >>> > >>> tool = "" > >>> xenstored = "" > >>> local = "" > >>> > >>> I tried this on a EL5-based distro running a 2.6.18.8-xen > >>> dom0 (with xend disabled on boot) and xenstore-ls gives > >>> me a long list of dom0 related registry entries. > >> > >> Those dom0 registry entries would be written by xend. If you just > run > >> xenstored, you''d expect the database to be pretty empty. Looks like > >> xenstored is running okay. Perhaps xend is trying to connect to > >> xenstored in > >> a different way to xenstore-ls. There are two different ways to > connect > >> -- > >> via a Unix domains socket /var/run/xenstored/socket; or via a kernel > >> device > >> /proc/xen/xenbus. > >> > >> -- Keir > >> > >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''m in the process of doing a fresh build which will > take awhile.Backing out to xen-4.0.1-rc3 did not help, though short of reinstalling the entire system from scratch, I''m not sure how to ensure I''ve backed _everything_ out to rc3... there''s too many flying parts.> > So I rebooted (with xend disabled), manually started xenstored, > > and ensured xenstored is indeed running with ps -ef. > > Then I tried "xenstore-ls" and all I get is: > > > > tool = "" > > xenstored = "" > > local = "" > > > > I tried this on a EL5-based distro running a 2.6.18.8-xen > > dom0 (with xend disabled on boot) and xenstore-ls gives > > me a long list of dom0 related registry entries. > > Those dom0 registry entries would be written by xend. If you just run > xenstored, you''d expect the database to be pretty empty. Looks like > xenstored is running okay.I had forgotten that the xenstored data is persistent across boot, so I removed /var/lib/xenstored/tdb on the working machine and rebooted. The behavior is still different between the two systems. This may or may not be relevant or provide a clue to someone who knows far more about xenstored and xend than I do. It appears to me that dom0 is "waiting" to connect to xenstored on the good system, but not on the bad system. Any chance this might be related to the renaming of evtchn (to xen_evtchn or xen-evtchn or whatever it is)? How might I check that? # GOOD system, xen-4.0-testing c/s 21209, 2.6.18.8-xen dom0, EL5 distro # ls /var/lib/xenstored/tdb ls: /var/lib/xenstored/tdb: No such file or directory # xenstore-ls xenstore-ls: xs_directory (/): No such device # xenstored # xenstore-ls tool = "" xenstored = "" local = "" domain = "" 0 = "" device = "" suspend = "" event-channel = "39" # /usr/lib/xen/bin/lsevtchn | grep 39 39: VCPU 0: Interdomain (Waiting connection) - Remote Domain 0 # # NOTE, xend not started yet ############################################# # BAD system, xen-4.0-testing c/s 21263, 2.6.32.16 pvops dom0, RHEL6b2 distro # ls /var/lib/xenstored/tdb ls: cannot access /var/lib/xenstored/tdb: No such file or directory # xenstore-ls ## HANGS until xenstored in another window!! # xenstored # xenstore-ls tool = "" xenstored = "" # ls /usr/lib/xen/bin/lsevtchn ls: cannot access /usr/lib/xen/bin/lsevtchn: No such file or directory # cd src/xen/xen-4.0-testing.hg # tools/xcutils/ adfljad;khgalkjgdhl 11: VCPU 0: Interdomain (Waiting connection) - Remote Domain 0, Port 28 28: VCPU 0: Interdomain (Waiting connection) - Remote Domain 0, Port 11 # # attempting to start xend prints a handful of dots, no errors, # # but /var/log/xen/xend.log shows the "Connection refused" failure # # and xend hasn''t started> Perhaps xend is trying to connect to xenstored in > a different way to xenstore-ls. There are two different ways to connect > -- > via a Unix domains socket /var/run/xenstored/socket; or via a kernel > device /proc/xen/xenbus.Could be, though I haven''t intentionally changed anything other than switching to the RHEL6b2 distro and 64-bit pvops dom0. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Jul-13 01:30 UTC
Re: [Xen-devel] Help booting Xen under RHEL6 beta2
On Mon, Jul 12, 2010 at 05:39:34PM -0700, Dan Magenheimer wrote:> > I''m in the process of doing a fresh build which will > > take awhile. > > Backing out to xen-4.0.1-rc3 did not help, though short > of reinstalling the entire system from scratch, I''m not > sure how to ensure I''ve backed _everything_ out to rc3... > there''s too many flying parts.Well, you can try to back-out the two changes in the pv-ops kernel. git revert 376d908f52427591cef4acd172db9c3ef28676ec and git revert 9d85e25edbcb8a684db8470119077475ac4d90b5 And that way the pv-ops kernel can run with Xen 4.0 and Xen 4.0-unstable as well. It could also be that you are mixing two versions of Xen. As in you might have the distro''s libs and then yours. Perhaps a bit of find / -name libxens*.so could shed some light? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/07/2010 22:36, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:>> Xen-4.0-testing:19983 would be the c/s being discussed there. > > Could you confirm that number? The changeset says > "xend: Fix use of set() for Python 2.3" and it was > 11 months ago which doesn''t match the "roll back to > xen-4.0.1-rc3" suggestion in the xen-users thread, > so I''m confused.Sorry it''s 21182. I was looking in 3.4 tree by mistake. -- Keir> The device-node minors do match between /dev/xen and /proc/misc. > > I''m in the process of doing a fresh build which will > take awhile. > > Thanks, > Dan > >> -----Original Message----- >> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >> Sent: Monday, July 12, 2010 3:10 PM >> To: Dan Magenheimer; Ian Campbell; Stefano Stabellini >> Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk >> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 >> >> Xen-4.0-testing:19983 would be the c/s being discussed there. You could >> try >> reverting it and see how that goes for you. But if you have >> /dev/xen/evtchn >> and /dev/xen/gntdev then (assuming the device nodes aren''t bogus for >> some >> reason, and there''s no reason they should be) you should be good to go >> in >> this respect. You could double check by ''ls -l /dev/xen'' and compare >> the >> device-node minors to the list in /proc/misc. >> >> -- Keir >> >> On 12/07/2010 21:44, "Dan Magenheimer" <dan.magenheimer@oracle.com> >> wrote: >> >>> This recent thread from xen-users with the exact symptoms: >>> http://www.gossamer-threads.com/lists/xen/users/177223 >>> (see last in thread) implies that this is the result of >>> a regression since xen-4.0.1-rc3, though it''s vague on >>> exactly what regressed ("evtchn stuff"?). >>> >>> I''ll try rebuilding to confirm. >>> >>>> -----Original Message----- >>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] >>>> Sent: Monday, July 12, 2010 2:39 PM >>>> To: Dan Magenheimer; Ian Campbell; Stefano Stabellini >>>> Cc: Young; xen-devel@lists.xensource.com; DonDutile; Konrad Wilk >>>> Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 >>>> >>>> On 12/07/2010 21:19, "Dan Magenheimer" <dan.magenheimer@oracle.com> >>>> wrote: >>>> >>>>> So I rebooted (with xend disabled), manually started xenstored, >>>>> and ensured xenstored is indeed running with ps -ef. >>>>> Then I tried "xenstore-ls" and all I get is: >>>>> >>>>> tool = "" >>>>> xenstored = "" >>>>> local = "" >>>>> >>>>> I tried this on a EL5-based distro running a 2.6.18.8-xen >>>>> dom0 (with xend disabled on boot) and xenstore-ls gives >>>>> me a long list of dom0 related registry entries. >>>> >>>> Those dom0 registry entries would be written by xend. If you just >> run >>>> xenstored, you''d expect the database to be pretty empty. Looks like >>>> xenstored is running okay. Perhaps xend is trying to connect to >>>> xenstored in >>>> a different way to xenstore-ls. There are two different ways to >> connect >>>> -- >>>> via a Unix domains socket /var/run/xenstored/socket; or via a kernel >>>> device >>>> /proc/xen/xenbus. >>>> >>>> -- Keir >>>> >>>> >> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Status update: Still no go. Konrad and I repeated some of the obvious xenstored related possible problems and everything looks fine. I tried rolling back the pvops evtchn/gntdev naming changes. No difference. I tried rolling back the tools evtchn changeset (21182). No difference. Best I can tell so far, this seems to be something unique to RHEL6b2 but don''t have the xenstored skills to debug any further. Dan> -----Original Message----- > From: Konrad Rzeszutek Wilk > Sent: Monday, July 12, 2010 7:30 PM > To: Dan Magenheimer > Cc: Keir Fraser; Ian Campbell; Stefano Stabellini; xen- > devel@lists.xensource.com; DonDutile; Young > Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 > > On Mon, Jul 12, 2010 at 05:39:34PM -0700, Dan Magenheimer wrote: > > > I''m in the process of doing a fresh build which will > > > take awhile. > > > > Backing out to xen-4.0.1-rc3 did not help, though short > > of reinstalling the entire system from scratch, I''m not > > sure how to ensure I''ve backed _everything_ out to rc3... > > there''s too many flying parts. > > Well, you can try to back-out the two changes in the pv-ops kernel. > > git revert 376d908f52427591cef4acd172db9c3ef28676ec > > and > git revert 9d85e25edbcb8a684db8470119077475ac4d90b5 > > And that way the pv-ops kernel can run with Xen 4.0 and Xen 4.0- > unstable > as well. > > It could also be that you are mixing two versions of Xen. As in you > might have the distro''s libs and then yours. Perhaps a bit of find / > -name libxens*.so could shed some light? >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just to close this for posterity: Move along, these are not the droids you are looking for. In days of debugging this problem (and also during much help from Konrad), I had always manually started xenstored before attempting to start xend. And every time I had tried xl, I had tried it after attempting to start xend, and it had failed. Apparently starting xend (in 4.0.1-rcX) AFTER starting xenstored puts xenstored into a very strange state, after which neither xm nor xl will work (but xenstore-ls and friends do work). In this state, all socket connections fail. So the classic joke: If it hurts when you do that, don''t do that. Bottom line: RHEL6b2 distro with 2.6.32.x pvops dom0 with xen-4.0-testing works. Thanks to everyone for helping on this! Dan> -----Original Message----- > From: Dan Magenheimer > Sent: Tuesday, July 13, 2010 5:55 PM > To: Konrad Wilk; Keir Fraser; Ian Campbell; Stefano Stabellini; xen- > devel@lists.xensource.com; DonDutile; Young > Subject: RE: [Xen-devel] Help booting Xen under RHEL6 beta2 > > Status update: Still no go. > > Konrad and I repeated some of the obvious xenstored related > possible problems and everything looks fine. > > I tried rolling back the pvops evtchn/gntdev naming changes. > No difference. I tried rolling back the tools evtchn changeset > (21182). No difference. > > Best I can tell so far, this seems to be something unique > to RHEL6b2 but don''t have the xenstored skills to debug > any further. > > Dan > > > -----Original Message----- > > From: Konrad Rzeszutek Wilk > > Sent: Monday, July 12, 2010 7:30 PM > > To: Dan Magenheimer > > Cc: Keir Fraser; Ian Campbell; Stefano Stabellini; xen- > > devel@lists.xensource.com; DonDutile; Young > > Subject: Re: [Xen-devel] Help booting Xen under RHEL6 beta2 > > > > On Mon, Jul 12, 2010 at 05:39:34PM -0700, Dan Magenheimer wrote: > > > > I''m in the process of doing a fresh build which will > > > > take awhile. > > > > > > Backing out to xen-4.0.1-rc3 did not help, though short > > > of reinstalling the entire system from scratch, I''m not > > > sure how to ensure I''ve backed _everything_ out to rc3... > > > there''s too many flying parts. > > > > Well, you can try to back-out the two changes in the pv-ops kernel. > > > > git revert 376d908f52427591cef4acd172db9c3ef28676ec > > > > and > > git revert 9d85e25edbcb8a684db8470119077475ac4d90b5 > > > > And that way the pv-ops kernel can run with Xen 4.0 and Xen 4.0- > > unstable > > as well. > > > > It could also be that you are mixing two versions of Xen. As in you > > might have the distro''s libs and then yours. Perhaps a bit of find / > > -name libxens*.so could shed some light? > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel