Byrne, John (HP Labs)
2008-Jul-19 00:52 UTC
[Xen-devel] HVM direct boot broken in xen-unstable
Hi! On x86_64, changeset 18081, running/building on rhel5, trying to use the HVM direct boot causes the domain to reboot immediately and then the log says the domain is restarting too rapidly. No obvious hints as to where the problem is. If I build the in-tree ioemu code, things work. Any clues as to where to look or how to get some useful debugging output? Thanks, John Byrne _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2008-Jul-19 08:35 UTC
Re: [Xen-devel] HVM direct boot broken in xen-unstable
On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:> Hi! > > On x86_64, changeset 18081, running/building on rhel5, trying to use > the HVM direct boot causes the domain to reboot immediately and then > the log says the domain is restarting too rapidly. No obvious hints > as to where the problem is. If I build the in-tree ioemu code, things > work. Any clues as to where to look or how to get some useful debugging > output?It appears the code is missing from the external ioemu code. The original changesets for HVM kernel boot were changeset: 16959:17cce0554151 user: Keir Fraser <keir.fraser@citrix.com> date: Fri Feb 01 11:14:53 2008 +0000 summary: Direct Linux boot: Support booting relocatable Linux kernels. changeset: 16960:67ca9c37ef02 user: Keir Fraser <keir.fraser@citrix.com> date: Fri Feb 01 11:16:37 2008 +0000 summary: Direct Linux boot: Support booting non-relocatable Linux kernels. Part of the first changeset is missing, and all of the second. So the kernel will be getting overwritten by the HVM firmware hence the crashes of the guest you see Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
xen-unstable"):> It appears the code is missing from the external ioemu code. The original
> changesets for HVM kernel boot were
Thanks for tracking that down. I will port those across.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
xen-unstable"):> On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote:
> > Hi!
> >
> > On x86_64, changeset 18081, running/building on rhel5, trying to use
> > the HVM direct boot causes the domain to reboot immediately and then
> > the log says the domain is restarting too rapidly. No obvious hints
> > as to where the problem is. If I build the in-tree ioemu code, things
> > work. Any clues as to where to look or how to get some useful
debugging
> > output?
Byrne, could you send me a copy of your config file so I can reproduce
this ? Also, some idea of how you made (or how you obtained) the
kernel image and initramfs. I don''t think I''ve ever done a
direct
kernel boot using this code (which is in patches supplied by Daniel)
and offhand I can''t seem to find the documentation.
> It appears the code is missing from the external ioemu code. The original
> changesets for HVM kernel boot were
I''ve looked at the code and these changes are ones I did attempt to
port across to the new qemu. You''ll see for example that the `evil
helper'' is still there, just slightly further down the file in pc.c.
I haven''t ever executed the resulting codepaths and evidently I broke
something, which is not all that surprising.
Thanks for reporting the bug. I look forward to fixing it :-).
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Daniel P. Berrange
2008-Jul-21 15:51 UTC
Re: [Xen-devel] HVM direct boot broken in xen-unstable
On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"): > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote: > > > Hi! > > > > > > On x86_64, changeset 18081, running/building on rhel5, trying to use > > > the HVM direct boot causes the domain to reboot immediately and then > > > the log says the domain is restarting too rapidly. No obvious hints > > > as to where the problem is. If I build the in-tree ioemu code, things > > > work. Any clues as to where to look or how to get some useful debugging > > > output? > > Byrne, could you send me a copy of your config file so I can reproduce > this ? Also, some idea of how you made (or how you obtained) the > kernel image and initramfs. I don''t think I''ve ever done a direct > kernel boot using this code (which is in patches supplied by Daniel) > and offhand I can''t seem to find the documentation.It works in the same way as booting PV off kernel+initrd - basically take a HVM guest config file and add loader="/usr/lib/xen/bin/hvmloader" kernel="/path/to/vmlinuz" ramdisk="/path/to/initrd" The vmlinuz & initrd would typically be the ones for installation of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i386/os/images/pxeboot/ It should boot your kernel & initrd and not pay attention to hardisk or CDROM devices. As mentioned in the original commit logs, older kernels need to use the evil helper, newer kernels should ''just work'' (thanks to the relocatable kernel support). Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Byrne, John (HP Labs)
2008-Jul-21 15:59 UTC
RE: [Xen-devel] HVM direct boot broken in xen-unstable
Daniel, Ian: The old ioemu version functions without the "loader" line which I was missing in my config. I''m testing to make sure that isn''t the issue I''m seeing with the git version. John> -----Original Message----- > From: Daniel P. Berrange [mailto:berrange@redhat.com] > Sent: Monday, July 21, 2008 8:52 AM > To: Ian Jackson > Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable > > On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote: > > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in > xen-unstable"): > > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) > wrote: > > > > Hi! > > > > > > > > On x86_64, changeset 18081, running/building on rhel5, trying to > use > > > > the HVM direct boot causes the domain to reboot immediately and > then > > > > the log says the domain is restarting too rapidly. No obvious > hints > > > > as to where the problem is. If I build the in-tree ioemu code, > things > > > > work. Any clues as to where to look or how to get some useful > debugging > > > > output? > > > > Byrne, could you send me a copy of your config file so I can > reproduce > > this ? Also, some idea of how you made (or how you obtained) the > > kernel image and initramfs. I don''t think I''ve ever done a direct > > kernel boot using this code (which is in patches supplied by Daniel) > > and offhand I can''t seem to find the documentation. > > It works in the same way as booting PV off kernel+initrd - basically > take a HVM guest config file and add > > loader="/usr/lib/xen/bin/hvmloader" > kernel="/path/to/vmlinuz" > ramdisk="/path/to/initrd" > > The vmlinuz & initrd would typically be the ones for installation > of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL > > http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i3 > 86/os/images/pxeboot/ > > It should boot your kernel & initrd and not pay attention to hardisk > or CDROM devices. > > As mentioned in the original commit logs, older kernels need to use the > evil helper, newer kernels should ''just work'' (thanks to the > relocatable > kernel support). > > Daniel > -- > |: Red Hat, Engineering, London -o- > http://people.redhat.com/berrange/ :| > |: http://libvirt.org -o- http://virt-manager.org -o- > http://ovirt.org :| > |: http://autobuild.org -o- > http://search.cpan.org/~danberr/ :| > |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B > 9505 :|_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Byrne, John (HP Labs)
2008-Jul-21 16:31 UTC
RE: [Xen-devel] HVM direct boot broken in xen-unstable
Daniel, Ian: No difference with "loader". The domain still disappears almost immediately. I''ll try to understand the original code (now that I know where to look) and see if I can understand what is happening. Thanks for your help, John Byrne> -----Original Message----- > From: Byrne, John (HP Labs) > Sent: Monday, July 21, 2008 8:59 AM > To: ''Daniel P. Berrange''; Ian Jackson > Cc: xen-devel@lists.xensource.com > Subject: RE: [Xen-devel] HVM direct boot broken in xen-unstable > > Daniel, Ian: > > The old ioemu version functions without the "loader" line which I was > missing in my config. I''m testing to make sure that isn''t the issue I''m > seeing with the git version. > > John > > > > -----Original Message----- > > From: Daniel P. Berrange [mailto:berrange@redhat.com] > > Sent: Monday, July 21, 2008 8:52 AM > > To: Ian Jackson > > Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable > > > > On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote: > > > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken > in > > xen-unstable"): > > > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) > > wrote: > > > > > Hi! > > > > > > > > > > On x86_64, changeset 18081, running/building on rhel5, trying > to > > use > > > > > the HVM direct boot causes the domain to reboot immediately and > > then > > > > > the log says the domain is restarting too rapidly. No obvious > > hints > > > > > as to where the problem is. If I build the in-tree ioemu code, > > things > > > > > work. Any clues as to where to look or how to get some useful > > debugging > > > > > output? > > > > > > Byrne, could you send me a copy of your config file so I can > > reproduce > > > this ? Also, some idea of how you made (or how you obtained) the > > > kernel image and initramfs. I don''t think I''ve ever done a direct > > > kernel boot using this code (which is in patches supplied by > Daniel) > > > and offhand I can''t seem to find the documentation. > > > > It works in the same way as booting PV off kernel+initrd - basically > > take a HVM guest config file and add > > > > loader="/usr/lib/xen/bin/hvmloader" > > kernel="/path/to/vmlinuz" > > ramdisk="/path/to/initrd" > > > > The vmlinuz & initrd would typically be the ones for installation > > of your distro, eg The $URL/images/pxeboot/ files for Fedora/RHEL > > > > > http://download.fedora.redhat.com/pub/fedora/linux/releases/9/Fedora/i3 > > 86/os/images/pxeboot/ > > > > It should boot your kernel & initrd and not pay attention to hardisk > > or CDROM devices. > > > > As mentioned in the original commit logs, older kernels need to use > the > > evil helper, newer kernels should ''just work'' (thanks to the > > relocatable > > kernel support). > > > > Daniel > > -- > > |: Red Hat, Engineering, London -o- > > http://people.redhat.com/berrange/ :| > > |: http://libvirt.org -o- http://virt-manager.org -o- > > http://ovirt.org :| > > |: http://autobuild.org -o- > > http://search.cpan.org/~danberr/ :| > > |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B > > 9505 :|_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange
2008-Jul-22 11:00 UTC
Re: [Xen-devel] HVM direct boot broken in xen-unstable
On Mon, Jul 21, 2008 at 04:46:17PM +0100, Ian Jackson wrote:> Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in xen-unstable"): > > On Sat, Jul 19, 2008 at 12:52:07AM +0000, Byrne, John (HP Labs) wrote: > > > Hi! > > > > > > On x86_64, changeset 18081, running/building on rhel5, trying to use > > > the HVM direct boot causes the domain to reboot immediately and then > > > the log says the domain is restarting too rapidly. No obvious hints > > > as to where the problem is. If I build the in-tree ioemu code, things > > > work. Any clues as to where to look or how to get some useful debugging > > > output? > > Byrne, could you send me a copy of your config file so I can reproduce > this ? Also, some idea of how you made (or how you obtained) the > kernel image and initramfs. I don''t think I''ve ever done a direct > kernel boot using this code (which is in patches supplied by Daniel) > and offhand I can''t seem to find the documentation. > > > It appears the code is missing from the external ioemu code. The original > > changesets for HVM kernel boot were > > I''ve looked at the code and these changes are ones I did attempt to > port across to the new qemu. You''ll see for example that the `evil > helper'' is still there, just slightly further down the file in pc.c. > > I haven''t ever executed the resulting codepaths and evidently I broke > something, which is not all that surprising.I think I''ve identified the problem. In the load_linux() method, the line where we copy the header into guest memory: /* store the finalized header and load the rest of the kernel */ cpu_physical_memory_write(real_addr, header, 1024); Is done before the Xen hook runs: xen_relocator_hook(prot_addr, protocol, header, kernel_size, real_addr, setup_size-1024); The xen_relocator_hook modifies byte 0x214 in the header to point to its hook, and so this change isn''t getting written into memory. The cpu_physical_memory_write(real_addr, header, 1024) call just needs to be moved down to be immediately after the call to the xen hook. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in
xen-unstable"):> It works in the same way as booting PV off kernel+initrd - basically
> take a HVM guest config file and add [...]
Thanks. With that I was able to reproduce the problem and I think I
have now fixed it in this changeset:
commit a7104be2d6a521dfdcf1ce9f82ae40525e223775
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Tue Jul 22 15:49:52 2008 +0100
Fix HVM direct kernel booting.
This is available now (untested other than ad-hoc by me) at
http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git
and will make it to the usual locations on xenbits in due course.
> I think I''ve identified the problem. In the load_linux() method,
the
> line where we copy the header into guest memory:
> [etc]
This was indeed a problem but not the only one.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Byrne, John (HP Labs)
2008-Jul-22 15:42 UTC
RE: [Xen-devel] HVM direct boot broken in xen-unstable
I will test this sometime today. (As soon as I get my test system back.) Thanks, John> -----Original Message----- > From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com] > Sent: Tuesday, July 22, 2008 7:52 AM > To: Daniel P. Berrange > Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable > > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in > xen-unstable"): > > It works in the same way as booting PV off kernel+initrd - basically > > take a HVM guest config file and add [...] > > Thanks. With that I was able to reproduce the problem and I think I > have now fixed it in this changeset: > > commit a7104be2d6a521dfdcf1ce9f82ae40525e223775 > Author: Ian Jackson <ian.jackson@eu.citrix.com> > Date: Tue Jul 22 15:49:52 2008 +0100 > > Fix HVM direct kernel booting. > > This is available now (untested other than ad-hoc by me) at > http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git > and will make it to the usual locations on xenbits in due course. > > > I think I''ve identified the problem. In the load_linux() method, the > > line where we copy the header into guest memory: > > [etc] > > This was indeed a problem but not the only one. > > Ian._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Byrne, John (HP Labs)
2008-Jul-22 18:57 UTC
RE: [Xen-devel] HVM direct boot broken in xen-unstable
There still seems to be an issue somewhere. If I boot a "router" kernel with a combined kernel/initramfs (no ramdisk line), it successfully boots. When I boot a rhel5 kernel, the kernel hangs at "checking if image is initramfs" saying it has bad gzip magic numbers. John> -----Original Message----- > From: Ian Jackson [mailto:Ian.Jackson@eu.citrix.com] > Sent: Tuesday, July 22, 2008 7:52 AM > To: Daniel P. Berrange > Cc: Byrne, John (HP Labs); xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] HVM direct boot broken in xen-unstable > > Daniel P. Berrange writes ("Re: [Xen-devel] HVM direct boot broken in > xen-unstable"): > > It works in the same way as booting PV off kernel+initrd - basically > > take a HVM guest config file and add [...] > > Thanks. With that I was able to reproduce the problem and I think I > have now fixed it in this changeset: > > commit a7104be2d6a521dfdcf1ce9f82ae40525e223775 > Author: Ian Jackson <ian.jackson@eu.citrix.com> > Date: Tue Jul 22 15:49:52 2008 +0100 > > Fix HVM direct kernel booting. > > This is available now (untested other than ad-hoc by me) at > http://www.chiark.greenend.org.uk/~ijackson/qemu-xen.git > and will make it to the usual locations on xenbits in due course. > > > I think I''ve identified the problem. In the load_linux() method, the > > line where we copy the header into guest memory: > > [etc] > > This was indeed a problem but not the only one. > > Ian._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Byrne, John (HP Labs) writes ("RE: [Xen-devel] HVM direct boot broken in
xen-unstable"):> There still seems to be an issue somewhere.
>
> If I boot a "router" kernel with a combined kernel/initramfs (no
ramdisk line), it successfully boots.
> When I boot a rhel5 kernel, the kernel hangs at "checking if image is
initramfs" saying it has bad gzip magic numbers.
I''ve tested Debian sarge 2.6.18-5-686 and Centos 5 2.6.23.1-42.fc8 and
both found the initramfs without any difficulty.
Can you perhaps email me your acutal kernel image and initramfs, and
of course your domain config file, and I could try to reproduce your
problem here ?
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel