Richard Blocker
2007-Jun-21 14:09 UTC
[Xen-users] Xen guest domain freezes when prelink runs
I''m new to running Xen, so maybe I missed something, but this is a puzzling problem that I didn''t find on any of the Xen lists. I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up to date as of today) and I have a single guest domain HVM also running RHEL 5. Whenever the guest OS runs the initial prelink job defined in /etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at 100% and the system stops responding. It never resumes (at least not for 12 hours). I can reboot the guest domain from the host machine, and it recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to see if it was just the load, but it was fine while that ran. If I run the prelink command manually, it immediately freezes. For the record, the hardware is a dual quad core Xeon system with 8GB of memory. The guest HVM uses a single CPU with 512MB of memory allocated. Has anyone else seen this? -- Richard Blocker Information Systems Manager Ecology and Evolution Organismal Biology and Anatomy University of Chicago Zoology 003 1101 E. 57th Street Chicago, IL 60637 rblocker@uchicago.edu telephone: 773-702-5135 facsimile: 773-702-9740 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tim Boyer
2007-Jun-22 00:33 UTC
[Xen-users] Re: Xen guest domain freezes when prelink runs
On Thu, 21 Jun 2007 09:09:39 -0500, Richard Blocker <rblocker@uchicago.edu> wrote:>I''m new to running Xen, so maybe I missed something, but this is a >puzzling problem that I didn''t find on any of the Xen lists. > >I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up >to date as of today) and I have a single guest domain HVM also running >RHEL 5. Whenever the guest OS runs the initial prelink job defined in >/etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at >100% and the system stops responding. It never resumes (at least not for >12 hours). I can reboot the guest domain from the host machine, and it >recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to >see if it was just the load, but it was fine while that ran. If I run >the prelink command manually, it immediately freezes. > >For the record, the hardware is a dual quad core Xeon system with 8GB of >memory. The guest HVM uses a single CPU with 512MB of memory allocated. > >Has anyone else seen this?Hah! Someone else with this... I''ve got a trouble report into RH on this one. It''s not just prelink; I think it''s tied to rpm. I can bring the system down doing a rpm -Va, or a sysreport without the -norpm switch. Also a dual quad core, with 16gb, and 1gb on the guest. -- tim boyer tim@denmantire.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Daniel P. Berrange
2007-Jun-22 02:40 UTC
Re: [Xen-users] Re: Xen guest domain freezes when prelink runs
On Thu, Jun 21, 2007 at 08:33:51PM -0400, Tim Boyer wrote:> On Thu, 21 Jun 2007 09:09:39 -0500, Richard Blocker <rblocker@uchicago.edu> > wrote: > > >I''m new to running Xen, so maybe I missed something, but this is a > >puzzling problem that I didn''t find on any of the Xen lists. > > > >I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up > >to date as of today) and I have a single guest domain HVM also running > >RHEL 5. Whenever the guest OS runs the initial prelink job defined in > >/etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at > >100% and the system stops responding. It never resumes (at least not for > >12 hours). I can reboot the guest domain from the host machine, and it > >recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to > >see if it was just the load, but it was fine while that ran. If I run > >the prelink command manually, it immediately freezes. > > > >For the record, the hardware is a dual quad core Xeon system with 8GB of > >memory. The guest HVM uses a single CPU with 512MB of memory allocated. > > > >Has anyone else seen this? > > Hah! Someone else with this... > > I''ve got a trouble report into RH on this one. It''s not just prelink; I think > it''s tied to rpm. I can bring the system down doing a rpm -Va, or a sysreport > without the -norpm switch. Also a dual quad core, with 16gb, and 1gb on the > guest.Actually it is prelink - rpm -Va will call out the prelink libraries when verifying IIRC. Anyway, this is ultimately a hypervisor bug in Xen 3.0.3 which should be fixed in the Xen 3.1.0 hypervisor. So should be working come RHEL-5.1 updates. Seriously though, you really really really don''t want to run any OS in full-virt if its capable of paravirt. You''ll get x10 -> x100 the I/O performance if you use paravirt and be able to scale up the number of guests per host much better. So I''d recommend using RHEL-5 paravirt at which point you won''t see the HVM bug anymore either... Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tim Boyer
2007-Jun-22 11:08 UTC
[Xen-users] Re: Xen guest domain freezes when prelink runs
On Fri, 22 Jun 2007 03:40:48 +0100, "Daniel P. Berrange" <berrange@redhat.com> wrote:>On Thu, Jun 21, 2007 at 08:33:51PM -0400, Tim Boyer wrote: >> On Thu, 21 Jun 2007 09:09:39 -0500, Richard Blocker <rblocker@uchicago.edu> >> wrote: >> >> >I''m new to running Xen, so maybe I missed something, but this is a >> >puzzling problem that I didn''t find on any of the Xen lists. >> > >> >I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up >> >to date as of today) and I have a single guest domain HVM also running >> >RHEL 5. Whenever the guest OS runs the initial prelink job defined in >> >/etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at >> >100% and the system stops responding. It never resumes (at least not for >> >12 hours). I can reboot the guest domain from the host machine, and it >> >recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to >> >see if it was just the load, but it was fine while that ran. If I run >> >the prelink command manually, it immediately freezes. >> > >> >For the record, the hardware is a dual quad core Xeon system with 8GB of >> >memory. The guest HVM uses a single CPU with 512MB of memory allocated. >> > >> >Has anyone else seen this? >> >> Hah! Someone else with this... >> >> I''ve got a trouble report into RH on this one. It''s not just prelink; I think >> it''s tied to rpm. I can bring the system down doing a rpm -Va, or a sysreport >> without the -norpm switch. Also a dual quad core, with 16gb, and 1gb on the >> guest. > >Actually it is prelink - rpm -Va will call out the prelink libraries when >verifying IIRC. Anyway, this is ultimately a hypervisor bug in Xen 3.0.3 >which should be fixed in the Xen 3.1.0 hypervisor. So should be working >come RHEL-5.1 updates. > >Seriously though, you really really really don''t want to run any OS in >full-virt if its capable of paravirt. You''ll get x10 -> x100 the I/O >performance if you use paravirt and be able to scale up the number of >guests per host much better. So I''d recommend using RHEL-5 paravirt >at which point you won''t see the HVM bug anymore either... > >Dan.Dan, thanks _very_ much for the info. Maybe you can answer my other open ticket while we''re here. :) I suspected that I made a Bad Choice when I did full virt on the RHEL5 systems instead of paravirt. I''ve got five systems running full - is there any good way to migrate a system from full virt to paravirt? -- tim boyer tim@denmantire.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Richard Blocker
2007-Jun-22 12:55 UTC
Re: [Xen-users] Re: Xen guest domain freezes when prelink runs
Ahhh, (deep sigh of relief) this is good to know. Thanks for the response, Dan. I disabled prelink and the VM has been running great for 24 hours, but I was afraid something else might be lurking to reach out and bite me. I need the full virtualization for better security and portability in this instance, and performance has not been an issue (so far). Rick Daniel P. Berrange wrote:> On Thu, Jun 21, 2007 at 08:33:51PM -0400, Tim Boyer wrote: >> On Thu, 21 Jun 2007 09:09:39 -0500, Richard Blocker <rblocker@uchicago.edu> >> wrote: >> >>> I''m new to running Xen, so maybe I missed something, but this is a >>> puzzling problem that I didn''t find on any of the Xen lists. >>> >>> I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up >>> to date as of today) and I have a single guest domain HVM also running >>> RHEL 5. Whenever the guest OS runs the initial prelink job defined in >>> /etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at >>> 100% and the system stops responding. It never resumes (at least not for >>> 12 hours). I can reboot the guest domain from the host machine, and it >>> recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to >>> see if it was just the load, but it was fine while that ran. If I run >>> the prelink command manually, it immediately freezes. >>> >>> For the record, the hardware is a dual quad core Xeon system with 8GB of >>> memory. The guest HVM uses a single CPU with 512MB of memory allocated. >>> >>> Has anyone else seen this? >> Hah! Someone else with this... >> >> I''ve got a trouble report into RH on this one. It''s not just prelink; I think >> it''s tied to rpm. I can bring the system down doing a rpm -Va, or a sysreport >> without the -norpm switch. Also a dual quad core, with 16gb, and 1gb on the >> guest. > > Actually it is prelink - rpm -Va will call out the prelink libraries when > verifying IIRC. Anyway, this is ultimately a hypervisor bug in Xen 3.0.3 > which should be fixed in the Xen 3.1.0 hypervisor. So should be working > come RHEL-5.1 updates. > > Seriously though, you really really really don''t want to run any OS in > full-virt if its capable of paravirt. You''ll get x10 -> x100 the I/O > performance if you use paravirt and be able to scale up the number of > guests per host much better. So I''d recommend using RHEL-5 paravirt > at which point you won''t see the HVM bug anymore either... > > Dan.-- Richard Blocker Information Systems Manager Ecology and Evolution Organismal Biology and Anatomy University of Chicago Zoology 003 1101 E. 57th Street Chicago, IL 60637 rblocker@uchicago.edu telephone: 773-702-5135 facsimile: 773-702-9740 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Richard Blocker
2007-Jun-22 14:35 UTC
Re: [Xen-users] Re: Xen guest domain freezes when prelink runs
So, 24 hours plus one and something did reach out and bite me. I got another 100% CPU usage and frozen guest HVM. This time there was no prelink running. Is full virtualization this full of bugs? Richard Blocker wrote:> Ahhh, (deep sigh of relief) this is good to know. Thanks for the > response, Dan. I disabled prelink and the VM has been running great for > 24 hours, but I was afraid something else might be lurking to reach out > and bite me. I need the full virtualization for better security and > portability in this instance, and performance has not been an issue (so > far). > > Rick > > Daniel P. Berrange wrote: >> On Thu, Jun 21, 2007 at 08:33:51PM -0400, Tim Boyer wrote: >>> On Thu, 21 Jun 2007 09:09:39 -0500, Richard Blocker <rblocker@uchicago.edu> >>> wrote: >>> >>>> I''m new to running Xen, so maybe I missed something, but this is a >>>> puzzling problem that I didn''t find on any of the Xen lists. >>>> >>>> I''m running the Xen 3.0.3 that comes with RedHat EL 5 (all stock, all up >>>> to date as of today) and I have a single guest domain HVM also running >>>> RHEL 5. Whenever the guest OS runs the initial prelink job defined in >>>> /etc/cron.daily (/usr/sbin/prelink -av -mR -f) the guest CPU pegs at >>>> 100% and the system stops responding. It never resumes (at least not for >>>> 12 hours). I can reboot the guest domain from the host machine, and it >>>> recovers fine, until prelink runs. I even ran cpu-burnin on the HVM to >>>> see if it was just the load, but it was fine while that ran. If I run >>>> the prelink command manually, it immediately freezes. >>>> >>>> For the record, the hardware is a dual quad core Xeon system with 8GB of >>>> memory. The guest HVM uses a single CPU with 512MB of memory allocated. >>>> >>>> Has anyone else seen this? >>> Hah! Someone else with this... >>> >>> I''ve got a trouble report into RH on this one. It''s not just prelink; I think >>> it''s tied to rpm. I can bring the system down doing a rpm -Va, or a sysreport >>> without the -norpm switch. Also a dual quad core, with 16gb, and 1gb on the >>> guest. >> Actually it is prelink - rpm -Va will call out the prelink libraries when >> verifying IIRC. Anyway, this is ultimately a hypervisor bug in Xen 3.0.3 >> which should be fixed in the Xen 3.1.0 hypervisor. So should be working >> come RHEL-5.1 updates. >> >> Seriously though, you really really really don''t want to run any OS in >> full-virt if its capable of paravirt. You''ll get x10 -> x100 the I/O >> performance if you use paravirt and be able to scale up the number of >> guests per host much better. So I''d recommend using RHEL-5 paravirt >> at which point you won''t see the HVM bug anymore either... >> >> Dan. > >-- Richard Blocker Information Systems Manager Ecology and Evolution Organismal Biology and Anatomy University of Chicago Zoology 003 1101 E. 57th Street Chicago, IL 60637 rblocker@uchicago.edu telephone: 773-702-5135 facsimile: 773-702-9740 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users