Xen devel - Dec 2009 - latest xen-unstable fails to boot on Dell D630 (likely hpet/Cstate problem)

In this thread, I observed that I was unable to
provoke deep C state (C3) on my Dell D630, which has
a Intel Merom (dual-core laptop) processor.  At that
time, when I tried enabling hpetbroadcast, dom0 boot failed.

http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01027.html

As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
default installation run /sbin/hwclock, which IIRC takes
the RTC away from Xen and gives it to dom0.  Since the
Xen hpet emulation does not do RTC emulation, bad things
then happen when a deep Cstate is entered (dom0 apparently
never wakes up).  I think Ke Yu has also reproduced this problem.

Sometime in the last few weeks, some patch in xen-unstable
apparently changed some defaults and xen-unstable will
no longer boot with this processor/dom0, with or without
hpetbroadcast on the Xen command line.  However, specifying
max_cstate=2 on the Xen command line allows a successful
dom0 boot, so I suspect the problem is the same (or at
least very similar).

I did a quick scan for hpet changes and found c/s 20497,
but backing it out made no difference.

I have a workaround for now, but since it is likely that
many customers (including all of Oracle''s OVS customers)
use a RHEL5-based dom0 boot sequence, and Merom processors
work fine otherwise, it would be nice to get this identified
and fixed before 4.0.

Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>-----Original Message-----
>In this thread, I observed that I was unable to
>provoke deep C state (C3) on my Dell D630, which has
>a Intel Merom (dual-core laptop) processor.  At that
>time, when I tried enabling hpetbroadcast, dom0 boot failed.
>
>http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01027.html
>
>As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
>default installation run /sbin/hwclock, which IIRC takes
>the RTC away from Xen and gives it to dom0.  Since the
>Xen hpet emulation does not do RTC emulation, bad things
>then happen when a deep Cstate is entered (dom0 apparently
>never wakes up).  I think Ke Yu has also reproduced this problem.
>
>Sometime in the last few weeks, some patch in xen-unstable
>apparently changed some defaults and xen-unstable will
>no longer boot with this processor/dom0, with or without
>hpetbroadcast on the Xen command line.  However, specifying
>max_cstate=2 on the Xen command line allows a successful
>dom0 boot, so I suspect the problem is the same (or at
>least very similar).
>
>I did a quick scan for hpet changes and found c/s 20497,
>but backing it out made no difference.
>
>I have a workaround for now, but since it is likely that
>many customers (including all of Oracle's OVS customers)
>use a RHEL5-based dom0 boot sequence, and Merom processors
>work fine otherwise, it would be nice to get this identified
>and fixed before 4.0.
Let's firstly figure out which component the issue resides. 

Firstly, in the default boot (i.e. without specifying max_cstate=2), when dom0
hangs, is xen still alive, E.g. can Xen response to three Ctrl-'A' in
serial?

If only dom0 hangs, it is probably that RTC malfunction make incorrect dom0 time
and lead dom0 fail to boot. Then RTC emulation in hypervisor should fix this
issue.

If Xen also hangs, it should be another bug, i.e. hpet broadcast does not wake
up CPU in deep C states. in this case, if convenient, could you help to do some
bisect to see which cset cause this bug?

Best Regards
Ke

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> case, if convenient, could you help to do some bisect to see 
> which cset cause this bug?
I can do this, but because it is often no longer easy to
bisect Xen because of interdependencies with other
components, I was hoping that Keir or you or someone might
have some idea of what changeset might have caused the regression.
But I''ll give bisecting a try.
> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> Xen response to three Ctrl-''A'' in serial?
Unfortunately, I can''t seem to get a Xen console working on
the Merom machine, and the problem can''t be reproduced on
my other machine where the Xen console is working (because
Conroe doesn''t support deep C).
> -----Original Message-----
> From: Yu, Ke [mailto:ke.yu@intel.com]
> Sent: Tuesday, December 08, 2009 12:08 AM
> To: Dan Magenheimer; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> >-----Original Message-----
> >In this thread, I observed that I was unable to
> >provoke deep C state (C3) on my Dell D630, which has
> >a Intel Merom (dual-core laptop) processor.  At that
> >time, when I tried enabling hpetbroadcast, dom0 boot failed.
> >
> >http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> g01027.html
> >
> >As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
> >default installation run /sbin/hwclock, which IIRC takes
> >the RTC away from Xen and gives it to dom0.  Since the
> >Xen hpet emulation does not do RTC emulation, bad things
> >then happen when a deep Cstate is entered (dom0 apparently
> >never wakes up).  I think Ke Yu has also reproduced this problem.
> >
> >Sometime in the last few weeks, some patch in xen-unstable
> >apparently changed some defaults and xen-unstable will
> >no longer boot with this processor/dom0, with or without
> >hpetbroadcast on the Xen command line.  However, specifying
> >max_cstate=2 on the Xen command line allows a successful
> >dom0 boot, so I suspect the problem is the same (or at
> >least very similar).
> >
> >I did a quick scan for hpet changes and found c/s 20497,
> >but backing it out made no difference.
> >
> >I have a workaround for now, but since it is likely that
> >many customers (including all of Oracle''s OVS customers)
> >use a RHEL5-based dom0 boot sequence, and Merom processors
> >work fine otherwise, it would be nice to get this identified
> >and fixed before 4.0.
> 
> Let''s firstly figure out which component the issue resides. 
> 
> Firstly, in the default boot (i.e. without specifying 
> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> Xen response to three Ctrl-''A'' in serial?
> 
> If only dom0 hangs, it is probably that RTC malfunction make 
> incorrect dom0 time and lead dom0 fail to boot. Then RTC 
> emulation in hypervisor should fix this issue.
> 
> If Xen also hangs, it should be another bug, i.e. hpet 
> broadcast does not wake up CPU in deep C states. in this 
> case, if convenient, could you help to do some bisect to see 
> which cset cause this bug?
> 
> Best Regards
> Ke
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> But I''ll give bisecting a try.
Looks like the problem has been around for awhile.  It appears
the problem starts at c/s 20073.  Xiantao cc''ed since 20073
was his patch.

20070 boots OK without max_cstate=2

20072 boots most of the way without max_cstate=2 but crashes
      before a login prompt (when xend is starting I think)

20073 FAILS to boot without max_cstate=2 but crashes
      before a login prompt

20082 FAILS to boot without max_cstate=2 but crashes
      before a login prompt with max_cstate=2

20143 FAILS to boot without max_cstate=2 but boots OK
      with max_cstate=2

Note that I have NOT bisected tools, just the hypervisor
so the crashes are likely due to a newer xend failing on
an older hypervisor (which is irrelevant to this problem).
> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Tuesday, December 08, 2009 10:42 AM
> To: Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> > case, if convenient, could you help to do some bisect to see 
> > which cset cause this bug?
> 
> I can do this, but because it is often no longer easy to
> bisect Xen because of interdependencies with other
> components, I was hoping that Keir or you or someone might
> have some idea of what changeset might have caused the regression.
> But I''ll give bisecting a try.
> 
> > max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> > Xen response to three Ctrl-''A'' in serial?
> 
> Unfortunately, I can''t seem to get a Xen console working on
> the Merom machine, and the problem can''t be reproduced on
> my other machine where the Xen console is working (because
> Conroe doesn''t support deep C).
> 
> > -----Original Message-----
> > From: Yu, Ke [mailto:ke.yu@intel.com]
> > Sent: Tuesday, December 08, 2009 12:08 AM
> > To: Dan Magenheimer; Xen-Devel (E-mail)
> > Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> > hpet/Cstate problem)
> > 
> > 
> > >-----Original Message-----
> > >In this thread, I observed that I was unable to
> > >provoke deep C state (C3) on my Dell D630, which has
> > >a Intel Merom (dual-core laptop) processor.  At that
> > >time, when I tried enabling hpetbroadcast, dom0 boot failed.
> > >
> > >http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> > g01027.html
> > >
> > >As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
> > >default installation run /sbin/hwclock, which IIRC takes
> > >the RTC away from Xen and gives it to dom0.  Since the
> > >Xen hpet emulation does not do RTC emulation, bad things
> > >then happen when a deep Cstate is entered (dom0 apparently
> > >never wakes up).  I think Ke Yu has also reproduced this problem.
> > >
> > >Sometime in the last few weeks, some patch in xen-unstable
> > >apparently changed some defaults and xen-unstable will
> > >no longer boot with this processor/dom0, with or without
> > >hpetbroadcast on the Xen command line.  However, specifying
> > >max_cstate=2 on the Xen command line allows a successful
> > >dom0 boot, so I suspect the problem is the same (or at
> > >least very similar).
> > >
> > >I did a quick scan for hpet changes and found c/s 20497,
> > >but backing it out made no difference.
> > >
> > >I have a workaround for now, but since it is likely that
> > >many customers (including all of Oracle''s OVS customers)
> > >use a RHEL5-based dom0 boot sequence, and Merom processors
> > >work fine otherwise, it would be nice to get this identified
> > >and fixed before 4.0.
> > 
> > Let''s firstly figure out which component the issue resides. 
> > 
> > Firstly, in the default boot (i.e. without specifying 
> > max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> > Xen response to three Ctrl-''A'' in serial?
> > 
> > If only dom0 hangs, it is probably that RTC malfunction make 
> > incorrect dom0 time and lead dom0 fail to boot. Then RTC 
> > emulation in hypervisor should fix this issue.
> > 
> > If Xen also hangs, it should be another bug, i.e. hpet 
> > broadcast does not wake up CPU in deep C states. in this 
> > case, if convenient, could you help to do some bisect to see 
> > which cset cause this bug?
> > 
> > Best Regards
> > Ke
> > 
> > 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>       before a login prompt (when xend is starting I think)
>
> Note that I have NOT bisected tools, just the hypervisor
> so the crashes are likely due to a newer xend failing on
> an older hypervisor (which is irrelevant to this problem).
I realized I just had to disable xend to test this and it
confirmed my hypothesis.

So 20072 boots fine without max_cstate=2, while 20073
fails to boot unless max_cstate=2 is a Xen boot parameter.
> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Tuesday, December 08, 2009 4:25 PM
> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> Cc: xiantao.zhang@intel.com
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> > But I''ll give bisecting a try.
> 
> Looks like the problem has been around for awhile.  It appears
> the problem starts at c/s 20073.  Xiantao cc''ed since 20073
> was his patch.
> 
> 20070 boots OK without max_cstate=2
> 
> 20072 boots most of the way without max_cstate=2 but crashes
>       before a login prompt (when xend is starting I think)
> 
> 20073 FAILS to boot without max_cstate=2 but crashes
>       before a login prompt
> 
> 20082 FAILS to boot without max_cstate=2 but crashes
>       before a login prompt with max_cstate=2
> 
> 20143 FAILS to boot without max_cstate=2 but boots OK
>       with max_cstate=2
> 
> Note that I have NOT bisected tools, just the hypervisor
> so the crashes are likely due to a newer xend failing on
> an older hypervisor (which is irrelevant to this problem).
> 
> > -----Original Message-----
> > From: Dan Magenheimer 
> > Sent: Tuesday, December 08, 2009 10:42 AM
> > To: Yu, Ke; Xen-Devel (E-mail)
> > Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> > hpet/Cstate problem)
> > 
> > 
> > > case, if convenient, could you help to do some bisect to see 
> > > which cset cause this bug?
> > 
> > I can do this, but because it is often no longer easy to
> > bisect Xen because of interdependencies with other
> > components, I was hoping that Keir or you or someone might
> > have some idea of what changeset might have caused the regression.
> > But I''ll give bisecting a try.
> > 
> > > max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> > > Xen response to three Ctrl-''A'' in serial?
> > 
> > Unfortunately, I can''t seem to get a Xen console working on
> > the Merom machine, and the problem can''t be reproduced on
> > my other machine where the Xen console is working (because
> > Conroe doesn''t support deep C).
> > 
> > > -----Original Message-----
> > > From: Yu, Ke [mailto:ke.yu@intel.com]
> > > Sent: Tuesday, December 08, 2009 12:08 AM
> > > To: Dan Magenheimer; Xen-Devel (E-mail)
> > > Subject: RE: latest xen-unstable fails to boot on Dell 
> D630 (likely
> > > hpet/Cstate problem)
> > > 
> > > 
> > > >-----Original Message-----
> > > >In this thread, I observed that I was unable to
> > > >provoke deep C state (C3) on my Dell D630, which has
> > > >a Intel Merom (dual-core laptop) processor.  At that
> > > >time, when I tried enabling hpetbroadcast, dom0 boot failed.
> > > >
> > > >http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> > > g01027.html
> > > >
> > > >As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
> > > >default installation run /sbin/hwclock, which IIRC takes
> > > >the RTC away from Xen and gives it to dom0.  Since the
> > > >Xen hpet emulation does not do RTC emulation, bad things
> > > >then happen when a deep Cstate is entered (dom0 apparently
> > > >never wakes up).  I think Ke Yu has also reproduced this
problem.
> > > >
> > > >Sometime in the last few weeks, some patch in xen-unstable
> > > >apparently changed some defaults and xen-unstable will
> > > >no longer boot with this processor/dom0, with or without
> > > >hpetbroadcast on the Xen command line.  However, specifying
> > > >max_cstate=2 on the Xen command line allows a successful
> > > >dom0 boot, so I suspect the problem is the same (or at
> > > >least very similar).
> > > >
> > > >I did a quick scan for hpet changes and found c/s 20497,
> > > >but backing it out made no difference.
> > > >
> > > >I have a workaround for now, but since it is likely that
> > > >many customers (including all of Oracle''s OVS
customers)
> > > >use a RHEL5-based dom0 boot sequence, and Merom processors
> > > >work fine otherwise, it would be nice to get this identified
> > > >and fixed before 4.0.
> > > 
> > > Let''s firstly figure out which component the issue
resides.
> > > 
> > > Firstly, in the default boot (i.e. without specifying 
> > > max_cstate=2), when dom0 hangs, is xen still alive, E.g. can 
> > > Xen response to three Ctrl-''A'' in serial?
> > > 
> > > If only dom0 hangs, it is probably that RTC malfunction make 
> > > incorrect dom0 time and lead dom0 fail to boot. Then RTC 
> > > emulation in hypervisor should fix this issue.
> > > 
> > > If Xen also hangs, it should be another bug, i.e. hpet 
> > > broadcast does not wake up CPU in deep C states. in this 
> > > case, if convenient, could you help to do some bisect to see 
> > > which cset cause this bug?
> > > 
> > > Best Regards
> > > Ke
> > > 
> > > 
> >
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan, 
Don''t use Cset20073 for testing separately, since it needs two minor
fixes check-ined by the Cset #20093 and #20149.  Except this, Keir also has a
typo in Cset #20076 fixed by Cset #20092. In addition, one serious issue is also
introduced in #Cset20084 which is fixed in Cset #20140.  I remembered Pod also
has issues which can crash hypervisor before Cset #20100. Thus, it is too hard
to identify this issue through bisect before #Cset20149, since these issues are
introduced and fixed crossedly.   Certainly, if you want to test Cset #20073,
you at least have to apply the Cset#20093 and #20149 on top of it.  :)
Xiantao


Dan Magenheimer wrote:
>> But I''ll give bisecting a try.
> 
> Looks like the problem has been around for awhile.  It appears
> the problem starts at c/s 20073.  Xiantao cc''ed since 20073
> was his patch.
> 
> 20070 boots OK without max_cstate=2
> 
> 20072 boots most of the way without max_cstate=2 but crashes
>       before a login prompt (when xend is starting I think)
> 
> 20073 FAILS to boot without max_cstate=2 but crashes
>       before a login prompt
> 
> 20082 FAILS to boot without max_cstate=2 but crashes
>       before a login prompt with max_cstate=2
> 
> 20143 FAILS to boot without max_cstate=2 but boots OK
>       with max_cstate=2
> 
> Note that I have NOT bisected tools, just the hypervisor
> so the crashes are likely due to a newer xend failing on
> an older hypervisor (which is irrelevant to this problem).
> 
>> -----Original Message-----
>> From: Dan Magenheimer
>> Sent: Tuesday, December 08, 2009 10:42 AM
>> To: Yu, Ke; Xen-Devel (E-mail)
>> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
>> hpet/Cstate problem) 
>> 
>> 
>>> case, if convenient, could you help to do some bisect to see
>>> which cset cause this bug?
>> 
>> I can do this, but because it is often no longer easy to
>> bisect Xen because of interdependencies with other
>> components, I was hoping that Keir or you or someone might
>> have some idea of what changeset might have caused the regression.
>> But I''ll give bisecting a try.
>> 
>>> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can
>>> Xen response to three Ctrl-''A'' in serial?
>> 
>> Unfortunately, I can''t seem to get a Xen console working on
>> the Merom machine, and the problem can''t be reproduced on
>> my other machine where the Xen console is working (because
>> Conroe doesn''t support deep C).
>> 
>>> -----Original Message-----
>>> From: Yu, Ke [mailto:ke.yu@intel.com]
>>> Sent: Tuesday, December 08, 2009 12:08 AM
>>> To: Dan Magenheimer; Xen-Devel (E-mail)
>>> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
>>> hpet/Cstate problem) 
>>> 
>>> 
>>>> -----Original Message-----
>>>> In this thread, I observed that I was unable to
>>>> provoke deep C state (C3) on my Dell D630, which has
>>>> a Intel Merom (dual-core laptop) processor.  At that
>>>> time, when I tried enabling hpetbroadcast, dom0 boot failed.
>>>> 
>>>> http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
>>>> g01027.html 
>>>> 
>>>> As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
>>>> default installation run /sbin/hwclock, which IIRC takes
>>>> the RTC away from Xen and gives it to dom0.  Since the
>>>> Xen hpet emulation does not do RTC emulation, bad things
>>>> then happen when a deep Cstate is entered (dom0 apparently
>>>> never wakes up).  I think Ke Yu has also reproduced this
problem.
>>>> 
>>>> Sometime in the last few weeks, some patch in xen-unstable
>>>> apparently changed some defaults and xen-unstable will
>>>> no longer boot with this processor/dom0, with or without
>>>> hpetbroadcast on the Xen command line.  However, specifying
>>>> max_cstate=2 on the Xen command line allows a successful
>>>> dom0 boot, so I suspect the problem is the same (or at
>>>> least very similar).
>>>> 
>>>> I did a quick scan for hpet changes and found c/s 20497,
>>>> but backing it out made no difference.
>>>> 
>>>> I have a workaround for now, but since it is likely that
>>>> many customers (including all of Oracle''s OVS
customers)
>>>> use a RHEL5-based dom0 boot sequence, and Merom processors
>>>> work fine otherwise, it would be nice to get this identified
>>>> and fixed before 4.0.
>>> 
>>> Let''s firstly figure out which component the issue
resides.
>>> 
>>> Firstly, in the default boot (i.e. without specifying
>>> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can
>>> Xen response to three Ctrl-''A'' in serial?
>>> 
>>> If only dom0 hangs, it is probably that RTC malfunction make
>>> incorrect dom0 time and lead dom0 fail to boot. Then RTC
>>> emulation in hypervisor should fix this issue.
>>> 
>>> If Xen also hangs, it should be another bug, i.e. hpet
>>> broadcast does not wake up CPU in deep C states. in this
>>> case, if convenient, could you help to do some bisect to see
>>> which cset cause this bug?
>>> 
>>> Best Regards
>>> Ke

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

FYI, 20073+20093+20149 boots properly and xend starts
WITH max_cstate=2, but dom0 FAILs to boot unless
max_cstate=2 is added as a Xen boot parameter.

So I still think something changed at 20073 that
causes Merom+RHEL5dom0 to fail to boot due to not
recovering from deep C-state (after dom0 runs
/sbin/hwclock ... Ke Yu knows how to reproduce
the problem).

Thanks,
Dan
> -----Original Message-----
> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> Sent: Tuesday, December 08, 2009 6:44 PM
> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> Dan, 
> Don''t use Cset20073 for testing separately, since it needs 
> two minor fixes check-ined by the Cset #20093 and #20149.  
> Except this, Keir also has a typo in Cset #20076 fixed by 
> Cset #20092. In addition, one serious issue is also 
> introduced in #Cset20084 which is fixed in Cset #20140.  I 
> remembered Pod also has issues which can crash hypervisor 
> before Cset #20100. Thus, it is too hard to identify this 
> issue through bisect before #Cset20149, since these issues 
> are introduced and fixed crossedly.   Certainly, if you want 
> to test Cset #20073, you at least have to apply the 
> Cset#20093 and #20149 on top of it.  :)   
> Xiantao
> 
> 
> Dan Magenheimer wrote:
> >> But I''ll give bisecting a try.
> > 
> > Looks like the problem has been around for awhile.  It appears
> > the problem starts at c/s 20073.  Xiantao cc''ed since 20073
> > was his patch.
> > 
> > 20070 boots OK without max_cstate=2
> > 
> > 20072 boots most of the way without max_cstate=2 but crashes
> >       before a login prompt (when xend is starting I think)
> > 
> > 20073 FAILS to boot without max_cstate=2 but crashes
> >       before a login prompt
> > 
> > 20082 FAILS to boot without max_cstate=2 but crashes
> >       before a login prompt with max_cstate=2
> > 
> > 20143 FAILS to boot without max_cstate=2 but boots OK
> >       with max_cstate=2
> > 
> > Note that I have NOT bisected tools, just the hypervisor
> > so the crashes are likely due to a newer xend failing on
> > an older hypervisor (which is irrelevant to this problem).
> > 
> >> -----Original Message-----
> >> From: Dan Magenheimer
> >> Sent: Tuesday, December 08, 2009 10:42 AM
> >> To: Yu, Ke; Xen-Devel (E-mail)
> >> Subject: RE: latest xen-unstable fails to boot on Dell D630
(likely
> >> hpet/Cstate problem) 
> >> 
> >> 
> >>> case, if convenient, could you help to do some bisect to see
> >>> which cset cause this bug?
> >> 
> >> I can do this, but because it is often no longer easy to
> >> bisect Xen because of interdependencies with other
> >> components, I was hoping that Keir or you or someone might
> >> have some idea of what changeset might have caused the regression.
> >> But I''ll give bisecting a try.
> >> 
> >>> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can
> >>> Xen response to three Ctrl-''A'' in serial?
> >> 
> >> Unfortunately, I can''t seem to get a Xen console working
on
> >> the Merom machine, and the problem can''t be reproduced on
> >> my other machine where the Xen console is working (because
> >> Conroe doesn''t support deep C).
> >> 
> >>> -----Original Message-----
> >>> From: Yu, Ke [mailto:ke.yu@intel.com]
> >>> Sent: Tuesday, December 08, 2009 12:08 AM
> >>> To: Dan Magenheimer; Xen-Devel (E-mail)
> >>> Subject: RE: latest xen-unstable fails to boot on Dell 
> D630 (likely
> >>> hpet/Cstate problem) 
> >>> 
> >>> 
> >>>> -----Original Message-----
> >>>> In this thread, I observed that I was unable to
> >>>> provoke deep C state (C3) on my Dell D630, which has
> >>>> a Intel Merom (dual-core laptop) processor.  At that
> >>>> time, when I tried enabling hpetbroadcast, dom0 boot
failed.
> >>>> 
> >>>>
http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> >>>> g01027.html 
> >>>> 
> >>>> As it turned out, all RHEL5-based (maybe RHEL4- also) dom0
> >>>> default installation run /sbin/hwclock, which IIRC takes
> >>>> the RTC away from Xen and gives it to dom0.  Since the
> >>>> Xen hpet emulation does not do RTC emulation, bad things
> >>>> then happen when a deep Cstate is entered (dom0 apparently
> >>>> never wakes up).  I think Ke Yu has also reproduced this
problem.
> >>>> 
> >>>> Sometime in the last few weeks, some patch in xen-unstable
> >>>> apparently changed some defaults and xen-unstable will
> >>>> no longer boot with this processor/dom0, with or without
> >>>> hpetbroadcast on the Xen command line.  However,
specifying
> >>>> max_cstate=2 on the Xen command line allows a successful
> >>>> dom0 boot, so I suspect the problem is the same (or at
> >>>> least very similar).
> >>>> 
> >>>> I did a quick scan for hpet changes and found c/s 20497,
> >>>> but backing it out made no difference.
> >>>> 
> >>>> I have a workaround for now, but since it is likely that
> >>>> many customers (including all of Oracle''s OVS
customers)
> >>>> use a RHEL5-based dom0 boot sequence, and Merom processors
> >>>> work fine otherwise, it would be nice to get this
identified
> >>>> and fixed before 4.0.
> >>> 
> >>> Let''s firstly figure out which component the issue
resides.
> >>> 
> >>> Firstly, in the default boot (i.e. without specifying
> >>> max_cstate=2), when dom0 hangs, is xen still alive, E.g. can
> >>> Xen response to three Ctrl-''A'' in serial?
> >>> 
> >>> If only dom0 hangs, it is probably that RTC malfunction make
> >>> incorrect dom0 time and lead dom0 fail to boot. Then RTC
> >>> emulation in hypervisor should fix this issue.
> >>> 
> >>> If Xen also hangs, it should be another bug, i.e. hpet
> >>> broadcast does not wake up CPU in deep C states. in this
> >>> case, if convenient, could you help to do some bisect to see
> >>> which cset cause this bug?
> >>> 
> >>> Best Regards
> >>> Ke
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dan Magenheimer wrote:
> FYI, 20073+20093+20149 boots properly and xend starts
> WITH max_cstate=2, but dom0 FAILs to boot unless
> max_cstate=2 is added as a Xen boot parameter. 
Could you attach the failure log ?  In addition, does this system have ioapic
support ? I think hpet doesn''t use MSI, right ?
Xiantao

> So I still think something changed at 20073 that
> causes Merom+RHEL5dom0 to fail to boot due to not
> recovering from deep C-state (after dom0 runs
> /sbin/hwclock ... Ke Yu knows how to reproduce
> the problem).
> 
> Thanks,
> Dan
> 
>> -----Original Message-----
>> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
>> Sent: Tuesday, December 08, 2009 6:44 PM
>> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
>> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
>> hpet/Cstate problem) 
>> 
>> 
>> Dan,
>> Don''t use Cset20073 for testing separately, since it needs
>> two minor fixes check-ined by the Cset #20093 and #20149.
>> Except this, Keir also has a typo in Cset #20076 fixed by
>> Cset #20092. In addition, one serious issue is also
>> introduced in #Cset20084 which is fixed in Cset #20140.  I
>> remembered Pod also has issues which can crash hypervisor
>> before Cset #20100. Thus, it is too hard to identify this
>> issue through bisect before #Cset20149, since these issues
>> are introduced and fixed crossedly.   Certainly, if you want
>> to test Cset #20073, you at least have to apply the
>> Cset#20093 and #20149 on top of it.  :)
>> Xiantao
>> 
>> 
>> Dan Magenheimer wrote:
>>>> But I''ll give bisecting a try.
>>> 
>>> Looks like the problem has been around for awhile.  It appears
>>> the problem starts at c/s 20073.  Xiantao cc''ed since
20073 was his
>>> patch. 
>>> 
>>> 20070 boots OK without max_cstate=2
>>> 
>>> 20072 boots most of the way without max_cstate=2 but crashes
>>>       before a login prompt (when xend is starting I think)
>>> 
>>> 20073 FAILS to boot without max_cstate=2 but crashes       before a
>>> login prompt 
>>> 
>>> 20082 FAILS to boot without max_cstate=2 but crashes
>>>       before a login prompt with max_cstate=2
>>> 
>>> 20143 FAILS to boot without max_cstate=2 but boots OK       with
>>> max_cstate=2 
>>> 
>>> Note that I have NOT bisected tools, just the hypervisor
>>> so the crashes are likely due to a newer xend failing on
>>> an older hypervisor (which is irrelevant to this problem).
>>> 
>>>> -----Original Message-----
>>>> From: Dan Magenheimer
>>>> Sent: Tuesday, December 08, 2009 10:42 AM
>>>> To: Yu, Ke; Xen-Devel (E-mail)
>>>> Subject: RE: latest xen-unstable fails to boot on Dell D630
>>>> (likely hpet/Cstate problem) 
>>>> 
>>>> 
>>>>> case, if convenient, could you help to do some bisect to
see
>>>>> which cset cause this bug?
>>>> 
>>>> I can do this, but because it is often no longer easy to
>>>> bisect Xen because of interdependencies with other
>>>> components, I was hoping that Keir or you or someone might
>>>> have some idea of what changeset might have caused the
regression.
>>>> But I''ll give bisecting a try.
>>>> 
>>>>> max_cstate=2), when dom0 hangs, is xen still alive, E.g.
can
>>>>> Xen response to three Ctrl-''A'' in serial?
>>>> 
>>>> Unfortunately, I can''t seem to get a Xen console
working on
>>>> the Merom machine, and the problem can''t be reproduced
on
>>>> my other machine where the Xen console is working (because
>>>> Conroe doesn''t support deep C).
>>>> 
>>>>> -----Original Message-----
>>>>> From: Yu, Ke [mailto:ke.yu@intel.com]
>>>>> Sent: Tuesday, December 08, 2009 12:08 AM
>>>>> To: Dan Magenheimer; Xen-Devel (E-mail)
>>>>> Subject: RE: latest xen-unstable fails to boot on Dell D630
>>>>> (likely hpet/Cstate problem) 
>>>>> 
>>>>> 
>>>>>> -----Original Message-----
>>>>>> In this thread, I observed that I was unable to
>>>>>> provoke deep C state (C3) on my Dell D630, which has
>>>>>> a Intel Merom (dual-core laptop) processor.  At that
>>>>>> time, when I tried enabling hpetbroadcast, dom0 boot
failed.
>>>>>> 
>>>>>>
http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
>>>>>> g01027.html 
>>>>>> 
>>>>>> As it turned out, all RHEL5-based (maybe RHEL4- also)
dom0
>>>>>> default installation run /sbin/hwclock, which IIRC
takes
>>>>>> the RTC away from Xen and gives it to dom0.  Since the
>>>>>> Xen hpet emulation does not do RTC emulation, bad
things
>>>>>> then happen when a deep Cstate is entered (dom0
apparently
>>>>>> never wakes up).  I think Ke Yu has also reproduced
this problem.
>>>>>> 
>>>>>> Sometime in the last few weeks, some patch in
xen-unstable
>>>>>> apparently changed some defaults and xen-unstable will
>>>>>> no longer boot with this processor/dom0, with or
without
>>>>>> hpetbroadcast on the Xen command line.  However,
specifying
>>>>>> max_cstate=2 on the Xen command line allows a
successful
>>>>>> dom0 boot, so I suspect the problem is the same (or at
>>>>>> least very similar).
>>>>>> 
>>>>>> I did a quick scan for hpet changes and found c/s
20497,
>>>>>> but backing it out made no difference.
>>>>>> 
>>>>>> I have a workaround for now, but since it is likely
that
>>>>>> many customers (including all of Oracle''s OVS
customers)
>>>>>> use a RHEL5-based dom0 boot sequence, and Merom
processors
>>>>>> work fine otherwise, it would be nice to get this
identified
>>>>>> and fixed before 4.0.
>>>>> 
>>>>> Let''s firstly figure out which component the issue
resides.
>>>>> 
>>>>> Firstly, in the default boot (i.e. without specifying
>>>>> max_cstate=2), when dom0 hangs, is xen still alive, E.g.
can
>>>>> Xen response to three Ctrl-''A'' in serial?
>>>>> 
>>>>> If only dom0 hangs, it is probably that RTC malfunction
make
>>>>> incorrect dom0 time and lead dom0 fail to boot. Then RTC
>>>>> emulation in hypervisor should fix this issue.
>>>>> 
>>>>> If Xen also hangs, it should be another bug, i.e. hpet
>>>>> broadcast does not wake up CPU in deep C states. in this
>>>>> case, if convenient, could you help to do some bisect to
see
>>>>> which cset cause this bug? 
>>>>> 
>>>>> Best Regards
>>>>> Ke

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Could you attach the failure log ?
I can''t get any failure logs because dom0 fails to boot.
The failure conditions are the same as described
here:

http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01027.html 

However, I have attached the xm dmesg output from
a successful boot (with max_cstate=2).
> In addition, does this system have ioapic support ?
I think so.  See attached log.
> I think hpet doesn''t use MSI, right ? 
I don''t think so.

Dan
> -----Original Message-----
> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> Sent: Tuesday, December 08, 2009 9:40 PM
> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> Dan Magenheimer wrote:
> > FYI, 20073+20093+20149 boots properly and xend starts
> > WITH max_cstate=2, but dom0 FAILs to boot unless
> > max_cstate=2 is added as a Xen boot parameter. 
> 
> Could you attach the failure log ?  In addition, does this 
> system have ioapic support ? I think hpet doesn''t use MSI, right ?
> Xiantao
> 
> 
> > So I still think something changed at 20073 that
> > causes Merom+RHEL5dom0 to fail to boot due to not
> > recovering from deep C-state (after dom0 runs
> > /sbin/hwclock ... Ke Yu knows how to reproduce
> > the problem).
> > 
> > Thanks,
> > Dan
> > 
> >> -----Original Message-----
> >> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> >> Sent: Tuesday, December 08, 2009 6:44 PM
> >> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> >> Subject: RE: latest xen-unstable fails to boot on Dell D630
(likely
> >> hpet/Cstate problem) 
> >> 
> >> 
> >> Dan,
> >> Don''t use Cset20073 for testing separately, since it
needs
> >> two minor fixes check-ined by the Cset #20093 and #20149.
> >> Except this, Keir also has a typo in Cset #20076 fixed by
> >> Cset #20092. In addition, one serious issue is also
> >> introduced in #Cset20084 which is fixed in Cset #20140.  I
> >> remembered Pod also has issues which can crash hypervisor
> >> before Cset #20100. Thus, it is too hard to identify this
> >> issue through bisect before #Cset20149, since these issues
> >> are introduced and fixed crossedly.   Certainly, if you want
> >> to test Cset #20073, you at least have to apply the
> >> Cset#20093 and #20149 on top of it.  :)
> >> Xiantao
> >> 
> >> 
> >> Dan Magenheimer wrote:
> >>>> But I''ll give bisecting a try.
> >>> 
> >>> Looks like the problem has been around for awhile.  It appears
> >>> the problem starts at c/s 20073.  Xiantao cc''ed since
> 20073 was his
> >>> patch. 
> >>> 
> >>> 20070 boots OK without max_cstate=2
> >>> 
> >>> 20072 boots most of the way without max_cstate=2 but crashes
> >>>       before a login prompt (when xend is starting I think)
> >>> 
> >>> 20073 FAILS to boot without max_cstate=2 but crashes      
>  before a
> >>> login prompt 
> >>> 
> >>> 20082 FAILS to boot without max_cstate=2 but crashes
> >>>       before a login prompt with max_cstate=2
> >>> 
> >>> 20143 FAILS to boot without max_cstate=2 but boots OK      
with
> >>> max_cstate=2 
> >>> 
> >>> Note that I have NOT bisected tools, just the hypervisor
> >>> so the crashes are likely due to a newer xend failing on
> >>> an older hypervisor (which is irrelevant to this problem).
> >>> 
> >>>> -----Original Message-----
> >>>> From: Dan Magenheimer
> >>>> Sent: Tuesday, December 08, 2009 10:42 AM
> >>>> To: Yu, Ke; Xen-Devel (E-mail)
> >>>> Subject: RE: latest xen-unstable fails to boot on Dell
D630
> >>>> (likely hpet/Cstate problem) 
> >>>> 
> >>>> 
> >>>>> case, if convenient, could you help to do some bisect
to see
> >>>>> which cset cause this bug?
> >>>> 
> >>>> I can do this, but because it is often no longer easy to
> >>>> bisect Xen because of interdependencies with other
> >>>> components, I was hoping that Keir or you or someone might
> >>>> have some idea of what changeset might have caused the 
> regression.
> >>>> But I''ll give bisecting a try.
> >>>> 
> >>>>> max_cstate=2), when dom0 hangs, is xen still alive,
E.g. can
> >>>>> Xen response to three Ctrl-''A'' in
serial?
> >>>> 
> >>>> Unfortunately, I can''t seem to get a Xen console
working on
> >>>> the Merom machine, and the problem can''t be
reproduced on
> >>>> my other machine where the Xen console is working (because
> >>>> Conroe doesn''t support deep C).
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: Yu, Ke [mailto:ke.yu@intel.com]
> >>>>> Sent: Tuesday, December 08, 2009 12:08 AM
> >>>>> To: Dan Magenheimer; Xen-Devel (E-mail)
> >>>>> Subject: RE: latest xen-unstable fails to boot on Dell
D630
> >>>>> (likely hpet/Cstate problem) 
> >>>>> 
> >>>>> 
> >>>>>> -----Original Message-----
> >>>>>> In this thread, I observed that I was unable to
> >>>>>> provoke deep C state (C3) on my Dell D630, which
has
> >>>>>> a Intel Merom (dual-core laptop) processor.  At
that
> >>>>>> time, when I tried enabling hpetbroadcast, dom0
boot failed.
> >>>>>> 
> >>>>>>
http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> >>>>>> g01027.html 
> >>>>>> 
> >>>>>> As it turned out, all RHEL5-based (maybe RHEL4-
also) dom0
> >>>>>> default installation run /sbin/hwclock, which IIRC
takes
> >>>>>> the RTC away from Xen and gives it to dom0.  Since
the
> >>>>>> Xen hpet emulation does not do RTC emulation, bad
things
> >>>>>> then happen when a deep Cstate is entered (dom0
apparently
> >>>>>> never wakes up).  I think Ke Yu has also
reproduced
> this problem.
> >>>>>> 
> >>>>>> Sometime in the last few weeks, some patch in
xen-unstable
> >>>>>> apparently changed some defaults and xen-unstable
will
> >>>>>> no longer boot with this processor/dom0, with or
without
> >>>>>> hpetbroadcast on the Xen command line.  However,
specifying
> >>>>>> max_cstate=2 on the Xen command line allows a
successful
> >>>>>> dom0 boot, so I suspect the problem is the same
(or at
> >>>>>> least very similar).
> >>>>>> 
> >>>>>> I did a quick scan for hpet changes and found c/s
20497,
> >>>>>> but backing it out made no difference.
> >>>>>> 
> >>>>>> I have a workaround for now, but since it is
likely that
> >>>>>> many customers (including all of Oracle''s
OVS customers)
> >>>>>> use a RHEL5-based dom0 boot sequence, and Merom
processors
> >>>>>> work fine otherwise, it would be nice to get this
identified
> >>>>>> and fixed before 4.0.
> >>>>> 
> >>>>> Let''s firstly figure out which component the
issue resides.
> >>>>> 
> >>>>> Firstly, in the default boot (i.e. without specifying
> >>>>> max_cstate=2), when dom0 hangs, is xen still alive,
E.g. can
> >>>>> Xen response to three Ctrl-''A'' in
serial?
> >>>>> 
> >>>>> If only dom0 hangs, it is probably that RTC
malfunction make
> >>>>> incorrect dom0 time and lead dom0 fail to boot. Then
RTC
> >>>>> emulation in hypervisor should fix this issue.
> >>>>> 
> >>>>> If Xen also hangs, it should be another bug, i.e. hpet
> >>>>> broadcast does not wake up CPU in deep C states. in
this
> >>>>> case, if convenient, could you help to do some bisect
to see
> >>>>> which cset cause this bug? 
> >>>>> 
> >>>>> Best Regards
> >>>>> Ke
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, Dan 
    We still can''t reproduce this failure locally, even with Merom
laptop. Do you have the dock with your Dell 630, and I think the dock should
have the serial port support, and maybe you can get the failure log through it. 
If we can get the failure log, it should be helpful to identify this issue. 
Also I have analyzed the Cset #20072 and Cst20073, and have no any clue which
can lead to this issue.  In addition, I also talked with Ke, he said he could
reproduce another issue related to hwclock, but for this issue, he also
can''t catch it in any platforms.  :(
Thanks!
Xiantao

-----Original Message-----
From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
Sent: Wednesday, December 09, 2009 11:50 PM
To: Zhang, Xiantao; Yu, Ke; Xen-Devel (E-mail)
Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely hpet/Cstate
problem)
> Could you attach the failure log ?
I can''t get any failure logs because dom0 fails to boot.
The failure conditions are the same as described
here:

http://lists.xensource.com/archives/html/xen-devel/2009-10/msg01027.html 

However, I have attached the xm dmesg output from
a successful boot (with max_cstate=2).
> In addition, does this system have ioapic support ?
I think so.  See attached log.
> I think hpet doesn''t use MSI, right ? 
I don''t think so.

Dan
> -----Original Message-----
> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> Sent: Tuesday, December 08, 2009 9:40 PM
> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> Dan Magenheimer wrote:
> > FYI, 20073+20093+20149 boots properly and xend starts
> > WITH max_cstate=2, but dom0 FAILs to boot unless
> > max_cstate=2 is added as a Xen boot parameter. 
> 
> Could you attach the failure log ?  In addition, does this 
> system have ioapic support ? I think hpet doesn''t use MSI, right ?
> Xiantao
> 
> 
> > So I still think something changed at 20073 that
> > causes Merom+RHEL5dom0 to fail to boot due to not
> > recovering from deep C-state (after dom0 runs
> > /sbin/hwclock ... Ke Yu knows how to reproduce
> > the problem).
> > 
> > Thanks,
> > Dan
> > 
> >> -----Original Message-----
> >> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> >> Sent: Tuesday, December 08, 2009 6:44 PM
> >> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> >> Subject: RE: latest xen-unstable fails to boot on Dell D630
(likely
> >> hpet/Cstate problem) 
> >> 
> >> 
> >> Dan,
> >> Don''t use Cset20073 for testing separately, since it
needs
> >> two minor fixes check-ined by the Cset #20093 and #20149.
> >> Except this, Keir also has a typo in Cset #20076 fixed by
> >> Cset #20092. In addition, one serious issue is also
> >> introduced in #Cset20084 which is fixed in Cset #20140.  I
> >> remembered Pod also has issues which can crash hypervisor
> >> before Cset #20100. Thus, it is too hard to identify this
> >> issue through bisect before #Cset20149, since these issues
> >> are introduced and fixed crossedly.   Certainly, if you want
> >> to test Cset #20073, you at least have to apply the
> >> Cset#20093 and #20149 on top of it.  :)
> >> Xiantao
> >> 
> >> 
> >> Dan Magenheimer wrote:
> >>>> But I''ll give bisecting a try.
> >>> 
> >>> Looks like the problem has been around for awhile.  It appears
> >>> the problem starts at c/s 20073.  Xiantao cc''ed since
> 20073 was his
> >>> patch. 
> >>> 
> >>> 20070 boots OK without max_cstate=2
> >>> 
> >>> 20072 boots most of the way without max_cstate=2 but crashes
> >>>       before a login prompt (when xend is starting I think)
> >>> 
> >>> 20073 FAILS to boot without max_cstate=2 but crashes      
>  before a
> >>> login prompt 
> >>> 
> >>> 20082 FAILS to boot without max_cstate=2 but crashes
> >>>       before a login prompt with max_cstate=2
> >>> 
> >>> 20143 FAILS to boot without max_cstate=2 but boots OK      
with
> >>> max_cstate=2 
> >>> 
> >>> Note that I have NOT bisected tools, just the hypervisor
> >>> so the crashes are likely due to a newer xend failing on
> >>> an older hypervisor (which is irrelevant to this problem).
> >>> 
> >>>> -----Original Message-----
> >>>> From: Dan Magenheimer
> >>>> Sent: Tuesday, December 08, 2009 10:42 AM
> >>>> To: Yu, Ke; Xen-Devel (E-mail)
> >>>> Subject: RE: latest xen-unstable fails to boot on Dell
D630
> >>>> (likely hpet/Cstate problem) 
> >>>> 
> >>>> 
> >>>>> case, if convenient, could you help to do some bisect
to see
> >>>>> which cset cause this bug?
> >>>> 
> >>>> I can do this, but because it is often no longer easy to
> >>>> bisect Xen because of interdependencies with other
> >>>> components, I was hoping that Keir or you or someone might
> >>>> have some idea of what changeset might have caused the 
> regression.
> >>>> But I''ll give bisecting a try.
> >>>> 
> >>>>> max_cstate=2), when dom0 hangs, is xen still alive,
E.g. can
> >>>>> Xen response to three Ctrl-''A'' in
serial?
> >>>> 
> >>>> Unfortunately, I can''t seem to get a Xen console
working on
> >>>> the Merom machine, and the problem can''t be
reproduced on
> >>>> my other machine where the Xen console is working (because
> >>>> Conroe doesn''t support deep C).
> >>>> 
> >>>>> -----Original Message-----
> >>>>> From: Yu, Ke [mailto:ke.yu@intel.com]
> >>>>> Sent: Tuesday, December 08, 2009 12:08 AM
> >>>>> To: Dan Magenheimer; Xen-Devel (E-mail)
> >>>>> Subject: RE: latest xen-unstable fails to boot on Dell
D630
> >>>>> (likely hpet/Cstate problem) 
> >>>>> 
> >>>>> 
> >>>>>> -----Original Message-----
> >>>>>> In this thread, I observed that I was unable to
> >>>>>> provoke deep C state (C3) on my Dell D630, which
has
> >>>>>> a Intel Merom (dual-core laptop) processor.  At
that
> >>>>>> time, when I tried enabling hpetbroadcast, dom0
boot failed.
> >>>>>> 
> >>>>>>
http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> >>>>>> g01027.html 
> >>>>>> 
> >>>>>> As it turned out, all RHEL5-based (maybe RHEL4-
also) dom0
> >>>>>> default installation run /sbin/hwclock, which IIRC
takes
> >>>>>> the RTC away from Xen and gives it to dom0.  Since
the
> >>>>>> Xen hpet emulation does not do RTC emulation, bad
things
> >>>>>> then happen when a deep Cstate is entered (dom0
apparently
> >>>>>> never wakes up).  I think Ke Yu has also
reproduced
> this problem.
> >>>>>> 
> >>>>>> Sometime in the last few weeks, some patch in
xen-unstable
> >>>>>> apparently changed some defaults and xen-unstable
will
> >>>>>> no longer boot with this processor/dom0, with or
without
> >>>>>> hpetbroadcast on the Xen command line.  However,
specifying
> >>>>>> max_cstate=2 on the Xen command line allows a
successful
> >>>>>> dom0 boot, so I suspect the problem is the same
(or at
> >>>>>> least very similar).
> >>>>>> 
> >>>>>> I did a quick scan for hpet changes and found c/s
20497,
> >>>>>> but backing it out made no difference.
> >>>>>> 
> >>>>>> I have a workaround for now, but since it is
likely that
> >>>>>> many customers (including all of Oracle''s
OVS customers)
> >>>>>> use a RHEL5-based dom0 boot sequence, and Merom
processors
> >>>>>> work fine otherwise, it would be nice to get this
identified
> >>>>>> and fixed before 4.0.
> >>>>> 
> >>>>> Let''s firstly figure out which component the
issue resides.
> >>>>> 
> >>>>> Firstly, in the default boot (i.e. without specifying
> >>>>> max_cstate=2), when dom0 hangs, is xen still alive,
E.g. can
> >>>>> Xen response to three Ctrl-''A'' in
serial?
> >>>>> 
> >>>>> If only dom0 hangs, it is probably that RTC
malfunction make
> >>>>> incorrect dom0 time and lead dom0 fail to boot. Then
RTC
> >>>>> emulation in hypervisor should fix this issue.
> >>>>> 
> >>>>> If Xen also hangs, it should be another bug, i.e. hpet
> >>>>> broadcast does not wake up CPU in deep C states. in
this
> >>>>> case, if convenient, could you help to do some bisect
to see
> >>>>> which cset cause this bug? 
> >>>>> 
> >>>>> Best Regards
> >>>>> Ke
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

OK, thanks for looking for the problem.  Since you
can''t reproduce it, it is likely a problem specific
to the Dell D630 motherboard or BIOS or HPET or something
like that.

Since I have a workaround (max_cstate=2), I will just
continue to use the workaround.

I don''t have a dock but if I get the serial port
working at some point, I will try to reproduce the
problem again.

Also, when Ke''s work on RTC emulation is completed,
let me know and I will give that a try.

Thanks,
Dan
> -----Original Message-----
> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> Sent: Sunday, December 13, 2009 1:26 AM
> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> hpet/Cstate problem)
> 
> 
> Hi, Dan 
>     We still can''t reproduce this failure locally, even with 
> Merom laptop. Do you have the dock with your Dell 630, and I 
> think the dock should have the serial port support, and maybe 
> you can get the failure log through it.  If we can get the 
> failure log, it should be helpful to identify this issue.  
> Also I have analyzed the Cset #20072 and Cst20073, and have 
> no any clue which can lead to this issue.  In addition, I 
> also talked with Ke, he said he could reproduce another issue 
> related to hwclock, but for this issue, he also can''t catch 
> it in any platforms.  :(
> Thanks!
> Xiantao
> 
> -----Original Message-----
> From: Dan Magenheimer [mailto:dan.magenheimer@oracle.com] 
> Sent: Wednesday, December 09, 2009 11:50 PM
> To: Zhang, Xiantao; Yu, Ke; Xen-Devel (E-mail)
> Subject: RE: latest xen-unstable fails to boot on Dell D630 
> (likely hpet/Cstate problem)
> 
> > Could you attach the failure log ?
> 
> I can''t get any failure logs because dom0 fails to boot.
> The failure conditions are the same as described
> here:
> 
> http://lists.xensource.com/archives/html/xen-devel/2009-10/msg
> 01027.html 
> 
> However, I have attached the xm dmesg output from
> a successful boot (with max_cstate=2).
> 
> > In addition, does this system have ioapic support ?
> 
> I think so.  See attached log.
> 
> > I think hpet doesn''t use MSI, right ? 
> 
> I don''t think so.
> 
> Dan
> 
> > -----Original Message-----
> > From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> > Sent: Tuesday, December 08, 2009 9:40 PM
> > To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> > Subject: RE: latest xen-unstable fails to boot on Dell D630 (likely
> > hpet/Cstate problem)
> > 
> > 
> > Dan Magenheimer wrote:
> > > FYI, 20073+20093+20149 boots properly and xend starts
> > > WITH max_cstate=2, but dom0 FAILs to boot unless
> > > max_cstate=2 is added as a Xen boot parameter. 
> > 
> > Could you attach the failure log ?  In addition, does this 
> > system have ioapic support ? I think hpet doesn''t use MSI,
right ?
> > Xiantao
> > 
> > 
> > > So I still think something changed at 20073 that
> > > causes Merom+RHEL5dom0 to fail to boot due to not
> > > recovering from deep C-state (after dom0 runs
> > > /sbin/hwclock ... Ke Yu knows how to reproduce
> > > the problem).
> > > 
> > > Thanks,
> > > Dan
> > > 
> > >> -----Original Message-----
> > >> From: Zhang, Xiantao [mailto:xiantao.zhang@intel.com]
> > >> Sent: Tuesday, December 08, 2009 6:44 PM
> > >> To: Dan Magenheimer; Yu, Ke; Xen-Devel (E-mail)
> > >> Subject: RE: latest xen-unstable fails to boot on Dell 
> D630 (likely
> > >> hpet/Cstate problem) 
> > >> 
> > >> 
> > >> Dan,
> > >> Don''t use Cset20073 for testing separately, since it
needs
> > >> two minor fixes check-ined by the Cset #20093 and #20149.
> > >> Except this, Keir also has a typo in Cset #20076 fixed by
> > >> Cset #20092. In addition, one serious issue is also
> > >> introduced in #Cset20084 which is fixed in Cset #20140.  I
> > >> remembered Pod also has issues which can crash hypervisor
> > >> before Cset #20100. Thus, it is too hard to identify this
> > >> issue through bisect before #Cset20149, since these issues
> > >> are introduced and fixed crossedly.   Certainly, if you want
> > >> to test Cset #20073, you at least have to apply the
> > >> Cset#20093 and #20149 on top of it.  :)
> > >> Xiantao
> > >> 
> > >> 
> > >> Dan Magenheimer wrote:
> > >>>> But I''ll give bisecting a try.
> > >>> 
> > >>> Looks like the problem has been around for awhile.  It
appears
> > >>> the problem starts at c/s 20073.  Xiantao cc''ed
since
> > 20073 was his
> > >>> patch. 
> > >>> 
> > >>> 20070 boots OK without max_cstate=2
> > >>> 
> > >>> 20072 boots most of the way without max_cstate=2 but
crashes
> > >>>       before a login prompt (when xend is starting I
think)
> > >>> 
> > >>> 20073 FAILS to boot without max_cstate=2 but crashes
> >  before a
> > >>> login prompt 
> > >>> 
> > >>> 20082 FAILS to boot without max_cstate=2 but crashes
> > >>>       before a login prompt with max_cstate=2
> > >>> 
> > >>> 20143 FAILS to boot without max_cstate=2 but boots OK    
with
> > >>> max_cstate=2 
> > >>> 
> > >>> Note that I have NOT bisected tools, just the hypervisor
> > >>> so the crashes are likely due to a newer xend failing on
> > >>> an older hypervisor (which is irrelevant to this
problem).
> > >>> 
> > >>>> -----Original Message-----
> > >>>> From: Dan Magenheimer
> > >>>> Sent: Tuesday, December 08, 2009 10:42 AM
> > >>>> To: Yu, Ke; Xen-Devel (E-mail)
> > >>>> Subject: RE: latest xen-unstable fails to boot on
Dell D630
> > >>>> (likely hpet/Cstate problem) 
> > >>>> 
> > >>>> 
> > >>>>> case, if convenient, could you help to do some
bisect to see
> > >>>>> which cset cause this bug?
> > >>>> 
> > >>>> I can do this, but because it is often no longer easy
to
> > >>>> bisect Xen because of interdependencies with other
> > >>>> components, I was hoping that Keir or you or someone
might
> > >>>> have some idea of what changeset might have caused
the
> > regression.
> > >>>> But I''ll give bisecting a try.
> > >>>> 
> > >>>>> max_cstate=2), when dom0 hangs, is xen still
alive, E.g. can
> > >>>>> Xen response to three Ctrl-''A''
in serial?
> > >>>> 
> > >>>> Unfortunately, I can''t seem to get a Xen
console working on
> > >>>> the Merom machine, and the problem can''t be
reproduced on
> > >>>> my other machine where the Xen console is working
(because
> > >>>> Conroe doesn''t support deep C).
> > >>>> 
> > >>>>> -----Original Message-----
> > >>>>> From: Yu, Ke [mailto:ke.yu@intel.com]
> > >>>>> Sent: Tuesday, December 08, 2009 12:08 AM
> > >>>>> To: Dan Magenheimer; Xen-Devel (E-mail)
> > >>>>> Subject: RE: latest xen-unstable fails to boot on
Dell D630
> > >>>>> (likely hpet/Cstate problem) 
> > >>>>> 
> > >>>>> 
> > >>>>>> -----Original Message-----
> > >>>>>> In this thread, I observed that I was unable
to
> > >>>>>> provoke deep C state (C3) on my Dell D630,
which has
> > >>>>>> a Intel Merom (dual-core laptop) processor. 
At that
> > >>>>>> time, when I tried enabling hpetbroadcast,
dom0 boot failed.
> > >>>>>> 
> > >>>>>>
http://lists.xensource.com/archives/html/xen-devel/2009-10/ms
> > >>>>>> g01027.html 
> > >>>>>> 
> > >>>>>> As it turned out, all RHEL5-based (maybe
RHEL4- also) dom0
> > >>>>>> default installation run /sbin/hwclock, which
IIRC takes
> > >>>>>> the RTC away from Xen and gives it to dom0. 
Since the
> > >>>>>> Xen hpet emulation does not do RTC emulation,
bad things
> > >>>>>> then happen when a deep Cstate is entered
(dom0 apparently
> > >>>>>> never wakes up).  I think Ke Yu has also
reproduced
> > this problem.
> > >>>>>> 
> > >>>>>> Sometime in the last few weeks, some patch in
xen-unstable
> > >>>>>> apparently changed some defaults and
xen-unstable will
> > >>>>>> no longer boot with this processor/dom0, with
or without
> > >>>>>> hpetbroadcast on the Xen command line. 
However, specifying
> > >>>>>> max_cstate=2 on the Xen command line allows a
successful
> > >>>>>> dom0 boot, so I suspect the problem is the
same (or at
> > >>>>>> least very similar).
> > >>>>>> 
> > >>>>>> I did a quick scan for hpet changes and found
c/s 20497,
> > >>>>>> but backing it out made no difference.
> > >>>>>> 
> > >>>>>> I have a workaround for now, but since it is
likely that
> > >>>>>> many customers (including all of
Oracle''s OVS customers)
> > >>>>>> use a RHEL5-based dom0 boot sequence, and
Merom processors
> > >>>>>> work fine otherwise, it would be nice to get
this identified
> > >>>>>> and fixed before 4.0.
> > >>>>> 
> > >>>>> Let''s firstly figure out which component
the issue resides.
> > >>>>> 
> > >>>>> Firstly, in the default boot (i.e. without
specifying
> > >>>>> max_cstate=2), when dom0 hangs, is xen still
alive, E.g. can
> > >>>>> Xen response to three Ctrl-''A''
in serial?
> > >>>>> 
> > >>>>> If only dom0 hangs, it is probably that RTC
malfunction make
> > >>>>> incorrect dom0 time and lead dom0 fail to boot.
Then RTC
> > >>>>> emulation in hypervisor should fix this issue.
> > >>>>> 
> > >>>>> If Xen also hangs, it should be another bug, i.e.
hpet
> > >>>>> broadcast does not wake up CPU in deep C states.
in this
> > >>>>> case, if convenient, could you help to do some
bisect to see
> > >>>>> which cset cause this bug? 
> > >>>>> 
> > >>>>> Best Regards
> > >>>>> Ke
> > 
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel