Xen devel - Oct 2003 - illegal page update with latest xen-unstable

Just checked out the latest xen-unstable this morning and am getting some 
interesting (read: bad) behavior...

The easiest way to reproduce the error is to just run ''make''
in the
xeno-unstable.bk tools directory.  The ant run hangs, and when I kill it
(both at console or from ssh) I get a machine reboot.

The build command (not very interesting):
"""
root@xeno:~/xeno-unstable/xeno-unstable.bk/tools/control# make
ant -buildfile build-cmdline.xml dist
"""

The xen serial console error:
"""
DOM0 FATAL ERROR: Illegal page update request
Domain 0 killed: rebooting machine!
"""

Xen 1.0 works fine for me, and I built unstable using the xen-clone script.

(clock on this machine is off, it was really built at 11am oct 14.)

Xen:
"""
Xen version 1.1-devel (root@is.marist.edu) (gcc version 3.2.2) Mon Oct 13
17:46:04 EDT 2003
"""

Dom0:
"""
DOM0: Linux version 2.4.22-xeno (root@xeno) (gcc version 3.2.2) #2 Mon Oct 13
17:50:12 EDT 2003
DOM0: Kernel command line: /boot/xenolinux-unstable.gz root=/dev/sda1 ro
console=xencons0 console=tty0 consoe=ttyS0 DOMID=0
"""

The box physically has 2 gig of ram and dom0 is limited to 128 meg.

On a different tangent, I also get a fatal error getting CPU performance 
counters over the serial link with the unstable tree:

"""
CPU performance counters for CPU 0 (current):
general protection fault: 0000,0000 
CPU:    0
EIP:    0808:[<fc507f8e>]       
EFLAGS: 00211282
eax: 00000000   ebx: fc5ad480   ecx: 000000c1   edx: 000003f8
esi: 000003fd   edi: 000003f8   ebp: fc502f60   esp: fc502ed0
ds: 0810   es: 0810   fs: 0810   gs: 0810   ss: 0810
Stack trace from ESP=fc502ed0:
fc58c860 00000000 00201092 00000000 00000063 000003fd 000003f8 [fc513c67]
       00000063 00000000 fc502f60 00000063 00000004 00000000 [fc57ba45] fc5f4a00
       00000000 00000004 fc5b2ac0 [fc57eb8c] 00000004 00000000 fc502f60 fc5f4a00
       00201246 00000000 [fc5038c3] fc5ad480 82d43916 fc5f4a00 00000000 fc5ad480
       00000000 fc5ad480 00000000 [fc57bf60] fc5ad480 00000000 00000ffc 00000000
       fc5ad480 00000000 00000000 fc5a0810 fc5a0810 00000810 fc5a0810 ffffff04
       [fc581b99] 00000808 00201246 c01999a0 00000000 [fc50c633] 0000000b
00000000
       df2c622a fc5ad480 00000000 00000000 00000000 00000000 00000000 00000000
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
       00000000 00000000 00000000 fc5ad480

****************************************
HYPERVISOR DEATH!!
Aieee! CPU0 is toast...
****************************************

Reboot in five seconds...
System Time went backwards: -104738496
"""

The hardware is a IBM x-series, P4 2.4ghz, adaptec scsi, tg3 gbit, 2 gig ram.

-m




-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Just checked out the latest xen-unstable this morning and am getting some 
> interesting (read: bad) behavior...
> 
> The easiest way to reproduce the error is to just run
''make'' in the
> xeno-unstable.bk tools directory.  The ant run hangs, and when I kill it
> (both at console or from ssh) I get a machine reboot.
Hmm... some nasty bugs :-(

Two things to try:

 1. Some more things have been fixed today. Try pulling and testing
    the latest tree. I can''t reproduce the problem but then
I''m using
    different binaries and libraries.
 
 2. If you still get the "illegal page update request", recompile
    xen/common/memory.c with MEM_LOG compiled in. This requires you to
    change ''#if 0'' in that file to ''#if 1''.

I hope this helps!

 Regards,
 Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Hmm... some nasty bugs :-(
Yup.  I was surprised, it''d been running pretty well with the 1.0, but
a lot
of what I''m interested in requires the vbd stuff.  
> Two things to try:
> 
>  1. Some more things have been fixed today. Try pulling and testing
>     the latest tree. I can''t reproduce the problem but then
I''m using
>     different binaries and libraries.
Done, no change, unfortunately.  For what it''s worth I''m using
libc 2.3.1
and gcc 3.2.2, from the slack 9.0 builds, in domain 0.
>  2. If you still get the "illegal page update request", recompile
>     xen/common/memory.c with MEM_LOG compiled in. This requires you to
>     change ''#if 0'' in that file to ''#if
1''.
Ok, the only new output that generated was:

DOM0: (file=memory.c, line=504) Bad page type/count (06000000!=07000000) cnt=1  
DOM0 FATAL ERROR: Illegal page update request  

Hope that helps track it down, if I have time I''ll poke in the source
too but
am just doing this as a hobby alongside work at the moment.

I can include complete boot logs if it''ll help, but didn''t
want to flood them
if it wasn''t going to be useful.

-m



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > Hmm... some nasty bugs :-(
> 
> Yup.  I was surprised, it''d been running pretty well with the 1.0,
but a lot
> of what I''m interested in requires the vbd stuff.  
The unstable tree is getting a lot of new stuff added each day. This
is bound to cause some problems. I''m hoping to stabilise it somewhat
now -- shake out some of the nastier bugs -- and then declare a
semi-stable release as a sanity checkpoint for future development.
 
> Done, no change, unfortunately.  For what it''s worth I''m
using libc 2.3.1
> and gcc 3.2.2, from the slack 9.0 builds, in domain 0.
>
> DOM0: (file=memory.c, line=504) Bad page type/count (06000000!=07000000)
cnt=1
> DOM0 FATAL ERROR: Illegal page update request  
> 
> Hope that helps track it down, if I have time I''ll poke in the
source too but
> am just doing this as a hobby alongside work at the moment.
This tells me that the bug is within the support for local descriptor
tables (LDTs), which are used by some versions of glibc
linuxthreads. Unfortunately, the versions we have installed on our
machines seem to not use LDTs :-(

It would be interesting to know what version of the tree you first saw
the bug on (''bk sccslog'' executed at the root of the
repository tells
you the last changeset that was checked in).

It''s possible you picked up a set which introduced a new bug, which
I''ve just a checked in a fix for. You may want to pull the fix and try
it out.

If the bug is still present then I can add some extra debugging to
help us sort this out.

 Cheers,
 Keir


-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> On a different tangent, I also get a fatal error getting CPU performance 
> counters over the serial link with the unstable tree:
Yes, those hey handlers are specific to P6, and actually crash P4 :-(

We''re only really interested in P4 perf counters (they''re much
more
useful than the P6 ones), so I''ve now removed those key handlers
from the development tree.

There''s a program called xen_cpuperf in tools/misc. This allows
access to P4 performance counters --- unfortunately the program isn''t
yet documented, and is in fact still under development :-(

 -- Keir

PS. Your other bug (with the illegal page update request) is I''m sure
now fixed in the development tree. It would be very useful if you
could confirm that this is the case. Thanks!



-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Changeset 1.523 seems to have eliminated the page error, at least with the
tests I''ve run so far.  Previously anything that ran java would kill it
(ant,
xenctl, etc), so far it''s been stable.

I''ll let you know if it shows up again. :)

-m

On Wed, Oct 15, 2003 at 12:49:22AM +0100, Keir Fraser
wrote:
> > > Hmm... some nasty bugs :-(
> > 
> > Yup.  I was surprised, it''d been running pretty well with the
1.0, but a lot
> > of what I''m interested in requires the vbd stuff.  
> 
> The unstable tree is getting a lot of new stuff added each day. This
> is bound to cause some problems. I''m hoping to stabilise it
somewhat
> now -- shake out some of the nastier bugs -- and then declare a
> semi-stable release as a sanity checkpoint for future development.
> 
> > Done, no change, unfortunately.  For what it''s worth
I''m using libc 2.3.1
> > and gcc 3.2.2, from the slack 9.0 builds, in domain 0.
> >
> > DOM0: (file=memory.c, line=504) Bad page type/count
(06000000!=07000000) cnt=1
> > DOM0 FATAL ERROR: Illegal page update request  
> > 
> > Hope that helps track it down, if I have time I''ll poke in
the source too but
> > am just doing this as a hobby alongside work at the moment.
> 
> This tells me that the bug is within the support for local descriptor
> tables (LDTs), which are used by some versions of glibc
> linuxthreads. Unfortunately, the versions we have installed on our
> machines seem to not use LDTs :-(
> 
> It would be interesting to know what version of the tree you first saw
> the bug on (''bk sccslog'' executed at the root of the
repository tells
> you the last changeset that was checked in).
> 
> It''s possible you picked up a set which introduced a new bug,
which
> I''ve just a checked in a fix for. You may want to pull the fix and
try
> it out.
> 
> If the bug is still present then I can add some extra debugging to
> help us sort this out.
> 
>  Cheers,
>  Keir

-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel