Xen devel - Aug 2008 - xenconsoled needs restart after long dom0 uptime otherwise domU console output hangs and domU doesn''t come up

Hello list!

I''ve noticed this problem a couple of times now.. so I''m
wondering if it''s a
known bug? 

Symptoms:

After "long" dom0 uptime (174 days when it happened now, was something
similar on previous times too) cannot start new pv domUs anymore.. 
pygrub runs and boots the kernel, but kernel newer gets to actually running
initrd
and the console output hangs/stalls..

"xm console" output might look like this:

checking if image is initramfs... it is
Freeing initrd memory: 12028k freed
Grant table initialized
NET: Registered protocol family 16
SMP alternatives: switching to SMP code
<hangs here, nothing happens anymore>

The point where it hangs might be a couple of lines earlier or later.. seems
to change on almost every run..

"xm list" in dom0 shows the domU as "-b----" and the Time(s)
stays at 1.9..
after 8 hours of being hanged/stalled Time(s) was at 5.0 for that domU, but
the console output had not changed, and the domU was still non-functional.

Already running domUs keep running fine when this happens.. 

Solution:

This problem gets fixed when you kill xenconsoled in dom0, and restart it. After
this new domUs can be started just fine, and everything works normally
again. "xm console" works, initrd is executed, domU comes up OK, and
you see
the full console output.

I''ve seen this happening on dom0s running RHEL/CentOS 5.0 (Xen 3.0.3)
and
RHEL/CentOS 5.1 (Xen 3.1.0). 

I just upgraded to 5.2 so can''t tell (yet) if it happens with Xen
3.1.2.

I''m running 32bit x86 PAE, and the domU kernels in question are Debian
etch
2.6.18-6-xen-686. I think it also happens RHEL/CentOS 5.x kernel-xen, but
not totally sure about that. I think I also saw it with Fedora 8/9
domU kernel-xen once.

Looks like a bug in xenconsoled.. 

-- Pasi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 14, 2008 at 02:22:10PM +0300, Pasi Kärkkäinen
wrote:
> Hello list!
> 
> I''ve noticed this problem a couple of times now.. so I''m
wondering if it''s a
> known bug? 
> 
> Symptoms:
> 
> After "long" dom0 uptime (174 days when it happened now, was
something
> similar on previous times too) cannot start new pv domUs anymore.. 
> pygrub runs and boots the kernel, but kernel newer gets to actually running
initrd
> and the console output hangs/stalls..
> 
> "xm console" output might look like this:
> 
> checking if image is initramfs... it is
> Freeing initrd memory: 12028k freed
> Grant table initialized
> NET: Registered protocol family 16
> SMP alternatives: switching to SMP code
> <hangs here, nothing happens anymore>
> 
> The point where it hangs might be a couple of lines earlier or later..
seems
> to change on almost every run..
> 
> "xm list" in dom0 shows the domU as "-b----" and the
Time(s) stays at 1.9..
> after 8 hours of being hanged/stalled Time(s) was at 5.0 for that domU, but
> the console output had not changed, and the domU was still non-functional.
> 
> Already running domUs keep running fine when this happens.. 
> 
> Solution:
> 
> This problem gets fixed when you kill xenconsoled in dom0, and restart it.
After
> this new domUs can be started just fine, and everything works normally
> again. "xm console" works, initrd is executed, domU comes up OK,
and you see
> the full console output.
> 
> I''ve seen this happening on dom0s running RHEL/CentOS 5.0 (Xen
3.0.3) and
> RHEL/CentOS 5.1 (Xen 3.1.0). 
> 
> I just upgraded to 5.2 so can''t tell (yet) if it happens with Xen
3.1.2.
> 
> I''m running 32bit x86 PAE, and the domU kernels in question are
Debian etch
> 2.6.18-6-xen-686. I think it also happens RHEL/CentOS 5.x kernel-xen, but
> not totally sure about that. I think I also saw it with Fedora 8/9
> domU kernel-xen once.
> 
> Looks like a bug in xenconsoled.. 
> 
Oh, and I forgot to mention there''s nothing unusual in "xm
log", "xm dmesg" or
in dom0 /var/log/*. 

xenconsoled doesn''t take 100% CPU or anything like that.. so it just
silently goes bad somehow. 

-- Pasi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen writes ("[Xen-devel] xenconsoled needs restart after long
dom0 uptime otherwise domU console output hangs and domU doesn''t come
up"):
> Looks like a bug in xenconsoled.. 
Can you tell from strace what xenconsoled is waiting for ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 14, 2008 at 03:20:06PM +0100, Ian Jackson
wrote:
> Pasi Kärkkäinen writes ("[Xen-devel] xenconsoled needs restart after
long dom0 uptime otherwise domU console output hangs and domU doesn''t
come up"):
> > Looks like a bug in xenconsoled.. 
> 
> Can you tell from strace what xenconsoled is waiting for ?
> 
Damn.. didn''t think of that when I still had the problem :(
I killed+restarted xenconsoled on the problematic dom0 already..

I''ll try to remember that the next time I see this problem.. 

-- Pasi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 14, 2008 at 7:22 PM, Pasi Kärkkäinen <pasik@iki.fi>
wrote:
> I''ve seen this happening on dom0s running RHEL/CentOS 5.0 (Xen
3.0.3) and
> RHEL/CentOS 5.1 (Xen 3.1.0).
>
> I just upgraded to 5.2 so can''t tell (yet) if it happens with Xen
3.1.2.
>
> I''m running 32bit x86 PAE, and the domU kernels in question are
Debian etch
> 2.6.18-6-xen-686. I think it also happens RHEL/CentOS 5.x kernel-xen, but
> not totally sure about that. I think I also saw it with Fedora 8/9
> domU kernel-xen once.
>
> Looks like a bug in xenconsoled..
FYI, I encountered similar issue about can''t get new domainU/xenU to
start or even existing running domainU/xenU to restart (meaning can
shutdown xenU but can''t start it back).  In the end, I noticed that
this was due to udev package being upgraded/updated.  Have you try to
execute /sbin/start_udev after your udev package being updated?  Maybe
this will solve your issue like mine :)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 14, 2008 at 11:21:12PM +0800, Teck Choon Giam
wrote:
> On Thu, Aug 14, 2008 at 7:22 PM, Pasi Kärkkäinen <pasik@iki.fi>
wrote:
> > I''ve seen this happening on dom0s running RHEL/CentOS 5.0
(Xen 3.0.3) and
> > RHEL/CentOS 5.1 (Xen 3.1.0).
> >
> > I just upgraded to 5.2 so can''t tell (yet) if it happens with
Xen 3.1.2.
> >
> > I''m running 32bit x86 PAE, and the domU kernels in question
are Debian etch
> > 2.6.18-6-xen-686. I think it also happens RHEL/CentOS 5.x kernel-xen,
but
> > not totally sure about that. I think I also saw it with Fedora 8/9
> > domU kernel-xen once.
> >
> > Looks like a bug in xenconsoled..
> 
> FYI, I encountered similar issue about can''t get new domainU/xenU
to
> start or even existing running domainU/xenU to restart (meaning can
> shutdown xenU but can''t start it back).  In the end, I noticed
that
> this was due to udev package being upgraded/updated.  Have you try to
> execute /sbin/start_udev after your udev package being updated?  Maybe
> this will solve your issue like mine :)
I didn''t upgrade any packages when this happened.. so it was not about
udev.

Killing+restarting xenconsoled fixed it for me.. 

-- Pasi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Aug 14, 2008 at 06:38:59PM +0300, Pasi Kärkkäinen
wrote:
> On Thu, Aug 14, 2008 at 11:21:12PM +0800, Teck Choon Giam wrote:
> > On Thu, Aug 14, 2008 at 7:22 PM, Pasi Kärkkäinen <pasik@iki.fi>
wrote:
> > > I''ve seen this happening on dom0s running RHEL/CentOS
5.0 (Xen 3.0.3) and
> > > RHEL/CentOS 5.1 (Xen 3.1.0).
> > >
> > > I just upgraded to 5.2 so can''t tell (yet) if it happens
with Xen 3.1.2.
> > >
> > > I''m running 32bit x86 PAE, and the domU kernels in
question are Debian etch
> > > 2.6.18-6-xen-686. I think it also happens RHEL/CentOS 5.x
kernel-xen, but
> > > not totally sure about that. I think I also saw it with Fedora
8/9
> > > domU kernel-xen once.
> > >
> > > Looks like a bug in xenconsoled..
> > 
> > FYI, I encountered similar issue about can''t get new
domainU/xenU to
> > start or even existing running domainU/xenU to restart (meaning can
> > shutdown xenU but can''t start it back).  In the end, I
noticed that
> > this was due to udev package being upgraded/updated.  Have you try to
> > execute /sbin/start_udev after your udev package being updated?  Maybe
> > this will solve your issue like mine :)
> 
> I didn''t upgrade any packages when this happened.. so it was not
about udev.
> 
> Killing+restarting xenconsoled fixed it for me.. 
> 
This happened again for me, but I forgot to take the strace.. doh.

I''ll try to remember the next time it happens.. it seems to happen
every now
and then, and on multiple machines.. 

I''m running CentOS 5.2, so it seems to happen with that version aswell.

Then again the userland Xen stuff in CentOS 5.2 is Xen 3.0.3 or so.. 
(even when the hypervisor in CentOS 5.2 is Xen 3.1.2.)

-- Pasi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel