Ben Thomas
2006-Mar-20 13:24 UTC
[Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
This may not be new with the latest source. I''ve been seeing the same
thing for some time, on certain hardware. It''s right after boot,
generally while printing the E820 map. I''ve noticed some patterns
to it, but haven''t spent the time to track it down. There''s
always
been something else more pressing to do. One slight difference is
between your example and what I''ve been doing is that I only use
com1 as the console for Xen. I''ve never seen it hang with only
vga as the console, but only with com1.
And, while it may seem unlikely that a simple serial UART is
having problems, it''s quite possible.
-b
Date: Mon, 20 Mar 2006 11:04:05 +0000
From: Keir Fraser <Keir.Fraser@cl.cam.ac.uk>
Subject: Re: [Xen-devel] Re: [Xen-users] xen hg tree from yesterday
hangs, when kernel console is directed to ttyS0
On 19 Mar 2006, at 15:10, Jussi Hamalainen wrote:
>>>> When I reboot my system with xen with the following options to
xen
>>>> and vmlinuz,
>>>>
>>>> xen.gz com1=115200,8n1 console=com1,vga
>>>> vmlinuz console=ttyS0,115200 console=tty0
>>>>
>>>> the system hangs up while booting the dom0 kernel.
>
>>
>> I think Xen has a major buffering/blocking IO issue with serial
>> console. It works for a while but when loads of lines get
>> flooded onto the console, the system freezes up. Adding a
>> "sync_console" to your Xen parameters might solve the
problem, or at
>> least it worked for me.
Maybe it''s hardware specific. I can echo a long string to the serial
console in a tight loop from dom0 shell and I see no problems. My setup
is similar to the OP''s (115200 baud).
-- Keir
--
------------------------------------------------------------------------
Ben Thomas Virtual Iron Software
bthomas@virtualiron.com Tower 1, Floor 2
978-849-1214 900 Chelmsford Street
Lowell, MA 01851
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2006-Mar-22 19:22 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Hi, On Mon, 2006-03-20 at 08:24 -0500, Ben Thomas wrote:> This may not be new with the latest source. I''ve been seeing the same > thing for some time, on certain hardware. It''s right after boot, > generally while printing the E820 map.I see this a lot too. In every case I''ve come across myself, using the HV "sync_console" option seems to cure it, but I have no idea why. --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ben Thomas
2006-Mar-22 20:22 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Oh, I wish, I wish... When I see this, I generally never get very far at all. Often, I''m at the "no output at all past grub" stage. Even with vga. Luckily, it''s confined to specific machines. At some point, I''ll slow down to warp 8 and be able to look more at it. I have some ideas on how to find it. -b Stephen C. Tweedie wrote:> Hi, > > On Mon, 2006-03-20 at 08:24 -0500, Ben Thomas wrote: > >>This may not be new with the latest source. I''ve been seeing the same >>thing for some time, on certain hardware. It''s right after boot, >>generally while printing the E820 map. > > > I see this a lot too. In every case I''ve come across myself, using the > HV "sync_console" option seems to cure it, but I have no idea why. > > --Stephen > >-- ------------------------------------------------------------------------ Ben Thomas Virtual Iron Software bthomas@virtualiron.com Tower 1, Floor 2 978-849-1214 900 Chelmsford Street Lowell, MA 01851 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2006-Mar-22 20:33 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Hi, On Wed, 2006-03-22 at 15:22 -0500, Ben Thomas wrote:> When I see this, I generally never get very far at all. Often, I''m > at the "no output at all past grub" stage. Even with vga. Luckily, > it''s confined to specific machines.Even with *only* VGA? I usually see exactly this with serial console enabled: if I attempt it without sync_console, dom0 always hangs solid very early in the boot process (though rarely at exactly the same location.) But I''ve never seen that sort of sporadic hang during boot using just VGA console exclusively; the bugs I''ve seen have always been either oopses or very predictable hangs at a specific location. --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ben Thomas
2006-Mar-22 21:37 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
No, sorry, implicit context here. It really feels like a serial line issue. When I talk about using vga, it''s concurrent usage with com1. I''ve never had a problem with using only vga. Although, that''s a very small percentage of my usage. -b Stephen C. Tweedie wrote:> Hi, > > On Wed, 2006-03-22 at 15:22 -0500, Ben Thomas wrote: > > >>When I see this, I generally never get very far at all. Often, I''m >>at the "no output at all past grub" stage. Even with vga. Luckily, >>it''s confined to specific machines. > > > Even with *only* VGA? I usually see exactly this with serial console > enabled: if I attempt it without sync_console, dom0 always hangs solid > very early in the boot process (though rarely at exactly the same > location.) But I''ve never seen that sort of sporadic hang during boot > using just VGA console exclusively; the bugs I''ve seen have always been > either oopses or very predictable hangs at a specific location. > > --Stephen > >-- ------------------------------------------------------------------------ Ben Thomas Virtual Iron Software bthomas@virtualiron.com Tower 1, Floor 2 978-849-1214 900 Chelmsford Street Lowell, MA 01851 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2006-Mar-22 21:46 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Hi, On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote:> No, sorry, implicit context here. It really feels like a > serial line issue. When I talk about using vga, it''s > concurrent usage with com1. I''ve never had a problem > with using only vga.OK, so it sounds very similar to what I''m seeing; does sync_console really not help? --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ben Thomas
2006-Mar-22 22:00 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Not one bit (pun intended). -b Stephen C. Tweedie wrote:> Hi, > > On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote: > >>No, sorry, implicit context here. It really feels like a >>serial line issue. When I talk about using vga, it''s >>concurrent usage with com1. I''ve never had a problem >>with using only vga. > > > OK, so it sounds very similar to what I''m seeing; does sync_console > really not help? > > --Stephen > >-- ------------------------------------------------------------------------ Ben Thomas Virtual Iron Software bthomas@virtualiron.com Tower 1, Floor 2 978-849-1214 900 Chelmsford Street Lowell, MA 01851 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Mar-23 17:55 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
In unstable changeset 9419 I''ve reduced contention for the serial-line spinlock, particularly in the serial ISR. It certainly has cleaned up some weird behaviour that I was able to reproduce when I hit the serial line hard from within Xen itself. It''s not clear it''ll fix your problems, but worth testing. -- Keir> Not one bit (pun intended). > > -b > > Stephen C. Tweedie wrote: > > Hi, > > > > On Wed, 2006-03-22 at 16:37 -0500, Ben Thomas wrote: > > > >>No, sorry, implicit context here. It really feels like a > >>serial line issue. When I talk about using vga, it''s > >>concurrent usage with com1. I''ve never had a problem > >>with using only vga. > > > > > > OK, so it sounds very similar to what I''m seeing; does sync_console > > really not help? > > > > --Stephen > > > > > > > -- > ------------------------------------------------------------------------ > Ben Thomas Virtual Iron Software > bthomas@virtualiron.com Tower 1, Floor 2 > 978-849-1214 900 Chelmsford Street > Lowell, MA 01851 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ben Thomas
2006-Mar-23 18:36 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
Hi Keir, Thanks for the heads-up. I''ll certainly give it a try. As this hang appears during boot, it''s not clear how much reducing spinlock contention will help. At times, it''s not even clear that I make it into __start_xen. Certainly, when I get a hang, I never make it out of __start_xen/init_e820/print_e820_memory_map. As it often stops in the middle of print_e820_memory_map output, that''s what lead to the suspicion about serial lines. Clearly, though, it''s merely a suspicion/guess. Thanks, -b Keir Fraser wrote:> In unstable changeset 9419 I''ve reduced contention for the serial-line > spinlock, particularly in the serial ISR. It certainly has cleaned up > some weird behaviour that I was able to reproduce when I hit the > serial line hard from within Xen itself. It''s not clear it''ll fix your > problems, but worth testing. > > -- Keir-- ------------------------------------------------------------------------ Ben Thomas Virtual Iron Software bthomas@virtualiron.com Tower 1, Floor 2 978-849-1214 900 Chelmsford Street Lowell, MA 01851 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2006-Mar-24 10:03 UTC
Re: [Xen-devel] Re: xen hg tree from yesterda hangs when kernel console is directed to ttyS0
> Hi Keir, > > Thanks for the heads-up. I''ll certainly give it a try. > > As this hang appears during boot, it''s not clear how much reducing > spinlock contention will help. At times, it''s not even clear that > I make it into __start_xen. Certainly, when I get a hang, I > never make it out of __start_xen/init_e820/print_e820_memory_map. > As it often stops in the middle of print_e820_memory_map > output, that''s what lead to the suspicion about serial lines. > Clearly, though, it''s merely a suspicion/guess.That''s a very interesting data point. It means that the problem you are seeing has nothing to do with asynchronous interrupt-driven transmit. At that point during boot characters are synchronously written to the transmit holding register when the status register indicates it is empty. It''s all amazingly straightforward. And of course there can be no SMP issues either. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel