Xen devel - Sep 2005 - A weird behavior of xenconsole on XEN/IA64

I saw a weird behavior related to xenconsole, when I''m working on domU
support on XEN/IA64:

After following sequence:
# lomount -t ext2 -diskimage /root/xenia64.img -partition2 /mnt
# xend start
# xm create -c

I can see domU boot messages flushed out immediately, but quiet at about
ACPI initialization. Then I have to do:

# Ctrl + ]
# xm console 1
# Ctrl + ]
# xm console 1
# Ctrl + ]
# xm console 1

Exactly at third "xm console 1", the subsequent messages continues to
show. Seems like this is not delayed message because domU''s progress
seems also halt for same long period and then continue to boot.

Definitely I don''t want to confuse anybody here, and it''s very
likely to
be an XEN/IA64 specific issue (out of sync for some time). I post it
here just in case any guys familiar with this part can point me some
possible clues about normal work flow of xenconsole, which then can help
me to find root cause. ;-)

Actually some time later, wait_for_blkif in blkfront also warns "timeout
to connect to device". Though there may be other reasons for the
connection, I''m also interested whether the reason for incontinuous
console message also relates to this one.

Is it possible from the lock of xenbus? One background is, xenstore has
already been setup correctly, and I can see some transaction succeeded
on that page.

Appreciate your comments.

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tian, Kevin wrote:
>I saw a weird behavior related to xenconsole, when I''m working on
domU
>support on XEN/IA64:
>
>After following sequence:
># lomount -t ext2 -diskimage /root/xenia64.img -partition2 /mnt
># xend start
># xm create -c
>
>I can see domU boot messages flushed out immediately, but quiet at about
>ACPI initialization. Then I have to do:
>
># Ctrl + ]
># xm console 1
># Ctrl + ]
># xm console 1
># Ctrl + ]
># xm console 1
>  
>
I''m not sure I fully understand, you''re hitting Ctrl + ] and
it''s not
taking you out of the console?

That code is in the client (tools/console/client/main.c) and hasn''t 
really changed.  Maybe Ctrl + ] generates a different scancode on your 
system?

Regards,

Anthony Liguori
>Exactly at third "xm console 1", the subsequent messages continues
to
>show. Seems like this is not delayed message because domU''s
progress
>seems also halt for same long period and then continue to boot.
>
>Definitely I don''t want to confuse anybody here, and it''s
very likely to
>be an XEN/IA64 specific issue (out of sync for some time). I post it
>here just in case any guys familiar with this part can point me some
>possible clues about normal work flow of xenconsole, which then can help
>me to find root cause. ;-)
>
>Actually some time later, wait_for_blkif in blkfront also warns
"timeout
>to connect to device". Though there may be other reasons for the
>connection, I''m also interested whether the reason for incontinuous
>console message also relates to this one.
>
>Is it possible from the lock of xenbus? One background is, xenstore has
>already been setup correctly, and I can see some transaction succeeded
>on that page.
>
>Appreciate your comments.
>
>Thanks,
>Kevin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> ># Ctrl + ]
> ># xm console 1
> ># Ctrl + ]
> ># xm console 1
> ># Ctrl + ]
> ># xm console 1
> >  
> >
> I''m not sure I fully understand, you''re hitting Ctrl + ]
and it''s not
> taking you out of the console?
No, I think the problem was that it''s only after he''s
connected to the
console a few times that the messages come through. It''s a complete
mystery to me. Does anyone with familiarity with that code know what
changes on IA64 might cause such a malfunction?

-- 
"I will not be pushed,  filed, stamped, indexed, briefed,  debriefed, or
numbered. My life is my own." <-- Number Six   http://surreal.istic.org/
"If you need an outward sign to know your own, you don''t really
belong."
"If the science fails,  the damn-foolishness may work." <--  ^--
Hagbard


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>>
>>I can see domU boot messages flushed out immediately, but quiet at
about
>>ACPI initialization. Then I have to do:
>>
>># Ctrl + ]
>># xm console 1
>># Ctrl + ]
>># xm console 1
>># Ctrl + ]
>># xm console 1
>>
>>
>I''m not sure I fully understand, you''re hitting Ctrl + ]
and it''s not
>taking you out of the console?
No, Ctrl+] can take me out of the console. What I mean is, when domU
boots to place around ACPI init, there''s no console output then. Then I
started to log on/off the console several times. However the first two
"xm console 1" got no output at all. Only the third "xm console
1" can
make domU continue to boot and output. If without the third, domU also
halt there and the xenconsole page also shows no updated content.

Thanks,
Kevin
>
>That code is in the client (tools/console/client/main.c) and hasn''t
>really changed.  Maybe Ctrl + ] generates a different scancode on your
>system?
>
>Regards,
>
>Anthony Liguori
>
>>Exactly at third "xm console 1", the subsequent messages
continues to
>>show. Seems like this is not delayed message because domU''s
progress
>>seems also halt for same long period and then continue to boot.
>>
>>Definitely I don''t want to confuse anybody here, and
it''s very likely
to
>>be an XEN/IA64 specific issue (out of sync for some time). I post it
>>here just in case any guys familiar with this part can point me some
>>possible clues about normal work flow of xenconsole, which then can
help
>>me to find root cause. ;-)
>>
>>Actually some time later, wait_for_blkif in blkfront also warns
"timeout
>>to connect to device". Though there may be other reasons for the
>>connection, I''m also interested whether the reason for
incontinuous
>>console message also relates to this one.
>>
>>Is it possible from the lock of xenbus? One background is, xenstore
has
>>already been setup correctly, and I can see some transaction succeeded
>>on that page.
>>
>>Appreciate your comments.
>>
>>Thanks,
>>Kevin
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>
>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Daniel Hulme
>Sent: 2005年9月15日 22:09
>
>> ># Ctrl + ]
>> ># xm console 1
>> ># Ctrl + ]
>> ># xm console 1
>> ># Ctrl + ]
>> ># xm console 1
>> >
>> >
>> I''m not sure I fully understand, you''re hitting Ctrl
+ ] and it''s not
>> taking you out of the console?
>No, I think the problem was that it''s only after he''s
connected to the
>console a few times that the messages come through. It''s a complete
>mystery to me. Does anyone with familiarity with that code know what
>changes on IA64 might cause such a malfunction?
Exactly right. Actually I''m doing the change on IA64 to make domU
working. However there''s a period of out-of-sync for xen/ia64
repository, and then after merging with mainstream, things become different as
previous when I did make multiple domains on IA64 working. So I posted the info
here, in case anyone can see some internal tricks from the phenomenon, based on
latest change for xenconsole. Is there any outstanding usage change regarding
xenconsole recently? I noted some days ago a thread related to xenconsole
timeout to allow creating pty, but seems different as my finding here. ;-(

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Tian, Kevin wrote:
>>I''m not sure I fully understand, you''re hitting Ctrl +
] and it''s not
>>taking you out of the console?
>>    
>>
>
>No, Ctrl+] can take me out of the console. What I mean is, when domU
>boots to place around ACPI init, there''s no console output then.
Then I
>started to log on/off the console several times. However the first two
>"xm console 1" got no output at all. Only the third "xm
console 1" can
>make domU continue to boot and output. If without the third, domU also
>halt there and the xenconsole page also shows no updated content.
>  
>
Ah, okay, sorry I''m still a bit groggy :-)

Sounds like event channel notifications aren''t happening.  Upon
startup,
the console client will check the console ring.  Otherwise, it will just 
wait for an event channel notification.

If you''re not seeing data unless you reconnect to the console,
it''s
likely because that notification isn''t being triggered (for whatever 
reason).

Hope this helps,

Anthony Liguori
>Thanks,
>Kevin
>  
>
>>That code is in the client (tools/console/client/main.c) and
hasn''t
>>really changed.  Maybe Ctrl + ] generates a different scancode on your
>>system?
>>
>>Regards,
>>
>>Anthony Liguori
>>
>>    
>>
>>>Exactly at third "xm console 1", the subsequent messages
continues to
>>>show. Seems like this is not delayed message because domU''s
progress
>>>seems also halt for same long period and then continue to boot.
>>>
>>>Definitely I don''t want to confuse anybody here, and
it''s very likely
>>>      
>>>
>to
>  
>
>>>be an XEN/IA64 specific issue (out of sync for some time). I post it
>>>here just in case any guys familiar with this part can point me some
>>>possible clues about normal work flow of xenconsole, which then can
>>>      
>>>
>help
>  
>
>>>me to find root cause. ;-)
>>>
>>>Actually some time later, wait_for_blkif in blkfront also warns
>>>      
>>>
>"timeout
>  
>
>>>to connect to device". Though there may be other reasons for
the
>>>connection, I''m also interested whether the reason for
incontinuous
>>>console message also relates to this one.
>>>
>>>Is it possible from the lock of xenbus? One background is, xenstore
>>>      
>>>
>has
>  
>
>>>already been setup correctly, and I can see some transaction
succeeded
>>>on that page.
>>>
>>>Appreciate your comments.
>>>
>>>Thanks,
>>>Kevin
>>>
>>>_______________________________________________
>>>Xen-devel mailing list
>>>Xen-devel@lists.xensource.com
>>>http://lists.xensource.com/xen-devel
>>>
>>>
>>>
>>>      
>>>
>
>
>  
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>>No, Ctrl+] can take me out of the console. What I mean is, when domU
>>boots to place around ACPI init, there''s no console output
then. Then
I
>>started to log on/off the console several times. However the first two
>>"xm console 1" got no output at all. Only the third "xm
console 1" can
>>make domU continue to boot and output. If without the third, domU also
>>halt there and the xenconsole page also shows no updated content.
>>
>>
>Ah, okay, sorry I''m still a bit groggy :-)
>
>Sounds like event channel notifications aren''t happening.  Upon
startup,
>the console client will check the console ring.  Otherwise, it will
just
>wait for an event channel notification.
>
>If you''re not seeing data unless you reconnect to the console,
it''s
>likely because that notification isn''t being triggered (for
whatever
>reason).
>
>Hope this helps,
>
>Anthony Liguori
>
Ha, that''s the useful info I want to look into now. Thanks a lot. 

- Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Finally the root cause is confirmed from event channel injection mechanism
broken under some corner case (Yes, IA64 specific). Once that condition is hit,
no event is injected to dom0 any more even when some event bits are pending.
This is why xenstored can''t receive message from another domain.

Later when issuing "xm console 1" and "Ctrl + ]", change of
connection status may cause xenstored to query store page actively and thus xenU
can forward progress a bit. Later when new transaction is coming, I have to do
same thing to trigger.

After fixing the evtchn part, now it works as desired and thanks for hints. ;-)

Thanks,
Kevin
>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin
>Sent: 2005年9月16日 9:21
>To: Anthony Liguori
>Cc: xen-devel@lists.xensource.com
>Subject: RE: [Xen-devel] A weird behavior of xenconsole on XEN/IA64
>
>>From: Anthony Liguori [mailto:aliguori@us.ibm.com]
>>>No, Ctrl+] can take me out of the console. What I mean is, when domU
>>>boots to place around ACPI init, there''s no console output
then. Then
>I
>>>started to log on/off the console several times. However the first
two
>>>"xm console 1" got no output at all. Only the third
"xm console 1" can
>>>make domU continue to boot and output. If without the third, domU
also
>>>halt there and the xenconsole page also shows no updated content.
>>>
>>>
>>Ah, okay, sorry I''m still a bit groggy :-)
>>
>>Sounds like event channel notifications aren''t happening.  Upon
>startup,
>>the console client will check the console ring.  Otherwise, it will
>just
>>wait for an event channel notification.
>>
>>If you''re not seeing data unless you reconnect to the console,
it''s
>>likely because that notification isn''t being triggered (for
whatever
>>reason).
>>
>>Hope this helps,
>>
>>Anthony Liguori
>>
>
>Ha, that''s the useful info I want to look into now. Thanks a lot.
>
>- Kevin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel