Xen devel - Feb 2005 - console driver - How domain0 processes console messages ?

Hello,

I am trying to undestand how the console driver works.
The files of interests are:
1.) drivers/xen/console/console.c
2.) arch/xen/kernel/ctrl_if.c

That is what I undestand so far. When an unpriveleged domain (say
domain1) wants to write to the console, the following happens:

1.) A function like kcons_write() is invoked to write the string to a
buffer "wbuf".
2.) Function __xencons_tx_flush() is  called to packet the characters
in the buffer "wbuf" into messages of type "ctrl_msg_t".
3.) A function like ctrl_if_send_message_noblock() is then called to
place the messages "ctrl_msg_t" into the communication ring that is
shared between dom1 and dom0.
4.) Function ctrl_if_notify_controller() is then called to send an
event notification to  domain0

Steps 1 to 4 occur in domain1 (the unprivileged domain). Now at the
over end of the event channel, I am trying to figure out where in
domain0 the event notification is being received and how it is being
processed. I am trying to undestand how domain0 processes console
messages  (i.e messages with type CMSG_CONSOLE and subtype
CMSG_CONSOLE_DATA).

Thanks
Richard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> I am trying to undestand how domain0 processes console
> messages  (i.e messages with type CMSG_CONSOLE and subtype
> CMSG_CONSOLE_DATA).
Console messages currently share the same shared memory rings that
control messages use.  As such, ctrl_if_send_message_noblock(), is a
non-blocking send on these rings, and ctrl_if_notify_controller() is
an event channel notification of a control event.  Aside from console
messages, control messages are exactly that -- things like driver
setup and connection.  See xen/include/public/io/domain_controller.h
for a full list.

The control message rings are mapped to user space in dom0 through
calls to libxc, while the notification is delivered to dom0 via
/dev/evtchn.  If you are using stable or testing, these rings are
accessed directly by xend, see tools/python/xen/lowlevel/xu/xu.c
(especially the notifier and port objects therein).  If you are using
the unstable tree, xcs (tools/xcs/xcs.c) demultiplexes the shared
rings/event channels.

In xend, consol messages are eventually handled in
tools/python/xen/xend/server/console.py.

Hope that helps.  We plan to separate the console messages from the
control rings in the very near future, allowing them to have a
separate driver and be throttled independent of control messages. 
This should make the current interface considerably simpler.

hth,
a.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 16 Feb 2005 07:54:22 +0000, Andrew Warfield
<andrew.warfield@gmail.com> wrote:
> Console messages currently share the same shared memory rings that
> control messages use.  As such, ctrl_if_send_message_noblock(), is a
> non-blocking send on these rings, and ctrl_if_notify_controller() is
> an event channel notification of a control event.  Aside from console
> messages, control messages are exactly that -- things like driver
> setup and connection.  See xen/include/public/io/domain_controller.h
> for a full list.
> 
> The control message rings are mapped to user space in dom0 through
> calls to libxc, while the notification is delivered to dom0 via
> /dev/evtchn.  If you are using stable or testing, these rings are
> accessed directly by xend, see tools/python/xen/lowlevel/xu/xu.c
> (especially the notifier and port objects therein).  If you are using
> the unstable tree, xcs (tools/xcs/xcs.c) demultiplexes the shared
> rings/event channels.
> 
> In xend, consol messages are eventually handled in
> tools/python/xen/xend/server/console.py.
> 
> Hope that helps.  We plan to separate the console messages from the
> control rings in the very near future, allowing them to have a
> separate driver and be throttled independent of control messages.
> This should make the current interface considerably simpler.
Hello Andrew,

Thanks for all the info, I am working on a stable tree of XEN. 
I am actually working on the mini-os. I am now porting some drivers to
the mini-os and I am starting with the console driver since it is the
simplest and uses the same communication ring as regular event
messages.

Whenever I try to output to the console of mini-os I get some garbage
(non ascii characters). In XenoLinux, the communication ring is found
at 2048 bytes offset from the
shared_info structure. I am refering to the following macro found in ctrl_if.c

************************************************************************
#define get_ctrl_if() ((control_if_t *)((char *)HYPERVISOR_shared_info + 2048))
************************************************************************

Are the control rings found at the same location if I boot something
else than Linux like for example the mini-os ? I am concern that my
console driver in mini-os is sending event notification to domain0 but
is however not writing the messages in the correct location of the
communication ring and that is why I am getting some  bogus output on
my console.

Next, I am going to look at tools/python/xen/lowlevel/xu/xu.c to see
if domain0 is  receiving the actual data being sent from mini-os
console driver.

Thanks
Richard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 16 Feb 2005 07:54:22 +0000, Andrew Warfield
<andrew.warfield@gmail.com> wrote:
> > I am trying to undestand how domain0 processes console
> > messages  (i.e messages with type CMSG_CONSOLE and subtype
> > CMSG_CONSOLE_DATA).
> 
> Console messages currently share the same shared memory rings that
> control messages use.  As such, ctrl_if_send_message_noblock(), is a
> non-blocking send on these rings, and ctrl_if_notify_controller() is
> an event channel notification of a control event.  Aside from console
> messages, control messages are exactly that -- things like driver
> setup and connection.  See xen/include/public/io/domain_controller.h
> for a full list.
> 
> The control message rings are mapped to user space in dom0 through
> calls to libxc, while the notification is delivered to dom0 via
> /dev/evtchn.  If you are using stable or testing, these rings are
> accessed directly by xend, see tools/python/xen/lowlevel/xu/xu.c
> (especially the notifier and port objects therein).  If you are using
> the unstable tree, xcs (tools/xcs/xcs.c) demultiplexes the shared
> rings/event channels.
Andrew, as I understand, xcs provides xend''s functionalities, and to
some extent (I expect that xcs is still under development) we can use
xcs instead of xend to manage domainUs. Is that correct?

Thank you,
AQ


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Andrew, as I understand, xcs provides xend''s functionalities, and
to
> some extent (I expect that xcs is still under development) we can use
> xcs instead of xend to manage domainUs. Is that correct?
Not exactly. ;)  As the control rings are shared memory, and indexed
using shared pointers, it isn''t really possible to have multiple
clients in dom0 using them concurrently.  With xend connecting
directly to these rings, it was impossible to add new control messages
(for instance if you were working on a new driver) without having to
get waist-deep in xend itself.

xcs (xen control switch) just acts as a mux/demux for the control
channels.  In the unstable tree, xend has been modified to bind to
only the control messages that it is actively handling.  This means
that other control messages may be handled by completely separate
daemons in dom0.  It hopefully will also allow us to start to move
away from the current monolithic xend code, to a new set of control
tools in the very near future.

If all you are trying to do is add a relatively simple service between
dom0 and your domains, then xcs should help.  One example here would
be to build a rough load-balancer that received cpu load announcements
from a set of VMs and then periodically repinned them to alternate
CPUs or migrated them to other physical hosts.  (Just in case anybody
is looking for a little project. ;) )

Several people are working on new control tools at the moment.
Anthony Liguori has posted some first-round c-based tools to the list
in the past week or so.  Mike Wray is also in the process of
redesigning xend itself.  If you are looking for something in the
short term, Anthony''s stuff may be helpful, and I''m sure
he''d
appreciate any feedback.

cheers,
a.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Whenever I try to output to the console of mini-os I get some garbage
> (non ascii characters). In XenoLinux, the communication ring is found
> at 2048 bytes offset from the
> shared_info structure. I am refering to the following macro found in
ctrl_if.c
> 
> ************************************************************************
> #define get_ctrl_if() ((control_if_t *)((char *)HYPERVISOR_shared_info +
2048))
> ************************************************************************
> 
> Are the control rings found at the same location if I boot something
> else than Linux like for example the mini-os ? I am concern that my
> console driver in mini-os is sending event notification to domain0 but
> is however not writing the messages in the correct location of the
> communication ring and that is why I am getting some  bogus output on
> my console.
The domain build tools tell a gues linux where its shared_info page is
at start of day.  Take a look at
tools/libxc/xc_linux_build.c:xc_linux_build() to see this in action --
the code doeas a GETDOMAININFO hypercall to find out the location of
the shared page for the newly created comain, and then passes this on
the start_info page to the new linux VM.  The startup code is all
called from tools/python/xen/xend/XendDomainInfo.py, which is a bit
convoluted.  Anthony''s c-based tools may be an easier reference for
what actually happens during domain startup -- it''s effectively just a
series of calls to libxc.
> Next, I am going to look at tools/python/xen/lowlevel/xu/xu.c to see
> if domain0 is  receiving the actual data being sent from mini-os
> console driver.
Sounds like a plan.  I''d thing that you should be able to pull most of
the control ring interaction code straight out of the console driver.

good luck!
a.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

>> Next, I am going to look at tools/python/xen/lowlevel/xu/xu.c to see
>>if domain0 is  receiving the actual data being sent from mini-os
>>console driver.
>>    
>>
>
>Sounds like a plan.  I''d thing that you should be able to pull most
of
>the control ring interaction code straight out of the console driver.
>  
>
With xcs you can run a utility called xcsdump which will show you most 
of the control traffic going on in your system.

The easiest thing to do though would probably be to launch vm-create 
(part of vm-tools) with the -i option (interactive) and just step 
through it with gdb.

The message flow isn''t terribly obvious just by looking at the code 
however it should be quite understandable in gdb.

Let me know when you get to implementing support for virtual devices (if 
you do).  I''ve got all sorts of xcs/xu debugging stuff that
I''m going to
eventually clean-up and submit.  They''re pretty granular for most of
the
important messages (dumping out every field in all the messages).

Good luck :-)

Regards,
>good luck!
>a.
>
>
>-------------------------------------------------------
>SF email is sponsored by - The IT Product Guide
>Read honest & candid reviews on hundreds of IT Products from real users.
>Discover which products truly live up to the hype. Start reading now.
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>  
>

-- 
Anthony Liguori
anthony@codemonkey.ws



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Hi,

On Wed, 16 Feb 2005 19:26:24 -0600, Anthony Liguori
<anthony@codemonkey.ws> wrote:
> The easiest thing to do though would probably be to launch vm-create
> (part of vm-tools) with the -i option (interactive) and just step
> through it with gdb.
> 
> The message flow isn''t terribly obvious just by looking at the
code
> however it should be quite understandable in gdb.
> 
> Let me know when you get to implementing support for virtual devices (if
> you do).  I''ve got all sorts of xcs/xu debugging stuff that
I''m going to
> eventually clean-up and submit.  They''re pretty granular for most
of the
> important messages (dumping out every field in all the messages).
> 
> Good luck :-)
Thanks Anthony.
Your debugging tools will be very helpful. 
I actually got the console driver to output correctly on mini-os.
However, I can only ouput a limited amount of data. After some time my
communication ring gets filled up and I cannot send anything. I''ve
tracked down my problem. The reason is because my mini-os does not
handle interrupts (or events) very well.

The backend driver in Domain0 sends an event to mini-os each time it
processes a message from the communication ring. Mini-os should
receive that event and update its communication ring pointers.
However, the entry.S file in mini-os is pretty outdated and mini-os
does not behave correctly after receiving its 1st interrupt. I''ve been
looking at XenoLinux''s entry.S file and trying to adapt the one in
mini-os but I don''t quite undestand what is going on in there and I
have not been successful so far.

Richard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel