Xen devel - Jul 2005 - Why have an idle domain? (Was: IDLE domain is scheduled more than dom0)

I wanted to start a new thread on this so it doesn''t get
buried in the old one.

Why is there an idle domain at all?  Certainly OS schedulers
need one and, yes, I understand that it makes it a tiny bit
easier to not have any special cases in the scheduler, but
Xen isn''t a (normal) OS and there is a very good reason NOT to
have an idle domain: Namely that context switching is not free.
It is very expensive on some machines; it may be cheap on an x86
but it is still not free.

The issue is what domain is running when a device interrupt arrives.
If it is the idle domain, then a domain switch is required before
it can be "delivered" to domain0.  This adds unnecessary latency
to interrupts... although small it CAN add up.  (The 1% measured
by Kevin in the previous thread is still unacceptable as far as
I''m concerned.)

What I propose is that domain0 become the idle domain.  Special
casing code would be added to the schedulers so that domain0
is always runnable and thus will absorb any "idle" cycles.

What if domain0 is not the target for the device interrupt?
Although some Xen configurations will have multiple driver
domains, this is probably an exception rather than the rule.
And "domain0 as the idle domain" will provide no worse interrupt
latency than "idle as the idle domain" for interrupts that
must be delivered to the non-domain0 driver domains.

So is there any good reason to have an idle domain other than
because that''s the way OS schedulers have always done it?

Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tuesday 12 July 2005 14:08, Magenheimer, Dan (HP Labs Fort Collins) 
wrote:
> I wanted to start a new thread on this so it doesn''t get
> buried in the old one.
>
> Why is there an idle domain at all?  Certainly OS schedulers
> need one and, yes, I understand that it makes it a tiny bit
> easier to not have any special cases in the scheduler, but
> Xen isn''t a (normal) OS and there is a very good reason NOT to
> have an idle domain: Namely that context switching is not free.
> It is very expensive on some machines; it may be cheap on an x86
> but it is still not free.
>
> The issue is what domain is running when a device interrupt arrives.
> If it is the idle domain, then a domain switch is required before
> it can be "delivered" to domain0.  This adds unnecessary latency
> to interrupts... although small it CAN add up.  (The 1% measured
> by Kevin in the previous thread is still unacceptable as far as
> I''m concerned.)
>
> What I propose is that domain0 become the idle domain.  Special
> casing code would be added to the schedulers so that domain0
> is always runnable and thus will absorb any "idle" cycles.
On dom0, will we then need a virtual cpu for each physical cpu in the 
box? I am assuming each idle cpu needs a virtual cpu from dom0 to 
"idle".  I usually use a uniprocessor kernel form dom0;
wouldn''t that
create a problem?

Also, if dom0 idled during cpu slack time, is this a polling idle?  If 
so, that would not be good for HT.

IMO, I''m not sure we need any domain at all to idle.  Can''t we
have some
sort of idle function in xen itself, which usually just sleeps the 
processor?

-Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Magenheimer, Dan (HP Labs Fort Collins) wrote:
>What I propose is that domain0 become the idle domain.  Special
>casing code would be added to the schedulers so that domain0
>is always runnable and thus will absorb any "idle" cycles.
>  
>
This is admittedly not a grave concern, but doing what you propose will 
screw up performance accounting, since we won''t know for sure how much 
cpu time dom0 is really using. And there is reason to believe that dom0 
cpu usage should be closely monitored.

Rob Gardner



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

I have a question about get_user_pages:

Does the get_user_pages return page_list that can be directly used by
dma_map_sg in xen0? (of course, we do sg[i].page = page_list[i] before
that.) What I actually mean is that if there is any tricky part wrt
pseudo physical address vs actual machine address here.

And I remember once I read that in xen0, the psedudo-physical address and
the actual machine address are the same. Is it still (or ever) the case
for xen-unstable?

If yes, that means nearly all memory operations, eg mapping, dma, etc. can
be performed in xen0 without any problem? Am I right?

Thanks.

Regards,
Wei


On Tue, 12 Jul 2005, Rob Gardner wrote:
> Magenheimer, Dan (HP Labs Fort Collins) wrote:
>
> >What I propose is that domain0 become the idle domain.  Special
> >casing code would be added to the schedulers so that domain0
> >is always runnable and thus will absorb any "idle" cycles.
> >
> >
>
> This is admittedly not a grave concern, but doing what you propose will
> screw up performance accounting, since we won''t know for sure how
much
> cpu time dom0 is really using. And there is reason to believe that dom0
> cpu usage should be closely monitored.
>
> Rob Gardner
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12 Jul 2005, at 20:08, Magenheimer, Dan (HP Labs Fort Collins) wrote:
> So is there any good reason to have an idle domain other than
> because that''s the way OS schedulers have always done it?
You can do whatever tricks you like in context_switch() when switching 
to/from the idle domain. arch/x86 already avoids switching most cpu 
state, so latency for short-blocking i/o domains is very small.

An idle domain is a convenient abstraction for thinking about the 
current scheduling state of a cpu -- it allows us to treat things more 
uniformly in the common code. Of course we can treat the idle domains 
very specially during state switch for performance.

I guess this will be even more important for arch/ia64, with the 
monstrous amount of architectural state to be switched.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel