Xen devel - Nov 2006 - [RFC]degradation on IPF due to hypercall set irq

Keir,

After moving from sharing PIC to hypercall set irq.
KB on UP VTI domain incurs > 10% degradation.

The root cause is hypercall is very expensive on IPF side
due to huge processor context.

I revert to sharing PIC in lastest Cset of IPF side,
Then We can get performance back.


I prepare to use shared IOSAPIC to deliver interrupt from 
Qemu to VTI domain.
In IPF side, PIC is not needed,
In the same time, we can assign more interrupt pins(24) to qemu.


What''s your opinion?


Thanks,
Anthony

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 3:33 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> After moving from sharing PIC to hypercall set irq.
> KB on UP VTI domain incurs > 10% degradation.
> 
> The root cause is hypercall is very expensive on IPF side
> due to huge processor context.
> 
> I revert to sharing PIC in lastest Cset of IPF side,
> Then We can get performance back.
We may well have similar degradation on x86 too. The cause is lots of
unnecessary calls to the set_level hypercall (when the level hasn''t
actually
changed). Qemu *definitely* needs to keep shadow wire state and only notify
Xen on transitions. If the rate of hypercalls is still too high (which I
think is unlikely) we can use batching multicalls.
> I prepare to use shared IOSAPIC to deliver interrupt from
> Qemu to VTI domain.
> In IPF side, PIC is not needed,
> In the same time, we can assign more interrupt pins(24) to qemu.
I moved x86 away from this on purpose, to obtain a clean abstraction. I
don''t think it''s a good idea for ia64 to step backwards here.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 15:07:
> On 22/11/06 3:33 am, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>> After moving from sharing PIC to hypercall set irq.
>> KB on UP VTI domain incurs > 10% degradation.
>> 
>> The root cause is hypercall is very expensive on IPF side
>> due to huge processor context.
>> 
>> I revert to sharing PIC in lastest Cset of IPF side,
>> Then We can get performance back.
> 
> We may well have similar degradation on x86 too. The cause is lots of
> unnecessary calls to the set_level hypercall (when the level
hasn''t
> actually changed). Qemu *definitely* needs to keep shadow wire state
> and only notify Xen on transitions. If the rate of hypercalls is
> still too high (which I think is unlikely) we can use batching
> multicalls. 


I have tried shadow wire state in IPF side, filtered out unnecessary set_level
hypercalls,
But I can only get about 50% performance degradation back.

In IPF side, I set all interrupt edge( there is no sharing interrupt in my
environment), so 1->0, 0->0 and 1->1
 is not Passed to xen by hypercall, only 0->1 is passed to xen by hypercall,
then about half of set_level
hypercalls are saved. But this can only get ~50% performance degradation back.

In previous shared PIC method, it is likely that interrupt and IO finish are
passed to xen by only one
hypercall xc_evtchn_notify.

But now we may need to use two hypercalls xc_evtchn_notify and set_level
hypercalls,
I think this is reason of another 50% performance degradation.

Batching multicall may be a good idea,
The only question is how and when we batch xc_evtchn_notify and set_level
hypercall,

When xc_evtchn_notify is called, there may be several set_level hypercall should
be called,
But set_level hypercall is based on irq line level, how to "remember"
several set_level hypercalls?
Maybe we need to change set_level hypercall interface.
> 
>> I prepare to use shared IOSAPIC to deliver interrupt from
>> Qemu to VTI domain.
>> In IPF side, PIC is not needed,
>> In the same time, we can assign more interrupt pins(24) to qemu.
> 
> I moved x86 away from this on purpose, to obtain a clean abstraction.
> I don''t think it''s a good idea for ia64 to step backwards
here.
Actually I don''t want to do this, if there are better solution to pull
back performance.

-- Anthony
> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 7:43 am, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> Batching multicall may be a good idea,
> The only question is how and when we batch xc_evtchn_notify and set_level
> hypercall,
> 
> When xc_evtchn_notify is called, there may be several set_level hypercall
> should be called,
> But set_level hypercall is based on irq line level, how to
"remember" several
> set_level hypercalls?
> Maybe we need to change set_level hypercall interface.
Have an array of set_level hypercall structures, and an array of multicall
structures. Fill them in at the point we currently do the hypercall. Flush
when:
 A) the array is already full; or
 B) when qemu passes through its event loop.

Make the arrays 16 entries large, for example, will be plenty.

Use the same mechanism for the notification (i.e., add to the multicall
array, to be flushed by qemu''s main loop).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 7:55 am, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk>
wrote:
> Have an array of set_level hypercall structures, and an array of multicall
> structures. Fill them in at the point we currently do the hypercall. Flush
> when:
>  A) the array is already full; or
>  B) when qemu passes through its event loop.
> 
> Make the arrays 16 entries large, for example, will be plenty.
> 
> Use the same mechanism for the notification (i.e., add to the multicall
> array, to be flushed by qemu''s main loop).
To clarify, by event/main loop I mean: Flush just before qemu blocks
(otherwise multicall can be held for unbounded time, unless we set a
batching timeout which hopefully we can avoid needing to do).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 15:59:
> On 22/11/06 7:55 am, "Keir Fraser"
<Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> Have an array of set_level hypercall structures, and an array of
>> multicall structures. Fill them in at the point we currently do the
>>  hypercall. Flush when: A) the array is already full; or
>>  B) when qemu passes through its event loop.
>> 
>> Make the arrays 16 entries large, for example, will be plenty.
>> 
>> Use the same mechanism for the notification (i.e., add to the
>> multicall array, to be flushed by qemu''s main loop).
> 
> To clarify, by event/main loop I mean: Flush just before qemu blocks
> (otherwise multicall can be held for unbounded time, unless we set a
> batching timeout which hopefully we can avoid needing to do).
Seems good,

Is it possible that we add some file descriptors for some interrupts in qemu
block (select)?
Then if there are any interrupts, qemu will be woke up.
In this case, we can only call multi-call just after qemu block.

--Anthony.



> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 08:16, "Xu, Anthony" <anthony.xu@intel.com> wrote:
>> To clarify, by event/main loop I mean: Flush just before qemu blocks
>> (otherwise multicall can be held for unbounded time, unless we set a
>> batching timeout which hopefully we can avoid needing to do).
> 
> Seems good,
> 
> Is it possible that we add some file descriptors for some interrupts in
qemu
> block (select)?
> Then if there are any interrupts, qemu will be woke up.
> In this case, we can only call multi-call just after qemu block.
I''m not sure what you mean.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 17:22:
> On 22/11/06 08:16, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>>> To clarify, by event/main loop I mean: Flush just before qemu
blocks
>>> (otherwise multicall can be held for unbounded time, unless we set
a
>>> batching timeout which hopefully we can avoid needing to do).
Why otherwise multicall can be held for unbounded time?

--Anthony

> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 09:24, "Xu, Anthony" <anthony.xu@intel.com> wrote:
>>>> To clarify, by event/main loop I mean: Flush just before qemu
blocks
>>>> (otherwise multicall can be held for unbounded time, unless we
set a
>>>> batching timeout which hopefully we can avoid needing to do).
> 
> Why otherwise multicall can be held for unbounded time?
Qemu only wakes up for device-model accesses. We don''t know when the
next of
those will be. So we should flush multicalls before the potentially blocking
select().

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 17:26:
> On 22/11/06 09:24, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>>>>> To clarify, by event/main loop I mean: Flush just before
qemu
>>>>> blocks (otherwise multicall can be held for unbounded time,
>>>>> unless we set a batching timeout which hopefully we can
avoid
>>>>> needing to do). 
>> 
>> Why otherwise multicall can be held for unbounded time?
> 
> Qemu only wakes up for device-model accesses. We don''t know when
the
> next of those will be. So we should flush multicalls before the
> potentially blocking select().
There are two threads, one is qemu thread, the other is IDE DMA thread,
In IDE DMA thread, when it finishing DMA opereration, it will set irq, but it
doesn''t  try to wakeup qemu thread. So if qemu thread is sleeping at
the same time,
this interrupt may be delivered until qemu thread wakes up, the time may be
10 msec.
So we need a mechanism for IDE DMA thread to wake up Qemu thread.

What''s your opinion?

Thanks,
Anthony



> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 09:38, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> There are two threads, one is qemu thread, the other is IDE DMA thread,
> In IDE DMA thread, when it finishing DMA opereration, it will set irq, but
it
> doesn''t  try to wakeup qemu thread. So if qemu thread is sleeping
at the same
> time,
> this interrupt may be delivered until qemu thread wakes up, the time may be
> 10 msec.
> So we need a mechanism for IDE DMA thread to wake up Qemu thread.
> 
> What''s your opinion?
Did the IDE code really need to made multithreaded? I suppose it''s a
better
model for the stub domain plans...  Anyway, it''s a pain here because it
will
require the shadow wire bitmap to be updated with atomic accesses and the
multicall state to be per-thread or to be protected with a mutex. Each
thread should flush multicall state before it blocks.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 17:48:
> On 22/11/06 09:38, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
> Did the IDE code really need to made multithreaded? 
This code was added about one year ago, the purpose is definitely to improve
IDE performance. I don''t have the performance data.
We can image,
If dom0 and hvm domain are running on different CPU,
It will improve parallel of hvm domain and qemu IDE device.

>I suppose it''s a
> better model for the stub domain plans...  Anyway, it''s a pain
here
Maybe we can let performance data to decide this.

> because it will require the shadow wire bitmap to be updated with
> atomic accesses and the multicall state to be per-thread or to be
> protected with a mutex. Each thread should flush multicall state
> before it blocks. 
I prefer atomic access, we used it in shared PIC.
If each thread flush multicall seperately,
There are some extra hypercalls.


-- Anthony
> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com> wrote:
>> because it will require the shadow wire bitmap to be updated with
>> atomic accesses and the multicall state to be per-thread or to be
>> protected with a mutex. Each thread should flush multicall state
>> before it blocks.
> 
> I prefer atomic access, we used it in shared PIC.
> If each thread flush multicall seperately,
> There are some extra hypercalls.
Since the threads run independently there seems little choice but for each
to be able to flush. If the IDE DMA support had been properly integrated
into the qemu select() event loop this would not be an issue.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 18:28:
> On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>> I prefer atomic access, we used it in shared PIC.
>> If each thread flush multicall seperately, There are some extra
>> hypercalls.
> 
> Since the threads run independently there seems little choice but for
> each to be able to flush. If the IDE DMA support had been properly
> integrated into the qemu select() event loop this would not be an
> issue.

Agree, we can use pipe to integrate IDE DMA support into select.
But we still need to use atomic access to shawdow line.

--Anthony
   
> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月22日 18:28:
> On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
> Since the threads run independently there seems little choice but for
> each to be able to flush. If the IDE DMA support had been properly
> integrated into the qemu select() event loop this would not be an
> issue. 
Hi keir,

I''m trying to use multi hypercall to resolve this issue.
But I found it is difficult to use multi hypercall in qemu.
Because a hypercall have different argument in qemu, libxc and dom0 kernel.

For example,
In qemu,
xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
In libxc
ioctl(xce_handle, IOCTL_EVTCHN_NOTIFY, &notify);
In dom0 kernel
HYPERVISOR_event_channel_op(EVTCHNOP_send, &send);

If we want to use multi hypercall in qemu, we need to get the interface
information
of dom0 kernel. We may need to add several more hypercall.

I propose to change xc_hvm_set_irq_level functionality to deliver interrupt and
wakeup cpu
at the same time.
For example,
 xc_hvm_set_irq_and_wakeup_cpu(int xc_handle, domid_t dom, int irq, int level,
int cpu)
This hypercall will deliver interrupt irq and wake up cpu.

What''s your opinion?


Thanks,
Anthony






> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir,

This patch is for comments, and it is based on IPF, it may not apply to IA32
side.

This patch delivers interrupt and IO finish in the same hypercall
xc_hvm_set_irq_and_wakeup_cpu,
This eliminate all unnecessary hypercalls.
In the meantime, I add a mechanism for IDE DMA thread to wakeup qemu main
block(select) to
Deliver IDE DMA interrupt.

Yes all this can improve KB performance greatly,
But KB is still incurring 2%~3% degradation compared to shared PIC irq line.

I guest the reason as following,

In shared PIC irq line method,
IDE DMA thread doesn''t wake up Qemu thread, after it has set irq line,
Select is only back when there is IO operation or other interrupt like keyborad,
mouse.
This is not likely to delay the IDE interrupt deliverring, because every time
xen returns to guest OS,
It will sync share irq line, and IDE interrupt can be delivered without dom0
waking up hvm domain.

There is no xc_evtchn_notify called due to IDE DMA interrupt deliver.


But In currently hypercall method,
There are many hypercalls called due to IDE DMA interrupt deliver.

So there are still 2%~3% performance degradation.


What''s your opinion?


Thanks,
Anthony




Xu, Anthony write on 2006年11月24日 10:00:
> Keir Fraser write on 2006年11月22日 18:28:
>> On 22/11/06 10:23, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
>> 
>> Since the threads run independently there seems little choice but for
>> each to be able to flush. If the IDE DMA support had been properly
>> integrated into the qemu select() event loop this would not be an
>> issue.
> 
> Hi keir,
> 
> I''m trying to use multi hypercall to resolve this issue.
> But I found it is difficult to use multi hypercall in qemu.
> Because a hypercall have different argument in qemu, libxc and dom0
> kernel. 
> 
> For example,
> In qemu,
> xc_evtchn_notify(xce_handle, ioreq_local_port[send_vcpu]);
> In libxc
> ioctl(xce_handle, IOCTL_EVTCHN_NOTIFY, &notify);
> In dom0 kernel
> HYPERVISOR_event_channel_op(EVTCHNOP_send, &send);
> 
> If we want to use multi hypercall in qemu, we need to get the
> interface information 
> of dom0 kernel. We may need to add several more hypercall.
> 
> I propose to change xc_hvm_set_irq_level functionality to deliver
> interrupt and wakeup cpu at the same time.
> For example,
>  xc_hvm_set_irq_and_wakeup_cpu(int xc_handle, domid_t dom, int irq,
> int level, int cpu) This hypercall will deliver interrupt irq and
> wake up cpu. 
> 
> What''s your opinion?
> 
> 
> Thanks,
> Anthony
> 
> 
> 
> 
> 
> 
> 
>> 
>>  -- Keir
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/11/06 07:08, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> This patch is for comments, and it is based on IPF, it may not apply to
IA32
> side.
> 
> This patch delivers interrupt and IO finish in the same hypercall
> xc_hvm_set_irq_and_wakeup_cpu,
> This eliminate all unnecessary hypercalls.
> In the meantime, I add a mechanism for IDE DMA thread to wakeup qemu main
> block(select) to
> Deliver IDE DMA interrupt.
Firstly, this patch is not against unstable tip.

Secondly, we should make multicalls work rather than kludge a
set_irq_and_notify_io_done type of hypercall. Applications are free to
include any of the Xen public headers. We really just need a xc_multicall()
API function.

Thirdly, either we should keep the independent IDE-DMA thread or it should
be entirely incorporated into the main qemu thread. Are pipe writes much
faster than just doing a hypercall? If much slower, why is that? Could you
work out a way of generically making IPC hypercalls faster (particularly
from privileged user space -- could you trap straight to the hypervisor from
user space rather than bouncing through guest kernel?).

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月24日 17:47:
> On 24/11/06 07:08, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>> This patch is for comments, and it is based on IPF, it may not apply
>> to IA32 side. 
>> 
>> This patch delivers interrupt and IO finish in the same hypercall
>> xc_hvm_set_irq_and_wakeup_cpu, This eliminate all unnecessary
>> hypercalls. In the meantime, I add a mechanism for IDE DMA thread to
>> wakeup qemu main block(select) to
>> Deliver IDE DMA interrupt.
> 
> Firstly, this patch is not against unstable tip.
> 
> Secondly, we should make multicalls work rather than kludge a
> set_irq_and_notify_io_done type of hypercall. Applications are free
> to include any of the Xen public headers. We really just need a
> xc_multicall() API function.   
This patch is not for checkin, and this patch is similar with multicall,
I just want to see if multicall can get same performance as share PIC line.
> 
> Thirdly, either we should keep the independent IDE-DMA thread or it
> should be entirely incorporated into the main qemu thread. Are pipe
> writes much faster than just doing a hypercall? If much slower, why
> is that? Could you work out a way of generically making IPC
> hypercalls faster (particularly from privileged user space -- could
> you trap straight to the hypervisor from user space rather than
> bouncing through guest kernel?).      
Trapping straight to the hypervisor from user space definitely can improve
performance,
But it will also improve performance with share PIC irq line.
The degradation is caused by extra hypercalls to deliver irq line status.
So the degradation should still exist.

Firstly I just want to verify whether hypercall can get similar or better
formance compared with
share PIC irq line.

After reading share PIC line code again, I have following finding.

 xc_evtchn_notify acctually can''t notify hvm domain interrupt
happening.
See below code segment, the event channel can only wake up hvm domain which
is blocked by IO operation.
So in fact dom0 doesn''t notify hvm domain interrupt happen,
Means some xc_evtchn_nofity, which are intended to notify hvm domain, are
unnecessary.
Hvm domain finds these pending interrupts just because it is trapped into
hypercall
at least HZ times per second, may be more frequent due to a lot of VMM_EXIT.


        if ( rchn->consumer_is_xen )
        {
            /* Xen consumers need notification only if they are blocked. */
            if ( test_and_clear_bit(_VCPUF_blocked_in_xen,
                                    &rvcpu->vcpu_flags) )
                vcpu_wake(rvcpu);
        }
        else


Until now, I can''t see hypercall is faster than share PIC line.

Can you enlighten me why we use hypercall?


> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/11/06 14:45, "Xu, Anthony" <anthony.xu@intel.com> wrote:
> Until now, I can''t see hypercall is faster than share PIC line.
> 
> Can you enlighten me why we use hypercall?
I didn''t change it for speed but because it''s cleaner. I
believe it can be
made no slower, with a bit of effort. x86 certainly won''t be reverting
to
the shared state.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser write on 2006年11月24日 23:29:
> On 24/11/06 14:45, "Xu, Anthony" <anthony.xu@intel.com>
wrote:
> 
>> Until now, I can''t see hypercall is faster than share PIC
line.
>> 
>> Can you enlighten me why we use hypercall?
> 
> I didn''t change it for speed but because it''s cleaner. I
believe it
> can be made no slower, with a bit of effort. x86 certainly won''t
be
> reverting to the shared state.  
Can''t share PIC line be done in a clean way?
I don''t think so.
I think which one get better performance wins.

--Anthony

> 
>  -- Keir
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel