Xu, Dongxiao
2009-Sep-30 07:21 UTC
[Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
One of the VNIF driver''s scalability issues is the high event channel frequency. It''s highly related to physical NIC''s interrupt frequency in dom0, which could be 20K HZ in some situation. The high frequency event channel notification makes the guest and dom0 CPU utilization at a high value, especially in multi-VM cases. The following two patches uses smart polling mechanism to replace event notification to reduce the CPU Utilization. Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Best Regards, -- Dongxiao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2009-Sep-30 07:54 UTC
Re: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
Are these patches against pv_ops? They don''t apply to 2.6.18 so I assume so. In which case you might want to Cc Jeremy as he manages the pv_ops tree. I can''t do anything with them. If you spin versions against 2.6.18 I will apply them. And don''t forget to patch xen-unstable''s version of ring.h -- that''s the ''master version'' of that public header. -- Keir On 30/09/2009 08:21, "Xu, Dongxiao" <dongxiao.xu@intel.com> wrote:> One of the VNIF driver''s scalability issues is the high event > channel frequency. It''s highly related to physical NIC''s interrupt > frequency in dom0, which could be 20K HZ in some situation. The > high frequency event channel notification makes the guest and dom0 > CPU utilization at a high value, especially in multi-VM cases. > The following two patches uses smart polling mechanism to > replace event notification to reduce the CPU Utilization. > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> > > Best Regards, > -- Dongxiao > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
James Harper
2009-Sep-30 09:17 UTC
RE: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead ofevent notification.
> > One of the VNIF driver''s scalability issues is the high event > channel frequency. It''s highly related to physical NIC''s interrupt > frequency in dom0, which could be 20K HZ in some situation. The > high frequency event channel notification makes the guest and dom0 > CPU utilization at a high value, especially in multi-VM cases. > The following two patches uses smart polling mechanism to > replace event notification to reduce the CPU Utilization. > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> >I really think that this problem would be better solved in the backend. In its simplest form, the backend simply needs to know the maximum acceptable packet latency between putting a packet on the rx ring and notifying the frontend (in addition to the existing event notification mechanism). The algorithm is something like: Send Packet to frontend: . put packet on ring . if prod crosses event then notify as normal . if event is > prod and there is no outstanding timer set then set the timer to now + 1ms (or whatever the maximum latency is set to). On Timer: . notify the frontend A more complicated form could also: . define the maximum amount of ''work'' per notify (eg always notify when there is (eg) 256k of data or more, regardless of timers or other criteria) . define a ''max time since last packet'' vs ''max time since first packet'' timers, to allow more packets to build up if they are coming in in a steady stream. A question though, a lot of hardware adapters already support interrupt moderation, which would result in ''bursty'' traffic in the backend, does that affect this sort of optimization? I have some comments on your patches too, I''ll follow up in a separate email. James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Xu, Dongxiao
2009-Sep-30 17:16 UTC
RE: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead ofevent notification.
James,
Putting this logic in netfront could almost eliminate all the event
notification, and from the data we can see that this approach brings more CPU
utilization downgrade. Also the flag set in xenstore give options to users, and
they can choose to use it or not. get_coalesce/set_coalesce interfaces give user
to balance the latency and CPU utilization.
The ''bursty'' traffic in netback will not affect our
optimization. The polling in netfront will automatically stop working if there
are no data packets sent/received during 100ms, which is much longer than the
interrupt interval.
Thanks!
Dongxiao
________________________________________
From: James Harper [james.harper@bendigoit.com.au]
Sent: Wednesday, September 30, 2009 2:17 AM
To: Xu, Dongxiao; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead
ofevent notification.
>
> One of the VNIF driver''s scalability issues is the high event
> channel frequency. It''s highly related to physical NIC''s
interrupt
> frequency in dom0, which could be 20K HZ in some situation. The
> high frequency event channel notification makes the guest and dom0
> CPU utilization at a high value, especially in multi-VM cases.
> The following two patches uses smart polling mechanism to
> replace event notification to reduce the CPU Utilization.
>
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
>
I really think that this problem would be better solved in the backend.
In its simplest form, the backend simply needs to know the maximum
acceptable packet latency between putting a packet on the rx ring and
notifying the frontend (in addition to the existing event notification
mechanism).
The algorithm is something like:
Send Packet to frontend:
. put packet on ring
. if prod crosses event then notify as normal
. if event is > prod and there is no outstanding timer set then set the
timer to now + 1ms (or whatever the maximum latency is set to).
On Timer:
. notify the frontend
A more complicated form could also:
. define the maximum amount of ''work'' per notify (eg always
notify when
there is (eg) 256k of data or more, regardless of timers or other
criteria)
. define a ''max time since last packet'' vs ''max time
since first packet''
timers, to allow more packets to build up if they are coming in in a
steady stream.
A question though, a lot of hardware adapters already support interrupt
moderation, which would result in ''bursty'' traffic in the
backend, does
that affect this sort of optimization?
I have some comments on your patches too, I''ll follow up in a separate
email.
James
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Xu, Dongxiao
2009-Sep-30 17:33 UTC
RE: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
Keir,
Thanks for reminder, currently this patch is for PV-ops tree. Later I will
send out patch for xen-unstable''s ring.h.
Thanks!
Dongxiao
________________________________________
From: Keir Fraser [keir.fraser@eu.citrix.com]
Sent: Wednesday, September 30, 2009 12:54 AM
To: Xu, Dongxiao; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of
event notification.
Are these patches against pv_ops? They don''t apply to 2.6.18 so I
assume so.
In which case you might want to Cc Jeremy as he manages the pv_ops tree. I
can''t do anything with them. If you spin versions against 2.6.18 I will
apply them. And don''t forget to patch xen-unstable''s version
of ring.h --
that''s the ''master version'' of that public header.
-- Keir
On 30/09/2009 08:21, "Xu, Dongxiao" <dongxiao.xu@intel.com>
wrote:
> One of the VNIF driver''s scalability issues is the high event
> channel frequency. It''s highly related to physical NIC''s
interrupt
> frequency in dom0, which could be 20K HZ in some situation. The
> high frequency event channel notification makes the guest and dom0
> CPU utilization at a high value, especially in multi-VM cases.
> The following two patches uses smart polling mechanism to
> replace event notification to reduce the CPU Utilization.
>
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
>
> Best Regards,
> -- Dongxiao
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Xu, Dongxiao
2009-Sep-30 18:17 UTC
[Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
Resend the patch and CC Jeremy. One of the VNIF driver''s scalability issues is the high event channel frequency. It''s highly related to physical NIC''s interrupt frequency in dom0, which could be 20K HZ in some situation. The high frequency event channel notification makes the guest and dom0 CPU utilization at a high value, especially in multi-VM cases. The following two patches uses smart polling mechanism to replace event notification to reduce the CPU Utilization. Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Best Regards, -- Dongxiao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Sep-30 23:41 UTC
Re: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
On 09/30/09 11:17, Xu, Dongxiao wrote:> Resend the patch and CC Jeremy. > > One of the VNIF driver''s scalability issues is the high event > channel frequency. It''s highly related to physical NIC''s interrupt > frequency in dom0, which could be 20K HZ in some situation. The > high frequency event channel notification makes the guest and dom0 > CPU utilization at a high value, especially in multi-VM cases. > The following two patches uses smart polling mechanism to > replace event notification to reduce the CPU Utilization. > > Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> >I''m having difficulty applying these because they''re in quoted-printable MIME format. Could you resend them as attachments? Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Xu, Dongxiao
2009-Oct-01 00:11 UTC
[Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
Send the patches in attachment. One of the VNIF driver''s scalability issues is the high event channel frequency. It''s highly related to physical NIC''s interrupt frequency in dom0, which could be 20K HZ in some situation. The high frequency event channel notification makes the guest and dom0 CPU utilization at a high value, especially in multi-VM cases. The following two patches uses smart polling mechanism to replace event notification to reduce the CPU Utilization. Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Best Regards, -- Dongxiao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Oct-01 00:21 UTC
Re: [Xen-devel][PV-ops][PATCH 0/2] VNIF: Using smart polling instead of event notification.
On 09/30/09 17:11, Xu, Dongxiao wrote:> Send the patches in attachment. > > One of the VNIF driver''s scalability issues is the high event > channel frequency. It''s highly related to physical NIC''s interrupt > frequency in dom0, which could be 20K HZ in some situation. The > high frequency event channel notification makes the guest and dom0 > CPU utilization at a high value, especially in multi-VM cases. > The following two patches uses smart polling mechanism to > replace event notification to reduce the CPU Utilization. >Thanks, applied. I fixed up some whitespace issues; please run kernel patches through scripts/checkpatch.pl before submission. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel