Xu, Dongxiao
2010-Apr-29 14:27 UTC
[Xen-devel][Pv-ops][PATCH 0/4 v2] Netback multiple threads support
Current netback uses one pair of tasklets for Tx/Rx data transaction. Netback tasklet could only run at one CPU at a time, and it is used to serve all the netfronts. Therefore it has become a performance bottle neck. This patch is to use multiple tasklet pairs to replace the current single pair in dom0. Assuming that Dom0 has CPUNR VCPUs, we define CPUNR kinds of tasklets pair (CPUNR for Tx, and CPUNR for Rx). Each pare of tasklets serve specific group of netfronts. Also for those global and static variables, we duplicated them for each group in order to avoid the spinlock. PATCH 01: Generilize static/global variables into ''struct xen_netbk''. PATCH 02: Introduce a new struct page_ext. PATCH 03: Multiple tasklets support. PATCH 04: Use Kernel thread to replace the tasklet. Recently I re-tested the patchset with Intel 10G multi-queue NIC device, and use 10 outside 1G NICs to do netperf tests with that 10G NIC. Case 1: Dom0 has more than 10 vcpus pinned with each physical CPU. With the patchset, the performance is 2x of the original throughput. Case 2: Dom0 has 4 vcpus pinned with 4 physical CPUs. With the patchset, the performance is 3.7x of the original throughput. when we test this patch, we found that the domain_lock in grant table operation (gnttab_copy()) becomes a bottle neck. We temporarily remove the global domain_lock to achieve good performance. Thanks, Dongxiao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Xu, Dongxiao
2010-Apr-29 14:53 UTC
[Xen-devel][Pv-ops][PATCH 0/4 v2] Netback multiple threads support
Seems that this file is missing... Resend it. Current netback uses one pair of tasklets for Tx/Rx data transaction. Netback tasklet could only run at one CPU at a time, and it is used to serve all the netfronts. Therefore it has become a performance bottle neck. This patch is to use multiple tasklet pairs to replace the current single pair in dom0. Assuming that Dom0 has CPUNR VCPUs, we define CPUNR kinds of tasklets pair (CPUNR for Tx, and CPUNR for Rx). Each pare of tasklets serve specific group of netfronts. Also for those global and static variables, we duplicated them for each group in order to avoid the spinlock. PATCH 01: Generilize static/global variables into ''struct xen_netbk''. PATCH 02: Introduce a new struct page_ext. PATCH 03: Multiple tasklets support. PATCH 04: Use Kernel thread to replace the tasklet. Recently I re-tested the patchset with Intel 10G multi-queue NIC device, and use 10 outside 1G NICs to do netperf tests with that 10G NIC. Case 1: Dom0 has more than 10 vcpus pinned with each physical CPU. With the patchset, the performance is 2x of the original throughput. Case 2: Dom0 has 4 vcpus pinned with 4 physical CPUs. With the patchset, the performance is 3.7x of the original throughput. when we test this patch, we found that the domain_lock in grant table operation (gnttab_copy()) becomes a bottle neck. We temporarily remove the global domain_lock to achieve good performance. Thanks, Dongxiao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-May-03 16:02 UTC
Re: [Xen-devel][Pv-ops][PATCH 0/4 v2] Netback multiple threads support
> when we test this patch, we found that the domain_lock in grant table > operation (gnttab_copy()) becomes a bottle neck. We temporarily > remove the global domain_lock to achieve good performance.Without that change (global domain_lock in place), what are the performance numbers? Is there a forthcoming patch to make the gnttab_copy have a much finer grained lock? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel