Hi, Keir To support more interrupt vectors in Xen for more devices, especially for SR-IOV devices in a large system, we implemented per-cpu vector for Xen like Linux does. For SR-IOV devices, since each VF needs several separate vectors for interrupt delivery and global ~200 vectors in Xen is insufficient and easily run out after installing two or three such devices. Becaue SR-IOV devices are becoming popular now, and from this point of view, we have to extend vector resource space to make these devices work. As linux does, we implemented per-cpu vector for Xen to extend and scale vector resource to nr_cpus x ~200 in a system. BTW, the core logic of the patches is ported from upstream linux and then adapted for Xen. Patch 0001: Change nr_irqs to nr_irqs_gsi and make nr_irqs_gsi only used for GSI interrupts. Patch 0002: Modify Xen from vector-based interrupt infrastructure to IRQ-based one, and the big change is that one irq number is also allocated for MSI interrupt source, and the idea is same as Linux''s. Patch 0003: Implement per-cpu vector for xen. Most core logic(such as vector allocation algorithm, IRQ migration logic...) is ported from upstream Linux. About the patch quality, we have done enough testings against upstream, and no any regression found after applying this patchset. Please help to review. Comments are very appreicated! Thanks! Signed-off-by : Xiantao Zhang <xiantao.zhang@Intel.com> Xiantao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Keir Another issue or limitation is also found when we tested Per-CPU vector patch at our side. Currenlty, Xen uses fixmap to access MSI-X resouce, (e.g. msi-x tables), 32 pages 512 pages are reserved for 32-pae and x86_64 separately in fixmap section, but each MSI-X capable device(regardless of real device or virtual function) at least needs one page to map its resource, so these pages may easily run out with these devices, espeically on 32-bit platforms. For 64-bit platforms, we can reserve more pages to fix the issue, but for 32-bit platforms, it is not always safe to increase the number of pages due to limited virtual address space. We have one optional solution to fix the issue through dynamically map/unmap MSI-X tables when access it, but the concern is that it may cost much due to frequent access for the resource. What''s suggestion or good idea to address the issue ? Thanks! Xiantao Zhang, Xiantao wrote:> Hi, Keir > To support more interrupt vectors in Xen for more devices, > especially for SR-IOV devices in a large system, we implemented > per-cpu vector for Xen like Linux does. For SR-IOV devices, since > each VF needs several separate vectors for interrupt delivery and > global ~200 vectors in Xen is insufficient and easily run out after > installing two or three such devices. Becaue SR-IOV devices are > becoming popular now, and from this point of view, we have to extend > vector resource space to make these devices work. As linux does, we > implemented per-cpu vector for Xen to extend and scale vector > resource to nr_cpus x ~200 in a system. BTW, the core logic of the > patches is ported from upstream linux and then adapted for Xen. > > Patch 0001: Change nr_irqs to nr_irqs_gsi and make nr_irqs_gsi only > used for GSI interrupts. > Patch 0002: Modify Xen from vector-based interrupt infrastructure to > IRQ-based one, and the big change is that one irq number is also > allocated for MSI interrupt source, and the idea is same as Linux''s. > Patch 0003: Implement per-cpu vector for xen. Most core logic(such > as vector allocation algorithm, IRQ migration logic...) is ported > from upstream Linux. > About the patch quality, we have done enough testings against > upstream, and no any regression found after applying this patchset. > Please help to review. Comments are very appreicated! Thanks! > > Signed-off-by : Xiantao Zhang <xiantao.zhang@Intel.com> > > Xiantao_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/08/2009 06:45, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:> Hi, Keir > Another issue or limitation is also found when we tested Per-CPU vector > patch at our side. Currenlty, Xen uses fixmap to access MSI-X resouce, (e.g. > msi-x tables), 32 pages 512 pages are reserved for 32-pae and x86_64 > separately in fixmap section, but each MSI-X capable device(regardless of > real device or virtual function) at least needs one page to map its resource, > so these pages may easily run out with these devices, espeically on 32-bit > platforms. For 64-bit platforms, we can reserve more pages to fix the issue, > but for 32-bit platforms, it is not always safe to increase the number of > pages due to limited virtual address space. We have one optional solution to > fix the issue through dynamically map/unmap MSI-X tables when access it, but > the concern is that it may cost much due to frequent access for the resource. > What''s suggestion or good idea to address the issue ? Thanks! > XiantaoA lower-performance option for 32-bit Xen would be fine by me. Very few people should have reason not to run 64-bit Xen now. Those who do may well have old systems with no/few MSI-x devices. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel