Rune and I just finished writing a report on using virtualization with Xen in clusters, with focus on network performance. You may download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf We hope you find it interesting! cheers, Hvard ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Jacob Gorm Hansen
2004-Nov-26 10:44 UTC
Re: [Xen-devel] Report on Virtualization in Clusters
Håvard Bjerke wrote:> Rune and I just finished writing a report on using virtualization with Xen in clusters, with focus on network performance. You may download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf > > We hope you find it interesting!Yes, very interesting. Good work. It seems from your tests without interrupt coalescing that Xen has a higher interrupt-handling overhead than native Linux. I suppose that with the new IO model, Xen needs to flush the TLBs for every interrupt that arrives. Do you think this is the reason for the overhead, or are the performance problems only due to your hardware being badly supported by Xen? Jacob ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Nov 26, 2004 at 11:44:03AM +0100, Jacob Gorm Hansen wrote:> Hvard Bjerke wrote: > >Rune and I just finished writing a report on using virtualization with Xen > >in clusters, with focus on network performance. You may download it from > >http://www.idi.ntnu.no/~havarbj/clust_virt.pdf > > > >We hope you find it interesting! > > Yes, very interesting. Good work. > > It seems from your tests without interrupt coalescing that Xen has a > higher interrupt-handling overhead than native Linux. I suppose that > with the new IO model, Xen needs to flush the TLBs for every interrupt > that arrives. Do you think this is the reason for the overhead, or are > the performance problems only due to your hardware being badly supported > by Xen?My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with ''ignorebiostables''. Thus using legacy PIC, everything seemed to behave normally except when not using IC. I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it''s the latter reason, otherwise I think it''s the former. It would be quite easy to verify on the other cluster. I just couldn''t find out how to disable IC on the tg3 driver. The Xen developers could probably give a better answer, though. Hvard ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Rune Johan Andresen
2004-Nov-26 15:27 UTC
Re: [Xen-devel] Report on Virtualization in Clusters
The application benchmarking behaved similar on the two clusters (with different HW), which makes it more probably it is the overhead. As mentioned we didn''t disable IC on the tg3 drive so we don''t really know. - Rune On Nov 26, 2004, at 11:44 AM, Jacob Gorm Hansen wrote:> Håvard Bjerke wrote: >> Rune and I just finished writing a report on using virtualization >> with Xen in clusters, with focus on network performance. You may >> download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf >> We hope you find it interesting! > > Yes, very interesting. Good work. > > It seems from your tests without interrupt coalescing that Xen has a > higher interrupt-handling overhead than native Linux. I suppose that > with the new IO model, Xen needs to flush the TLBs for every interrupt > that arrives. Do you think this is the reason for the overhead, or are > the performance problems only due to your hardware being badly > supported by Xen? > > Jacob------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Please note that the application benchmarking was performed with IC enabled. The application benchmarking results do not reflect the problems that were experienced with IC disabled. Hvard On Fri, Nov 26, 2004 at 04:27:10PM +0100, Rune Johan Andresen wrote:> > The application benchmarking behaved similar on the two clusters (with > different HW), which makes > it more probably it is the overhead. As mentioned we didn''t disable IC > on the tg3 drive so we don''t > really know. > > - Rune > > On Nov 26, 2004, at 11:44 AM, Jacob Gorm Hansen wrote: > > >Hvard Bjerke wrote: > >>Rune and I just finished writing a report on using virtualization > >>with Xen in clusters, with focus on network performance. You may > >>download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf > >>We hope you find it interesting! > > > >Yes, very interesting. Good work. > > > >It seems from your tests without interrupt coalescing that Xen has a > >higher interrupt-handling overhead than native Linux. I suppose that > >with the new IO model, Xen needs to flush the TLBs for every interrupt > >that arrives. Do you think this is the reason for the overhead, or are > >the performance problems only due to your hardware being badly > >supported by Xen? > > > >Jacob >------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> > It seems from your tests without interrupt coalescing that Xen has a > > higher interrupt-handling overhead than native Linux. I suppose that > > with the new IO model, Xen needs to flush the TLBs for every interrupt > > that arrives. Do you think this is the reason for the overhead, or are > > the performance problems only due to your hardware being badly supported > > by Xen? > > My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with ''ignorebiostables''. Thus using legacy PIC, everything seemed to behave normally except when not using IC. > > I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it''s the latter reason, otherwise I think it''s the former.The fact that you had to disable the IOAPIC with ignorebiostables is rather concerning. It certainly sounds like Xen is really struggling on your platform. What motherboard chipset are the machines in your cluster? We''re hoping that this kind of problem will go away when Xen 2.1 gets released, as we''re planing on leaving IOAPIC initialisation to domain 0, hence using the existing Linux code. Domain 0 will then have an interface to tell Xen just what it needs to know about the IOAPIC configuration (i.e. how to ACK interrupts). Ian ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Sun, Nov 28, 2004 at 11:17:03AM +0000, Ian Pratt wrote:> > > > It seems from your tests without interrupt coalescing that Xen has a > > > higher interrupt-handling overhead than native Linux. I suppose that > > > with the new IO model, Xen needs to flush the TLBs for every interrupt > > > that arrives. Do you think this is the reason for the overhead, or are > > > the performance problems only due to your hardware being badly supported > > > by Xen? > > > > My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with ''ignorebiostables''. Thus using legacy PIC, everything seemed to behave normally except when not using IC. > > > > I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it''s the latter reason, otherwise I think it''s the former. > > The fact that you had to disable the IOAPIC with ignorebiostables > is rather concerning. It certainly sounds like Xen is really > struggling on your platform. What motherboard chipset are the > machines in your cluster?They are Intel 875P chipsets Hvard ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel