Here are some details on our horrible benchmark results: System setup: Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller Benchmark: PARKBENCH low-level pingpong benchmark comms1_mpi MPI library: Scali MPI (http://www.scali.com/) Xen distribution: 2.0 stable, downloaded with bitkeeper Xen configuration: default, i.e. we just leave config-2.6.8.1-xen0 the way it is CPU: Single Intel(R) Pentium(R) 4 CPU 3.40GHz Memory: 1 GB Bandwidth measurements: Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL: 786.034 MByte/s Between two nodes each running only xen domain 0: 56.480 MByte/s A graph over the measurements - x = message length, y = time: http://www.idi.ntnu.no/~havarbj/tmp/plot.png Cheers, Havard ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
Mark A. Williamson
2004-Oct-15 14:02 UTC
[Xen-devel] Re: Xen & I/O in clusters - problems!
> Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL: > > 786.034 MByte/s > > Between two nodes each running only xen domain 0: > > 56.480 MByte/sThat''s surprising - I''d have expected any performance problems to involve unpriv domains somehow. We''ve never had any performance problems when just running domain 0, even when the code was still under development... It''d be interesting to see your config file (I know it''s just the default but it''d be interesting for comparison as there''s no obvious reason for your problems). Cheers, Mark ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Fri, Oct 15, 2004 at 02:02:31PM +0000, Mark A. Williamson wrote:> > Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL: > > > > 786.034 MByte/s > > > > Between two nodes each running only xen domain 0: > > > > 56.480 MByte/s > > That''s surprising - I''d have expected any performance problems to involve > unpriv domains somehow. We''ve never had any performance problems when just > running domain 0, even when the code was still under development... >What''s possibly even more funny is that when I do the same benchmark localhost <-> localhost, ie. through the loopback interface, on domain 0, the bandwidth is halved. CPU use is ~100% during both these benchmarks on domain 0 (50% per process in the last benchmark). This indicates to me that the bandwidth depends on CPU resources. Some heavy processing is happening somewhere. I suspect this might have something to do with the MPI library scaMPI, which is supposed to be more closely linked with the lower layers of the OSI protocol stack or something. I will investigate it further. Another thing that might be worth noting is that we''re using the 2.6.8.1 kernel on domain 0 as opposed to a 2.4 kernel. I don''t think that would make any difference, though (except for the fact that modules aren''t loaded, but I don''t think we need any of them anyway).> It''d be interesting to see your config file (I know it''s just the default but > it''d be interesting for comparison as there''s no obvious reason for your > problems). >Attached Cheers, Havard
John Enok Vollestad
2004-Oct-16 20:05 UTC
[Xen-devel] Re: Xen & I/O in clusters - problems!
Håvard Bjerke <Havard.Bjerke <at> idi.ntnu.no> writes:> > On Fri, Oct 15, 2004 at 02:02:31PM +0000, Mark A. Williamson wrote: > > > Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL: > > > > > > 786.034 MByte/s > > > > > > Between two nodes each running only xen domain 0: > > > > > > 56.480 MByte/s > > > > That''s surprising - I''d have expected any performance problems to involve > > unpriv domains somehow. We''ve never had any performance problems when just > > running domain 0, even when the code was still under development... > > > > What''s possibly even more funny is that when I do the same benchmark localhost<-> localhost, ie. through> the loopback interface, on domain 0, the bandwidth is halved. CPU use is ~100%during both these> benchmarks on domain 0 (50% per process in the last benchmark). This indicatesto me that the bandwidth> depends on CPU resources. Some heavy processing is happening somewhere. > > I suspect this might have something to do with the MPI library scaMPI, whichis supposed to be more closely> linked with the lower layers of the OSI protocol stack or something. I willinvestigate it further. Around 2000 the driver used a mix of polling and interrupts to get the latency down. If I remember correctly there done polling for abount half the time of an interrupt. To get the adapters manipulate memory directly the driver also have to allocate memory in physical continuous blocks. Exact how the reading and writing of this areas is done through the drivers I do not know but this should not make any performance hit except if Xen have any issues with MMU manipulation. You could send SCALI an email.. -- John Enok ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel