Xen devel - Oct 2004 - Re: Xen & I/O in clusters - problems!

Here are some details on our horrible benchmark results:


System setup:

Ethernet controller: 
	Intel Corp. 82547GI Gigabit Ethernet Controller

Benchmark: 
	PARKBENCH low-level pingpong benchmark comms1_mpi

MPI library: 
	Scali MPI (http://www.scali.com/)

Xen distribution: 
	2.0 stable, downloaded with bitkeeper

Xen configuration: 
	default, i.e. we just leave config-2.6.8.1-xen0 the way it is

CPU: 
	Single Intel(R) Pentium(R) 4 CPU 3.40GHz

Memory: 
	1 GB


Bandwidth measurements:

Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL:

	786.034 MByte/s

Between two nodes each running only xen domain 0:

	56.480 MByte/s

A graph over the measurements - x = message length, y = time:

	http://www.idi.ntnu.no/~havarbj/tmp/plot.png


Cheers,
Havard


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL:
>
>  786.034 MByte/s
>
> Between two nodes each running only xen domain 0:
>
>  56.480 MByte/s
That''s surprising - I''d have expected any performance problems
to involve
unpriv domains somehow.  We''ve never had any performance problems when
just
running domain 0, even when the code was still under development...

It''d be interesting to see your config file (I know it''s just
the default but
it''d be interesting for comparison as there''s no obvious
reason for your
problems).

Cheers,
Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, Oct 15, 2004 at 02:02:31PM +0000, Mark A. Williamson
wrote:
> > Between two nodes running plain redhat EL3 with kernel 2.4.21-15.EL:
> >
> >  786.034 MByte/s
> >
> > Between two nodes each running only xen domain 0:
> >
> >  56.480 MByte/s
> 
> That''s surprising - I''d have expected any performance
problems to involve
> unpriv domains somehow.  We''ve never had any performance problems
when just
> running domain 0, even when the code was still under development...
> 
What''s possibly even more funny is that when I do the same benchmark
localhost <-> localhost, ie. through the loopback interface, on domain 0,
the bandwidth is halved. CPU use is ~100% during both these benchmarks on domain
0 (50% per process in the last benchmark). This indicates to me that the
bandwidth depends on CPU resources. Some heavy processing is happening
somewhere.

I suspect this might have something to do with the MPI library scaMPI, which is
supposed to be more closely linked with the lower layers of the OSI protocol
stack or something. I will investigate it further.

Another thing that might be worth noting is that we''re using the
2.6.8.1 kernel on domain 0 as opposed to a 2.4 kernel. I don''t think
that would make any difference, though (except for the fact that modules
aren''t loaded, but I don''t think we need any of them anyway).
> It''d be interesting to see your config file (I know it''s
just the default but
> it''d be interesting for comparison as there''s no obvious
reason for your
> problems).
> 
Attached

Cheers,
Havard

Håvard Bjerke <Havard.Bjerke <at> idi.ntnu.no> writes:
> 
> On Fri, Oct 15, 2004 at 02:02:31PM +0000, Mark A. Williamson wrote:
> > > Between two nodes running plain redhat EL3 with kernel
2.4.21-15.EL:
> > >
> > >  786.034 MByte/s
> > >
> > > Between two nodes each running only xen domain 0:
> > >
> > >  56.480 MByte/s
> > 
> > That''s surprising - I''d have expected any
performance problems to involve
> > unpriv domains somehow.  We''ve never had any performance
problems when just
> > running domain 0, even when the code was still under development...
> > 
> 
> What''s possibly even more funny is that when I do the same
benchmark localhost
<-> localhost, ie. through
> the loopback interface, on domain 0, the bandwidth is halved. CPU use is
~100%
during both these
> benchmarks on domain 0 (50% per process in the last benchmark). This
indicates
to me that the bandwidth
> depends on CPU resources. Some heavy processing is happening somewhere.
> 
> I suspect this might have something to do with the MPI library scaMPI,
which
is supposed to be more closely
> linked with the lower layers of the OSI protocol stack or something. I will
investigate it further.

Around 2000 the driver used a mix of polling and interrupts to get the latency
down.  If I remember correctly there done polling for abount half the time of an
interrupt.

To get the adapters manipulate memory directly the driver also have to allocate
memory in physical continuous blocks.  Exact how the reading and writing of this
areas is done through the drivers I do not know but this should not make any
performance hit except if Xen have any issues with MMU manipulation.

You could send SCALI an email..


-- 
John Enok



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel