Hi!
This should probably be added to "best practices"[1] or "common
problems"[2].
Two servers with 10x1GBit ethernet each were able to achieve around
1150MB/s (byte not bit) in network benchmarks conducted with iperf
(2.6.26/stock Debian Lenny).
Booting Xen (3.4.3-rc3 from Debian/unstable) with kernel 2.6.32.11 (pv-ops)
as Dom0 it turned out that the servers could not do more than about
700-800MB/s. (A long way of debugging/testing/benchmarking followed -- I
tried lots of things including Xen-4.0.0 ;-)
Looking at /proc/interrupts lead to the solution:
CPU0 CPU1 CPU2 CPU3
1500: 1859518 0 0 0 xen-pirq-msi eth9
1501: 2295607 0 0 0 xen-pirq-msi eth8
1502: 2720851 0 0 0 xen-pirq-msi eth7
1503: 3064408 0 0 0 xen-pirq-msi eth6
1504: 4573347 0 0 0 xen-pirq-msi eth5
1505: 4672864 0 0 0 xen-pirq-msi eth4
all interrups are pinned on CPU0; booting the system without Xen shows that
interrupts are equaly distributed over all the cores.
So the solution is quite easy: either manually balance interrupts by eg:
echo <BITMASK> > /proc/irq/<INTERRUPT>/smp_affinity
where <BITMASK> (in hex) is
1 -> CPU0
2 -> CPU1
4 -> CPU2
8 -> CPU3
and so on. This is actually a bit mask that allows to attach several
CPUs/cores to handle interrupts on native linux. In Xen this is disabled:
If you specify a mask like "3" (=1+2 which means CPU0 and CPU1) the
first
matching CPU will be chosen -- CPU0 in this case.
The easier and for most cases better solution will be to install
irqbalance[3] which Debian Lenny did not do automatically.
After installation of irqbalance I measured values around 1100MB/s which is
about 50MB/s less than native. I am not sure if manual balancing will bring
better results or if I should do more thorrow testing. On the other hand,
having 1100MB/s seems to be fast enough for what I plan to do! ;-)
There was however an interesting difference between Xen-3.4.3-rc3 and
Xen-4.0.0: With the very same kernel (2.6.32.11 pv-ops/Debian) Xen-3.4.3
got stuck for fractions of seconds while under high load whereas with
Xen-4.0.0 I did not experience anything similar. In terms of numbers
Xen-4.0.0 was between 1 and 10MB/s faster -- which is less than 1%.
best regards,
Adi Kriegisch
[1] http://wiki.xensource.com/xenwiki/XenBestPractices
[2] http://wiki.xensource.com/xenwiki/XenCommonProblems
[3] http://www.irqbalance.org
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users