Pete Kay
2010-Feb-24 19:30 UTC
[CentOS] Desperately need help with multi-core NIC performance
Hi, I am running a VOIP application on Centos 5.3. It is a 16 core box with 12 G of mem and all what it does is passing packets. What happens is that at around 2K channels running g711 ( 64k) codec, all eth0 is used up and no more traffic can go through. I have checked google and it talked about interrupt scheduler. does anyone know how to configure the kernel to allow it to use all CPSs for socket transmission of UDP packets? Any pointer will be greatly appreciated. pete
Kwan Lowe
2010-Feb-24 19:48 UTC
[CentOS] Desperately need help with multi-core NIC performance
On Wed, Feb 24, 2010 at 2:30 PM, Pete Kay <petedao at gmail.com> wrote:> Hi, > > I am running a VOIP application on Centos 5.3. ?It is a 16 core box > with 12 G of mem and all what it does is passing packets. > > What happens is that at around 2K channels running g711 ( 64k) codec, > all eth0 is used up and no more traffic can go through. > > I have checked google and it talked about interrupt scheduler. > > does anyone know how to configure the kernel to allow it to use all > CPSs for socket transmission of UDP packets? > > Any pointer will be greatly appreciated.There are some configuration parameters in the xinetd.conf file that relate to CPS for services handled through xinetd. However, it may be worthwhile to configure the service to listen directly to the port in very high usage cases. Not sure if this is what you mean..
John R Pierce
2010-Feb-24 19:55 UTC
[CentOS] Desperately need help with multi-core NIC performance
Pete Kay wrote:> What happens is that at around 2K channels running g711 ( 64k) codec, > all eth0 is used up and no more traffic can go through. >2000 * 64kbit/sec is 128Mbit/sec, not counting any additional protocol overhead are you sending and receiving this data on the same ethernet port? then its double that total traffic. what speed is your ethernet ? if all this data is flowing through a single UDP port, then its pretty hard to spread it across multiple CPUs as only one application thread is reading or writing from that socket and doing whatever sorts of routing you're doing. you may well be better off with a server that has fewer faster cores.
Tim Nelson
2010-Feb-24 20:59 UTC
[CentOS] Desperately need help with multi-core NIC performance
----- "Pete Kay" <petedao at gmail.com> wrote:> Hi, > > I am running a VOIP application on Centos 5.3. It is a 16 core box > with 12 G of mem and all what it does is passing packets. > > What happens is that at around 2K channels running g711 ( 64k) codec, > all eth0 is used up and no more traffic can go through. > > I have checked google and it talked about interrupt scheduler. > > does anyone know how to configure the kernel to allow it to use all > CPSs for socket transmission of UDP packets? > > Any pointer will be greatly appreciated. > > peteWhat kind of NICs are you using? --Tim
Rob Kampen
2010-Feb-24 21:02 UTC
[CentOS] Desperately need help with multi-core NIC performance
Pete Kay wrote:> Hi, > > I am running a VOIP application on Centos 5.3. It is a 16 core box > with 12 G of mem and all what it does is passing packets. > > What happens is that at around 2K channels running g711 ( 64k) codec, > all eth0 is used up and no more traffic can go through. >So that's about 128Mbits per second not counting UDP packet overheads I guesstimate that you cannot get much more than that through a 1G ethernet NIC - it is not CP bound, rather you are nearing the capacity of a single NIC. HTH Rob> I have checked google and it talked about interrupt scheduler. > > does anyone know how to configure the kernel to allow it to use all > CPSs for socket transmission of UDP packets? > > Any pointer will be greatly appreciated. > > pete > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-------------- next part -------------- A non-text attachment was scrubbed... Name: rkampen.vcf Type: text/x-vcard Size: 121 bytes Desc: not available URL: <http://lists.centos.org/pipermail/centos/attachments/20100224/70290305/attachment.vcf>
Bobby
2010-Feb-24 22:39 UTC
[CentOS] Desperately need help with multi-core NIC performance
On Wednesday 24 February 2010 14:55:22 John R Pierce wrote:> Pete Kay wrote: > > What happens is that at around 2K channels running g711 ( 64k) codec, > > all eth0 is used up and no more traffic can go through. > > 2000 * 64kbit/sec is 128Mbit/sec, not counting any additional protocolActually if you are running g.711 over VoIP, the overhead brings it up to something like 87Kbps per direction. Bobby
J.Witvliet at MINDEF.NL
2010-Feb-25 10:11 UTC
[CentOS] Desperately need help with multi-core NIC performance
-----Original Message----- From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On Behalf Of nate Sent: Thursday, February 25, 2010 1:03 AM To: centos at centos.org Subject: Re: [CentOS] Desperately need help with multi-core NIC performance Pete Kay wrote:> Hi > > So is that the limit? I have heard people being able to run like 10K > call channels before max out CPU cap.I would verify the network throughput of your system to make sure the NIC/switch/etc are functioning normally, I use iperf to do this, really simple tool to use just need two systems. On a good network you should be able to sustain roughly 900+Mbit/s with standard frame sizes and iperf on a single gigE link(hopefully with no tuning) ------------------------------------------------------------ Just for reference... On a slightly less-then-optimal network i could get 900Mbps between an HP DL380 and an old IBM-T43, both with 1Gb nics and no tuning at all, through corporate network. Between two DL380 with 10GB nics i got a performance boost (2500Mb -> 6500Mb) after i raised the MTU to 8K hw ______________________________________________________________________ Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages.