asterisk users - Mar 2006 - Asterisk & SMP: Is irqbalance Redundant on 2.6 Kernels?

Asterisk users,

I posted the following email to the Fedora users list 
<https://www.redhat.com/archives/fedora-list/2006-March/msg04154.html> 
and it got no responses, so now I'm calling on your expertise.  Please 
take a look at it and share your knowledge on the subject with me.

Additionally, let me know if you believe what I am trying to do is 
critically flawed. It's possible that it would be better to allocate one 
processor to handle all of the IRQs and reserve the others for running 
processes.  I have some concerns about retransmits if packet data is 
split across the CPUs and re-assembled in a different order than 
expected.  I'm open to any suggestions on how to best balance Asterisk's
load on an SMP system.

===========================================================
List users,

I have a server running Fedora Core 3 with the 2.6.12-1.1376_FC3smp
kernel.  It has four physical CPUs and I would like to balance the IRQs
across them.  It is a large scale VOIP server running Asterisk, so my
main interest is balancing the IRQs generated by the ethernet device.
Obviously, there is heavy network activity, so if the IRQs are not
balanced the server will be CPU bound by the lone processor assigned to
handle them.

Initially, I solved this problem by adding the following line to
"/etc/rc.local" (82 is the IRQ of the ethernet device):

  echo 0f > /proc/irq/82/smp_affinity

This worked like a charm.  The IRQs from the ethernet device were
balanced across the four processors and each of their idle times was
comparable.  Unfortunately, when I duplicated this on our backup machine
(adjusting the IRQ accordingly), I discovered that something was
overwriting the value that I echoed to the "smp_affinity" file.  The
problem was that the primary machine had a service that included a
fifteen second sleep in its init script.  This service was started
*after* the irqbalance daemon and *before* rc.local.  Apparently
irqbalance runs on a ten second interval and (this part is speculation
based on my experience, as it is not the most well documented daemon out
there) it only updates the smp affinity of ethernet devices on the first
poll.  All this led to irqbalance overwriting the value I echoed to the
"smp_affinity" file on the backup machine, because it did not have the
service with the sleep in its init script installed.

My first question is, what is the point of the irqbalance daemon if the
kernel can handle balancing IRQs by itself?

Googling for irqbalance brought me to the documentation for the Debian
package <http://packages.debian.org/unstable/utils/irqbalance>, which
states "Useful mostly just for 2.4 kernels, or 2.6 kernels with
CONFIG_IRQBALANCE turned off."  I grepped through the kernel source, but
could not find this constant so as an experiment I turned off the
irqbalance daemon and rebooted the system.  Every "smp_affinity" file
under "/proc/irq/*/" now contained the default value of
"ffffffff" and I
confirmed that all IRQs were being balanced across the CPUs by checking
"/proc/interrupts".

Is it possible that irqbalance is just a holdover from the 2.4 kernel
days?  Can it be turned off on my kernel without any adverse affects?

I also ran the same experiment on a multi-processor SUSE box running the
2.6.5-52-smp kernel.  This machine did not have irqbalance installed and
the values in the "smp_affinity" files were all "ff".  On
this machine,
all of the interrupts were being handled by CPU0. Apparently, SUSE's
kernels are compiled without IRQ balancing enabled.

I've also done a little digging through "kernel-parameters.txt"
and I
came across a few interesting lines:

  acpi=     noirq -- do not use ACPI for IRQ routing

  acpi_irq_balance   [HW,ACPI] ACPI will balance active IRQs -- default
in APIC mode

  pci=      noacpi   [IA-32] Do not use ACPI for IRQ routing or for PCI
scanning.

  noapic             [SMP,APIC] Tells the kernel to not make use of any
IOAPICs that may be present in the system.

  noirqbalance       [IA-32,SMP,KNL] Disable kernel irq balancing

This is all more than a little confusing to me.  It appears that the
kernel's IRQ balancing is contingent on ACPI and APIC, but I'm far from
positive about this conclusion.  If someone could explain to me how
these different components interact with IRQ balancing, I would
appreciate it.  My fear is that in an attempt to slim down the server I
will turn off a component that will disable or silently cripple IRQ
balancing by the kernel.

Thank you,

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

I contacted the developer of irqbalance (Eric Dorland) and he responded 
as follows:

"Certainly it can't cause any serious problems, just performance ones. 
And yes, it is my understanding that later 2.6 can do irq balancing 
themselves if the right option is set when compiling it."

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

Matt Roth wrote:
> Asterisk users,
>
> I posted the following email to the Fedora users list 
>
<https://www.redhat.com/archives/fedora-list/2006-March/msg04154.html>
> and it got no responses, so now I'm calling on your expertise.  Please 
> take a look at it and share your knowledge on the subject with me.
>
> Additionally, let me know if you believe what I am trying to do is 
> critically flawed. It's possible that it would be better to allocate 
> one processor to handle all of the IRQs and reserve the others for 
> running processes.  I have some concerns about retransmits if packet 
> data is split across the CPUs and re-assembled in a different order 
> than expected.  I'm open to any suggestions on how to best balance 
> Asterisk's load on an SMP system.
>
> ===========================================================>
> List users,
>
> I have a server running Fedora Core 3 with the 2.6.12-1.1376_FC3smp
> kernel.  It has four physical CPUs and I would like to balance the IRQs
> across them.  It is a large scale VOIP server running Asterisk, so my
> main interest is balancing the IRQs generated by the ethernet device.
> Obviously, there is heavy network activity, so if the IRQs are not
> balanced the server will be CPU bound by the lone processor assigned to
> handle them.
>
> Initially, I solved this problem by adding the following line to
> "/etc/rc.local" (82 is the IRQ of the ethernet device):
>
>  echo 0f > /proc/irq/82/smp_affinity
>
> This worked like a charm.  The IRQs from the ethernet device were
> balanced across the four processors and each of their idle times was
> comparable.  Unfortunately, when I duplicated this on our backup machine
> (adjusting the IRQ accordingly), I discovered that something was
> overwriting the value that I echoed to the "smp_affinity" file. 
The
> problem was that the primary machine had a service that included a
> fifteen second sleep in its init script.  This service was started
> *after* the irqbalance daemon and *before* rc.local.  Apparently
> irqbalance runs on a ten second interval and (this part is speculation
> based on my experience, as it is not the most well documented daemon out
> there) it only updates the smp affinity of ethernet devices on the first
> poll.  All this led to irqbalance overwriting the value I echoed to the
> "smp_affinity" file on the backup machine, because it did not
have the
> service with the sleep in its init script installed.
>
> My first question is, what is the point of the irqbalance daemon if the
> kernel can handle balancing IRQs by itself?
>
> Googling for irqbalance brought me to the documentation for the Debian
> package <http://packages.debian.org/unstable/utils/irqbalance>, which
> states "Useful mostly just for 2.4 kernels, or 2.6 kernels with
> CONFIG_IRQBALANCE turned off."  I grepped through the kernel source,
but
> could not find this constant so as an experiment I turned off the
> irqbalance daemon and rebooted the system.  Every "smp_affinity"
file
> under "/proc/irq/*/" now contained the default value of
"ffffffff" and I
> confirmed that all IRQs were being balanced across the CPUs by checking
> "/proc/interrupts".
>
> Is it possible that irqbalance is just a holdover from the 2.4 kernel
> days?  Can it be turned off on my kernel without any adverse affects?
>
> I also ran the same experiment on a multi-processor SUSE box running the
> 2.6.5-52-smp kernel.  This machine did not have irqbalance installed and
> the values in the "smp_affinity" files were all "ff". 
On this machine,
> all of the interrupts were being handled by CPU0. Apparently, SUSE's
> kernels are compiled without IRQ balancing enabled.
>
> I've also done a little digging through
"kernel-parameters.txt" and I
> came across a few interesting lines:
>
>  acpi=     noirq -- do not use ACPI for IRQ routing
>
>  acpi_irq_balance   [HW,ACPI] ACPI will balance active IRQs -- default
> in APIC mode
>
>  pci=      noacpi   [IA-32] Do not use ACPI for IRQ routing or for PCI
> scanning.
>
>  noapic             [SMP,APIC] Tells the kernel to not make use of any
> IOAPICs that may be present in the system.
>
>  noirqbalance       [IA-32,SMP,KNL] Disable kernel irq balancing
>
> This is all more than a little confusing to me.  It appears that the
> kernel's IRQ balancing is contingent on ACPI and APIC, but I'm far
from
> positive about this conclusion.  If someone could explain to me how
> these different components interact with IRQ balancing, I would
> appreciate it.  My fear is that in an attempt to slim down the server I
> will turn off a component that will disable or silently cripple IRQ
> balancing by the kernel.
>
> Thank you,
>
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
> _______________________________________________
> --Bandwidth and Colocation provided by Easynews.com --
>
> Asterisk-Users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>

I contacted the developer of irqbalance (Eric Dorland) and he responded
as follows:

"Certainly it can't cause any serious problems, just performance ones.
And yes, it is my understanding that later 2.6 can do irq balancing
themselves if the right option is set when compiling it."

Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer

Matt Roth wrote:
> Asterisk users,
>
> I posted the following email to the Fedora users list 
>
<https://www.redhat.com/archives/fedora-list/2006-March/msg04154.html>
> and it got no responses, so now I'm calling on your expertise.  Please 
> take a look at it and share your knowledge on the subject with me.
>
> Additionally, let me know if you believe what I am trying to do is 
> critically flawed. It's possible that it would be better to allocate 
> one processor to handle all of the IRQs and reserve the others for 
> running processes.  I have some concerns about retransmits if packet 
> data is split across the CPUs and re-assembled in a different order 
> than expected.  I'm open to any suggestions on how to best balance 
> Asterisk's load on an SMP system.
>
> ===========================================================>
> List users,
>
> I have a server running Fedora Core 3 with the 2.6.12-1.1376_FC3smp
> kernel.  It has four physical CPUs and I would like to balance the IRQs
> across them.  It is a large scale VOIP server running Asterisk, so my
> main interest is balancing the IRQs generated by the ethernet device.
> Obviously, there is heavy network activity, so if the IRQs are not
> balanced the server will be CPU bound by the lone processor assigned to
> handle them.
>
> Initially, I solved this problem by adding the following line to
> "/etc/rc.local" (82 is the IRQ of the ethernet device):
>
>  echo 0f > /proc/irq/82/smp_affinity
>
> This worked like a charm.  The IRQs from the ethernet device were
> balanced across the four processors and each of their idle times was
> comparable.  Unfortunately, when I duplicated this on our backup machine
> (adjusting the IRQ accordingly), I discovered that something was
> overwriting the value that I echoed to the "smp_affinity" file. 
The
> problem was that the primary machine had a service that included a
> fifteen second sleep in its init script.  This service was started
> *after* the irqbalance daemon and *before* rc.local.  Apparently
> irqbalance runs on a ten second interval and (this part is speculation
> based on my experience, as it is not the most well documented daemon out
> there) it only updates the smp affinity of ethernet devices on the first
> poll.  All this led to irqbalance overwriting the value I echoed to the
> "smp_affinity" file on the backup machine, because it did not
have the
> service with the sleep in its init script installed.
>
> My first question is, what is the point of the irqbalance daemon if the
> kernel can handle balancing IRQs by itself?
>
> Googling for irqbalance brought me to the documentation for the Debian
> package <http://packages.debian.org/unstable/utils/irqbalance>, which
> states "Useful mostly just for 2.4 kernels, or 2.6 kernels with
> CONFIG_IRQBALANCE turned off."  I grepped through the kernel source,
but
> could not find this constant so as an experiment I turned off the
> irqbalance daemon and rebooted the system.  Every "smp_affinity"
file
> under "/proc/irq/*/" now contained the default value of
"ffffffff" and I
> confirmed that all IRQs were being balanced across the CPUs by checking
> "/proc/interrupts".
>
> Is it possible that irqbalance is just a holdover from the 2.4 kernel
> days?  Can it be turned off on my kernel without any adverse affects?
>
> I also ran the same experiment on a multi-processor SUSE box running the
> 2.6.5-52-smp kernel.  This machine did not have irqbalance installed and
> the values in the "smp_affinity" files were all "ff". 
On this machine,
> all of the interrupts were being handled by CPU0. Apparently, SUSE's
> kernels are compiled without IRQ balancing enabled.
>
> I've also done a little digging through
"kernel-parameters.txt" and I
> came across a few interesting lines:
>
>  acpi=     noirq -- do not use ACPI for IRQ routing
>
>  acpi_irq_balance   [HW,ACPI] ACPI will balance active IRQs -- default
> in APIC mode
>
>  pci=      noacpi   [IA-32] Do not use ACPI for IRQ routing or for PCI
> scanning.
>
>  noapic             [SMP,APIC] Tells the kernel to not make use of any
> IOAPICs that may be present in the system.
>
>  noirqbalance       [IA-32,SMP,KNL] Disable kernel irq balancing
>
> This is all more than a little confusing to me.  It appears that the
> kernel's IRQ balancing is contingent on ACPI and APIC, but I'm far
from
> positive about this conclusion.  If someone could explain to me how
> these different components interact with IRQ balancing, I would
> appreciate it.  My fear is that in an attempt to slim down the server I
> will turn off a component that will disable or silently cripple IRQ
> balancing by the kernel.
>
> Thank you,
>
> Matthew Roth
> InterMedia Marketing Solutions
> Software Engineer and Systems Developer
> _______________________________________________
> --Bandwidth and Colocation provided by Easynews.com --
>
> Asterisk-Users mailing list
> To UNSUBSCRIBE or update options visit:
>   http://lists.digium.com/mailman/listinfo/asterisk-users
>