Alexandru Dragoi
2006-Sep-16 18:39 UTC
Performance problem on a linux bridge used for shaping.
Hello, Here is the situation. There is a machine with 3 intel gigabit card, 2 of them on PCI-X and in bridge, the 3rd is used only for management access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel from debian (testing) and htb with u32 on, i usually get about 30-40% software interrupts on CPU0 and CPU2, and without htb and u32, 10% less. Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I managed with smp_afinity to "move" irq of one card to a different CPU. In these circumstances, I have about 20% or a little less soft interrupts on each CPU without shaping, but about 60-70% os soft interrupts with shapping, and sometimes there is packet loss, also dropped packets are shown on ifconfig. The htb script is same. I have u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI, of course. I can''t think on anything else that can cause the problem, but seem to be something in the kernel. Here is the output of lspci: # lspci 0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub (rev 0a) 0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers (rev 0a) 0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A (rev 0a) 0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express Port A1 (rev 0a) 0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge (rev 02) 0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal Host Controller (rev 02) 0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal Host Controller (rev 02) 0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer (rev 02) 0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable Interrupt Controller (rev 02) 0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced Host Controller (rev 02) 0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a) 0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface Controller (rev 02) 0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage Controller (rev 02) 0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02) 0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02) 0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit Ethernet Controller (Copper) (rev 02) 0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) 0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05) The traffic is somewhere at 40kpps of traffic and 120mbit up, 120mbit/down. Suggestions about better hardware and kernel, or links with docs about these are really welcomed. Bye
Simon Lodal
2006-Sep-16 21:17 UTC
Re: Performance problem on a linux bridge used for shaping.
I have similar hardware, load and trouble. Interrupts are only sent to one cpu, instead of all of them, because that was only overhead. I think the default was changed somewhere around 2.6.10 or .12, but I have forgotten the url. There is a CONFIG_IRQBALANCE option in the kernel, but last time I checked (2.6.16) it did not work very well; almost never does anything. So I have turned it off. I use the userspace irqbalance daemon that periodically sets the smp_affinity''s, with the effect that ingoing and outgoing traffic are handled by each their cpu (assuming that no other interrupts or processes are significant). That helps some. But we only shape in one direction, and it can not help spread the shaping load between the CPU''s. There is also an acpi_irq_balance kernel parameter (not related to the kernel irq balancer), which apparently uses the APIC to do interrupt round-robin. It worked surprisingly well (perfect, actually) on an old dual celeron that I tested; the network interrupts are spread nicely and evenly across the cpu''s. It is probably very chipset dependent, and I have not yet tested it on the firewalls. But as I understand from this: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2006/09/14#netconf2006_day2 it might not even be an advantage, since the current shaping code would just make the cpu''s step on each others toes. Regards, Simon On Saturday 16 September 2006 20:39, Alexandru Dragoi wrote:> Hello, > > Here is the situation. There is a machine with 3 intel gigabit card, 2 > of them on PCI-X and in bridge, the 3rd is used only for management > access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel > from debian (testing) and htb with u32 on, i usually get about 30-40% > software interrupts on CPU0 and CPU2, and without htb and u32, 10% less. > Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I > managed with smp_afinity to "move" irq of one card to a different CPU. > In these circumstances, I have about 20% or a little less soft > interrupts on each CPU without shaping, but about 60-70% os soft > interrupts with shapping, and sometimes there is packet loss, also > dropped packets are shown on ifconfig. The htb script is same. I have > u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI, > of course. I can''t think on anything else that can cause the problem, > but seem to be something in the kernel. Here is the output of lspci: > > # lspci > 0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub > (rev 0a) > 0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers > (rev 0a) > 0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express > Port A (rev 0a) > 0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express > Port A1 (rev 0a) > 0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge > (rev 02) > 0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal > Host Controller (rev 02) > 0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal > Host Controller (rev 02) > 0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer > (rev 02) > 0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable > Interrupt Controller (rev 02) > 0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced > Host Controller (rev 02) > 0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a) > 0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface > Controller (rev 02) > 0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage > Controller (rev 02) > 0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02) > 0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit > Ethernet Controller (Copper) (rev 02) > 0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit > Ethernet Controller (Copper) (rev 02) > 0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL > (rev 27) > 0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit > Ethernet Controller (rev 05) > > The traffic is somewhere at 40kpps of traffic and 120mbit up, > 120mbit/down. Suggestions about better hardware and kernel, or links > with docs about these are really welcomed. > > Bye > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Alexandru Dragoi
2006-Sep-17 16:33 UTC
Re: Performance problem on a linux bridge used for shaping.
Simon Lodal wrote:>I have similar hardware, load and trouble. > >Interrupts are only sent to one cpu, instead of all of them, because that was >only overhead. I think the default was changed somewhere around 2.6.10 >or .12, but I have forgotten the url. > >There is a CONFIG_IRQBALANCE option in the kernel, but last time I checked >(2.6.16) it did not work very well; almost never does anything. So I have >turned it off. I use the userspace irqbalance daemon that periodically sets >the smp_affinity''s, with the effect that ingoing and outgoing traffic are >handled by each their cpu (assuming that no other interrupts or processes are >significant). That helps some. But we only shape in one direction, and it can >not help spread the shaping load between the CPU''s. > >There is also an acpi_irq_balance kernel parameter (not related to the kernel >irq balancer), which apparently uses the APIC to do interrupt round-robin. It >worked surprisingly well (perfect, actually) on an old dual celeron that I >tested; the network interrupts are spread nicely and evenly across the cpu''s. >It is probably very chipset dependent, and I have not yet tested it on the >firewalls. > >But as I understand from this: >http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2006/09/14#netconf2006_day2 >it might not even be an advantage, since the current shaping code would just >make the cpu''s step on each others toes. > > >Regards, >Simon > > >On Saturday 16 September 2006 20:39, Alexandru Dragoi wrote: > > >>Hello, >> >>Here is the situation. There is a machine with 3 intel gigabit card, 2 >>of them on PCI-X and in bridge, the 3rd is used only for management >>access. The machine is a dual Xeon 2.8GHz with HT. With 2.6.8 kernel >>from debian (testing) and htb with u32 on, i usually get about 30-40% >>software interrupts on CPU0 and CPU2, and without htb and u32, 10% less. >>Now, if I boot with 2.6.17.9 kernel, first all irqs are on same CPU. I >>managed with smp_afinity to "move" irq of one card to a different CPU. >>In these circumstances, I have about 20% or a little less soft >>interrupts on each CPU without shaping, but about 60-70% os soft >>interrupts with shapping, and sometimes there is packet loss, also >>dropped packets are shown on ifconfig. The htb script is same. I have >>u32 performance counters enabled on u32 in 2.6.17.9. I also have NAPI, >>of course. I can''t think on anything else that can cause the problem, >>but seem to be something in the kernel. Here is the output of lspci: >> >># lspci >>0000:00:00.0 Host bridge: Intel Corporation E7320 Memory Controller Hub >>(rev 0a) >>0000:00:00.1 ff00: Intel Corporation E7320 Error Reporting Registers >>(rev 0a) >>0000:00:02.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express >>Port A (rev 0a) >>0000:00:03.0 PCI bridge: Intel Corporation E7525/E7520/E7320 PCI Express >>Port A1 (rev 0a) >>0000:00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge >>(rev 02) >>0000:00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal >>Host Controller (rev 02) >>0000:00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal >>Host Controller (rev 02) >>0000:00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer >>(rev 02) >>0000:00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable >>Interrupt Controller (rev 02) >>0000:00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced >>Host Controller (rev 02) >>0000:00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a) >>0000:00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface >>Controller (rev 02) >>0000:00:1f.2 IDE interface: Intel Corporation 6300ESB SATA Storage >>Controller (rev 02) >>0000:00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02) >>0000:03:02.0 Ethernet controller: Intel Corporation 82544EI Gigabit >>Ethernet Controller (Copper) (rev 02) >>0000:03:03.0 Ethernet controller: Intel Corporation 82544EI Gigabit >>Ethernet Controller (Copper) (rev 02) >>0000:04:02.0 VGA compatible controller: ATI Technologies Inc Rage XL >>(rev 27) >>0000:04:03.0 Ethernet controller: Intel Corporation 82541GI Gigabit >>Ethernet Controller (rev 05) >> >>The traffic is somewhere at 40kpps of traffic and 120mbit up, >>120mbit/down. Suggestions about better hardware and kernel, or links >>with docs about these are really welcomed. >> >>Bye >> >>_______________________________________________ >>LARTC mailing list >>LARTC@mailman.ds9a.nl >>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >> >>Somebody recomended me to use Gigabit Intel dual port and kernel 2.6.9, because some distrubutions stopped at 2.6.9. Now, if things changed on SMP since 2.6.10 or 2.6.12, I believe those distributions (perhaps some Redhat and CentOS) know what are they doing.