Hartmut Brandt
2014-Apr-10 11:22 UTC
Some gruesome moments with performance of FreeBSD at over 20K interfaces
On Thu, 10 Apr 2014, Vladislav Prodan wrote: VP>> On Wed, 9 Apr 2014, Vladislav Prodan wrote: VP>> VP>> VP>b) Service bsnmpd started at 12K interfaces, but immediately loaded CPU VP>> VP>at 80-100% VP>> VP>> I could imagine that this is because of the statistics polling. bsnmp VP>> implements 64-bit interface statistics but we have only 32-bit statistics VP>> in the kernel. So it polls the kernel statistics for each interface on a VP>> rate that ensures that 32-bit don't overflow. If the interfaces are GBit VP>> or, worse, 10GBit interfaces the polling rate is rather high (in the order VP>> of seconds). VP>> VP>> You should either make sure that the interfaces report sensible bitrates VP>> (I doubt that 20k interfaces could all be GBit interfaces) or force a slower VP>> polling interval by setting begemotIfForcePoll.0 to some large value. VP>> VP>> harti VP>> VP> VP>Thanks for the tip. VP> VP>At least 10 interfaces to be 1Gb, and the rest no more than 50M. VP>BegemotIfForcePoll parameter in this case a little help, you will be forced to stand another value for Gigabit Interface begemotIfForcePoll ... Yeah. There is only one parameter. You are running -STABLE, right? On current the statistics are 64-bit (I wonder whether the operation on these values is automatically atomic on all our platforms, though). harti
Ermal Luçi
2014-Apr-10 12:38 UTC
Some gruesome moments with performance of FreeBSD at over 20K interfaces
Another note related to Q-in-Q. You would probably be better of creating standard vlans for the first vlan layer and use ng_vlan for the second++ part of the Q-in-Q on top of the first ones. This also give better usability and will speedup a bit your times. On Thu, Apr 10, 2014 at 1:22 PM, Hartmut Brandt <hartmut.brandt at dlr.de>wrote:> On Thu, 10 Apr 2014, Vladislav Prodan wrote: > > VP>> On Wed, 9 Apr 2014, Vladislav Prodan wrote: > VP>> > VP>> VP>b) Service bsnmpd started at 12K interfaces, but immediately > loaded CPU > VP>> VP>at 80-100% > VP>> > VP>> I could imagine that this is because of the statistics polling. bsnmp > VP>> implements 64-bit interface statistics but we have only 32-bit > statistics > VP>> in the kernel. So it polls the kernel statistics for each interface > on a > VP>> rate that ensures that 32-bit don't overflow. If the interfaces are > GBit > VP>> or, worse, 10GBit interfaces the polling rate is rather high (in the > order > VP>> of seconds). > VP>> > VP>> You should either make sure that the interfaces report sensible > bitrates > VP>> (I doubt that 20k interfaces could all be GBit interfaces) or force a > slower > VP>> polling interval by setting begemotIfForcePoll.0 to some large value. > VP>> > VP>> harti > VP>> > VP> > VP>Thanks for the tip. > VP> > VP>At least 10 interfaces to be 1Gb, and the rest no more than 50M. > VP>BegemotIfForcePoll parameter in this case a little help, you will be > forced to stand another value for Gigabit Interface begemotIfForcePoll ... > > Yeah. There is only one parameter. > > You are running -STABLE, right? On current the statistics are 64-bit > (I wonder whether the operation on these values is automatically atomic on > all our platforms, though). > > harti > _______________________________________________ > freebsd-net at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org" >-- Ermal