Hi there! I am working a lot with VoIP in my company, so I thought to use linux bridge functionality together with tc to emulate delay, jitter, packet loss, duplication, reordering etc. for testing purposes in our lab against our VoIP products. I just recognized, that a basic bridge just with it''s minumum configuration of 2 network interfaces creates latency of approx. 5ms on very low traffic. This seems to be independent on CPU speed. I tried on 2 GHz PC while having just 64kBit traffic with packet size of about 300bytes. I am using Knoppix 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For some reason they put iproute2 041019 on this distro, which is intended to be used for kernel 2.6.9. I am aware of remastering the CD, but have to check if it is possible to recompile the kernel for the remaster. back to my question: where does this latency come from? "top" shows almost no load while the bridge is handling traffic, so how come? is there some timer-granularity which can be set in the kernel, is the latency normal, or what could cause it else? Thank you very much in advance! /Christian
Hi, It becomes possible to play with a bunch of CPU-related timers in the 2.6.13-rc series, which MAY help (but no guarantees). The latest tree also has some scheduling fixes which probably won''t make much of a difference to you. Standard distro kernels tend to be compromises, which means they''ll be OK for everything but great at nothing. If you want to squeeze every last bit of performance out of the machine, you''ll need to do some kernel configuration work. The latest "vanilla" kernel is 2.6.13-rc3 and the latest Andrew Morton release is 2.6.13-rc3-mm1 (The differences that MAY be useful to you is that the -mm release has some driver fixes for ethernet cards.) If you roll your own kernel and are wanting to use it for a bridge setup, my guess is you want to use the server settings for preempt (no forced preemption - ie: pretty much disable it) but would likely want to use the desktop settings for the timer frequency (1000 Hz) as that gives the fastest response to events. (If you''re using an SMP machine, 250 Hz might be better, as SMP doesn''t like lots of interrupts.) Depending on who you ask and what phase the moon is in, different people give different opinions about whether to compile in or use modules. Compiled-in drivers MAY be marginally faster and MAY eat fractionally less kernel memory, which MIGHT trim down latency a little. If that''s not serious voodoo enough, don''t compile in any network layers you''re not using. Every layer is absolutely going to add latency, because it is extra code to run. Finally, and this is going to be the hardest step, it MAY be possible to get the Linux kernel to compile with the latest Intel C compiler. If you''re using a genuine Intel processor, the speedup is something like 40% - for AMD or other ix86 processors, GCC is either equal to or faster than Intel''s compiler. The problem with using Intel''s C compiler is that it has very different ideas on what is ok than GCC, so the kernel won''t necessarily compile. Sometimes people put icc patches out, to fix this, but not all kernels have patches and the kernel of interest is a pre-release, making patches less likely in the event they are needed. Any of these steps should trim a little latency off, and if you somehow manage all of them, you should get quite a decent improvement. Whether the improvement is worth the effort required is another matter. --- "Christian Konecny (VI/SEA)" <christian.konecny@ericsson.com> wrote:> Hi there! > > I am working a lot with VoIP in my company, so I > thought to use linux bridge functionality together > with tc to emulate delay, jitter, packet loss, > duplication, reordering etc. for testing purposes in > our lab against our VoIP products. > I just recognized, that a basic bridge just with > it''s minumum configuration of 2 network interfaces > creates latency of approx. 5ms on very low traffic. > This seems to be independent on CPU speed. I tried > on 2 GHz PC while having just 64kBit traffic with > packet size of about 300bytes. > I am using Knoppix 3.82 which is actually a debian > Live-CD Linux, Kernel 2.6.11. > For some reason they put iproute2 041019 on this > distro, which is intended to be used for kernel > 2.6.9. > I am aware of remastering the CD, but have to check > if it is possible to recompile the kernel for the > remaster. > > back to my question: where does this latency come > from? > "top" shows almost no load while the bridge is > handling traffic, so how come? > is there some timer-granularity which can be set in > the kernel, is the latency normal, or what could > cause it else? > > Thank you very much in advance! > > /Christian > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc>__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Thanks a lot for these very detailed tuning tips. I am not sure whether this effort really becomes necesary. Maybe there is still some module active in my distro causing this latency issue... I would highly appreciate if anyone of you could either confirm similar latency issues or report about no latence on your bridge ? Actually I check this with ethereal by making an RTP stream analysis on my voice media stream. The VoIP device generates continously data packets every 30ms. If I capture a trace on either side of the bridge, I can see that the bridge will add jitter of about 5ms. You could use also iperf or something similar to generate traffic.
Hi, Christian Konecny (VI/SEA) schrieb:> I just recognized, that a basic bridge just with it''s > minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. > This seems to be independent on CPU speed. I tried on > 2 GHz PC while having just 64kBit traffic with packet > size of about 300bytes.That''s strange. On my bridge with 4 network interfaces, the additional latency is always below 0.5 ms, even if I''m pushing 400 MBit/s through the machine and a kernel compile is running at 100% CPU. Network interfaces are PCIe GBit from Syskonnect, the machine is an Athlon64 at 2 GHz. Even if the clock speed is halved by powersave the additional latency will not go above 0.9 ms. Kernel is vanilla 2.6.11.x. Regards, Carl-Daniel -- http://www.hailfinger.org/
On Fri, 22 Jul 2005 09:55:25 +0200 "Christian Konecny (VI/SEA)" <christian.konecny@ericsson.com> wrote:> Hi there! > > I am working a lot with VoIP in my company, so I thought to use linux > bridge functionality together with tc to emulate delay, jitter, > packet loss, duplication, reordering etc. for testing purposes in our > lab against our VoIP products. I just recognized, that a basic bridge > just with it''s minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. This seems to be > independent on CPU speed. I tried on 2 GHz PC while having just > 64kBit traffic with packet size of about 300bytes. I am using Knoppix > 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For > some reason they put iproute2 041019 on this distro, which is > intended to be used for kernel 2.6.9. I am aware of remastering the > CD, but have to check if it is possible to recompile the kernel for > the remaster. > > back to my question: where does this latency come from? > "top" shows almost no load while the bridge is handling traffic, so > how come? is there some timer-granularity which can be set in the > kernel, is the latency normal, or what could cause it else? > > Thank you very much in advance! > > /Christian >What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration.
-----Original Message----- From: Carl-Daniel Hailfinger [mailto:c-d.hailfinger.devel.2005@gmx.net] Sent: Freitag, 22. Juli 2005 17:28 To: Christian Konecny (VI/SEA) Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Latency of Linux Bridge Carl-Daniel wrote: ============That''s strange. On my bridge with 4 network interfaces, the additional latency is always below 0.5 ms, even if I''m pushing 400 MBit/s through the machine and a kernel compile is running at 100% CPU. Network interfaces are PCIe GBit from Syskonnect, the machine is an Athlon64 at 2 GHz. Even if the clock speed is halved by powersave the additional latency will not go above 0.9 ms. Kernel is vanilla 2.6.11.x. ============= Did you use certain specific compile options? I have changed now from (Knoppix) Debian to (Slax) Slackware running now 2.6.12.2 and have exactly the same on each machine. top shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec. I can still measure a variable delta between packets of 5ms-10ms. How did you measure your latency values? I am using 3 PCs, one is the Linux-Box acting as bridge (brctl addbr br0; brctl addif br0 eth0; brctl addif br0 eth1), the other 2 PCs are connected to either side of the bridge, running ethereal. In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms. If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge. The difference is always in steps of 5ms. so, on sending side is always 30ms difference between each packet on receiving side - after the bridge - the delta is then 25,30,35, or 40ms. Is this really different in your setup? thanks, Christian PS: Servus aus Wien!
you wrote: ===============What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. =============== the latency is already there without having netem running. Only kernel + brctl to configure the bridge shows the described latency of 5ms. Hardware does not matter. I tried on a 500MHz Pentium and also on Pentium III with 3GHz and 1GB RAM. top always shows me a CPU usage less than 0.5%, load average 0.1 while the bridge is handling roughly 64kBit/sec. I can still measure a variable delta between packets of 5ms-10ms. I am using 3 PCs, one is the Linux-Box acting as bridge setup scripts, nothing in special: brctl addbr br0 brctl addif br0 eth0 brctl addif br0 eth1 ifconfig eth0 up ifconfig eth1 up ifconfig br0 up the other 2 PCs are connected to either side of the bridge, running ethereal. In my case I have 2 Phone systems generating continous traffic. Each system is sending out packets every 30ms. If I compare the 2 traces I can see that after the bridge the timings are different then before the bridge. The difference is always in steps of 5ms. so, on sending side is always 30ms difference between each packet on receiving side - after the bridge - the delta is then 25,30,35, or 40ms. i am not sure, what you mean with "The amount depends on the requested delay". I would require an accuracy of about 0.5ms - 1ms on top of that I will then start with "tc qdisc netem" to start my emulation lateron. But as long as the accuracy of the bridge as such is that imprecise, this is not the best basis for my target. you wrote: ==============The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration. ==============Sorry, but I am not so good in kernel compiling. Where can I find these values. I was grep-ing the whole kernel-sources and could not find it. also make menuconfig did not provide me these settings (at least I could not find them). Could you please give me a hint where to start searching? thanks, Christian -----Original Message----- From: Stephen Hemminger [mailto:shemminger@osdl.org] Sent: Samstag, 23. Juli 2005 01:49 To: Christian Konecny (VI/SEA) Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Latency of Linux Bridge On Fri, 22 Jul 2005 09:55:25 +0200 "Christian Konecny (VI/SEA)" <christian.konecny@ericsson.com> wrote:> Hi there! > > I am working a lot with VoIP in my company, so I thought to use linux > bridge functionality together with tc to emulate delay, jitter, > packet loss, duplication, reordering etc. for testing purposes in our > lab against our VoIP products. I just recognized, that a basic bridge > just with it''s minumum configuration of 2 network interfaces creates > latency of approx. 5ms on very low traffic. This seems to be > independent on CPU speed. I tried on 2 GHz PC while having just > 64kBit traffic with packet size of about 300bytes. I am using Knoppix > 3.82 which is actually a debian Live-CD Linux, Kernel 2.6.11. For > some reason they put iproute2 041019 on this distro, which is > intended to be used for kernel 2.6.9. I am aware of remastering the > CD, but have to check if it is possible to recompile the kernel for > the remaster. > > back to my question: where does this latency come from? > "top" shows almost no load while the bridge is handling traffic, so > how come? is there some timer-granularity which can be set in the > kernel, is the latency normal, or what could cause it else? > > Thank you very much in advance! > > /Christian >What hardware, and setup scripts? If you are using netem then it can add latency. The amount depends on the requested delay and the HZ value of the kernel and the choice of PSCHED_CLOCK_SOURCE in the kernel configuration.
Thank you very much for your feedback! Carl-Daniel Hailfinger wrote: ========================> How did you measure your latency values? (with bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 963ms rtt min/avg/max/mdev = 0.642/0.914/2.487/0.277 ms, ipg/ewma 0.964/1.023 ms (same config without bridge in between) linux:~ # ping -f -c 1000 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. --- 192.168.0.1 ping statistics --- 1000 packets transmitted, 1000 received, 0% packet loss, time 883ms rtt min/avg/max/mdev = 0.767/0.834/4.925/0.173 ms, ipg/ewma 0.884/0.814 ms ======================== Same here. Ping delivers me the same values. I also tried following ping -f -c 1000 192.168.0.1 -i 0.03 this produces pretty the same traffic like the one I have from my telephony systems. I did a sniffer-trace on that and could not see any deviations as big as 5ms. Therefore I assume the bridge will handle ICMP easier/faster than normal UDP-traffic. I also did another test with iperf (http://dast.nlanr.net/Projects/Iperf/) with similar values: iperf -c 192.168.0.1 -u -len 90 -t 7200 20k This produces UDP traffic with 90ms packets every 30ms. Now I have again the deviations in steps of 5ms. So, icmp might not be a good tool to measure jitter in this terms. I am pretty sure you will have the same with iperf or any other method to produce UDP-traffic. Carl-Daniel Hailfinger wrote: ========================It is very possible that a bridge changes the timing distribtion (but the drastic effect you''re seeing shouldn''t happen). ========================I fully agree ;-) thanks, Christian PS: What I am doing wrong? Each mail from me is not put correctly into the corresponding position within the thread of the mailing list. Have M$ Outlook here...