Hello everyone, I''m a newbie experimenting with CBQ shaping and am facing a few problems. Can any of you please help? TEST SETUP: +---------------+ +----------------+ | 10.0.0.103 |----------->| 10.0.0.102 | +---------------+ +----------------+ 10.0.0.103: Linux, 100Mbit/s NIC 10.0.0.102: Windows, 100Mbit/s NIC, iperf tcp server (ports 2000 and 2001) WHAT I WANT TO DO: 1. Traffic from 10.0.0.103 to 10.0.0.102 port 2000 should always receive at least 60Mbit/s regardless of presence of other traffic. 2. In the absence of traffic to 10.0.0.102 port 2000, all other traffic should use all available bandwidth CBQ SETUP 1: #!/bin/bash # rate1 = 60Mbit/s RATE1=614400000 PRIO="prio 1" DEV="dev eth0" OPTION="allot 1514 maxburst 20 avpkt 1000" # reset qdiscs tc qdisc del $DEV root # root CBQ tc qdisc add $DEV root handle 10: cbq bandwidth 100mbit avpkt 1000 # 60 Mbit/s class tc class add $DEV parent 10:0 classid 10:1 cbq bandwidth 100mbit rate $RATE1 $OPTION $PRIO borrow # add filter tc filter add $DEV parent 10:0 protocol ip prio 3 handle 1 fw flowid 10:1 # mark packets iptables -A OUTPUT -t mangle -p tcp --dport 2000 -d 10.0.0.102 -j MARK --set-mark 1 OBSERVED RESULTS FOR SETUP 1: 1. A single iperf session to 10.0.0.102 port 2000 for 40 seconds reports 93.1 Mbit/s 2. Two simultaneous iperf sessions to 10.0.0.102 on ports 2000 and 2001 for 40 seconds each report 48.4 Mbit/s and 44.3 Mbit/s respectively 3. "tc -s -d class show dev eth0" shows the 10:1 class processing packets and I assume port 2001 traffic uses the root qdisc. Is this assumption right? CBQ SETUP 2: I added a CBQ class 10:2 with a rate of 10240000, prio 3, borrow parameter, a filter to direct port 2001 traffic to 10:2 and iptables rules to assign fwmark. OBSERVED RESULTS FOR SETUP 2: Almost the same as above results with traffic being directed to appropriate classes. QUESTIONS: 1. What am I doing wrong? Why doesn''t port 2000 traffic always receive at least 60 Mbit/s? 2. prio 1 offers higher priority than prio 3, right? Lower the number, higher the priority? 3. bandwidth parameter: I''ve seen examples where people always use the NIC bandwidth (100 Mbit/s) and some examples where people use the link bandwidth (say 6 Mbit/s for a DSL link). Which is right? 4. is it recommended that I have a class below the root CBQ and all other classes as sub classes of that class? Thanks! Muthu
[...]> RATE1=614400000tc(8) manpage extract: UNITS All parameters accept a floating point number, possibly followed by a unit. Bandwidths or rates can be specified in: kbps Kilobytes per second mbps Megabytes per second kbit Kilobits per second mbit Megabits per second bps or a bare number Bytes per second [...]> tc qdisc add $DEV root handle 10: cbq bandwidth 100mbit avpkt 1000Bandwidth is 100 Megabits per second.> # 60 Mbit/s class > tc class add $DEV parent 10:0 classid 10:1 cbq bandwidth 100mbit rate > $RATE1 $OPTION $PRIO borrowRate is 614400000 Bytes per second, roughly about 6 Gigabits per second. I hope it helps. -- DO4-UANIC
Thank you Denis. I modified my script to use "mbit" uniformly but I still don''t understand the results: #!/bin/bash RATE1=60mbit RATE2=10mbit PRIO1="prio 1" PRIO2="prio 3" OPTION="allot 1514 maxburst 20 avpkt 1000" tc qdisc del dev $DEV root tc qdisc add $DEV root handle 10: cbq bandwidth 100mbit avpkt 1000 tc class add $DEV parent 10:0 handle 10:1 cbq bandwidth 100mbit rate $RATE1 $OPTION $PRIO1 borrow tc class add $DEV parent 10:0 handle 10:2 cbq bandwidth 100mbit rate $RATE2 $OPTION $PRIO2 borrow tc filter add $DEV parent 10:0 protocol ip prio 1 handle 1 fw flowid 10:1 tc filter add $DEV parent 10:0 protocol ip prio 1 handle 2 fw flowid 10:2 iptables -A OUTPUT -t mangle -p tcp --dport 2000 -d 10.0.0.102 -j mark --set-mark 1 iptables -A OUTPUT -t mangle -p tcp --dport 2001 -d 10.0.0.102 -j mark --set-mark 2 OBSERVED RESULTS: - With both 10:1 and 10:2 as "borrow" Single iperf session to port 2000 reports 93.7 Mbits/s Two simultaneous iperf sessions to ports 2000 and 2001 report 45.8 and 45.7 Mbits/s respectively Single iperf session to port 2001 reports 89.1 Mbits/s - With 10:2 as "bounded" Single iperf session to port 2000 reports 88.9 Mbits/s Two simultaneous iperf sessions to ports 2000 and 2001 report 72.8 and 19.3 Mbits/s respectively Single iperf session to port 2001 reports 11.2 Mbits/s QUESTIONS: 1. When 10:2 is "bounded", is the reported 19.3 Mbits/s usage within acceptable limits for CBQ accuracy? 2. When both 10:1 and 10:2 are "borrow", why doesn''t port 2001 receive all bandwidth when it is the only session present? 3. When both 10:1 and 10:2 are "borrow", why do they share the available bandwidth equally? 4. Is my goal of having port 2000 receive at least 60mbits all the time AND port 2001 receiving all bandwidth when there is no port 2000 traffic not feasible? I appreciate any help members can provide. Thanks! Muthu On 1/4/06, Denis Ovsienko <linux@pilot.org.ua> wrote:> [...] > > RATE1=614400000 > tc(8) manpage extract: > UNITS > All parameters accept a floating point number, possibly > followed by a unit. > > Bandwidths or rates can be specified in: > > kbps Kilobytes per second > > mbps Megabytes per second > > kbit Kilobits per second > > mbit Megabits per second > > bps or a bare number > Bytes per second > > > [...] > > tc qdisc add $DEV root handle 10: cbq bandwidth 100mbit avpkt 1000 > Bandwidth is 100 Megabits per second. > > > # 60 Mbit/s class > > tc class add $DEV parent 10:0 classid 10:1 cbq bandwidth 100mbit rate > > $RATE1 $OPTION $PRIO borrow > Rate is 614400000 Bytes per second, roughly about 6 Gigabits per second. > I hope it helps. > > -- > DO4-UANIC > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >
> It doesn''t seem to help. I still see bandwidth being shared equally > between classes. Both iperf sessions report approximately 46 to 47 > Mbits/sec. Do you think what I am trying to achieve is possible with > CBQ?Okay... I have spent some time trying different CBQ trees until I found that I can''t divide 100mbits with precision better than 10 mbit. I managed to do it before though, but with different proportions... Most surprisingly precision goes worse when avpkt goes closer to 1500 (iperf sends quite big packets). I tried HTB and found that it doesn''t divide 100mbit well, but if we try to limit outgoing interface by 50 mbit and divide it, it works just fine (see below). I guess that CBQ would do it too, if we don''t try to control 100% of interface bandwidth and set all qdisc/class attributes precisely. -------------------------------------------------------------------------- tc qdisc add dev home root handle 10: htb default 3 tc class add dev home parent 10: classid 10:1 htb rate 50mbit tc class add dev home parent 10:1 classid 10:2 htb rate 40mbit ceil 50mbit tc class add dev home parent 10:1 classid 10:3 htb rate 10mbit ceil 50mbit tc filter add dev home parent 10: prio 1 protocol ip u32 match ip dst 10.0.0.102 match ip dport 2000 0xffff flowid 10:2 Now we have iperf server at 10.0.0.102, ports 2000 and 2001. 1. port 2000 traffic only $ iperf --client 10.0.0.102 --port 2000 --format k --time 10 --interval 1 ------------------------------------------------------------ Client connecting to 10.0.0.102, TCP port 2000 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 5] local 10.0.0.103 port 44056 connected with 10.0.0.102 port 2000 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 1.0 sec 6376 KBytes 52232 Kbits/sec [ 5] 1.0- 2.0 sec 6120 KBytes 50135 Kbits/sec [ 5] 2.0- 3.0 sec 6184 KBytes 50659 Kbits/sec [ 5] 3.0- 4.0 sec 6224 KBytes 50987 Kbits/sec [ 5] 4.0- 5.0 sec 6168 KBytes 50528 Kbits/sec [ 5] 5.0- 6.0 sec 6192 KBytes 50725 Kbits/sec [ 5] 6.0- 7.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 7.0- 8.0 sec 6216 KBytes 50921 Kbits/sec [ 5] 8.0- 9.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 9.0-10.0 sec 6200 KBytes 50790 Kbits/sec [ 5] 0.0-10.0 sec 62104 KBytes 50780 Kbits/sec 2. port 2001 traffic only iperf --client 10.0.0.102 --port 2001 --format k --time 10 --interval 1 ------------------------------------------------------------ Client connecting to 10.0.0.102, TCP port 2001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 5] local 10.0.0.103 port 48914 connected with 10.0.0.102 port 2001 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 1.0 sec 6280 KBytes 51446 Kbits/sec [ 5] 1.0- 2.0 sec 6176 KBytes 50594 Kbits/sec [ 5] 2.0- 3.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 3.0- 4.0 sec 6192 KBytes 50725 Kbits/sec [ 5] 4.0- 5.0 sec 6224 KBytes 50987 Kbits/sec [ 5] 5.0- 6.0 sec 6216 KBytes 50921 Kbits/sec [ 5] 6.0- 7.0 sec 6192 KBytes 50725 Kbits/sec [ 5] 7.0- 8.0 sec 6192 KBytes 50725 Kbits/sec [ 5] 8.0- 9.0 sec 6200 KBytes 50790 Kbits/sec [ 5] 9.0-10.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 0.0-10.0 sec 62096 KBytes 50787 Kbits/sec 3. port 2001 traffic is sent alone, then mixed with port 2000 traffic, then is sent alone again. 3.1 xterm 1: $ date; iperf --client 10.0.0.102 --port 2001 --format k --time 20 --interval 1 Птн Янв 6 04:31:54 MSK 2006 ------------------------------------------------------------ Client connecting to 10.0.0.102, TCP port 2001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 5] local 10.0.0.103 port 53131 connected with 10.0.0.102 port 2001 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 1.0 sec 6336 KBytes 51905 Kbits/sec [ 5] 1.0- 2.0 sec 6184 KBytes 50659 Kbits/sec [ 5] 2.0- 3.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 3.0- 4.0 sec 6200 KBytes 50790 Kbits/sec [ 5] 4.0- 5.0 sec 6192 KBytes 50725 Kbits/sec [ 5] 5.0- 6.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 6.0- 7.0 sec 6200 KBytes 50790 Kbits/sec [ 5] 7.0- 8.0 sec 3360 KBytes 27525 Kbits/sec [ 5] 8.0- 9.0 sec 1216 KBytes 9961 Kbits/sec [ 5] 9.0-10.0 sec 1232 KBytes 10093 Kbits/sec [ 5] 10.0-11.0 sec 1248 KBytes 10224 Kbits/sec [ 5] 11.0-12.0 sec 1312 KBytes 10748 Kbits/sec [ 5] 12.0-13.0 sec 1224 KBytes 10027 Kbits/sec [ 5] 13.0-14.0 sec 1208 KBytes 9896 Kbits/sec [ 5] 14.0-15.0 sec 4088 KBytes 33489 Kbits/sec [ 5] 15.0-16.0 sec 6208 KBytes 50856 Kbits/sec [ 5] 16.0-17.0 sec 6184 KBytes 50659 Kbits/sec [ 5] 17.0-18.0 sec 6216 KBytes 50921 Kbits/sec [ 5] 18.0-19.0 sec 6224 KBytes 50987 Kbits/sec [ 5] 19.0-20.0 sec 6200 KBytes 50790 Kbits/sec [ ID] Interval Transfer Bandwidth [ 5] 0.0-20.0 sec 89456 KBytes 36613 Kbits/sec 3.2 xterm 2: $ date; iperf --client 10.0.0.102 --port 2000 --format k --time 7 --interval 1 Птн Янв 6 04:32:02 MSK 2006 ------------------------------------------------------------ Client connecting to 10.0.0.102, TCP port 2000 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 5] local 10.0.0.103 port 41592 connected with 10.0.0.102 port 2000 [ ID] Interval Transfer Bandwidth [ 5] 0.0- 1.0 sec 4928 KBytes 40370 Kbits/sec [ 5] 1.0- 2.0 sec 5008 KBytes 41026 Kbits/sec [ 5] 2.0- 3.0 sec 4960 KBytes 40632 Kbits/sec [ 5] 3.0- 4.0 sec 4960 KBytes 40632 Kbits/sec [ 5] 4.0- 5.0 sec 4944 KBytes 40501 Kbits/sec [ 5] 5.0- 6.0 sec 4968 KBytes 40698 Kbits/sec [ 5] 6.0- 7.0 sec 4960 KBytes 40632 Kbits/sec [ 5] 0.0- 7.0 sec 34736 KBytes 40547 Kbits/sec -- DO4-UANIC