I''m in the process of building a new gateway to use the newer complex traffic controls in Shorewall instead of hand-crafted tc commands. At present I have (considering only outbound traffic, and omitting actual rates for clarity) : # External I/F run_tc qdisc add dev ethext root handle 1: htb default 12 run_tc class add dev ethext parent 1: classid 1:1 htb rate ...
On 10/29/2012 10:19 AM, Simon Hobson wrote:> I''m in the process of building a new gateway to use the newer complex > traffic controls in Shorewall instead of hand-crafted tc commands. > > At present I have (considering only outbound traffic, and omitting > actual rates for clarity) : > > # External I/F > > run_tc qdisc add dev ethext root handle 1: htb default 12 > run_tc class add dev ethext parent 1: classid 1:1 htb rate ... > € > run_tc class add dev ethext parent 1:1 classid 1:10 htb rate ... prio 1 > > run_tc class add dev ethext parent 1:10 classid 1:11 htb rate ... prio 1 > run_tc class add dev ethext parent 1:10 classid 1:12 htb rate ... prio 2 > run_tc class add dev ethext parent 1:10 classid 1:13 htb rate ... prio 3 > run_tc class add dev ethext parent 1:10 classid 1:14 htb rate ... prio 4 > > run_tc qdisc add dev ethext parent 1:11 handle 11: sfq perturb 10 > run_tc qdisc add dev ethext parent 1:12 handle 12: sfq perturb 10 > run_tc qdisc add dev ethext parent 1:13 handle 13: sfq perturb 10 > run_tc qdisc add dev ethext parent 1:14 handle 14: sfq perturb 10 > € > And a load more groups of classes for different customers or groups of servers. > > Am I right in thinking that this would translate to the following in > tcclasses ? > > #INTERFACE MARK RATE CEIL PRIORITY OPTIONS > €€ethext:1 - ... ... 1 > ethext:1:10 - ... ... 1 > ethext:10:11 - ... ... 1 tcp-ack,tos-minimize-delay > ethext:10:12 - ... ... 2 default > ethext:10:13 - ... ... 3 > €ethext:10:14 - ... ... 4 > € > €ethext:1:20 - ... ... 1 > ethext:20:11 - ... ... 1 tcp-ack,tos-minimize-delay > > and so on ... >Yes.> > As for devices, at present I do inbound control on the internal > interface of the router. As I read it, to use an IFB device I just > need this in tcdevices : > #INTERFACE IN-BANDWIDTH OUT-BANDWIDTH OPTIONS REDIRECTED > ethext - 1024kbit classify > ifb0 - 1024kbit classify ethext > €and then use IFB0 as the interface in tcclasses. > Is that correct ?Yes, assuming that you can classify incoming traffic based on packets as they look on the wire.> > > Also, is there an error in > http://shorewall.net/manpages/shorewall-tcclasses.html€ ? It says : >> Name of interface. Each interface may be listed only once in this file.€ > while in fact it needs to be listed for each line. Copy and paste > oversight from tcdevices manage ? >Yes. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Windows 8 Center - In partnership with Sourceforge Your idea - your app - 30 days. Get started! http://windows8center.sourceforge.net/ what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/
Tom Eastep wrote:> > Am I right in thinking that this would translate to the following in > > tcclasses ? > > ... > >Yes.Champion.
On 10/30/2012 03:48 AM, Simon Hobson wrote:> Tom Eastep wrote: > >> > Am I right in thinking that this would translate to the following in >> > tcclasses ? >> > ... >> >> Yes. > > > Champion.€ > > > > >> Yes, assuming that you can classify incoming traffic based on packets as >> they look on the wire. > > > I''m currently using loads of tc filters like this : > run_tc filter add dev ethint parent 100:0 protocol ip prio 1 > u32 match ip dst a.b.c.x/32 flowid 100:117 > run_tc filter add dev ethext parent 1:0 protocol ip prio 1 > u32 match ip src a.b.c.x/32 flowid 1:17 > run_tc filter add dev ethint parent 100:0 protocol ip prio 1 > u32 match ip dst a.b.c.x/32 match ip dport 25 0xffff flowid 100:118 > run_tc filter add dev ethext parent 1:0 protocol ip prio 1 > u32 match ip src a.b.c.x/32 match ip dport 25 0xffff flowid 1:18 > > > It''s not too bad where there''s a separate bandwidth allocation for a > single customer or device, but in some cases there are multiple > customers/devices sharing one allocation - so it''s gets somewhat > repetitive and I suspect inefficient. > > I still haven''t fully got my head around > http://shorewall.net/traffic_shaping.htm€, but as I read it, I should > be able to do something like this : > > #MARK SOURCE DESTINATION PROTOCOL PORT(S) CLIENT USER/ TEST > # PORT(S) GROUP > > # If the connection already has a mark, use it and skip the rest of the rules > RESTORE 0.0.0.0/0 0.0.0.0/0 all - - - 0 > CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - !0 > > > # Customer/group A > 12 a.b.c.x/32 0.0.0.0/0 all - - - !0 > 12 a.b.c.y/32 0.0.0.0/0 all - - - !0 > 12 a.b.c.z/32 0.0.0.0/0 all - - - !0 > 11 0.0.0.0/0 0.0.0.0/0 icmp echo-request - - 12 > 11 0.0.0.0/0 0.0.0.0/0 icmp echo-reply - - 12 > 13 0.0.0.0/0 0.0.0.0/0 tcp 110 - - 12 > 13 0.0.0.0/0 0.0.0.0/0 tcp - 110 - 12 > > 13 0.0.0.0/0 0.0.0.0/0 tcp 143 - - 12 > 13 0.0.0.0/0 0.0.0.0/0 tcp - 143 - 12 > > 14 0.0.0.0/0 0.0.0.0/0 tcp 25 - - 12 > 14 0.0.0.0/0 0.0.0.0/0 tcp - 25 - 12 > > > # Repeat above for each group of hosts ... > > # Assuming we''ve marked the packet, now save the mark to the connection > SAVE 0.0.0.0/0 0.0.0.0/0 all - - - !0 > > One aspect I''m unclear about ... > From the above page, I see these two bits of information : > >> classify >> If specified, classification of traffic into the various classes is >> done by CLASSIFY entries in /etc/shorewall/tcrules or by entries in >> /etc/shorewall/tcfilters. No MARK value will be associated with >> classes on this interface. > >> IFB devices automatically get the classify option. > > €So as I read that, because I''m using an IFB device, I can''t use the > packet marks as above. So would this work to then ''convert'' marks to > flows : > > 1:11 0.0.0.0/0 0.0.0.0/0 all - - - 11 > CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - 11 > > 1:12 0.0.0.0/0 0.0.0.0/0 all - - - 12 > CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - 12 > > 1:13 0.0.0.0/0 0.0.0.0/0 all - - - 13 > CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - 13 > > 1:14 0.0.0.0/0 0.0.0.0/0 all - - - 14 > CONTINUE 0.0.0.0/0 0.0.0.0/0 all - - - 14 > > > Or is there a more efficient way of doing things ?When you use an IFB, you must use filters (/etc/shorewall/tcfilters) to do the classification of inbound traffic. There is no Netfilter hook prior to the traffic being passed to the IFB, so tcrules in any form won''t work. For outbound traffic, your tcrules approach works fine. It also works fine if you continue to do the shaping on your internal interface rather than on an IFB.> > NB - I''m hoping to script at least part of this - at least tcclasses, > and possibly the host matching rules of tcrules. This is because I > have three systems to keep in sync : > two routers (active and standby), both firewall rules and data > collection scripts > data collection and graphing (rrd) on third machine > > The idea is to have a text file with a list of class groups with > their traffic rates, and a list of hosts and the class group they > should be in. Then a script can build part of the config for both > routers and the graphing from these lists. > > > PS - while I''m struggling to get my head round the user end of this, > I am in awe that you can keep track of how this works behind the > scenes ! >I struggle to keep this traffic shaping stuff straight in my head as well. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct
Tom Eastep wrote:>When you use an IFB, you must use filters (/etc/shorewall/tcfilters) to >do the classification of inbound traffic. There is no Netfilter hook >prior to the traffic being passed to the IFB, so tcrules in any form >won''t work. > >For outbound traffic, your tcrules approach works fine. It also works >fine if you continue to do the shaping on your internal interface rather >than on an IFB.Ah, so decision time then I''ll probably stick with IFB since I''m also looking at potentially adding an additional internal interface in the future. But I''ll have to have a chat with others first and see where things are likely to go before I finalise that. Which do you think is likely to be most efficient - CPU load wise ? tcrules as I''ve been looking at (and don''t use IFB), or tcfilters as I''ve been doing them up till now ? In particular, I''m thinking about the case where I might have <some number> of IP addresses to include in one set of classes - so potentially duplicating "address <something> and port <something>" rules many times in tcfilters. The particular group that''s in mind at the moment is about 16 discrete IPs (not a simple address/mask set).>I struggle to keep this traffic shaping stuff straight in my head as well.:D So it''s not just me then ! -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct
On 10/30/2012 09:05 AM, Simon Hobson wrote:> Tom Eastep wrote: > >> When you use an IFB, you must use filters (/etc/shorewall/tcfilters) to >> do the classification of inbound traffic. There is no Netfilter hook >> prior to the traffic being passed to the IFB, so tcrules in any form >> won''t work. >> >> For outbound traffic, your tcrules approach works fine. It also works >> fine if you continue to do the shaping on your internal interface rather >> than on an IFB. > > Ah, so decision time then > > I''ll probably stick with IFB since I''m also looking at potentially > adding an additional internal interface in the future. But I''ll have > to have a chat with others first and see where things are likely to > go before I finalise that. > > Which do you think is likely to be most efficient - CPU load wise ? > tcrules as I''ve been looking at (and don''t use IFB), or tcfilters as > I''ve been doing them up till now ? > > In particular, I''m thinking about the case where I might have <some > number> of IP addresses to include in one set of classes - so > potentially duplicating "address <something> and port <something>" > rules many times in tcfilters. The particular group that''s in mind at > the moment is about 16 discrete IPs (not a simple address/mask set). >With your tcrules approach, you only have to evaluate the long list of tcrules once for any given connections (because you are using SAVE/RESTORE). With tcfilters, you have to evaluate them on every packet. So the tcrules approach is definitely a win from a CPU utilization perspective. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct
Tom Eastep wrote:> > Which do you think is likely to be most efficient - CPU load wise ? > > tcrules as I''ve been looking at (and don''t use IFB), or tcfilters as > > I''ve been doing them up till now ? > > >> In particular, I''m thinking about the case where I might have <some >> number> of IP addresses to include in one set of classes - so >> potentially duplicating "address <something> and port <something>" >> rules many times in tcfilters. The particular group that''s in mind at >> the moment is about 16 discrete IPs (not a simple address/mask set). >> > >With your tcrules approach, you only have to evaluate the long list of >tcrules once for any given connections (because you are using >SAVE/RESTORE). With tcfilters, you have to evaluate them on every >packet. So the tcrules approach is definitely a win from a CPU >utilization perspective.That''s what I thought. I think it will also be easier to maintain - not necessarily for me, I always have to consider whoever is "lucky" enough to take over if I move on. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct
Tom Eastep wrote:> > In particular, I''m thinking about the case where I might have <some >> number> of IP addresses to include in one set of classes - so >> potentially duplicating "address <something> and port <something>" >> rules many times in tcfilters. The particular group that''s in mind at > > the moment is about 16 discrete IPs (not a simple address/mask set).>With your tcrules approach, you only have to evaluate the long list of >tcrules once for any given connections (because you are using >SAVE/RESTORE). With tcfilters, you have to evaluate them on every >packet. So the tcrules approach is definitely a win from a CPU >utilization perspective.So I decided on the TC rules option in the end, and hit a few "gotchas" along the way - at least one of which was of my own making ! The first was a simple typo : 12 192.168.2.123/32 0.0.0.0/0 all - - - !0
On 11/06/2012 05:30 AM, Simon Hobson wrote:> Tom Eastep wrote: > >> > In particular, I''m thinking about the case where I might have <some >>> number> of IP addresses to include in one set of classes - so >>> potentially duplicating "address <something> and port <something>" >>> rules many times in tcfilters. The particular group that''s in mind at >> > the moment is about 16 discrete IPs (not a simple address/mask set). > >> With your tcrules approach, you only have to evaluate the long list of >> tcrules once for any given connections (because you are using >> SAVE/RESTORE). With tcfilters, you have to evaluate them on every >> packet. So the tcrules approach is definitely a win from a CPU >> utilization perspective. > > So I decided on the TC rules option in the end, and hit a few > "gotchas" along the way - at least one of which was of my own making ! > > The first was a simple typo : > 12 192.168.2.123/32 0.0.0.0/0 all - - - !0 > €doesn''t match packets with a mark value of zero. Doh. > > Then, next issue I found was that my classifier rules weren''t working > - because it seems that mark values don''t get preserved between tcpre > and tcpost. In hindsight, and once I''d realised that, it makes sense > - the packet will have been through the routing process, so it''s no > longer the same packet, and hence will have lost it''s packet mark in > the process.Actually, that is the result of setting FORWARD_CLEAR_MARK=Yes in shorewall.conf. If you set FORWARD_CLEAR_MARK=No, then the packet marks established in tcpre are preserved.> Putting the restore on the start of the post-routing chain restores > the connection mark I''ve saved earlier, and it works just fine.And is efficient.> Of course, I''d also forgotten that I need rules to classify both the > inbound and outbound traffic - hence two classify rules per mark > value. I noticed that Shorewall automatically substitutes the > destination interface if the destination is 0.0.0.0/0 which saves a > little more effort. > > > So now to duplicate the existing rules, and find a quiet time to test > it in the live network. > > > I do have one further question though, about the mark column in tcclasses. > The manpage says "Must be specified as ''-'' if the classify option is > given for the interface".>> If I put a number in that column, then I get "WARNING: Class NUMBER > ignored -- INTERFACE ethint does not have the ''classify'' option > /etc/shorewall/tcclasses€". > I was looking to see just what Shorewall does with it - ie if it > would automate the latter part of my tcrules. >That warning is bogus; a filter is still generated that classifies packets using the packet mark. I''ll remove the warning in 4.5.9.1. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ LogMeIn Central: Instant, anywhere, Remote PC access and management. Stay in control, update software, and manage PCs from one command center Diagnose problems and improve visibility into emerging IT issues Automate, monitor and manage. Do more in less time with Central http://p.sf.net/sfu/logmein12331_d2d
Some feedback on this, and a bit of a performance indicator which might be of use to someone else. I got the new box put in our live network on Sunday, so it''s now running with real traffic. At the moment we''re running with around 10Mbps in and out of general hosting traffic (not large media/video files). The traffic is accounted for by perhaps a dozen or two hosts, with a fair mix of stuff going on - mostly web based access to business systems, and public web servers. We sell on some of our bandwidth to other tenants in the building, and also to another nearby building via a wireless link. Some of these get dedicated bandwidth, others share bandwidth. Also, some of our hosting customers get dedicated bandwidth. In all, at the moment we have 6 groups, and each group has four priority levels - so it''s very much like the "wondershaper" setup, but multiplied by 6 and working on both inbound (as best it''s possible) and outbound traffic. The System is an Alix 2D2 LX800 board. 500MHz AMD Geode processor, 256M RAM, Debian Wheezy OS & 3.2.0 kernel. The system is routing (no NAT) at the network boundary, and doing traffic shaping and accounting. It runs from a 4G Compact Flash card, logging (syslog) is sent to another server, and RRD updates are sent to another server using RRDCached. Overall, excluding my collection & RRD update scripts, the system runs at around 40-50% idle and 50-60% system interrupt. When collection scripts are running, idle can drop to 10-20%, so not a huge amount of headroom, but enough for what we need now - and it takes a lot less power than the old 1.1GHz Pentium 3 rack mount server it replaces ! I also costs a lot less than a replacement for the old server which had died. If we significantly increase out bandwidth, then I''ll need to look for an alternative board with a little more horsepower - the Alix boards are attractive as they have multiple real NICs on them. I''ve used some includes, and the intention is that some of these files will be script generated once I''ve written the script. Eg, the traffic shaping classes in particular will have a text file listing the groups and the rates allowed - the script will then do some checks (make sure the committed rates don''t add up to more than the parents and so on) before building the include file and sending it to both routers - as well as updating the graphing we have. The accounting comprises :># Outside global stats >outside:COUNT - ethext - >outside:COUNT - - ethext >DONE outside > ># Do acocunting by IP address >account-ip - - - >total-ip:COUNT account-ip ethext - >total-ip:COUNT account-ip - ethext >DONE total-ip > >acc-serv >total-serv:COUNT acc-serv ethext - >total-serv:COUNT acc-serv - ethext >DONE total-serv > > >INCLUDE accounting.ip >INCLUDE accounting.service