Hello guys,
I past the last days trying to configure my shorewall 4.06 firewall to
allow openvpn bridging connection.
My scenario is the following:
roadwarrior (openvpn client) -------------> Internet ------------>
(X.Y.W.Z - eth0) Firewall/Gateway (10.x.x.254 - eth1) --------> Local
Lan -------> OpenVPN Server (10.x.x.249 - br0)
where 10.x.x.0-254 is my private lan
X.Y.Z.W is the public IP address of the firewall
I have to perform the following issues:
1) The only bridge i want to allow is the openvpn tunnel between road
warriors and OpenVPN Server.
2) The firewall has OpenSwan installed on it to allow old, but still
living, ipsec tunnel.
OpenVPN works if i run command ''shorewall clear''
Firewall configuration is the following:
/etc/shorewall/conf
BRIDGING = Yes
/etc/shorewall/zones
###############################################################################
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
fw firewall
# ipsec vpn zone
vpn ipv4
# OpenVPN zone
vpn1 ipv4
net ipv4
loc ipv4
/etc/shorewall/interfaces
###############################################################################
#ZONE INTERFACE BROADCAST OPTIONS
loc eth1 10.x.x.255 routeback
- br0 detect
vpn ipsec0
/etc/shorewall/hosts
###############################################################################
#ZONE HOST(S) OPTIONS
vpn br0:eth0:<ip subnet 1>/24,<ip subnet 2</30
net br0:eth0
vpn1 br0:tap0
where <ip subnet 1> and <ip subnet 2> have real ip subnet address
/etc/shorewall/tunnels
###############################################################################
#TYPE ZONE GATEWAY GATEWAY
#
ipsec net a.b.c.d
ipsec net e.f.g.h
openvpn net 0.0.0.0/0 vpn1
/etc/shorewall/masq
br0:eth0:!<ip subnet 1>/24 10.x.x.0/24
br0:eth0:!<ip subnet 2>/30 10.x.x.0/24
br0:eth0 eth1
/etc/shorewall/rules
###############################################################################
#ACTION SOURCE DEST PROTO DEST SOURCE
ORIGINAL RATE USER/ MARK
# PORT PORT(S) DEST LIMIT
GROUP
SECTION ESTABLISHED
#SECTION RELATED
SECTION NEW
# OpenVPN listen port forwarding
DNAT:info net:X.Y.W.Z loc:10.x.x.249 udp 1194 - -
OpenVPN client conf (Road Warrior Side)
C:\Programmi\OpenVPN\config\client1.conf
###############################################################################
dev tap
proto udp
# change this to your server''s address
remote X.Y.Z.W
port 1194
resolv-retry infinite
nobind
persist-key
persist-tun
# Point the key and crt files to
tls-client
ca ca.crt
cert client1.crt
key client1.key
#ensure that we are talking to a server
ns-cert-type server
#confirm we are talking to the correct server
tls-auth ta.key 1
# Select a cryptographic cipher.
# If the cipher option is used on the server
# then you must also specify it here.
cipher AES-128-CBC
# Enable compression on the VPN link.
comp-lzo
#fragment large packets
# I found I needed this for some games but it is
# not required
fragment 1400
# enable user/pass authentication
#auth-user-pass
pull
verb 4
OpenVPN server conf (OpenVPN Internal LAN server)
/etc/openvpn/server.conf
###############################################################################
# Which local IP address should OpenVPN
# listen on? (optional)
local 10.x.x.249
port 1194
# TCP or UDP server?
proto udp
#This is key to configuring our bridge
dev tap0
#direct these to your generated files
ca keys/ca.crt
cert keys/server.crt
key keys/server.key
dh keys/dh2048.pem
ifconfig-pool-persist ip-clients.txt
#ensure the range of ip addresses you use in the last two arguments
# of this statement are not in use by either the DHCP server or any other
# device on your internal network.
server-bridge 10.x.x.249 255.255.255.0 10.x.x.180 10.x.x.199
#needed to allow communication to internal network
client-to-client
keepalive 10 120
#encryption - very important ;)
#AES encryption is backed by many security firms
#however if you are concerned about speed use blowfish: "BF-CB"
cipher AES-128-CBC
#if you have another subnet you need to provide the route
push "route 192.168.x.0 255.255.255.0 10.x.x.1"
push "route 192.168.y.0 255.255.255.0 10.x.x.1"
#server id protection
tls-auth ta.key 0
#compression for network speed
comp-lzo
# if packets are too large fragment them (only really useful if you have
an old router)
#fragment 1400
#limit the number of connections
max-clients 5
#some secuurity settings
# do not use if running server on Windows
user nobody
group nogroup
persist-key
persist-tun
#log file settings
status /var/log/openvpn/openvpn-status.log
log /var/log/openvpn/openvpn.log
verb 3
# authentication plugin
#forces client to have a linux acount in order to connect
#plugin /usr/lib/openvpn/openvpn-auth-pam.so login
The above configuration doesn''t work correctly
Any Ideas hot to solve the problem?
Thanks in advance for the reply.
Regards.
--
_________________________________________________
*/Gianni Socionovo/*
//
Le informazioni contenute nella presente comunicazione e i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone o alla Società sopraindicate. La diffusione,
distribuzione e/o copiatura del documento trasmesso da parte di
qualsiasi soggetto diverso dal destinatario è proibita, sia ai sensi
dell''art. 616 c.p. , che ai sensi del D.Lgs. n. 196/2003. Se avete
ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di
informare il mittente.
The information in this e-mail is confidential and may also be legally
privileged. It is intended for the addressee only. Unauthorized
recipients are required to maintain confidentiality. If you have
received this e-mail in error please notify us immediately, destroy any
copies. Any use, dissemination, forwarding, printing or copying of this
e-mail is prohibited in accordance with art. 616 of the Penal Code and
Legislative Decree N° 196 of 2003.
------------------------------------------------------------------------------
Register Now & Save for Velocity, the Web Performance & Operations
Conference from O''Reilly Media. Velocity features a full day of
expert-led, hands-on workshops and two days of sessions from industry
leaders in dedicated Performance & Operations tracks. Use code vel09scf
and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Gianni Socionovo wrote:> Hello guys, > > I past the last days trying to configure my shorewall 4.06 firewall to > allow openvpn bridging connection. > > My scenario is the following: > > roadwarrior (openvpn client) -------------> Internet ------------> > (X.Y.W.Z - eth0) Firewall/Gateway (10.x.x.254 - eth1) --------> Local > Lan -------> OpenVPN Server (10.x.x.249 - br0) > > where 10.x.x.0-254 is my private lan > > X.Y.Z.W is the public IP address of the firewall > > I have to perform the following issues: > > 1) The only bridge i want to allow is the openvpn tunnel between road > warriors and OpenVPN Server. > > 2) The firewall has OpenSwan installed on it to allow old, but still > living, ipsec tunnel. > > OpenVPN works if i run command ''shorewall clear'' > > Firewall configuration is the following:At http://www.shorewall.net/support.htm, we specifically ask that you *not* include your configuration files but rather submit the output of ''shorewall dump'' collected as described in the article.> > /etc/shorewall/conf > > BRIDGING = YesBRIDGING=Yes does not work with kernel 2.6.20 and later. See http://www.shorewall.net/Notices.html#Notice1.> > /etc/shorewall/zones > ############################################################################### > #ZONE TYPE OPTIONS IN OUT > # OPTIONS OPTIONS > fw firewall > # ipsec vpn zone > vpn ipv4 > # OpenVPN zone > vpn1 ipv4 > net ipv4 > loc ipv4 > > /etc/shorewall/interfaces > ############################################################################### > #ZONE INTERFACE BROADCAST OPTIONS > loc eth1 10.x.x.255 routeback > - br0 detect > vpn ipsec0Which computer does this configuration run on? eth1 is on the the Firewall/Gateway while br0 is on the OpenVPN server. And if Firewall/Gateway is the same as OpenVPN server then your network diagram is completely misleading.> > /etc/shorewall/hosts > ############################################################################### > #ZONE HOST(S) OPTIONS > vpn br0:eth0:<ip subnet 1>/24,<ip subnet 2</30 > net br0:eth0 > vpn1 br0:tap0 > > where <ip subnet 1> and <ip subnet 2> have real ip subnet address > > /etc/shorewall/tunnels > ############################################################################### > #TYPE ZONE GATEWAY GATEWAY > # > ipsec net a.b.c.d > ipsec net e.f.g.h > openvpn net 0.0.0.0/0 vpn1 > > /etc/shorewall/masq > br0:eth0:!<ip subnet 1>/24 10.x.x.0/24 > br0:eth0:!<ip subnet 2>/30 10.x.x.0/24 > br0:eth0 eth1 > > /etc/shorewall/rules > ############################################################################### > #ACTION SOURCE DEST PROTO DEST SOURCE > ORIGINAL RATE USER/ MARK > # PORT PORT(S) DEST LIMIT > GROUP > SECTION ESTABLISHED > #SECTION RELATED > SECTION NEW > # OpenVPN listen port forwarding > DNAT:info net:X.Y.W.Z loc:10.x.x.249 udp 1194 - - > >I have no clue what this configuration is trying to do. As I mentioned above, it looks like you have one Shorewall configuration for two computers. That won''t work. And unless you are running an old kernel, BRIDGING=Yes won''t work. Plus, you will have to change the configuration when you upgrade to kernel 2.6.20 or later. If you need to post again, please provide the output of ''shorewall dump'' as described in the article linked above. If you don''t wish to divulge the details of your network to the list, then please send the problem report itself to the list and send the dump, as an attachment, to upload@shorewall.net. Thanks, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Tom Eastep ha scritto:> Gianni Socionovo wrote: > >> Hello guys, >> >> I past the last days trying to configure my shorewall 4.06 firewall to >> allow openvpn bridging connection. >> >> My scenario is the following: >> >> roadwarrior (openvpn client) -------------> Internet ------------> >> (X.Y.W.Z - eth0) Firewall/Gateway (10.x.x.254 - eth1) --------> Local >> Lan -------> OpenVPN Server (10.x.x.249 - br0) >> >> where 10.x.x.0-254 is my private lan >> >> X.Y.Z.W is the public IP address of the firewall >> >> I have to perform the following issues: >> >> 1) The only bridge i want to allow is the openvpn tunnel between road >> warriors and OpenVPN Server. >> >> 2) The firewall has OpenSwan installed on it to allow old, but still >> living, ipsec tunnel. >> >> OpenVPN works if i run command ''shorewall clear'' >> >> Firewall configuration is the following: >> > > At http://www.shorewall.net/support.htm, we specifically ask that you > *not* include your configuration files but rather submit the output of > ''shorewall dump'' collected as described in the article. > > >> /etc/shorewall/conf >> >> BRIDGING = Yes >> > > BRIDGING=Yes does not work with kernel 2.6.20 and later. See > http://www.shorewall.net/Notices.html#Notice1. >Ok thank you for the notice. I will set br0 with option bridge.> >> /etc/shorewall/zones >> ############################################################################### >> #ZONE TYPE OPTIONS IN OUT >> # OPTIONS OPTIONS >> fw firewall >> # ipsec vpn zone >> vpn ipv4 >> # OpenVPN zone >> vpn1 ipv4 >> net ipv4 >> loc ipv4 >> >> /etc/shorewall/interfaces >> ############################################################################### >> #ZONE INTERFACE BROADCAST OPTIONS >> loc eth1 10.x.x.255 routeback >> - br0 detect >> vpn ipsec0 >> > > Which computer does this configuration run on? eth1 is on the the > Firewall/Gateway while br0 is on the OpenVPN server. And if > Firewall/Gateway is the same as OpenVPN server then your network diagram > is completely misleading. >The computer with the configuration is the firewall/router, the OpenVPN server is in the internal lan and it has the interface br0. It is protected by firewall/gateway shorewall, but i cannot understand how to pass bridging packets trough the firewall/gateway> >> /etc/shorewall/hosts >> ############################################################################### >> #ZONE HOST(S) OPTIONS >> vpn br0:eth0:<ip subnet 1>/24,<ip subnet 2</30 >> net br0:eth0 >> vpn1 br0:tap0 >> >> where <ip subnet 1> and <ip subnet 2> have real ip subnet address >> >> /etc/shorewall/tunnels >> ############################################################################### >> #TYPE ZONE GATEWAY GATEWAY >> # >> ipsec net a.b.c.d >> ipsec net e.f.g.h >> openvpn net 0.0.0.0/0 vpn1 >> >> /etc/shorewall/masq >> br0:eth0:!<ip subnet 1>/24 10.x.x.0/24 >> br0:eth0:!<ip subnet 2>/30 10.x.x.0/24 >> br0:eth0 eth1 >> >> /etc/shorewall/rules >> ############################################################################### >> #ACTION SOURCE DEST PROTO DEST SOURCE >> ORIGINAL RATE USER/ MARK >> # PORT PORT(S) DEST LIMIT >> GROUP >> SECTION ESTABLISHED >> #SECTION RELATED >> SECTION NEW >> # OpenVPN listen port forwarding >> DNAT:info net:X.Y.W.Z loc:10.x.x.249 udp 1194 - - >> >> >> > > I have no clue what this configuration is trying to do. As I mentioned > above, it looks like you have one Shorewall configuration for two > computers. That won''t work. And unless you are running an old kernel, > BRIDGING=Yes won''t work. Plus, you will have to change the configuration > when you upgrade to kernel 2.6.20 or later. > > If you need to post again, please provide the output of ''shorewall dump'' > as described in the article linked above. If you don''t wish to divulge > the details of your network to the list, then please send the problem > report itself to the list and send the dump, as an attachment, to > upload@shorewall.net. > > Thanks, > -Tom > > ------------------------------------------------------------------------ > > ------------------------------------------------------------------------------ > Register Now & Save for Velocity, the Web Performance & Operations > Conference from O''Reilly Media. Velocity features a full day of > expert-led, hands-on workshops and two days of sessions from industry > leaders in dedicated Performance & Operations tracks. Use code vel09scf > and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf > > > __________ Informazioni da ESET NOD32 Antivirus, versione del database delle firme digitali 4046 (20090430) __________ > > Il messaggio è stato controllato da ESET NOD32 Antivirus. > > www.nod32.it > > > ------------------------------------------------------------------------ > > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > > > > __________ Informazioni da ESET NOD32 Antivirus, versione del database delle firme digitali 4046 (20090430) __________ > > Il messaggio è stato controllato da ESET NOD32 Antivirus. > > www.nod32.it > >------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Gianni Socionovo wrote:> > Tom Eastep ha scritto: >> >> Which computer does this configuration run on? eth1 is on the the >> Firewall/Gateway while br0 is on the OpenVPN server. And if >> Firewall/Gateway is the same as OpenVPN server then your network diagram >> is completely misleading. >> > The computer with the configuration is the firewall/router, the OpenVPN > server is in the internal lan and it has the interface br0. It is > protected by firewall/gateway shorewall, but i cannot understand how to > pass bridging packets trough the firewall/gatewayThe packets that pass through the firewall are simple UDP port 1194 packets. The *only configuration needed on the Firewall/Gateway is the DNAT rule*. No bridges, no zones, NOTHING. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tom, Was hoping you could point me in the right direction. I'm an iptables novice and have been trusting my configuration to Shorewall but have run into some pushback from a colleague. I've never taken the time to learn iptables so I don’t know the meaning of all the rules or what is really going on. My problem is I have to explain why Shorewall is doing or figure out a way to make the output more basic (be able to leave things out). My colleague has suggested the rule set produced by Shorewall is too "busy and inefficient" (whatever is meant by that). I believe my colleague to be as familiar with iptables as I am and is getting hung up by all the chains. I think he would prefer to see the default 3 (input, forward, and output)and nothing more. Are there any switches I can use to have a more basic firewall produced? If not is there any documentation which shows the logic of how and why all the chains are created? I want to continue using Shorewall because I think it's fantastic. I suspect having been doing this for what appears to be a decade you have more focused skill set and I trust what Shorewall is producing I just need to be able to describe the output a little better. Apologies in advance if the information is out there, my googling just wasn’t producing the answers I was seeking. I'm sure your busy and I appreciate your time in pointing me in the right direction. Regards, Tom Campion USMax Corporation Senior DNS/Unix Engineer NOAA NOC (contractor) (301) 713-0600 x 152 (240) 338-0944 (cell) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) iD8DBQFJ+egboeGLZ7Sw9vsRArOgAJ0fk2ffv9vWeZJqruRYMzGAeDV1QACglNs9 Q33lLBZs3CW8BtSx35ribU0=n/X3 -----END PGP SIGNATURE----- ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O'Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Tom Campion wrote:> Tom, > > Was hoping you could point me in the right direction. I''m an iptables > novice and have been trusting my configuration to Shorewall but have > run into some pushback from a colleague. I''ve never taken the time to > learn iptables so I dont know the meaning of all the rules or what > is really going on. My problem is I have to explain why Shorewall is > doing or figure out a way to make the output more basic (be able to > leave things out). My colleague has suggested the rule set produced > by Shorewall is too "busy and inefficient" (whatever is meant by > that).I believe my colleague to be as familiar with iptables as I am > and is getting hung up by all the chains. I think he would prefer to > see the default 3 (input, forward, and output)and nothing more.The Shorewall configuration is designed to scale to 100s of zones with many rules in each coordinate of the zones->zone matrix. A newbie configuration of INPUT, FORWARD and OUTPUT scales miserably.> Are > there any switches I can use to have a more basic firewall produced?No.> If not is there any documentation which shows the logic of how and > why all the chains are created?The only article on this subject is http://www.shorewall.net/PacketHandling.html. I haven''t tried to keep that up very well.> I want to continue using Shorewall > because I think it''s fantastic. I suspect having been doing this for > what appears to be a decade you have more focused skill set and I > trust what Shorewall is producing I just need to be able to describe > the output a little better.Reading the output of ''Shorewall dump'' is not that difficult if you start out by assuming that you *can* understand it rather than by assuming that you can''t. Use the above article along with http://www.shorewall.net/NetfilterOverview.html. Good luck, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Tom Eastep wrote:> Tom Campion wrote: >> Tom, >> >> Was hoping you could point me in the right direction. I''m an iptables >> novice and have been trusting my configuration to Shorewall but have >> run into some pushback from a colleague. I''ve never taken the time to >> learn iptables so I dont know the meaning of all the rules or what >> is really going on. My problem is I have to explain why Shorewall is >> doing or figure out a way to make the output more basic (be able to >> leave things out). My colleague has suggested the rule set produced >> by Shorewall is too "busy and inefficient" (whatever is meant by >> that).I believe my colleague to be as familiar with iptables as I am >> and is getting hung up by all the chains. I think he would prefer to >> see the default 3 (input, forward, and output)and nothing more. > > The Shorewall configuration is designed to scale to 100s of zones with > many rules in each coordinate of the zones->zone matrix. A newbie > configuration of INPUT, FORWARD and OUTPUT scales miserably.Two questions that your colleague should ask when evaluating Shorewall''s ruleset against the ''100s of rules in FORWARD'' approach are: a) How many rules, on average, does a packet that is part of an established connection traverse? b) How many rules, on average, does a packet representing a connection request traverse. When answering those questions, I think the approach taken in Shorewall becomes clearer. Now if you have very simple-minded firewall requirements, Shorewall''s approach is probably overkill. But if your requirements are that simple, maybe Shorewall itself is overkill for your particular application. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Tom Eastep wrote:>Now if you have very simple-minded firewall requirements, Shorewall''s >approach is probably overkill. But if your requirements are that simple, >maybe Shorewall itself is overkill for your particular application.Quite right. If it''s any help to the OP, I personally manage all the Linux boxes at work in our hosting facility. The second package I install (on the same line as openssh-server) is Shorewall - even for those boxes where the rules are so simple that I could now cobble together a few lines if iptables. As Tom says, the rules might look complicated, but once you figure out what''s happening, they aren''t that bad, and packets shouldn''t need to go through that many steps. On the other hand, Shorewall will allow you to have a capable firewall that others can manage - just try handing over a file of iptables rules to a relative novice and ask them "just add a rule for ..." ! I think anyone competent enough to be allowed to administer one of your servers should be able to cope with Shorewall. At the other extreme to the "could do it in a few lines of iptables" boxes, our boundary routers run Linux, and I configure them with Shorewall - including full accounting and traffic shaping. Quite a low powered box will handle our traffic whilst still showing 99.something% idle. Before I settled on Shorewall, I tried a few other options - they were either too restricting or too hard :-/ -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Simon Hobson wrote:> At the other extreme to the "could do it in a few lines of iptables" > boxes, our boundary routers run Linux, and I configure them with > Shorewall - including full accounting and traffic shaping. Quite a > low powered box will handle our traffic whilst still showing > 99.something% idle.Do you have any step by step HowTo for that or script that is not proprietary? I am planing to something like that, but it will take a lot of reading an probably developing before production version. I am small local WISP and I need ability to HTB my customers in as easiest way possible. My focus is on throttling down bandwidth hogger''s and my choice is Shorewall base with web interface (or RADIUS based script communicating with Shorewall). Thanks, Ljubomir ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Ljubomir Ljubojevic wrote:> > At the other extreme to the "could do it in a few lines of iptables" >> boxes, our boundary routers run Linux, and I configure them with >> Shorewall - including full accounting and traffic shaping. Quite a >> low powered box will handle our traffic whilst still showing >> 99.something% idle. > > >Do you have any step by step HowTo for that or script that is not >proprietary? I am planing to something like that, but it will take a lot >of reading an probably developing before production version. I am small >local WISP and I need ability to HTB my customers in as easiest way >possible. My focus is on throttling down bandwidth hogger''s and my >choice is Shorewall base with web interface (or RADIUS based script >communicating with Shorewall).Well it''s not all Shorewall, but this is the business end of what we have - anonimised for obvious reasons. Like many things, it''s not the only way to do it, and there are probably other ways as well, but "it works for us". In terms of resources required, this runs on a 1GHz Pentium III, and memory requirements aren''t high :>cat /proc/meminfo >MemTotal: 1036092 kB >MemFree: 499152 kB >Buffers: 399816 kB >Cached: 101296 kBI''ll correct the statement I made earlier - it generally runs between 97% and 99% idle, it will use more cpu when it''s updating the RRDs, and when the frontend is drawing graphs. I share the directory containing the rrd files via NFS. A typical output from top is :>top - 10:54:31 up 44 days, 8:17, 1 user, load average: 0.00, 0.01, 0.00 >Tasks: 65 total, 2 running, 63 sleeping, 0 stopped, 0 zombie >Cpu(s): 1.0%us, 0.0%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st >Mem: 1036092k total, 537172k used, 498920k free, 399816k buffers >Swap: 2939884k total, 0k used, 2939884k free, 101296k cachedThis is the traffic accounting stuff. We have a full class C at work, so I log traffic for all 254 addresses. Before I put this in place, we had problem with running out of bandwidth, but we had no idea how badly, which servers were consuming it, or whether we could mitigate the problem. As it happens, we were able to mitigate the problem (and make VoIP usable again) by application of traffic shaping, and we put off upgrading the bandwidth for about a year. Since then, it''s allowed us to spot "problems", and along with Nagios to monitor stuff, it''s often allowed us to fix things before the customers phone us up to say it''s broken :-) The data is stuffed into an RRD database, and from there it is graphed with some custom scripts. Since anyone in the office can call up the graphs, it means a lot of troubleshooting can be done without me getting involved. accounting :>outside-in:COUNT - ethext - >outside-out:COUNT - - ethext >DONE outside > ># Do acocunting by IP address >account-ip - - - >total-ip-in:COUNT account-ip ethext - >total-ip-out:COUNT account-ip - ethext >DONE total-ip > >acc-serv >total-serv-in:COUNT acc-serv ethext - >total-serv-out:COUNT acc-serv - ethext >DONE total-serv > > >INCLUDE accounting.ip >INCLUDE accounting.serviceaccounting.ip :>acc1-in:COUNT account-ip ethext a.b.c.1 >acc1-out:COUNT account-ip a.b.c.1 ethext >DONE acc1 > >acc2-in:COUNT account-ip ethext a.b.c.2 >acc2-out:COUNT account-ip a.b.c.2 ethext >DONE acc2 > >acc3-in:COUNT account-ip ethext a.b.c.3 >acc3-out:COUNT account-ip a.b.c.3 ethext >DONE acc3 > >... > >acc252-in:COUNT account-ip ethext a.b.c.252 >acc252-out:COUNT account-ip a.b.c.252 ethext >DONE acc252 > >acc253-in:COUNT account-ip ethext a.b.c.253 >acc253-out:COUNT account-ip a.b.c.253 ethext >DONE acc253 > >acc254-in:COUNT account-ip ethext a.b.c.254 >acc254-out:COUNT account-ip a.b.c.254 ethext >DONE acc254accounting.service is currently empty, it''s there to support future plans to add accounting by IP and service - eg so we can see the split between http, https, smtp, ftp etc for an IP address. Getting the data out is done with a shell script run every few minutes from cron :>#/bin/bash ># Script to extract values from shorewall output > >cd /var/rrd > >/usr/bin/rrdtool update ip-stats.rrd N:`/sbin/iptables -L account-ip -vxn | \ > /usr/bin/awk ''BEGIN { getline ; getline } > { print $2 }'' | \ > /usr/bin/tr '' >'' '':'' | /bin/sed -e ''s/:$//''`And the rrd definition for this is :>rrdtool create ip-stats.rrd -s 300 \ > DS:total-in:DERIVE:600:0:U \ > DS:total-out:DERIVE:600:0:U \ > \ > DS:ip1-in:DERIVE:600:0:U \ > DS:ip1-out:DERIVE:600:0:U \ > DS:ip2-in:DERIVE:600:0:U \ > DS:ip2-out:DERIVE:600:0:U \ > DS:ip3-in:DERIVE:600:0:U \ > DS:ip3-out:DERIVE:600:0:U \ >... > DS:ip253-in:DERIVE:600:0:U \ > DS:ip253-out:DERIVE:600:0:U \ > DS:ip254-in:DERIVE:600:0:U \ > DS:ip254-out:DERIVE:600:0:U \ > \ > RRA:AVERAGE:0.5:1:576 \ > RRA:MAX:0.5:1:576 \ > RRA:AVERAGE:0.5:6:672 \ > RRA:MAX:0.5:6:672 \ > RRA:AVERAGE:0.5:24:732 \ > RRA:MAX:0.5:24:732 \ > RRA:AVERAGE:0.5:288:730 \ > RRA:MAX:0.5:288:730 > ># CFs for : ># 1 x 576 48hrx 5m ># 6 x 672 14d x 1/2hr ># 24 x 732 61d x 2hr ># 288 x 730 730d x 24hrI''ll do the traffic shaping in a separate email. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
This is exactly what I need to get me started. When I absorb your mail''s I plan to go/play with the ip+services part as well. At that point I should be able to expand/enhance it on my own. This is VERY appreciated, it will shave a lot of time I would spend reading documentation and testing. Thanks Ljubomir ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
>I''ll do the traffic shaping in a separate email.And here it is, though it stands a great chance of getting mangled by line wrapping :-( I''ve done my best to manually break long lines - so everyhwere there is a line of the form :>blah blah blah \ > blahthat should all be on one line as :>blah blah blah blahThe downside though is that a lot of it is harder to read. tcstart :>INCLUDE tcstart-class >INCLUDE tcstart-ruletcstart-class :># clean existing down- and uplink qdiscs, hide errors >tc qdisc del dev ethint root 2> /dev/null > /dev/null >tc qdisc del dev ethint ingress 2> /dev/null > /dev/null >tc qdisc del dev ethext root 2> /dev/null > /dev/null >tc qdisc del dev ethext ingress 2> /dev/null > /dev/null > ># External I/F > ># install root HTB, point default traffic to 1:12: >run_tc qdisc add dev ethext root handle 1: htb default 12 ># shape everything at uplink speed >run_tc class add dev ethext parent 1: classid 1:1 htb rate \ > $OutSpeed burst 20k cburst 20k > ># Internal I/F ># First, an overall queue/classes to split firewall and net traffic ># install root HTB, point default traffic to 100:102: >run_tc qdisc add dev ethint root handle 100: htb default 112 >run_tc class add dev ethint parent 100: classid 100:100 htb \ > rate 95000kbit ># Class for firewall traffic - effectively unlimited >run_tc class add dev ethint parent 100:100 classid 100:101 htb \ > rate 75000kbit prio 1 >run_tc qdisc add dev ethint parent 100:101 handle 102: sfq \ > perturb 10 ># Class for net traffic - limit to line speed >run_tc class add dev ethint parent 100:100 classid 100:102 \ > htb rate $InSpeed burst 20k cburst 20k prio 1 > ># Need to filter FW generated traffic to 100:111 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 \ > u32 match ip src x.y.z.154/32 flowid 100:101 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 \ > u32 match ip src a.b.c.254/32 flowid 100:101 > > > ># Main traffic ># Out >run_tc class add dev ethext parent 1:1 classid 1:10 htb rate \ > 1400kbit ceil $OutCeilDef burst 16k cburst 16k prio 1 > >run_tc class add dev ethext parent 1:10 classid 1:11 htb rate \ > 500kbit ceil $OutCeilDef burst 12k cburst 12k prio 1 >run_tc class add dev ethext parent 1:10 classid 1:12 htb rate \ > 600kbit ceil $OutCeilDef burst 12k cburst 12k prio 2 >run_tc class add dev ethext parent 1:10 classid 1:13 htb rate \ > 200kbit ceil $OutCeilDef burst 12k cburst 12k prio 3 >run_tc class add dev ethext parent 1:10 classid 1:14 htb rate \ > 100kbit ceil 3072kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethext parent 1:11 handle 11: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:12 handle 12: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:13 handle 13: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:14 handle 14: sfq perturb 10 > ># In >run_tc class add dev ethint parent 100:102 classid 100:110 htb \ > rate 1400kbit ceil $InCeilDef burst 16k cburst 16k prio 1 > >run_tc class add dev ethint parent 100:110 classid 100:111 htb \ > rate 500kbit ceil $InCeilDef burst 12k cburst 12k prio 1 >run_tc class add dev ethint parent 100:110 classid 100:112 htb \ > rate 600kbit ceil $InCeilDef burst 12k cburst 12k prio 2 >run_tc class add dev ethint parent 100:110 classid 100:113 htb \ > rate 200kbit ceil $InCeilDef burst 12k cburst 12k prio 3 >run_tc class add dev ethint parent 100:110 classid 100:114 htb \ > rate 100kbit ceil 3072kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethint parent 100:111 handle 111: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:112 handle 112: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:113 handle 113: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:114 handle 114: sfq perturb 10 > > > ># Misc customers ># Out >run_tc class add dev ethext parent 1:1 classid 1:15 htb \ > rate 500kbit ceil 4096kbit burst 16k cburst 16k prio 1 > >run_tc class add dev ethext parent 1:15 classid 1:16 htb \ > rate 250kbit ceil 4096kbit burst 12k cburst 12k prio 1 >run_tc class add dev ethext parent 1:15 classid 1:17 htb \ > rate 240kbit ceil 4096kbit burst 12k cburst 12k prio 2 >run_tc class add dev ethext parent 1:15 classid 1:18 htb \ > rate 5kbit ceil 4096kbit burst 12k cburst 12k prio 3 >run_tc class add dev ethext parent 1:15 classid 1:19 htb \ > rate 5kbit ceil 4096kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethext parent 1:16 handle 16: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:17 handle 17: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:18 handle 18: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:19 handle 19: sfq perturb 10 > ># In >run_tc class add dev ethint parent 100:102 classid 100:115 \ > htb rate 500kbit ceil 4096kbit burst 16k cburst 16k prio 1 > >run_tc class add dev ethint parent 100:115 classid 100:116 \ > htb rate 250kbit ceil 4096kbit burst 12k cburst 12k prio 1 >run_tc class add dev ethint parent 100:115 classid 100:117 \ > htb rate 240kbit ceil 4096kbit burst 12k cburst 12k prio 2 >run_tc class add dev ethint parent 100:115 classid 100:118 \ > htb rate 5kbit ceil 4096kbit burst 12k cburst 12k prio 3 >run_tc class add dev ethint parent 100:115 classid 100:119 \ > htb rate 5kbit ceil 4096kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethint parent 100:116 handle 116: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:117 handle 117: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:118 handle 118: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:119 handle 119: sfq perturb 10 > > > ># Customer 1 (128kbps) ># Out >run_tc class add dev ethext parent 1:1 classid 1:20 htb \ > rate 128kbit ceil 1024kbit burst 16k cburst 16k prio 1 > >run_tc class add dev ethext parent 1:20 classid 1:21 htb \ > rate 120kbit ceil 1024kbit burst 12k cburst 12k prio 1 >run_tc class add dev ethext parent 1:20 classid 1:22 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 2 >run_tc class add dev ethext parent 1:20 classid 1:23 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 3 >run_tc class add dev ethext parent 1:20 classid 1:24 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethext parent 1:21 handle 21: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:22 handle 22: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:23 handle 23: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:24 handle 24: sfq perturb 10 > ># In >run_tc class add dev ethint parent 100:102 classid 100:120 \ > htb rate 1024kbit ceil 1024kbit burst 16k cburst 16k prio 1 > >run_tc class add dev ethint parent 100:120 classid 100:121 \ > htb rate 120kbit ceil 1024kbit burst 12k cburst 12k prio 1 >run_tc class add dev ethint parent 100:120 classid 100:122 \ > htb rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 2 >run_tc class add dev ethint parent 100:120 classid 100:123 \ > htb rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 3 >run_tc class add dev ethint parent 100:120 classid 100:124 \ > htb rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethint parent 100:121 handle 121: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:122 handle 122: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:123 handle 123: sfq perturb 10 >run_tc qdisc add dev ethint parent 100:124 handle 124: sfq perturb 10 > >...That *should* be enough to work out what''s going on, but I''ll try and break it down a bit as it does look a bit daunting at first. I''ll just deal with outbound traffic, the inbound is much the same but with the added complication of not throttling traffic from the router itself. When this was set up, there wasn''t a facility for an intermediate, internal, virtual interface (Intermediate Queuing Device, IQD ?) so I''m shaping egress on the internal network. If you have more than on internal interface, then you''d need to use an IQD to shape traffic.># install root HTB, point default traffic to 1:12: >run_tc qdisc add dev ethext root handle 1: htb default 12 ># shape everything at uplink speed >run_tc class add dev ethext parent 1: classid 1:1 htb \ > rate $OutSpeed burst 20k cburst 20kSelf explanatory - setup the root of the class heirarchy.># Main traffic ># Out >run_tc class add dev ethext parent 1:1 classid 1:10 htb \ > rate 1400kbit ceil $OutCeilDef burst 16k cburst 16k prio 1Here we add a class for our general traffic - ie everything that doesn''t belong to a specific customers allocation. $OutCeilDef is defined in the params file, as is $OutSpeed. rate sets the limit on the bandwidth allowed through the class, while ceil sets a limit on what may be borrowed from other classes that aren''t using all of their bandwidth. run_tc class add dev ethext parent 1:10 classid 1:11 htb \> rate 500kbit ceil $OutCeilDef burst 12k cburst 12k prio 1 >run_tc class add dev ethext parent 1:10 classid 1:12 htb \ > rate 600kbit ceil $OutCeilDef burst 12k cburst 12k prio 2 >run_tc class add dev ethext parent 1:10 classid 1:13 htb \ > rate 200kbit ceil $OutCeilDef burst 12k cburst 12k prio 3 >run_tc class add dev ethext parent 1:10 classid 1:14 htb \ > rate 100kbit ceil 3072kbit burst 12k cburst 12k prio 4And within that, we add four further classes - just like the "Wondershaper" setup>run_tc qdisc add dev ethext parent 1:11 handle 11: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:12 handle 12: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:13 handle 13: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:14 handle 14: sfq perturb 10And within each class, we use SFQ (Statistical Fair Queueing) which I believe does a reasonable job of splitting bandwidth between streams. Now we add some traffic control for individual customers :># Customer 1 (128kbps) ># Out >run_tc class add dev ethext parent 1:1 classid 1:20 htb \ > rate 128kbit ceil 1024kbit burst 16k cburst 16k prio 1 > >run_tc class add dev ethext parent 1:20 classid 1:21 htb \ > rate 120kbit ceil 1024kbit burst 12k cburst 12k prio 1 >run_tc class add dev ethext parent 1:20 classid 1:22 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 2 >run_tc class add dev ethext parent 1:20 classid 1:23 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 3 >run_tc class add dev ethext parent 1:20 classid 1:24 htb \ > rate 2kbit ceil 1024kbit burst 12k cburst 12k prio 4 > >run_tc qdisc add dev ethext parent 1:21 handle 21: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:22 handle 22: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:23 handle 23: sfq perturb 10 >run_tc qdisc add dev ethext parent 1:24 handle 24: sfq perturb 10This works exactly the same way as our general traffic setup - only the rates are different. In this case, the customer is guaranteed 128kbps, and allowed to burst up to 1Mbps. If you are following, you will realise that we now have a tree that (if it doesn''t get mangled !) looks like this : ethext - root htb - class 1:1 - + class 1:10 - + class 1:11 - SFQ . | + class 1:12 - SFQ . | + class 1:13 - SFQ . | + class 1:14 - SFQ . | . + class 1:20 - + class 1:21 - SFQ . | + class 1:22 - SFQ . | + class 1:23 - SFQ . | + class 1:24 - SFQ . | Note that you need to be able to do basic arithmetic when setting your rates. The sum of the rates for classes 1:11-1:14 must NOT exceed the rate for class 1:10. Non of the ceiling rates for classes 1:11-1:14 can exceed the ceiling for class 1:10. Similarly, the sum of the rates for classes 1:10, 1:20, ... must not exceed the rate for class 1:1, and their ceilings must not exceed the ceiling for class 1:1. If you ignore this, then I believe the result is that the queuing takes place in the wrong class and you lose the prioritisation under heavy traffic conditions. That''s the classes set up, now for some rules. tcstart-rule :># Note - order of rules is significant, first matching rule applies > ># Customers rules come first, then other rules. > ># Misc Customers ># XYZ ># mail >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.123/32 match ip dport 25 0xffff flowid 100:117 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.123/32 match ip sport 25 0xffff flowid 1:17 ># everything else >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.123/32 flowid 100:117 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.123/32 flowid 1:17 > ># Customer 1 ># VOIP (SIP 5060-5072 (5056-5087 = 0x13c0/ffe0), \ > RTP 8000-8051 (8000-8063 = 0x1f40/ffc0)) >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.157 match ip dport 5056 0xffe0 flowid 100:121 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.157 match ip sport 5056 0xffe0 flowid 1:21 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.157 match ip dport 8000 0xffc0 flowid 100:121 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.157 match ip sport 8000 0xffc0 flowid 1:21 ># Mail >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.157/32 match ip dport 25 0xffff flowid 100:123 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.157/32 match ip sport 25 0xffff flowid 1:23 ># Everything else >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.157/32 flowid 100:122 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.157/32 flowid 1:22 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.42/32 flowid 100:122 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.42/32 flowid 1:22 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.19/32 flowid 100:122 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.19/32 flowid 1:22 > ># General Filters > ># VoIP (SIP 5060, RTP 10240-11263, IAX2 4569 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.110 match ip dport 5060 0xffff flowid 100:111 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.110 match ip dport 10240 0xfc00 flowid 100:111 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dst a.b.c.110 match ip dport 4569 0xffff flowid 100:111 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.110 match ip sport 5060 0xffff flowid 1:11 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.110 match ip sport 10240 0xfc00 flowid 1:11 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip src a.b.c.110 match ip sport 4569 0xffff flowid 1:11 > ># DNS >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 53 0xffff flowid 100:111 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 53 0xffff flowid 1:11 > > ># Mail (SMTP 25 & 465, Submisstion 587, POP3 110 & 995, \ > IMAP 143 & 993) is priority 3 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 25 0xffff flowid 100:114 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 25 0xffff flowid 1:14 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 465 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 465 0xffff flowid 1:13 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 587 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 587 0xffff flowid 1:13 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 110 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 110 0xffff flowid 1:13 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 995 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 995 0xffff flowid 1:13 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 143 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 143 0xffff flowid 1:13 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 993 0xffff flowid 100:113 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 993 0xffff flowid 1:13 > > ># RSync traffic (873) priority 4 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip sport 873 0xffff flowid 100:114 >run_tc filter add dev ethint parent 100:0 protocol ip prio 1 u32 \ > match ip dport 873 0xffff flowid 100:114 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip sport 873 0xffff flowid 1:14 >run_tc filter add dev ethext parent 1:0 protocol ip prio 1 u32 \ > match ip dport 873 0xffff flowid 1:14 > > ># TOS Minimum Delay (ssh, NOT scp) in 1:11: >run_tc filter add dev ethext parent 1:0 protocol ip prio 10 u32 \ > match ip src a.b.c.0/24 match ip tos 0x10 0xff flowid 1:11 > ># ICMP (ip protocol 1) in the interactive class 1:11 so we ># can do measurements & impress our friends: >run_tc filter add dev ethext parent 1:0 protocol ip prio 10 u32 \ > match ip src a.b.c.0/24 match ip protocol 1 0xff flowid 1:11 > ># To speed up downloads while an upload is going on, put ACK packets in ># the interactive class: >run_tc filter add dev ethext parent 1:0 protocol ip prio 10 u32 \ > match ip src a.b.c.0/24 match ip protocol 6 0xff \ > match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 \ > match u8 0x10 0xff at 33 flowid 1:11 > > ># Internal I/F > ># TOS Minimum Delay (ssh, NOT scp) >run_tc filter add dev ethint parent 100:0 protocol ip prio 10 u32 \ > match ip dst a.b.c.0/24 match ip tos 0x10 0xff flowid 1:111 > ># ICMP (ip protocol 1) in the interactive class so we can do \ > measurements & impress our friends: >run_tc filter add dev ethint parent 100:0 protocol ip prio 10 u32 \ > match ip dst a.b.c.0/24 match ip protocol 1 0xff flowid 1:111 > ># To speed up downloads while an upload is going on, put ACK \ > packets in the interactive class: >run_tc filter add dev ethint parent 100:0 protocol ip prio 10 u32 \ > match ip dst a.b.c.0/24 match ip protocol 6 0xff \ > match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 \ > match u8 0x10 0xff at 33 flowid 1:111This should also be reasonably easy to decipher. Here we have a customer XYZ that goes in the "misc customers" traffic allocation - basically these customers get to share a traffic allocation and only the SFQ will protect from one of them hogging the bandwidth. In practice, they are light users and it''s not an issue. Then we have Customer 1 that has their own allocation, and we have rules to allocate their traffic to their own classes (note that they have 3 IP addresses in this example). Finally we have the "anything not already classified" rules - VoIP and DNS go into the priority class, mail goes into the low priority class, and rsync goes into the very low priority class. That sets up the traffic control, now you need to monitor and test it ! You can type something like "/sbin/tc -s class show dev ethext" and you''ll get several pages of stats. I knocked up a script that would extract just the basic info to help with testing : cat show_stats_tc>#/bin/bash > >( /sbin/tc -s class show dev ethext > /sbin/tc -s class show dev ethint parent 100: > /sbin/tc -s class show dev ethint parent 101: ) | \ > /bin/sed -e :a -e ''$!N;s/\n / /;ta'' -e ''P;D'' | \ > /bin/sed -r -e "s/^class htb ([0-9]+):([0-9]+) .* [0-9]+ \ > pkt .dropped ([0-9]+),.* rate ([0-9K]+)bit .*$/\1 \2 \3 > \4/" | \ > /bin/grep -v ''^$'' | \ > sort -nFor graphing, I collect the data and stuff it into a number of rrd files using a script run from cron : get_stats_tc :>#/bin/bash ># Script to extract values from shorewall output > >cd /var/rrd > >Now=`date +%s` > >( /sbin/tc -s class show dev ethext > /sbin/tc -s class show dev ethint ) | \ > /bin/sed -e :a -e ''$!N;s/\n / /;ta'' -e ''P;D'' | \ > /bin/sed -r -e "s/^class htb [0-9]+:([0-9]+) .* Sent ([0-9]+) >bytes [0-9]+ \ > pkt .dropped ([0-9]+),.*$/\1 \2 \3/" | \ > /bin/grep -v ''^$'' | \ > ( > while read Class ByteCount DropCount > do > Bytes[${Class}]=${ByteCount} > Dropped[${Class}]=${DropCount} > done > > ># Main link >/usr/bin/rrdtool update tc-main-in.rrd ${Now}:${Bytes[110]:-"U"}:\ > ${Dropped[110]:="U"}:${Bytes[111]:-"U"}:${Dropped[111]:="U"}:\ > ${Bytes[112]:-"U"}:${Dropped[112]:="U"}:${Bytes[113]:-"U"}:\ > ${Dropped[113]:="U"}:${Bytes[114]:-"U"}:${Dropped[114]:="U"} >/usr/bin/rrdtool update tc-main-out.rrd ${Now}:${Bytes[10]:-"U"}:\ > ${Dropped[10]:="U"}:${Bytes[11]:-"U"}:${Dropped[11]:="U"}:\ > ${Bytes[12]:-"U"}:${Dropped[12]:="U"}:${Bytes[13]:-"U"}:\ > ${Dropped[13]:="U"}:${Bytes[14]:-"U"}:${Dropped[14]:="U"} > ># Misc Customers >/usr/bin/rrdtool update tc-misc-cust-in.rrd ${Now}:${Bytes[115]:-"U"}:\ > ${Dropped[115]:="U"}:${Bytes[116]:-"U"}:${Dropped[116]:="U"}:\ > ${Bytes[117]:-"U"}:${Dropped[117]:="U"}:${Bytes[118]:-"U"}:\ > ${Dropped[118]:="U"}:${Bytes[119]:-"U"}:${Dropped[119]:="U"} >/usr/bin/rrdtool update tc-misc-cust-out.rrd ${Now}:${Bytes[15]:-"U"}:\ > ${Dropped[15]:="U"}:${Bytes[16]:-"U"}:${Dropped[16]:="U"}:\ > ${Bytes[17]:-"U"}:${Dropped[17]:="U"}:${Bytes[18]:-"U"}:\ > ${Dropped[18]:="U"}:${Bytes[19]:-"U"}:${Dropped[19]:="U"} > ># Customer 1 >/usr/bin/rrdtool update tc-tag-in.rrd ${Now}:${Bytes[120]:-"U"}:\ > ${Dropped[120]:="U"}:${Bytes[121]:-"U"}:${Dropped[121]:="U"}:\ > ${Bytes[122]:-"U"}:${Dropped[122]:="U"}:${Bytes[123]:-"U"}:\ > ${Dropped[123]:="U"}:${Bytes[124]:-"U"}:${Dropped[124]:="U"} >/usr/bin/rrdtool update tc-tag-out.rrd ${Now}:${Bytes[20]:-"U"}:\ > ${Dropped[20]:="U"}:${Bytes[21]:-"U"}:${Dropped[21]:="U"}:\ > ${Bytes[22]:-"U"}:${Dropped[22]:="U"}:${Bytes[23]:-"U"}:\ > ${Dropped[23]:="U"}:${Bytes[24]:-"U"}:${Dropped[24]:="U"} > >... >)This is a bit of the system I''m particularly proud of, having managed to separate the collection of the stats from the tc counters and the insertion fo those stats into rrd files. Ie, if the actual classes and this script don''t agree, then nothing breaks :-) Eg, if a customer leaves and we delete the classes etc, then updates simply put "U" (unknown) into the RRD database until the script gets modified. And finally, a script to create the rrd databases make_tc :>#!/bin/bash ># Make rrd file for Traffic Shaping stats ># ># cx = traffic count ># dx = drop count ># x=g (global),1-4 > >[ $# -ne 1 ] && ( echo "usage: $0 <filename>" ; exit 1 ) > >rrdtool create $1.rrd -s 300 \ > DS:cg:DERIVE:600:0:U \ > DS:dg:DERIVE:600:0:U \ > DS:c1:DERIVE:600:0:U \ > DS:d1:DERIVE:600:0:U \ > DS:c2:DERIVE:600:0:U \ > DS:d2:DERIVE:600:0:U \ > DS:c3:DERIVE:600:0:U \ > DS:d3:DERIVE:600:0:U \ > DS:c4:DERIVE:600:0:U \ > DS:d4:DERIVE:600:0:U \ > \ > RRA:AVERAGE:0.5:1:576 \ > RRA:MAX:0.5:1:576 \ > RRA:AVERAGE:0.5:6:672 \ > RRA:MAX:0.5:6:672 \ > RRA:AVERAGE:0.5:24:732 \ > RRA:MAX:0.5:24:732 \ > RRA:AVERAGE:0.5:288:730 \ > RRA:MAX:0.5:288:730 > ># CFs for : ># 1 x 576 48hrx 5m ># 6 x 672 14d x 1/2hr ># 24 x 732 61d x 2hr ># 288 x 730 730d x 24hrSo if you''ve followed that through, and are still awake, then you are now a traffic shaping and accounting guru ;-) - or at least you are now equipped to impress your boss :-) -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Oh yes, and a couple of links that are highly recommended : http://lartc.org/howto/ Linux Advanced Routing & Traffic Control HOWTO This is a must read for anyone getting into the advanced networking in Linux http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm HTB Linux queuing discipline manual - user guide As the adverts go, it does exactly what it says in the title. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Simon Hobson wrote:> Oh yes, and a couple of links that are highly recommended : > > http://lartc.org/howto/ > Linux Advanced Routing & Traffic Control HOWTO > > This is a must read for anyone getting into the advanced networking in Linux > > > http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm > HTB Linux queuing discipline manual - user guide > > As the adverts go, it does exactly what it says in the title. > >Thanks Simon, as I said, it is very appreciated. I have read HTB manuals and Lartc, and (for my sake) it was not necessary to explain the code, I like dissecting examples my self ;-) totaly another thing is that I have no experience with HTB, always lacked the time to experiment. And that is where your code comes like the rain in the desert. My plan is to (maybe) try building web frontend to easily manage shorewall''s advanced functions so I can fine tune my network and those networks I manage. Thanks again, many, many, thanks. Ljubomir ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Ljubomir Ljubojevic wrote:>I have read HTB manuals and Lartc, and (for my sake) it was not >necessary to explain the code, I like dissecting examples my self ;-)Well I guess there will be others for whom it might be helpful.>totaly another thing is that I have no experience with HTB, always >lacked the time to experiment. And that is where your code comes like >the rain in the desert.I''ve always found that a working example can save a lot of headache. As I said, this setup didn''t spring up overnight, and I had a few "roadblocks" along the way. A funny (in hindsight) one was when we upgraded the connection at work, and I built a new box to handle routing - the old setup had my traffic shaping box as a bridge. The traffic logging worked fine, but when I turned on the traffic shaping, throughput fell "dramatically". I couldn''t work it out, had a couple of friends offering advice, and then it finally dawned on me in a Homer-esque "Doh" moment - I''d got confused between bits and bytes, and was shaping at 750 kbits/s instead of 750kbytes/s !>My plan is to (maybe) try building web frontend to easily manage >shorewall''s advanced functions so I can fine tune my network and those >networks I manage.I keep thinking I could do with knocking up a script to build the tc scripts from a config file. It''s so easy to make a mistake and screw things up, plus a script could easily check things like "do the rates add up correctly". -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf
Simon Hobson wrote:> > I keep thinking I could do with knocking up a script to build the tc > scripts from a config file. It''s so easy to make a mistake and screw > things up, plus a script could easily check things like "do the rates > add up correctly". >At first glance, I didn''t see anything there that Shorewall''s inbuilt shaper can''t do. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Register Now & Save for Velocity, the Web Performance & Operations Conference from O''Reilly Media. Velocity features a full day of expert-led, hands-on workshops and two days of sessions from industry leaders in dedicated Performance & Operations tracks. Use code vel09scf and Save an extra 15% before 5/3. http://p.sf.net/sfu/velocityconf