as seems typical in my life, my first need is whats almost considered the exceptional or hard case... first, as the dump shows, I''m 3.4.4 ubuntu gutsy uname -a -> Linux hq.tarbox.org 2.6.22-14-server #1 SMP Sun Oct 14 22:09:15 GMT 2007 x86_64 GNU/Linux I have a central server and want to do multi-provider shaping / routing and have services on the firewall (the server is hefty) which need to talk to the network and local servers and be shaped (in particular for asterisk voip and ssh like I see often discussed) ... so I have badness all around... shaping, multiISP, and services on the firewall.... ugh... So, reading the information on the pages and searching yield information... but I don''t know what to do with some of it and definitely can''t tell if its working. its kinda working... things are flowing... I''ve effectively undone balancing... which I should be able to do..(getting balancing working was the easy part... funny that way... turning it off, now thats another thing entirely :-) I''ve included a dump as requested... its pretty huge so I figure that just about anything you might need is in there... So, the dump is for the simple case. I have a couplea rules to define how I want internal traffic routed to providers. I have other rules to provide bandwidth control in later chains through classification / prioritization / queues... (of course, this didn''t really make sense to me... seems like with HIGH_MARKS one could "or" the low bits used for prioritization and get it all done in prefilter in one shot... but whatever... I''m sure theres a reason) but, then comes the question of whats happening with multi-provider priority assignment. Looking at the mangle table, it almost looks like it wants to classify... but none of the packets appear to be getting to the queues in tcpost... at least from what I can tell... so, it could be working, I guess, but I have no real way of knowing... so, do I need to somehow use the "connection" marking stuff? There''s no explanation as to what that really is... conntrack really wants to use it (its reporting something... but not my stuff... at least a lot of the time)... but I''ve played around some with it and it didn''t appear to help... perhaps by not doing the "save" and "restore" in tcrules I''m not saving state between the prefilter / tcout / tcfor / tcpost... I see some reference to this being all "connection" based... but that can''t be right... we still need the data to end up in the queues to be processed in a prioritized fashion... but, while some traffic appears to get "classified" there... its nowhere near what I''m pumping... of course, the connection stuff seems related to prefilter (meaning our prefilter rules won''t get run to insure that stuff gets routed back correctly... ok, I''m cool with that)... after which we still should be packet based... but then I read that just the first packet identifies a connection... and there''s other stuff doing save / restore (e.g. conntrack) so I don''t know if thats in the way... are my marks geting cleared before getting to the next chain? Perhaps going to the default queue at the root? looks that way... but, who knows... and, furthermore, while I can set rules in the prefilter, forward, and post chains, the $FW stuff gets done in output... so, if I needed to do a restore or save (again, whatever they are and why one might need to use them) with that packet information... how would I reference the tcout table to execute the store (only have :P, :F, :T)? so, I can''t route properly there perhaps due to strangeness with iptables... ok, I set the bindaddr in asterisk... but I still need it prioritized... doesn''t appear to be happening even though it looks like the marks want to be set in the tcout chain Then I search and find these wonderful tools like IPTstate... which, unfortunately, fall just shy of usefulness here... sure, iptstate shows connections... and that''s great... but, all I see is source and destination... what about how its going through the firewall? like, what interface is the data going out on? I''m really surprised there''s no method to define which interface one might be interested in in such an outstanding tool... of course, i have bmon and all the other similar tools... so I know when bulk data going out the right interface... but no idea of the relationships back to connections... or source / destination... I could use wireshark... but I still won''t see the queueing used to put the packets out... only by inferring that things are "better"... but that seems kinda out there... I have read about trace... and even thought about using accounting... but that didn''t seem a no brainer either.. Also, the docs are pretty thin in explaining what really goes on in the flow. i''ve read all the shorewall docs, and just about everything referenced... but when it comes to actually explaining what the routing is, how packet marketing is supposed to work between chains in various tables, what a connection mark is, why it matters, how one might use it... I''m baffled... of course, much of this is an issue with iptables... Generally, I wouldn''t need to know all this at the beginning... but, it doesn''t seem to be working... and I can''t figure out what tests to run to determine inter-chain behavior given that the packet counts don''t seem right... So, I''m hugely impressed with iptables, shorewall as a kinda rule based generator to simplify... and some of neat things which I''d like to do eventually in user space to really twiddle the bits live to give real-time fine grained control.. (perhaps not tonight :-)... but wow, even the easy stuff is hard... I can just imagine how wild the hard stuff is... thanks for listening and hopefully you can help. -glenn -- Glenn H. Tarbox, PhD ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
Glenn Tarbox, PhD wrote:> > I''ve included a dump as requested... its pretty huge so I figure that > just about anything you might need is in there...> > So, the dump is for the simple case. I have a couplea rules to define > how I want internal traffic routed to providers. I have other rules to > provide bandwidth control in later chains through classification / > prioritization / queues... (of course, this didn''t really make sense to > me... seems like with HIGH_MARKS one could "or" the low bits used for > prioritization and get it all done in prefilter in one shot... but > whatever... I''m sure theres a reason)Two reasons: a) MARK rules aren''t terminating. So ''OR'' rules are cumulative and must be used with extreme care. b) The routing subsystem looks at the whole mark (there''s no provision for supplying a mask).> > but, then comes the question of whats happening with multi-provider > priority assignment. Looking at the mangle table, it almost looks like > it wants to classify... but none of the packets appear to be getting to > the queues in tcpost... at least from what I can tell... so, it could be > working, I guess, but I have no real way of knowing...You can see how traffic is flowing through the TC classes using "shorewall show tc".> > so, do I need to somehow use the "connection" marking stuff?No.> There''s no explanation as to what that really is...Have you read http://www.shorewall.net/PacketMarking.html?> conntrack really wants to use it (its reporting somethingbut not my stuff... at least a lot of the> time)It is not conntrack that uses it -- it is the ''track'' option in /etc/shorewall/providers -- see http://www.shorewall.net/MultiISP.html#Providers. but I''ve played around some with it and it didn''t appear to> help... perhaps by not doing the "save" and "restore" in tcrules I''m > not saving state between the prefilter / tcout / tcfor / tcpost...You don''t have to. As explained at http://www.shorewall.net/PacketMarking.html, the packet mark is a field in the SKB which follows the packet while it is in the system.> > I see some reference to this being all "connection" based... but that > can''t be right... we still need the data to end up in the queues to be > processed in a prioritized fashion... but, while some traffic appears to > get "classified" there... its nowhere near what I''m pumping... of > course, the connection stuff seems related to prefilter (meaning our > prefilter rules won''t get run to insure that stuff gets routed back > correctly... ok, I''m cool with that)... after which we still should be > packet based... but then I read that just the first packet identifies a > connection... and there''s other stuff doing save / restore ( e.g. > conntrack) so I don''t know if thats in the way... are my marks geting > cleared before getting to the next chain? Perhaps going to the default > queue at the root? looks that way... but, who knows...The only time that you need to do connection marking in conjunction with Traffic Shaping is when you need to ''remember'' that a connection has a particular characteristic. A good example is at http://www.shorewall.net/IPP2P.html.> > and, furthermore, while I can set rules in the prefilter, forward, and > post chains, the $FW stuff gets done in output... so, if I needed to do > a restore or save (again, whatever they are and why one might need to > use them) with that packet information... how would I reference the > tcout table to execute the store (only have :P, :F, :T)? so, I can''t > route properly there perhaps due to strangeness with iptables... ok, I > set the bindaddr in asterisk... but I still need it prioritized... > doesn''t appear to be happening even though it looks like the marks want > to be set in the tcout chainIf you place your marking rules in the POSTROUTING chain (tcpost), then they will apply to both forwarded traffic and to traffic originating on the firewall. Refer to the diagram at the top of http://www.shorewall.net/NetfilterOverview.html. <rant about unrelated product deleted>> > I could use wireshark... but I still won''t see the queueing used to put > the packets out... only by inferring that things are "better"... but > that seems kinda out there... I have read about trace... and even > thought about using accounting... but that didn''t seem a no brainer > either..Again, "shorewall show tc" shows you how many packets have been sent through each of the TC classes. While traffic is flowing through a class, you can also see the bit-rate.> > Also, the docs are pretty thin in explaining what really goes on in the > flow. i''ve read all the shorewall docs, and just about everything > referenced... but when it comes to actually explaining what the routing > is, how packet marketing is supposed to work between chains in various > tables, what a connection mark is, why it matters, how one might use > it... I''m baffled...If you have read http://www1.shorewall.net/NetfilterOverview.html and http://www.shorewall.net/PacketMarking.html and still don''t understand it, then anything more that I would write here probably won''t help. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
On Mon, 2007-12-03 at 02:58 -0800, Glenn Tarbox, PhD wrote:> > I''ve included a dump as requested... its pretty huge so I figure that > just about anything you might need is in there... > > So, the dump is for the simple case. I have a couplea rules to define how I want internal traffic routed to providers.Here''s what I see: Chain tcfor (1 references) pkts bytes target prot opt in out source destination 0 0 MARK udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:9998 MARK set 0xa 0 0 MARK udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:9999 MARK set 0xa 1925 101K MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 MARK set 0x1e 0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8802 MARK set 0x32 The first two rules could be combined by using a port range. In /etc/shorewall/tcrules: 0xa 0.0.0.0/0 0.0.0.0/0 udp 9998:9999 19955 packets were forwarded during the period covered by the dump: Chain FORWARD (policy ACCEPT 19956 packets, 13M bytes) 19955 13M tcfor 0 -- * * 0.0.0.0/0 0.0.0.0/0 So you are only marking about 10 percent of them (1925 out of 19955). For OUTPUT, you are mistakenly trying to set traffic control marks in the tcout chain. This is actually a bug in Shorewall-shell (all versions including 4.0.6). The tcout chain should have the same restriction as the PREROUTING chain -- namely that with HIGH_ROUTE_MARKS=Yes, only high mark values can be assigned in that chain. Shorewall-shell doesn''t enforce that. At any rate, none of the traffic matched any of your rules. The requirements seem similar enough that you might must move all of the rules to POSTROUTING and only have to code them once. In summary, you marked only 10% of the forwarded traffic and none of the OUTPUT traffic. This leads, of course, to most of the traffic going into the default classes. Chain tcpost (1 references) pkts bytes target prot opt in out source destination 0 0 CLASSIFY 0 -- * eth0 0.0.0.0/0 0.0.0.0/0 MARK match 0xa/0xff CLASSIFY set 1:110 0 0 CLASSIFY 0 -- * eth0 0.0.0.0/0 0.0.0.0/0 MARK match 0x14/0xff CLASSIFY set 1:120 1922 100K CLASSIFY 0 -- * eth0 0.0.0.0/0 0.0.0.0/0 MARK match 0x1e/0xff CLASSIFY set 1:130 0 0 CLASSIFY 0 -- * eth0 0.0.0.0/0 0.0.0.0/0 MARK match 0x28/0xff CLASSIFY set 1:140 0 0 CLASSIFY 0 -- * eth0 0.0.0.0/0 0.0.0.0/0 MARK match 0x32/0xff CLASSIFY set 1:150 0 0 CLASSIFY 0 -- * eth2 0.0.0.0/0 0.0.0.0/0 MARK match 0xa/0xff CLASSIFY set 2:110 0 0 CLASSIFY 0 -- * eth2 0.0.0.0/0 0.0.0.0/0 MARK match 0x14/0xff CLASSIFY set 2:120 2 104 CLASSIFY 0 -- * eth2 0.0.0.0/0 0.0.0.0/0 MARK match 0x1e/0xff CLASSIFY set 2:130 0 0 CLASSIFY 0 -- * eth2 0.0.0.0/0 0.0.0.0/0 MARK match 0x28/0xff CLASSIFY set 2:140 0 0 CLASSIFY 0 -- * eth2 0.0.0.0/0 0.0.0.0/0 MARK match 0x32/0xff CLASSIFY set 2:150 Looking at the Traffic control part of the dump, here is the interesting part of eth0: class htb 1:130 parent 1:1 leaf 130: prio 3 quantum 1500 rate 100000bit ceil 1000Kbit burst 1624b/8 mpu 0b overhead 0b cburst 2749b/8 mpu 0b overhead 0b level 0 Sent 423522 bytes 6419 pkt (dropped 0, overlimits 0 requeues 0) rate 496bit 0pps backlog 0b 0p requeues 0 lended: 4299 borrowed: 2120 giants: 0 tokens: 121953 ctokens: 20984 class htb 1:140 parent 1:1 leaf 140: prio 4 quantum 1500 rate 100000bit ceil 1000Kbit burst 1624b/8 mpu 0b overhead 0b cburst 2749b/8 mpu 0b overhead 0b level 0 Sent 2258870 bytes 2519 pkt (dropped 0, overlimits 0 requeues 0) rate 2232bit 0pps backlog 0b 0p requeues 0 lended: 1523 borrowed: 996 giants: 0 tokens: 14453 ctokens: 10234 These two classes are identically defined from the point of view of HTB except that 1:140 appears to be the default. These two are getting all of the traffic. For eth4: class htb 2:130 parent 2:1 leaf 130: prio 3 quantum 1500 rate 60000bit ceil 600000bit burst 1574b/8 mpu 0b overhead 0b cburst 2249b/8 mpu 0b overhead 0b level 0 Sent 132 bytes 2 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 lended: 2 borrowed: 0 giants: 0 tokens: 188422 ctokens: 27640 class htb 2:140 parent 2:1 leaf 140: prio 4 quantum 1500 rate 60000bit ceil 600000bit burst 1574b/8 mpu 0b overhead 0b cburst 2249b/8 mpu 0b overhead 0b level 0 Sent 11250307 bytes 142912 pkt (dropped 0, overlimits 0 requeues 0) rate 80bit 0pps backlog 0b 0p requeues 0 lended: 142912 borrowed: 0 giants: 0 tokens: 198828 ctokens: 28671 Note that the lonely two packets that were classified to 2:130 were actually sent through that class. The rest went into the default class. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
On Mon, 2007-12-03 at 15:11 -0800, Tom Eastep wrote:> > At any rate, none of the traffic matched any of your rules. The > requirements seem similar enough that you might must move all of the > rules to POSTROUTING and only have to code them once. >And of course, in POSTROUTING the rules could be changed to CLASSIFY rules so that you wouldn''t have to do any packet marking at all. Just a thought, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
On Mon, 2007-12-03 at 15:11 -0800, Tom Eastep wrote:> For OUTPUT, you are mistakenly trying to set traffic control marks in > the tcout chain. This is actually a bug in Shorewall-shell (all versions > including 4.0.6). The tcout chain should have the same restriction as > the PREROUTING chain -- namely that with HIGH_ROUTE_MARKS=Yes, only high > mark values can be assigned in that chain. Shorewall-shell doesn''t > enforce that.Neither does Shorewall-perl, for that matter. Both compilers will enforce this beginning with Shorewall 4.2. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4
On Mon, 2007-12-03 at 15:11 -0800, Tom Eastep wrote:> > For OUTPUT, you are mistakenly trying to set traffic control marks in > the tcout chain. This is actually a bug in Shorewall-shell (all versions > including 4.0.6). The tcout chain should have the same restriction as > the PREROUTING chain -- namely that with HIGH_ROUTE_MARKS=Yes, only high > mark values can be assigned in that chain. Shorewall-shell doesn''t > enforce that.I should add that you might find that a lot more OUTPUT traffic matches your rules, once you move them to POSTROUTING. Here''s the mangle OUTPUT chain: Chain OUTPUT (policy ACCEPT 342K packets, 34M bytes) pkts bytes target prot opt in out source destination 143K 9243K CONNMARK 0 -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00 192K 20M tcout 0 -- * * 0.0.0.0/0 0.0.0.0/0 MARK match 0x0/0xff00 Note that only traffic whose connection isn''t marked in the upper byte goes through the tcout chain. So no traffic associated with external connections is going through that chain. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- SF.Net email is sponsored by: The Future of Linux Business White Paper from Novell. From the desktop to the data center, Linux is going mainstream. Let it simplify your IT future. http://altfarm.mediaplex.com/ad/ck/8857-50307-18918-4