I''ve been lurking for a while, trying to figure out this traffic control stuff. We have 3 ADSL modems upstream, with 2, 2, and 1 megabit total bandwidth, 1/4 of it up, 3/4 down. We also have about 200 workstations downstream, connected through wifi. We are presently using tc to create about 200 buckets with HTB and SFQ. It seems like this is too many buckets, since usually only about 80 of the 200 are active, meaning the bucket size for each workstation is small. We have trouble with P2P uploads, but the P2P solutions I have seen, if I understand them, get rather unwieldy if you have to keep a couple hundred clients separate rather than grouping them by protocol, etc. We absolutely need to be able to keep RRD stats on bandwidth usage per client, but we also need to switch from IP to MAC-based access. It would really help if I could get a few comments on my feeble attempts to develop a strategy here: Does it sound feasible to keep a database of MAC addresses and client numbers and flush/regenerate tc buckets about every other minute? First, iptables would check authorized MAC addresses and mark packets with a client number. This client number would be the basis for throttling uploads. I would run a Perl script that would fetch the ARP table to see what MAC addresses are on, and pair them with a client number and an IP number. In some cases there would be multiple IPs with the same MAC, or multiple IPs and MACs that are the same client, depending on the router/bridge in use. The Perl script would run maybe once every other minute and flush/recreate HTB/SFQ buckets for all current active clients, plus one "misc" bucket, that would get freshly logged on clients, and the occasional client who swapped their wifi card without telling us. Does this strategy seem ... reasonable? Other solutions? Thanks lots. -- Jan Wilson, SysAdmin _/*]; jan@corozal.com Corozal Junior College | |:'' corozal.com corozal.bz Corozal Town, Belize | /'' chetumal.com & linux.bz Reg. Linux user #151611 |_/ Network, PHP, Perl, HTML _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>Does this strategy seem ... reasonable? Other solutions? Thanks lots. > >What is your goal? Why isn''t 2 buckets, one for P2P and the everything else bucket (with an SFQ on each) enough for what you want (ie what''s your target?) What''s wrong with the current setup (apart from wanting to switch to MAC access)? I think I saw that there was something called ESFQ which does something similar to SFQ but on a per IP basis, rather than a per port/IP basis. Perhaps that would help? The HOWTO has an example of setting up a hash type ruleset, perhaps that could be modified to work with MAC addresses...? _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
* Ed Wildgoose <lists@wildgooses.com> [040518 08:57]:> What is your goal? Why isn''t 2 buckets, one for P2P and the everything > else bucket (with an SFQ on each) enough for what you want (ie what''s > your target?) What''s wrong with the current setup (apart from wanting > to switch to MAC access)?Well, as I see it (which could easily be wrong), the problem is with setting up over 200 buckets, even though only 60 to 90, maybe, are active at any one time. With the limited bandwidth we have available (it is very expensive here), it makes rather tiny buckets. If we further divide so there is a bucket for P2P and a regular bucket for each client, now you have over 600 buckets, right? One class per user and two leaves? Unfortunately, we HAVE to break this down by client, and be able to track bandwidth by client. In fact, by client, rather than by machine, which is why I thought of marking by a client number, on upload, at least. Then, on download, IP numbers could translate BACK into client marks by the same Perl script and lookups. What I was really asking is whether redistributing the buckets every 2 minutes is a problem. I know the SFQ perturbs every 5 or 10 seconds, so 2 minutes seemed conservative, comparatively. But I know I might be looking at this all wrong, so I''d appreciate any hint I could get.> I think I saw that there was something called ESFQ which does something > similar to SFQ but on a per IP basis, rather than a per port/IP basis. > Perhaps that would help? The HOWTO has an example of setting up a hash > type ruleset, perhaps that could be modified to work with MAC addresses...?Thanks, I''ll take a look at that also. While the traffic control documentation is pretty good, it could use more commented examples. -- Jan Wilson, SysAdmin _/*]; jan@corozal.com Corozal Junior College | |:'' corozal.com corozal.bz Corozal Town, Belize | /'' chetumal.com & linux.bz Reg. Linux user #151611 |_/ Network, PHP, Perl, HTML _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Am Tuesday 18 May 2004 17:42 schrieb Jan Wilson:> Unfortunately, we HAVE to break this down by client, and be able to track > bandwidth by client. In fact, by client, rather than by machineIf I get this right, you want to know how much traffic each client caused, because they have to pay for it. This should be done independent of your actual traffic shaping strategy. You shouldn''t need tc to count traffic?> What I was really asking is whether redistributing the buckets every 2 > minutes is a problem. I know the SFQ perturbs every 5 or 10 seconds, > so 2 minutes seemed conservative, comparatively.You could add/remove classes on the fly as required. You probably should avoid removing all classes everytime a client becomes (in)active. But if you want to enhance quality and P2P traffic really is a problem, I don''t see another way than to move this traffic away from the user classes. In my own script, I put P2P into user class, because it''s intended for small home network. Imagine you have 5 users, even if 4 have P2P running to max the line out, the 5th user still gets 1/5th of bandwidth guaranteed, which is probably enough. With 80 user, he''d only get 1/80th, which probably isn''t enough to do anything useful. Of course, it also depends on the total bandwidth of your line and your client''s requirements. I''ve never worked with big networks, but I imagine that P2P traffic there is pretty much unwanted. So why do you want to allow P2P traffic eating 99% of your line? Sure, if you limit P2P to say 25% of your line, paying users may complain about bad P2P performance - but that''s still better than all users complaining about bad overall performance in general. Andreas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
* Andreas Klauer <Andreas.Klauer@metamorpher.de> [040518 10:32]:> If I get this right, you want to know how much traffic each client caused, > because they have to pay for it. This should be done independent of your > actual traffic shaping strategy. You shouldn''t need tc to count traffic?Actually, we charge most clients the same, but we would like to restrict the bandwidth hogs somewhat, and in a throttling situation, be able to give some priority to clients who "behave". Do you mean that we should NOT be using the tc to count traffic? What instead? We are using a Python script that reads values using tc -s -d class show <device> Is there a better way?> You could add/remove classes on the fly as required. You probably should > avoid removing all classes everytime a client becomes (in)active. But if > you want to enhance quality and P2P traffic really is a problem, I don''t > see another way than to move this traffic away from the user classes.Maybe a compromise ... I could have the script set up x number of buckets and add/delete up to or down to a certain number of classes, then flush all and recalculate and redistribute the bandwidth only after the number of classes has exceeded or fallen below a certain number.> In my own script, I put P2P into user class, because it''s intended for > small home network. Imagine you have 5 users, even if 4 have P2P running > to max the line out, the 5th user still gets 1/5th of bandwidth > guaranteed, which is probably enough. With 80 user, he''d only get 1/80th, > which probably isn''t enough to do anything useful. Of course, it also > depends on the total bandwidth of your line and your client''s > requirements.We don''t want to just drop all P2P packets, so I guess we will have to throttle it, since our P2P users will suck up all the upload that is available, 24 hours a day if allowed. We would like to place the heavy uploaders into a tighter upload bucket, without regard to whether the upload is P2P or something else (don''t know what). What I''ve seen on this list is that if we have to double or triple our HTB buckets in order to use ipp2p, it might not be worth it for a large network with limited bandwidth. If this is not accurate, I hope someone will correct me.> I''ve never worked with big networks, but I imagine that P2P traffic there > is pretty much unwanted. So why do you want to allow P2P traffic eating > 99% of your line? Sure, if you limit P2P to say 25% of your line, paying > users may complain about bad P2P performance - but that''s still better > than all users complaining about bad overall performance in general.What we would prefer is for the P2P user to complain because he/she is not downloading very fast, and then we can explain it is because they leave P2P running ... that may lead to some social engineering. Our bandwidth is SO expensive, we want to avoid adding additional bandwidth as long as possible. It is also why we are using ADSL rather than T1 or something. Thanks for your comments. -- Jan Wilson, SysAdmin _/*]; jan@corozal.com Corozal Junior College | |:'' corozal.com corozal.bz Corozal Town, Belize | /'' chetumal.com & linux.bz Reg. Linux user #151611 |_/ Network, PHP, Perl, HTML _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Jan Wilson wrote:>* Ed Wildgoose <lists@wildgooses.com> [040518 08:57]: > > >>What is your goal? Why isn''t 2 buckets, one for P2P and the everything >>else bucket (with an SFQ on each) enough for what you want (ie what''s >>your target?) What''s wrong with the current setup (apart from wanting >>to switch to MAC access)? >> >> > >Well, as I see it (which could easily be wrong), the problem is with >setting up over 200 buckets, even though only 60 to 90, maybe, are >active at any one time. With the limited bandwidth we have available >(it is very expensive here), it makes rather tiny buckets. > >Yeah, but the buckets have a guaranteed bandwidth, and also get to share the remaining bandwidth between them. So if you have 600 buckets, and only two are active, then those two buckets should get half the bandwidth each (if you set it up that way). Perhaps that''s really obvious, but just wanted to be sure Also, I think you can do your accounting with firewall rules as well as tc rules. It may be more flexible to seperate accounting from throttling - perhaps it will give you more options to set this up. For example, you could setup the rules with the class of traffic at the top and the users underneath, eg top level is "P2P", "bulk", "interactive", and under each is a user class (or SFQ, etc). Now you could limit the P2P to be only 1/3 of total bandwidth, which is then fought over by the users (each will get a fraction depending on how many others are using it, not how many classes you have). Perhaps for the other classes even SFQ will be enough? The howto mentions some examples of using hashing which may allow you a concise way to setup these rules by IP for example (even though you won''t be accounting that way). You then need to look to iptables to see if you can do some standard rules to account for your traffic. Perhaps someone here could suggest a neat way? Good luck. Would be interested to hear your final solution Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
* Ed Wildgoose <lists@wildgooses.com> [040518 14:20]:> Jan Wilson wrote: > >Well, as I see it (which could easily be wrong), the problem is with > >setting up over 200 buckets, even though only 60 to 90, maybe, are > >active at any one time. With the limited bandwidth we have available > >(it is very expensive here), it makes rather tiny buckets. > > Yeah, but the buckets have a guaranteed bandwidth, and also get to share > the remaining bandwidth between them. So if you have 600 buckets, and > only two are active, then those two buckets should get half the > bandwidth each (if you set it up that way). Perhaps that''s really > obvious, but just wanted to be sureI never mind anyone stating the "obvious" ... one person''s obvious is another''s "eureka" ;-) But I find it awkward, for our purposes, the way rate and ceil seem to work. If we theoretically have 200 clients, and all 200 are on, then we can set rate for 1/200 of the total ... in our case only about 10 kilobits per user. They CAN borrow up to ceil, yes. But since we are REALLY busy, we can''t really do much to give one client more than another. However, if we KNOW only 50 are on right now, then we can give each of those a rate of 1/50, guaranteed. We can set ceil a little differently. And if we have bandwidth hogs, we can adjust things so it is fairer for all. Suppose we have invited 200 guests for dessert. So we make 200 tarts to be sure everyone gets one. But only 50 guests show up. So they eat their own tart, and then fight over the other 150 tarts. What if you could look at the total number of guests before they enter the dining room, make a quick assessment, and put two or three tarts on each plate instead of one. They still might squabble a bit over the remaining tarts, but not as much of a free-for-all as before. That is what I was thinking to do with my Perl script.> Also, I think you can do your accounting with firewall rules as well as > tc rules. It may be more flexible to seperate accounting from > throttling - perhaps it will give you more options to set this up.This is a really interesting idea ... if we can keep our accounting separate then it WOULD open up more possibilities. Maybe someone can point me in the direction of IP traffic accounting that we could use separate from the tc.> For example, you could setup the rules with the class of traffic at the > top and the users underneath, eg top level is "P2P", "bulk", > "interactive", and under each is a user class (or SFQ, etc). Now you > could limit the P2P to be only 1/3 of total bandwidth, which is then > fought over by the users (each will get a fraction depending on how many > others are using it, not how many classes you have). Perhaps for the > other classes even SFQ will be enough?I think I can see that ... if we don''t have to put each customer into one (or two or three) different buckets. If we could even put groups of customers into buckets together, that might be good. But again, it seems like it would be handy to be able to dynamically change bucket sizes and whose packets go through them.> The howto mentions some examples of using hashing which may allow you a > concise way to setup these rules by IP for example (even though you > won''t be accounting that way).I looked at that again, and I do see that it might be useful, especially if we can handle our bandwidth accounting separately. If anyone has a working example I could look at, I might actually have a chance of understanding it ;-) I have no trouble with the hex math, being an old assembly language programmer, but it isn''t quite sinking in.> You then need to look to iptables to see if you can do some standard > rules to account for your traffic. Perhaps someone here could suggest a > neat way?That would be REALLY helpful.> Good luck. Would be interested to hear your final solutionI promise there will be a link to a well-documented Perl script when I get something working. :-) Even if it is a static arrangement, I want to set it up with a script that can be adjusted easily when we have new customers, MAC address changes, additional ADSL modems, etc. Thanks lots for your suggestions. -- Jan Wilson, SysAdmin _/*]; jan@corozal.com Corozal Junior College | |:'' corozal.com corozal.bz Corozal Town, Belize | /'' chetumal.com & linux.bz Reg. Linux user #151611 |_/ Network, PHP, Perl, HTML _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>But I find it awkward, for our purposes, the way rate and ceil seem to >work. If we theoretically have 200 clients, and all 200 are on, then >we can set rate for 1/200 of the total ... in our case only about 10 >kilobits per user. They CAN borrow up to ceil, yes. But since we are >REALLY busy, we can''t really do much to give one client more than >another. > >Hey, I just read the docs, I don''t have any experience of using this. BUT, there appears to be a priority option (it''s not called priority, but I don''t have the docs in front of me) on the HTB qdisc which seems to let you bias the squabble towards certain users and away from others. I vaguely recall that it might be that the excess gets divided as a ratio of the priorities?>However, if we KNOW only 50 are on right now, then we can give each of >those a rate of 1/50, guaranteed. We can set ceil a little >differently. And if we have bandwidth hogs, we can adjust things so >it is fairer for all. > >Suppose we have invited 200 guests for dessert. So we make 200 tarts >to be sure everyone gets one. But only 50 guests show up. So they >eat their own tart, and then fight over the other 150 tarts. > >What if you could look at the total number of guests before they enter >the dining room, make a quick assessment, and put two or three tarts >on each plate instead of one. They still might squabble a bit over >the remaining tarts, but not as much of a free-for-all as before. > >Hmm, perhaps I''m being stupid, but those two situations seem to be identical to me? I can''t see the difference between 1 "tart", and a 1/50 share of the remaining 150 tarts, or starting with 1/50 of 200 tarts...? Also, you appear to be more interested in priortising based on the *type* of traffic. So that appears to be more logically at the top of your tree. Its only like this that you can limit the P2P people I think. Otherwise all you can say is that each P2P user can use their full allocation of bandwidth to download junk. However, if you put P2P at the top and limit it to 20% of total, then each user will be able to use *up to* their full quota, but if there are lots of P2P users, then each may not even be able to reach that level, and so will find they are throttled back below even their personal download quota. This appears to be what you are after?>This is a really interesting idea ... if we can keep our accounting >separate then it WOULD open up more possibilities. Maybe someone can >point me in the direction of IP traffic accounting that we could use >separate from the tc. > >I thought that there were accounting logs on all iptables chains. There is also an accounting option in iptables I seem to remember (check man pages). Perhaps there are other options involving a user space queue which doesn''t actually queue, but counts stuff... I''m sure there are other options as well>>For example, you could setup the rules with the class of traffic at the >>top and the users underneath, eg top level is "P2P", "bulk", >>"interactive", and under each is a user class (or SFQ, etc). Now you >>could limit the P2P to be only 1/3 of total bandwidth, which is then >>fought over by the users (each will get a fraction depending on how many >>others are using it, not how many classes you have). Perhaps for the >>other classes even SFQ will be enough? >> >> > >I think I can see that ... if we don''t have to put each customer into >one (or two or three) different buckets. If we could even put groups >of customers into buckets together, that might be good. But again, it >seems like it would be handy to be able to dynamically change bucket >sizes and whose packets go through them. > >I think this method will make that easier? Remember that each customer will turn up multiple times now, ie for each type of traffic. You can more easily tune the buckets determining the amount of each sort of traffic, eg turning up P2P in the evening, and down in the day simply by adjusting the single top bucket (not each individual one). Good luck Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Tuesday 18 May 2004 17:41, Jan Wilson wrote: <snip>> I promise there will be a link to a well-documented Perl script when I > get something working. :-) Even if it is a static arrangement, I > want to set it up with a script that can be adjusted easily when we > have new customers, MAC address changes, additional ADSL modems, etc.Then indeed it does sound like you have two different but related problem sets here. 1) You want to do bandwidth accounting for customers. 2) You want traffic shaping to provide the smoothest service possible. The former must have some search hits on Google I''d imagine. Bandwidth accounting is a popular thing.> Thanks lots for your suggestions.I have been following your posts with interest. I look forward to hearing about the solutions you devise. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Am Tuesday 18 May 2004 19:43 schrieb Jan Wilson:> Maybe a compromise ... I could have the script set up x number of > buckets and add/delete up to or down to a certain number of classes, > then flush all and recalculate and redistribute the bandwidth only > after the number of classes has exceeded or fallen below a certain > number.Well - HTB will work too even if the children rates added together don''t equal the parent rate (Wondershaper is living proof ;-P), so probably recreating all classes is not necessary. It''s more important that there are as few classes as possible (no class for inactive user). So you could try just adding a class when a user gets active and removing a class when a user gets inactive. I think there was someone on this list who had a script that worked like that with ~1000 users in total. People notice in my LAN (which has only 5 users) when I re-run my fairnat script (the machine it runs on is quite old, so it takes several second until everything is re-created, and interactive connections tend to get sloppy for a while after that), so I think it would be really bad if you did the same every 2 or somewhat minutes with so many users.> We would like to place the heavy uploaders into a tighter upload bucket, > without regard to whether the upload is P2P or something else (don''t know > what).Sure, that sounds like a good idea. However, what exactly do you intend to do to ''tighten'' things?> What I''ve seen on this list is that if we have to double or triple our > HTB buckets in order to use ipp2p, it might not be worth it for a > large network with limited bandwidth.Shaping on a user basis will only affect the P2P users. If you make an IPP2P class as child of the user class with low priority, the effect will be that the P2P user can still use interactive connections (because his own P2P traffic has lower priority). But for other users it won''t matter, they don''t know wether the traffic caused by users is P2P or something else. The only thing you could do on a user basis would be setting a hard ceil for P2P. That way, P2P traffic of a user could never cause the user class to borrow. But I don''t recommend this setup - prioritizing interactive over P2P traffic is something the user should do at home.> Our bandwidth is SO expensive, we want to avoid adding additional > bandwidth as long as possible. It is also why we are using ADSL > rather than T1 or something.Adding bandwidth wouldn''t change much anyway, as long as P2P and similar takes all the bandwidth there is. So this way or another you need a good solution to clean this traffic mess up. HTH Andreas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Jan Wilson wrote:> I''ve been lurking for a while, trying to figure out this traffic > control stuff. We have 3 ADSL modems upstream, with 2, 2, and 1 > megabit total bandwidth, 1/4 of it up, 3/4 down.Once you have your strategy/policy sorted the detail here could be important. What are the bitrates of each connection, are you loadbalancing or choosing your own routes, what interface do you (plan to) shape on? Andy. <snip> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/