Looking through all the advanced routing docs I can find, I think I have dug up enough parts to cobble something together to do what I need but I am befuddled. Here is a description of my problem: Two providers. A primary I will call provider-A and a backup that I will call provider-B. I collect full routes from both by BGP. My aggregate traffic output varies from about 130MB in the middle of the night up to about 300MB during the day ... a little lower on the weekends. Provider-B is more expensive and has a 50MB minimum. I have fiddled with my BGP so that I end up sending about 45-50MB of traffic to provider-B during my peak time of the day. What I would like to do is pretty much nail provider-B to 50MB at all times using a Linux box in the traffic path. A bit more detail on what I am trying to do: A packet arriving from inside my network has 4 possible dispositions. 1. There is a route to the destination from both providers (most likely). 2. There is a route only from Provider-A. 3. There is a route only from Provider-B. 4. There is no route from either provider. I can make zebra put routes into realms. I can then check arriving packets to see if a realm has a route to the destination. Packets in disposition 2 must go to provider-A, packets in disposition 3 must go to provider-B. Packets in disposition 1 are what I call "the pool" and may go to either A or B to get to their distination. What I want to do is create three streams ... A, B, and Pool. I need to mark A so that it gets routed to provider-A (with FWMARK or some other means ... say TOS), mark stream B so that it is nailed to provider B, BUT when stream B is below 50MB, I want to pull in packets from the pool to bring it up to 50. I do NOT want to rate-limit at 50 because if I loose my link to provider-A or they have a peering issue, more than 50MB might need to go to B, I just want to stop pulling traffic from the pool at that point. Any traffic in the pool remaining after B has pulled what it wants would be marked for provider-A. What I end up with is something like this: Traffic that can only be routed to A goes to A Traffic that can only be routed to B goes to B If B is less than 50MB, get some of the traffic that could go to either A or B to make up the difference Remaining traffic goes to A. Is anyone here ever done this?
On Tue, Dec 11, 2001 at 02:10:12PM -0800, George Bonser wrote:> Two providers. A primary I will call provider-A and a backup that I will > call provider-B. I collect full routes from both by BGP. My aggregate > traffic output varies from about 130MB in the middle of the night up to > about 300MB during the day ... a little lower on the weekends. Provider-B > is more expensive and has a 50MB minimum. I have fiddled with my BGP so > that I end up sending about 45-50MB of traffic to provider-B during my > peak time of the day. What I would like to do is pretty much nail > provider-B to 50MB at all times using a Linux box in the traffic path.So you send 300mbit/s? Wow.> A bit more detail on what I am trying to do: > > A packet arriving from inside my network has 4 possible dispositions. > > 1. There is a route to the destination from both providers (most likely). > 2. There is a route only from Provider-A. > 3. There is a route only from Provider-B. > 4. There is no route from either provider. > > I can make zebra put routes into realms. I can then check arriving packets > to see if a realm has a route to the destination. Packets in disposition 2 > must go to provider-A, packets in disposition 3 must go to provider-B. > Packets in disposition 1 are what I call "the pool" and may go to either A > or B to get to their distination.I get this.> What I want to do is create three streams ... A, B, and Pool. I need to > mark A so that it gets routed to provider-A (with FWMARK or some other > means ... say TOS), mark stream B so that it is nailed to provider B, BUT > when stream B is below 50MB, I want to pull in packets from the pool to > bring it up to 50. I do NOT want to rate-limit at 50 because if I loose my > link to provider-A or they have a peering issue, more than 50MB might need > to go to B, I just want to stop pulling traffic from the pool at that > point. Any traffic in the pool remaining after B has pulled what it wants > would be marked for provider-A.Hmm. Hmmm. I think this can be done! You should attach an ingress shaper to all interfaces that receive traffic. I hope this is only one, or you will be in trouble. To attach an ingress shaper to multiple interfaces, you need it loaded as a module, btw, otherwise Bad Things will happen. Now, you must use a policing filter that will tag all traffic below 50mbit/s. Later on, you will use this tag to route with. All traffic above gets no tag, or another tag. The policing filter isn''t the hard part, see the ''synflood'' section in the HOWTO. The hard part is placing the mark. I think DSMARK does what you want. Later on, you must find a way to route based on the tc_index, as set by DSMARK. Perhaps by matching on it in iptables FORWARD and replacing it with an fwmark. I''m not sure. But I think the answer lies in DSMARK. Otherwise I can whip up a tc filter that does fwmark directly. Let me know. [ Ok, I reread what you want and it''s a bit different, but these are the tools available. You may need to create lots and lots of rules because you need to be able to tell at ingress time where a packet is going to be sent. Tc filters can be hashed & are then lighting fast. ] If you get this working you can give a talk about it on whatever routing conference you choose, btw. Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
Got to ask.. (this is kind of cool btw) .. What machine will he need to shape traffic at those speeds ? Our ISP is probably interested.. We need to shape traffic at 155Mbit/s speeds to do it though.. And, I mentioned that they probably should consider to use a linuxbox to prioritize cusomer like us in favour of those private customers on their fibre network... Or atleast do fair quing.. / P> -----Ursprungligt meddelande----- > Fran: lartc-admin@mailman.ds9a.nl > [mailto:lartc-admin@mailman.ds9a.nl]For bert hubert > Skickat: den 12 december 2001 00:01 > Till: lartc@mailman.ds9a.nl > Amne: Re: [LARTC] I know there must be a way ... > > > On Tue, Dec 11, 2001 at 02:10:12PM -0800, George Bonser wrote: > > > Two providers. A primary I will call provider-A and a backup that I will > > call provider-B. I collect full routes from both by BGP. My aggregate > > traffic output varies from about 130MB in the middle of the night up to > > about 300MB during the day ... a little lower on the weekends. > Provider-B > > is more expensive and has a 50MB minimum. I have fiddled with my BGP so > > that I end up sending about 45-50MB of traffic to provider-B during my > > peak time of the day. What I would like to do is pretty much nail > > provider-B to 50MB at all times using a Linux box in the traffic path. > > So you send 300mbit/s? Wow. > > > A bit more detail on what I am trying to do: > > > > A packet arriving from inside my network has 4 possible dispositions. > > > > 1. There is a route to the destination from both providers > (most likely). > > 2. There is a route only from Provider-A. > > 3. There is a route only from Provider-B. > > 4. There is no route from either provider. > > > > I can make zebra put routes into realms. I can then check > arriving packets > > to see if a realm has a route to the destination. Packets in > disposition 2 > > must go to provider-A, packets in disposition 3 must go to provider-B. > > Packets in disposition 1 are what I call "the pool" and may go > to either A > > or B to get to their distination. > > I get this. > > > What I want to do is create three streams ... A, B, and Pool. I need to > > mark A so that it gets routed to provider-A (with FWMARK or some other > > means ... say TOS), mark stream B so that it is nailed to > provider B, BUT > > when stream B is below 50MB, I want to pull in packets from the pool to > > bring it up to 50. I do NOT want to rate-limit at 50 because if > I loose my > > link to provider-A or they have a peering issue, more than 50MB > might need > > to go to B, I just want to stop pulling traffic from the pool at that > > point. Any traffic in the pool remaining after B has pulled > what it wants > > would be marked for provider-A. > > Hmm. Hmmm. I think this can be done! > > You should attach an ingress shaper to all interfaces that > receive traffic. > I hope this is only one, or you will be in trouble. To attach an ingress > shaper to multiple interfaces, you need it loaded as a module, btw, > otherwise Bad Things will happen. > > Now, you must use a policing filter that will tag all traffic below > 50mbit/s. Later on, you will use this tag to route with. All traffic above > gets no tag, or another tag. > > The policing filter isn''t the hard part, see the ''synflood'' section in the > HOWTO. The hard part is placing the mark. I think DSMARK does > what you want. > Later on, you must find a way to route based on the tc_index, as set by > DSMARK. Perhaps by matching on it in iptables FORWARD and > replacing it with > an fwmark. > > I''m not sure. But I think the answer lies in DSMARK. Otherwise I > can whip up > a tc filter that does fwmark directly. Let me know. > > [ Ok, I reread what you want and it''s a bit different, but these are the > tools available. You may need to create lots and lots of rules > because you > need to be able to tell at ingress time where a packet is going to be > sent. Tc filters can be hashed & are then lighting fast. ] > > If you get this working you can give a talk about it on whatever routing > conference you choose, btw. > > Regards, > > bert > > -- > http://www.PowerDNS.com Versatile DNS Software & Services > Trilab The Technology People > Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - > ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/lartc/ >
On Tue, Dec 11, 2001 at 02:10:12PM -0800, George Bonser wrote:> dug up enough parts to cobble something together to do what I need but I > am befuddled. Here is a description of my problem: > > Two providers. A primary I will call provider-A and a backup that I will > call provider-B. I collect full routes from both by BGP. My aggregate > traffic output varies from about 130MB in the middle of the night up to > about 300MB during the day ... a little lower on the weekends. Provider-B > is more expensive and has a 50MB minimum. I have fiddled with my BGP so > that I end up sending about 45-50MB of traffic to provider-B during my > peak time of the day. What I would like to do is pretty much nail > provider-B to 50MB at all times using a Linux box in the traffic path.Ok - I missed this. So we have this A / [ your network ] - [ linux machine ] - [ router ] \ B The Linux machine also has a full view and knows where traffic will go. But the router does its own routing?> A bit more detail on what I am trying to do: > > A packet arriving from inside my network has 4 possible dispositions. > > 1. There is a route to the destination from both providers (most likely).Should be so yes. Your router routes on AS path length however.> I can make zebra put routes into realms. I can then check arriving packets > to see if a realm has a route to the destination. Packets in disposition 2 > must go to provider-A, packets in disposition 3 must go to provider-B. > Packets in disposition 1 are what I call "the pool" and may go to either A > or B to get to their distination.Well, tricky! Even if the Linux box knows, it should communicate this to the router. Perhaps you can do measurements in the FORWARD chain and reconfigure your router based on those measurements? The linux machine only knows at egress where traffic will go, by then it''s really too late to do anything about it, except possibly DSMARK stuff. Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
On Wed, Dec 12, 2001 at 12:19:09AM +0100, Paul Wisen wrote:> Got to ask.. (this is kind of cool btw) .. > > What machine will he need to shape traffic at those speeds ?Hehe, yeah, that is a problem. 300mbit in + 300mbit out requires a quality machine, probably with multiple PCI busses, or 64bit if possible.> Our ISP is probably interested.. We need to shape traffic at 155Mbit/s > speeds to do it though.. And, I mentioned that they probably should consider > to use a linuxbox to prioritize cusomer like us in favour of those private > customers on their fibre network... Or atleast do fair quing..Well, some things can be done at that rate and other things can''t. 155Mbit/s is well within Linux'' range, we''ve routed 90mbit/s (90 in, 90 out) on a low end machine. RED certainly works at those rates as it is very simpleminded. SFQ is probably out, unless you recompile your kernel. At those speeds you may want larger queues than 128 packets, which is the SFQ default. PRIO would also not be a problem. I think a PRIO with RED qdiscs may be what you want. And who knows, CBQ is also *very* lightweight in its determination of link utilization. I would say that this is most likely possible at 155Mbit/s. You will need to ask around - lots of people are doing gigabit experiments, some configurations work far better than others. Regards, bert -- http://www.PowerDNS.com Versatile DNS Software & Services Trilab The Technology People Netherlabs BV / Rent-a-Nerd.nl - Nerd Available - ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
> Ok - I missed this. So we have this > A > / > [ your network ] - [ linux machine ] - [ router ] > \Or possibly this: A / [ my network ] - [ linux machine ] \ B> The Linux machine also has a full view and knows where traffic will go. > > But the router does its own routing?I currently do not have a linux box in the path. I have two GigE links to Provider-A and a GigE link to provider-B. Each GigE terminates on an Extreme Summit 1i layer 2/3 switch. What I currently do is have each of these units peer with internal BGP to a linux box running Zebra. I use it to set the policies and aggregate everything into one single routing table that I pass to a router just outside the firewall. The effect there is to provide that router with the best next hop for traffic so it goes to the router associated with the egress link without having to peer with all three of the routers. The linux box acts as a route server but is not in the traffic path. But enough of that ... What I really need is a different kind of policing filter. Rather than one that sets a maximum bandwidth, I need one where I can set a MINIMUM bandwidth and something that will pull traffic from another queue to keep it at that minumim OR A filter that can query ANOTHER queue''s rate monitor and reclassify traffic into that queue whenever the target queue''s traffic rate is below some configurable number. It is probably going to involve writing something new. I really do not see a way to do it yet with the pieces I see. I am going to do some more reading over the weekend and see what I can come up with before I start hacking on stuff, I really hate reinventing wheels.