List I need some help, advice or just a starting point on the following situation: Link A - 64kbps leased line Link B - 512kbps ADSL line Is it possible to have Link A saturated constantly and have the excess traffic "spill over" onto Link B? I know it''s possible to have packets sent down links in a round-robin fashion and I''ve read in the howto on load sharing over multiple interfaces ( http://lartc.org/howto/lartc.loadshare.html), but I do not have control over the termination of the link at the ISP''s (two different one as well). Also note that splitting different protocols over each of these links are not possible in our case. Reason being, Link A is a more reliable and more expensive link, so I need to over-use it''s capacity if it we''re, and use the cheaper ADSL (link B) offering to keep al services running when the leased line (A) is saturated. Any tips, suggestions and comments would be welcomed. Regards -- Kenneth Kalmer kenneth.kalmer@gmail.com http://opensourcery.blogspot.com _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> List > > I need some help, advice or just a starting point on the following > situation: > > Link A - 64kbps leased line > Link B - 512kbps ADSL line > > Is it possible to have Link A saturated constantly and have the excess > traffic "spill over" onto Link B? I know it''s possible to have packets > sent down links in a round-robin fashion and I''ve read in the howto on > load sharing over multiple interfaces > (http://lartc.org/howto/lartc.loadshare.html), but I do not have control > over the termination of the link at the ISP''s (two different one as > well). Also note that splitting different protocols over each of these > links are not possible in our case. > > Reason being, Link A is a more reliable and more expensive link, so I > need to over-use it''s capacity if it we''re, and use the cheaper ADSL > (link B) offering to keep al services running when the leased line (A) > is saturated. > > Any tips, suggestions and comments would be welcomed. > > RegardsOff hand the thing that comes to mind would be to use an IPTables rule to estimate the rate of flow (I''m not sure what match this is (limit?) but I do think there is one.) and reset the route or mark traffic and have the routing table reroute traffic that is marked. Keep in mind that this will only roughly saturate your 64 kbps link on your outbound traffic. It will do nothing to control the % utilization on the traffic that comes back in to you. Can I ask why you are wanting to saturate tech 64 kbps leased line? Are you trying to encourage management that you need a faster leased line by going to them with graphs stating that the ADSL they purchased did not really solve the problem like they were wanting it to? ;) Another not so nice trick that you could do is just send bogus traffic, via packet gen, out the 64k at a lower priority than the rest of your legitimate traffic thus insurring that the 64 kbps line is full all the time even if you don''t have that much legitimate traffic on it. Yes I should probably be thumped or at least pelted with Nerf Darts for this idea, but it is an answer if you are just trying to saturate the 64 kbps line. (Time to run and hide as I hear the Nerf guns being pumped up!) Grant. . . .
You can''t split a particular IP connection between two links, but can instead only determine which link a particular connection will occur on. Given this, it sounds like you want to have some way to detect that Link A is already saturated and then send all further connections to Link B until Link A is no longer saturated. Maybe someone can tell you how to do that if that''s really what you want to do (others here know far more about this than me), but my guess is you really don''t want to do that. With the huge bandwidth disparity between the two links, route cacheing, and the inability know how much bandwidth any particular conneciton will consume, I think you''d end up with a giant mess... those people with connections unlucky enough to end up on Link A would probably be very unhappy people indeed. Generally speaking I think it would make more sense to put all traffic over Link B, and then use Link A only for emergencies. Maybe route the most critical traffic over Link A if you really want to feel like its being utilized as something other than a pure backup, but personally I wouldn''t even do that. Just because Link A is more reliable and more expensive doesn''t mean it makes sense to use it as your primary conduit. With Link B having eight times the bandwidth, it seems the obvious choice as the primary. Use it, and keep the users happy most of the time (instead of making them miserable most of the time). On the rare ocassions it goes down, use bandwidth shaping to make sure the highest priority traffic gets access to Link A first. In all the time I''ve used DSL, I''ve had severe outages twice for reasons other than standard maintenance. In both cases (in two separate locations), the cause was the ILEC phone company mistakingly dropping the wire pair while doing other work (freakin took over a week in each case to get my connectivity back!!). This sort of thing could just as easily happen with a leased line though, so I''m not really sure I buy that the leased line is really more reliable than DSL line from a high quality ISP. Although maybe a particular SLA makes it so in some legal sense since you can then sue someone. Personally, if your leased line really costs more than the DSL, I''d get rid of it and get a 2nd DSL line from another provider and use that as your backup instead. Anyway, I guess my main point is that the high cost of your leased line might be clouding your thinking on this. I wouldn''t let the comparitive cost be your guiding light here. Go with what makes sense from a technology perspective, and don''t guilt yourself into trying to get full utilization out of the slow link just because it costs more. ----- Original Message ----- From: Kenneth Kalmer To: lartc Sent: Saturday, April 23, 2005 4:34 PM Subject: [LARTC] Spill over List I need some help, advice or just a starting point on the following situation: Link A - 64kbps leased line Link B - 512kbps ADSL line Is it possible to have Link A saturated constantly and have the excess traffic "spill over" onto Link B? I know it''s possible to have packets sent down links in a round-robin fashion and I''ve read in the howto on load sharing over multiple interfaces (http://lartc.org/howto/lartc.loadshare.html), but I do not have control over the termination of the link at the ISP''s (two different one as well). Also note that splitting different protocols over each of these links are not possible in our case. Reason being, Link A is a more reliable and more expensive link, so I need to over-use it''s capacity if it we''re, and use the cheaper ADSL (link B) offering to keep al services running when the leased line (A) is saturated. Any tips, suggestions and comments would be welcomed. Regards -- Kenneth Kalmer kenneth.kalmer@gmail.com http://opensourcery.blogspot.com ------------------------------------------------------------------------------ _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Taylor & Chris (and the list) The arguments behind my choice here is cost driven, the 64kbps line is a fixed monthly rate for unlimited use, the 512kbps line costs us roughly ZAR250 per 3GB of usage. This can get quite expensive as the lines in question is for a college and we all know what students do to bandwidth :) Taken the amount we pay every month for the 64kbps line it''s more economical to over utilize the link as a primary connection than to have it lying around as a backup. South Africa and data connections don''t go well in the same sentence... As Chris suggested, I need something that can detect when Link A is saturated and then redirect the traffic over Link B until there is available bandwidth on Link A again. The rate limit trick of Taylor might work once I get to understand the usage patterns of these students. But for at least the first 3 months I won''t have proper data at my disposal. Thanks for your replies! On 4/24/05, Chris Bennett <chris@symbio.com> wrote:> > You can''t split a particular IP connection between two links, but can > instead only determine which link a particular connection will occur on. > Given this, it sounds like you want to have some way to detect that Link A > is already saturated and then send all further connections to Link B until > Link A is no longer saturated. > Maybe someone can tell you how to do that if that''s really what you want > to do (others here know far more about this than me), but my guess is you > really don''t want to do that. With the huge bandwidth disparity between the > two links, route cacheing, and the inability know how much bandwidth any > particular conneciton will consume, I think you''d end up with a giant > mess... those people with connections unlucky enough to end up on Link A > would probably be very unhappy people indeed. > Generally speaking I think it would make more sense to put all traffic > over Link B, and then use Link A only for emergencies. Maybe route the most > critical traffic over Link A if you really want to feel like its being > utilized as something other than a pure backup, but personally I wouldn''t > even do that. > Just because Link A is more reliable and more expensive doesn''t mean it > makes sense to use it as your primary conduit. With Link B having eight > times the bandwidth, it seems the obvious choice as the primary. Use it, and > keep the users happy most of the time (instead of making them miserable most > of the time). On the rare ocassions it goes down, use bandwidth shaping to > make sure the highest priority traffic gets access to Link A first. > In all the time I''ve used DSL, I''ve had severe outages twice for reasons > other than standard maintenance. In both cases (in two separate locations), > the cause was the ILEC phone company mistakingly dropping the wire pair > while doing other work (freakin took over a week in each case to get my > connectivity back!!). This sort of thing could just as easily happen with a > leased line though, so I''m not really sure I buy that the leased line is > really more reliable than DSL line from a high quality ISP. Although maybe a > particular SLA makes it so in some legal sense since you can then sue > someone. Personally, if your leased line really costs more than the DSL, I''d > get rid of it and get a 2nd DSL line from another provider and use that as > your backup instead. > Anyway, I guess my main point is that the high cost of your leased line > might be clouding your thinking on this. I wouldn''t let the comparitive cost > be your guiding light here. Go with what makes sense from a technology > perspective, and don''t guilt yourself into trying to get full utilization > out of the slow link just because it costs more. > > ----- Original Message ----- > *From:* Kenneth Kalmer <kenneth.kalmer@gmail.com> > *To:* lartc <lartc@mailman.ds9a.nl> > *Sent:* Saturday, April 23, 2005 4:34 PM > *Subject:* [LARTC] Spill over > > List > > I need some help, advice or just a starting point on the following > situation: > > Link A - 64kbps leased line > Link B - 512kbps ADSL line > > Is it possible to have Link A saturated constantly and have the excess > traffic "spill over" onto Link B? I know it''s possible to have packets sent > down links in a round-robin fashion and I''ve read in the howto on load > sharing over multiple interfaces ( > http://lartc.org/howto/lartc.loadshare.html), but I do not have control > over the termination of the link at the ISP''s (two different one as well). > Also note that splitting different protocols over each of these links are > not possible in our case. > > Reason being, Link A is a more reliable and more expensive link, so I need > to over-use it''s capacity if it we''re, and use the cheaper ADSL (link B) > offering to keep al services running when the leased line (A) is saturated. > > Any tips, suggestions and comments would be welcomed. > > Regards > > -- > > Kenneth Kalmer > kenneth.kalmer@gmail.com > http://opensourcery.blogspot.com > > ------------------------------ > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > >-- Kenneth Kalmer kenneth.kalmer@gmail.com http://opensourcery.blogspot.com _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Kenneth Kalmer wrote:> Taylor & Chris (and the list) > > The arguments behind my choice here is cost driven, the 64kbps line is a > fixed monthly rate for unlimited use, the 512kbps line costs us roughly > ZAR250 per 3GB of usage. This can get quite expensive as the lines in > question is for a college and we all know what students do to bandwidth :) > > Taken the amount we pay every month for the 64kbps line it''s more > economical to over utilize the link as a primary connection than to have > it lying around as a backup. South Africa and data connections don''t go > well in the same sentence... > > As Chris suggested, I need something that can detect when Link A is > saturated and then redirect the traffic over Link B until there is > available bandwidth on Link A again. The rate limit trick of Taylor > might work once I get to understand the usage patterns of these > students. But for at least the first 3 months I won''t have proper data > at my disposal. > > Thanks for your replies!Seeing as you are trying to avoid excessive bandwidth charges you find you self in a rather interesting position. Aside form what ever you end up using to control what route your traffic goes out you will ultimately want to have a fairly tight set up on what type of traffic is allowed. I have a client here in my town that managed to saturate an ADSL connection in the first 72 hours that we had the network in a testing state before we even went live with it. Ultimately we had to keep the traffic under 20 GB worth every month or they incurred extra charges. To this end I ended up setting up the router to only allow outbound traffic if it was destined to the following ports: (20) 21, 22, 23, 25, 53, 80, 110, 119, 143, and 443. I told the client that we would run with that set up until they told me that they needed another port opened for any given reason. I took the mentality of ever ything else will be shut down unless you can give me a reasonable request that is backed by someone with political authority (House monitor, Board Members, etc) that says that it is ok for you to be using this type of traffic. After I did this things have been GREAT. We have not had any more problems save for the month that they did not pay the ISP. The real problem that I see in trying to keep one link saturated is that you can control the outbound traffic but you have no way to control the inbound traffic. Well there are ways that you can attempt to control the inbound traffic but they are more a responsive control method (which is less likely to succeed) than a proactive control method like you can do if you control the sending end of the connection. A really drastic idea that I have that might not be too hard to implement would be to establish a PPP multilink session out both connections to a server somewhere on the internet that has HIGH bandwidth (more than your aggregate bandwidth). What this will allow you to do is control what link the traffic will come in and out. All your traffic would appear to be coming from the IP of the server out on the net but I don''t see that as a problem really. I think if you do this you can set up routing and traffic control to try to use the 64 kbps link as the primary link and role over to the 512 kbps ASDL link if the first is saturated. I''m not sure how to go about doing this as this would be either in side of the PPP set up or some other complex method. You cold have traffic split across both links too as you control both ends and can reassemble it the way that it needs to be before it is sent out to the world or in to your LAN. One really nasty what that I could think to make this work would be to set up a default gateway to the IP of the ISP side of the 64 kbps link with a metric of 1 and a default gateway to the IP of the ISP side of the 512 kbps link with a metric of 2. You would need to watch the rate of traffic flowing out the 64 kbps link and any traffic that would be over it you would need to reject with no route to host which would cause your router to choose the other default route out, in this case the 512 kbps link. One MAJOR draw back to this that I see is that you would need / want to flush your routing cache fairly often, at least once per minute to make sure that your router would not end up learning that the 64 kbps link had problems and thus start using the 512 kbps link as a de-fac-to standard by remembering that the 64 kbps link did not have a route to any specific host. Something else that you should consider is that the subnets directly on the other side of each respective link have the best route to them of that said link vs going out the other link and hopping around on the internet to get back in to the first links IPSs network. Why take the long way around the building around the back to get from one front corner to the other? This is the type of paradigm that you will be creating. If this is not an issue you can disregard this statement. Needless to say there are a lot of issues involved here. This is not just a simple here you go type of thing. Grant. . . .
A little googling tells me 250 ZAR ~ 42 USD. Is this correct? If so, ouch.. that''s pricey. 3GB (assuming B in this case is BYTE) comes out to about 9kbit / second over a month, if I did my math correctly. Ouch again. Does the 3GB apply to the total of up and down traffic, or just down? Because you can''t control traffic coming to you very well. You can try to control TCP traffic with policing, but UDP traffic does its own thing. Not to mention jokers who decide to flood the link for the hell of it. Given this new info, it sounds more like you shouldn''t try to use the 512kbit link at all unless the 64kbit link goes down. If you do try to push "excess" traffic onto it, all that does is encourage the use of applications that will consume the entire bandwidth available. If that is really beyond your budget, it doesn''t seem like something you''d want to do. Better to set the expectations at 64kbit so the users don''t get the idea of tuning into Internet radio or something. In fact, if the 64kbit link does go down, it could be a good idea to police the 512kbit link down to 64kbit, just so the users don''t jump for joy when the 64kbit link goes down... (keeping in mind that policing is no guarantee that you''ll actually stay below 64kbit usage, especially if a lot of the traffic is UDP). ----- Original Message ----- From: Kenneth Kalmer To: Chris Bennett ; Taylor Grant Cc: lartc Sent: Monday, April 25, 2005 2:48 AM Subject: Re: [LARTC] Spill over Taylor & Chris (and the list) The arguments behind my choice here is cost driven, the 64kbps line is a fixed monthly rate for unlimited use, the 512kbps line costs us roughly ZAR250 per 3GB of usage. This can get quite expensive as the lines in question is for a college and we all know what students do to bandwidth :) Taken the amount we pay every month for the 64kbps line it''s more economical to over utilize the link as a primary connection than to have it lying around as a backup. South Africa and data connections don''t go well in the same sentence... As Chris suggested, I need something that can detect when Link A is saturated and then redirect the traffic over Link B until there is available bandwidth on Link A again. The rate limit trick of Taylor might work once I get to understand the usage patterns of these students. But for at least the first 3 months I won''t have proper data at my disposal. Thanks for your replies! _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Chris, Taylor and the list Sorry for the late reply, had a much needed holiday and did some research on the topic. I''m willing to test Taylor''s suggestion using some ugly route hacking to balance traffic over the links but I need an application that I can call to get the current throughput on any link, then match this with the links known max and then working on the 95th percentile start changing routes to try and keep each link maxed at around 95%. I know this will not be easy to achieve, but if nothing else it will be a great learning experience. Another thing I stumbled upon is the TEQL scheduling algorithm, http://www.docum.org/docum.org/docs/kernel/teql.php, and it seems to do just what I need. Considering all my connections (except the 64kbps) is PPP connections I can easily use ip-down and ip-up to enslave,deslave devices from the qdisc that does the traffic. I can also easily adjust all the iptables rules from here as well to make sure the source natting is correct. I also understand that in this situation I need to turn off return path filtering in the kernel to avoid the kernel rejecting the packets as martian packets, or is this only the case when I have control over both ends of the link and do channel trunking? Another project that came up is balance, http://balance.sourceforge.net, but this is only a TCP proxy that performs round-robin loads balancing with failover. BalanceNG, the commercial version does have weighted averages and least-used paths and I don''t know what else. A few commercial alternatives, one from Sangoma called Wanpipe uses the TEQL as it''s backed and it only provides the fail-over and alternative configurations and monitoring IIRC. In reply to Chris, yes, South African bandwidth prices has long been a topic that brings fear into the heart of the average South African. Read more on http://www.hellkom.com.co.za, I have nothing to do with this site but they do tell it like it is... Also, all the users are already shaped behind the box so they won''t know when they are using the faster links. We don''t only have these two connections, we have plenty more including wireless links with seperate providers, multiple ADSL and leased line combinations. The 64kbps and 512kbps where only examples, but thanks, I appreciate your opinions on this! As usual, any comments, suggestions, even insults are welcomed and appreciated! On 4/25/05, Chris Bennett <chris@symbio.com> wrote:> > A little googling tells me 250 ZAR ~ 42 USD. Is this correct? If so, ouch.. that''s pricey. > > 3GB (assuming B in this case is BYTE) comes out to about 9kbit / second over a month, if I did my math correctly. Ouch again. > > Does the 3GB apply to the total of up and down traffic, or just down? Because you can''t control traffic coming to you very well. You can try to control TCP traffic with policing, but UDP traffic does its own thing. Not to mention jokers who decide to flood the link for the hell of it. > > Given this new info, it sounds more like you shouldn''t try to use the 512kbit link at all unless the 64kbit link goes down. If you do try to push "excess" traffic onto it, all that does is encourage the use of applications that will consume the entire bandwidth available. If that is really beyond your budget, it doesn''t seem like something you''d want to do. Better to set the expectations at 64kbit so the users don''t get the idea of tuning into Internet radio or something. In fact, if the 64kbit link does go down, it could be a good idea to police the 512kbit link down to 64kbit, just so the users don''t jump for joy when the 64kbit link goes down... (keeping in mind that policing is no guarantee that you''ll actually stay below 64kbit usage, especially if a lot of the traffic is UDP). > > ----- Original Message ----- > From: Kenneth Kalmer > To: Chris Bennett ; Taylor Grant > Cc: lartc > Sent: Monday, April 25, 2005 2:48 AM > Subject: Re: [LARTC] Spill over > > Taylor & Chris (and the list) > > The arguments behind my choice here is cost driven, the 64kbps line is a fixed monthly rate for unlimited use, the 512kbps line costs us roughly ZAR250 per 3GB of usage. This can get quite expensive as the lines in question is for a college and we all know what students do to bandwidth :) > > Taken the amount we pay every month for the 64kbps line it''s more economical to over utilize the link as a primary connection than to have it lying around as a backup. South Africa and data connections don''t go well in the same sentence... > > As Chris suggested, I need something that can detect when Link A is saturated and then redirect the traffic over Link B until there is available bandwidth on Link A again. The rate limit trick of Taylor might work once I get to understand the usage patterns of these students. But for at least the first 3 months I won''t have proper data at my disposal. > > Thanks for your replies! > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > >-- Kenneth Kalmer kenneth.kalmer@gmail.com http://opensourcery.blogspot.com
Chris, Taylor and the list Sorry for the late reply, had a much needed holiday and did some research on the topic. I''m willing to test Taylor''s suggestion using some ugly route hacking to balance traffic over the links but I need an application that I can call to get the current throughput on any link, then match this with the links known max and then working on the 95th percentile start changing routes to try and keep each link maxed at around 95%. I know this will not be easy to achieve, but if nothing else it will be a great learning experience. Another thing I stumbled upon is the TEQL scheduling algorithm, http://www.docum.org/docum.org/docs/kernel/teql.php, and it seems to do just what I need. Considering all my connections (except the 64kbps) is PPP connections I can easily use ip-down and ip-up to enslave,deslave devices from the qdisc that does the traffic. I can also easily adjust all the iptables rules from here as well to make sure the source natting is correct. I also understand that in this situation I need to turn off return path filtering in the kernel to avoid the kernel rejecting the packets as martian packets, or is this only the case when I have control over both ends of the link and do channel trunking? Another project that came up is balance, http://balance.sourceforge.net, but this is only a TCP proxy that performs round-robin loads balancing with failover. BalanceNG, the commercial version does have weighted averages and least-used paths and I don''t know what else. A few commercial alternatives, one from Sangoma called Wanpipe uses the TEQL as it''s backed and it only provides the fail-over and alternative configurations and monitoring IIRC. In reply to Chris, yes, South African bandwidth prices has long been a topic that brings fear into the heart of the average South African. Read more on http://www.hellkom.com.co.za, I have nothing to do with this site but they do tell it like it is... Also, all the users are already shaped behind the box so they won''t know when they are using the faster links. We don''t only have these two connections, we have plenty more including wireless links with seperate providers, multiple ADSL and leased line combinations. The 64kbps and 512kbps where only examples, but thanks, I appreciate your opinions on this! As usual, any comments, suggestions, even insults are welcomed and appreciated! On 4/25/05, Chris Bennett <chris@symbio.com> wrote:> > A little googling tells me 250 ZAR ~ 42 USD. Is this correct? If so, ouch.. that''s pricey. > > 3GB (assuming B in this case is BYTE) comes out to about 9kbit / second over a month, if I did my math correctly. Ouch again. > > Does the 3GB apply to the total of up and down traffic, or just down? Because you can''t control traffic coming to you very well. You can try to control TCP traffic with policing, but UDP traffic does its own thing. Not to mention jokers who decide to flood the link for the hell of it. > > Given this new info, it sounds more like you shouldn''t try to use the 512kbit link at all unless the 64kbit link goes down. If you do try to push "excess" traffic onto it, all that does is encourage the use of applications that will consume the entire bandwidth available. If that is really beyond your budget, it doesn''t seem like something you''d want to do. Better to set the expectations at 64kbit so the users don''t get the idea of tuning into Internet radio or something. In fact, if the 64kbit link does go down, it could be a good idea to police the 512kbit link down to 64kbit, just so the users don''t jump for joy when the 64kbit link goes down... (keeping in mind that policing is no guarantee that you''ll actually stay below 64kbit usage, especially if a lot of the traffic is UDP). > > ----- Original Message ----- > From: Kenneth Kalmer > To: Chris Bennett ; Taylor Grant > Cc: lartc > Sent: Monday, April 25, 2005 2:48 AM > Subject: Re: [LARTC] Spill over > > Taylor & Chris (and the list) > > The arguments behind my choice here is cost driven, the 64kbps line is a fixed monthly rate for unlimited use, the 512kbps line costs us roughly ZAR250 per 3GB of usage. This can get quite expensive as the lines in question is for a college and we all know what students do to bandwidth :) > > Taken the amount we pay every month for the 64kbps line it''s more economical to over utilize the link as a primary connection than to have it lying around as a backup. South Africa and data connections don''t go well in the same sentence... > > As Chris suggested, I need something that can detect when Link A is saturated and then redirect the traffic over Link B until there is available bandwidth on Link A again. The rate limit trick of Taylor might work once I get to understand the usage patterns of these students. But for at least the first 3 months I won''t have proper data at my disposal. > > Thanks for your replies! > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > >-- Kenneth Kalmer kenneth.kalmer@gmail.com http://opensourcery.blogspot.com