Hello there, I''m having lots of problems with my setup here. Let me explain: I am network administrator for my university dorm. We are about 300 users, and we have 2 ADSL connections doing load balancing with 300kbits upstream and 2Mbit downstream. The load balancing is working great, we are doing connection tracking so I can mark and hence prioritize interactive traffic and ACKS on the upstream, and with ipp2p I mark p2p traffic allocating it under the non-interactive queue. The problem comes when there is more than 70 users + or -, when interactive traffic stops working at all, or it has a very VERY high latency. I have a setup based on HTB and WRR. I have only two possible explanations, or either the CONNMARK module introduces a very high latency when the number of entries on /pron/net/ip_conntrack rises above lets say 800, or maybe the Ethernets I have (cheap Ethernets) are getting saturated with that amount of traffic. My server is a AMD Duron 800MHz with 768Mb of RAM. Anyone knowing the marvellous solution? :) Thank you guys.. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
You did not say, what kind of interactive traffic you have, and is your dsl capable to hold it 800 connections should not be a problem for such cpu, if you are sure you did not satureted your dsl, in both directions, and how about http trafiic? then here is another possibility, even if tcp can be shaped on forward, there is still one problem, if your link as almost full, new tcp connections are not controlable at start. all you can do for now is to limit your max rate to 80% of link speed. this problem usualy happens when someone is using bittorent. that software creates 20 or more connections for each file each tcp connection initiation takes about 3kb of data which cant be shaped. At first try to limit your link speed to 50% of capacity , then check if it helps to reduce latency. I am working on imq driver with tcp prediction, which should fix some problems. ----- Original Message ----- From: "GoMi" <gomi@perezoso.net> To: <lartc@mailman.ds9a.nl> Sent: Friday, May 14, 2004 6:18 PM Subject: [LARTC] RV: LATENCY PROBLEMS> Hello there, > I''m having lots of problems with my setup here. Let me explain: > > I am network administrator for my university dorm. We are about 300 users, > and we have 2 ADSL connections doing load balancing with 300kbits upstream > and 2Mbit downstream. > > The load balancing is working great, we are doing connection tracking so I > can mark and hence prioritize interactive traffic and ACKS on theupstream,> and with ipp2p I mark p2p traffic allocating it under the non-interactive > queue. > > The problem comes when there is more than 70 users + or -, wheninteractive> traffic stops working at all, or it has a very VERY high latency. > > I have a setup based on HTB and WRR. > > I have only two possible explanations, or either the CONNMARK module > introduces a very high latency when the number of entries on > /pron/net/ip_conntrack rises above lets say 800, or maybe the Ethernets I > have (cheap Ethernets) are getting saturated with that amount of traffic. > > My server is a AMD Duron 800MHz with 768Mb of RAM. > > Anyone knowing the marvellous solution? :) > > Thank you guys.. > > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Am Friday 14 May 2004 17:18 schrieb GoMi:> I am network administrator for my university dorm. We are about 300 > users, and we have 2 ADSL connections doing load balancing with 300kbits > upstream and 2Mbit downstream.That''s not much bandwidth if there are a lot of p2p users.> and with ipp2p I mark p2p traffic allocating it under the > non-interactive queue.That''s good - but you didn''t write anything about your setup. Make sure that p2p traffic is in a class that gets restricted aggressively. If you''re using HTB, the P2P class should have to borrow everything and have lower ceil and prio than other classes.> I have only two possible explanations, or either the CONNMARK module > introduces a very high latency when the number of entries on > /pron/net/ip_conntrack rises above lets say 800, or maybe the Ethernets > I have (cheap Ethernets) are getting saturated with that amount of > traffic. > > My server is a AMD Duron 800MHz with 768Mb of RAM.I doubt that it''s CONNMARK fault. I only got a 233MHz machine doing the routing and still the CPU has loads of idle time. That little network traffic isn''t so bad and 800 connections are few for 70 users. Andreas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi there Roy, first of all, thanks :) I am talking about http, smtp, pop3 interactive traffic for example. As I said my DSL has 300kbit upstream and 2Mbit downstream, and I have an HTB rate of 150kbit and an ingress policy running at 1500kbit for downstream control. That way I can get sure there are no queues either at my ISP or my DSL modem, and the queues are managed at the Linux box right? I also have iplimit module and each user is limited to 15 parallel connections, so they can not cope with all the BW. -----Mensaje original----- De: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl] En nombre de Roy Enviado el: viernes, 14 de mayo de 2004 18:20 Para: GoMi; lartc@mailman.ds9a.nl Asunto: Re: [LARTC] RV: LATENCY PROBLEMS You did not say, what kind of interactive traffic you have, and is your dsl capable to hold it 800 connections should not be a problem for such cpu, if you are sure you did not satureted your dsl, in both directions, and how about http trafiic? then here is another possibility, even if tcp can be shaped on forward, there is still one problem, if your link as almost full, new tcp connections are not controlable at start. all you can do for now is to limit your max rate to 80% of link speed. this problem usualy happens when someone is using bittorent. that software creates 20 or more connections for each file each tcp connection initiation takes about 3kb of data which cant be shaped. At first try to limit your link speed to 50% of capacity , then check if it helps to reduce latency. I am working on imq driver with tcp prediction, which should fix some problems. ----- Original Message ----- From: "GoMi" <gomi@perezoso.net> To: <lartc@mailman.ds9a.nl> Sent: Friday, May 14, 2004 6:18 PM Subject: [LARTC] RV: LATENCY PROBLEMS> Hello there, > I''m having lots of problems with my setup here. Let me explain: > > I am network administrator for my university dorm. We are about 300 users, > and we have 2 ADSL connections doing load balancing with 300kbits upstream > and 2Mbit downstream. > > The load balancing is working great, we are doing connection tracking so I > can mark and hence prioritize interactive traffic and ACKS on theupstream,> and with ipp2p I mark p2p traffic allocating it under the non-interactive > queue. > > The problem comes when there is more than 70 users + or -, wheninteractive> traffic stops working at all, or it has a very VERY high latency. > > I have a setup based on HTB and WRR. > > I have only two possible explanations, or either the CONNMARK module > introduces a very high latency when the number of entries on > /pron/net/ip_conntrack rises above lets say 800, or maybe the Ethernets I > have (cheap Ethernets) are getting saturated with that amount of traffic. > > My server is a AMD Duron 800MHz with 768Mb of RAM. > > Anyone knowing the marvellous solution? :) > > Thank you guys.. > > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Friday 14 May 2004 13:26, Andreas Klauer wrote:> Am Friday 14 May 2004 17:18 schrieb GoMi: > > I am network administrator for my university dorm. We are about 300 > > users, and we have 2 ADSL connections doing load balancing with 300kbits > > upstream and 2Mbit downstream. > > That''s not much bandwidth if there are a lot of p2p users.Indeed.> > and with ipp2p I mark p2p traffic allocating it under the > > non-interactive queue. > > That''s good - but you didn''t write anything about your setup. > Make sure that p2p traffic is in a class that gets restricted > aggressively. If you''re using HTB, the P2P class should have > to borrow everything and have lower ceil and prio than other classes.Precisely. I have mine limited to 8kbit on a 256kbit connection. The HTB manual mentions some interesting results[1] when you mess with prio, though. (I just lowered the ceil for my p2p bulk class to 16kbit less than my max, 192kbit, and I''m experiencing a significant decrease in lag. Thanks Andreas!) [1] http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm#prio When using IPP2P[2] with 2.4.24 I found that, at least with the edonkey2000 network, it frequently missed enough packets to kill interactivity on my link. Packets that should not have were finding their way into the class for TCP ACKs and ICMP and into the default class, indicating packets were being missed. I was using CONNMARK was suggested to maintain state on which connections were edonkey2000. I upgraded to 2.6.6 last night and patched my kernel and IPTables with L7-Filter[3]. I''m using the edonkey2000 layer 7 match and it''s proving to be extremely effective, even with the Overnet protocol for edonkey2000 enabled which is when I started having problems with IPP2P[2]. A quick snippet of the pattern match[4] for edonkey2000 from L7 indicates just how nasty the match is, and why IPP2P[2] might be missing some packets: rebecca:~# cat /etc/l7-protocols/edonkey.pat ... ^[\xe3\xc5\xe5\xd4].?.?.?.? \ ([\x01\x02\x05\x14\x15\x16\x18\x19\x1a\x1b\x1c\x20\x21\x32\x33\x34 \ \x35\x46\x38\x40\x41\x42\x43\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f \ \x50\x51\x52\x53\x54\x55\x56\x57\x58\x5b\x5c\x60\x81\x82\x90\x91\x93 \ \x96\x97\x98\x99\x9a\x9b\x9c\x9e\xa0\xa1\xa2\xa3\xa4]| \ \x59................?[ -~]|\x96....$) ... #ipp2p essentially uses "\xe3....\x47", which doesn''t seem at all right to me. ... EOF [2] http://rnvs.informatik.uni-leipzig.de/ipp2p/index_en.html [3] http://l7-filter.sourceforge.net/ [4] http://sourceforge.net/project/showfiles.php?\ group_id=80085&package_id=81756 <snip>> Andreas_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Am Friday 14 May 2004 21:06 schrieb Jason Boxman:> Precisely. I have mine limited to 8kbit on a 256kbit connection. The > HTB manual mentions some interesting results[1] when you mess with prio, > though.Yes, prio is very interesting... ;-)> rebecca:~# cat /etc/l7-protocols/edonkey.pat > ... > ^[\xe3\xc5\xe5\xd4].?.?.?.? \ > ([\x01\x02\x05\x14\x15\x16\x18\x19\x1a\x1b\x1c\x20\x21\x32\x33\x34 \ > \x35\x46\x38\x40\x41\x42\x43\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f \ > \x50\x51\x52\x53\x54\x55\x56\x57\x58\x5b\x5c\x60\x81\x82\x90\x91\x93 \ > \x96\x97\x98\x99\x9a\x9b\x9c\x9e\xa0\xa1\xa2\xa3\xa4]| \ > \x59................?[ -~]|\x96....$) > ... > #ipp2p essentially uses "\xe3....\x47", which doesn''t seem at all right > to me. ...It seems that you have done quite a research here. You should definitely tell the IPP2P author about this. Anyway - even if I weren''t using IPP2P, P2P traffic wouldn''t really matter since I put all traffic into user classes. So the only person who''s suffering would be the P2P user. And I don''t really care about that. ;-) I''m not specially limiting P2P traffic either. I just put it into a fourth prio band of a prio qdisc, which works okay. Probably I should try creating two HTB classes per user instead. But this would mean having 3 HTB classes per user (one user class, one default child, one p2p child) which is a bit much. Andreas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Friday 14 May 2004 16:57, Andreas Klauer wrote: <snip>> > rebecca:~# cat /etc/l7-protocols/edonkey.pat > > ... > > ^[\xe3\xc5\xe5\xd4].?.?.?.? \ > > ([\x01\x02\x05\x14\x15\x16\x18\x19\x1a\x1b\x1c\x20\x21\x32\x33\x34 \ > > \x35\x46\x38\x40\x41\x42\x43\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f \ > > \x50\x51\x52\x53\x54\x55\x56\x57\x58\x5b\x5c\x60\x81\x82\x90\x91\x93 \ > > \x96\x97\x98\x99\x9a\x9b\x9c\x9e\xa0\xa1\xa2\xa3\xa4]| \ > > \x59................?[ -~]|\x96....$) > > ... > > #ipp2p essentially uses "\xe3....\x47", which doesn''t seem at all right > > to me. ... > > It seems that you have done quite a research here. > You should definitely tell the IPP2P author about this.Research, yeah. But I didn''t write the above filter. :) I just found it works a lot better than the one IPP2P is presently using.> Anyway - even if I weren''t using IPP2P, P2P traffic wouldn''t really matter > since I put all traffic into user classes. So the only person who''s > suffering would be the P2P user. And I don''t really care about that. ;-)Interesting. How did you accomplish that? I''m used to using HTB where you apportion fixed amounts of bandwidth per class. Does each user have a fixed rate? Sounds like a cool configuration.> I''m not specially limiting P2P traffic either. I just put it into a fourth > prio band of a prio qdisc, which works okay. Probably I should try > creating two HTB classes per user instead. But this would mean having 3 > HTB classes per user (one user class, one default child, one p2p child) > which is a bit much.So right now each user is getting a prio disc?> Andreas_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Am Friday 14 May 2004 23:59 schrieb Jason Boxman:> > Anyway - even if I weren''t using IPP2P, P2P traffic wouldn''t really > > matter since I put all traffic into user classes. So the only person > > who''s suffering would be the P2P user. And I don''t really care about > > that. ;-) > > Interesting. How did you accomplish that?It''s nothing special. Each user gets his/her own HTB class. No matter what kind of traffic the user generates (Interactive, P2P, ...), it shouldn''t affect the other users. This works well in small networks. It wouldn''t work well if I had to create 500 classes for a ADSL line, because the guaranteed rates per user would be too low.> So right now each user is getting a prio disc?It looks like this: http://www.metamorpher.de/files/fairnat_ipp2p.png (warning, big image: ~5000x2000 pixels) Blue = qdisc, Green = class; Blue arrow = filter. Please note that 1:3 is a class for local LAN traffic. The other are user classes. Every user gets a prio qdisc with 4 bands (4th band for P2P, everything else gets classified by the TOS priomap). The prio leafs get SFQ on top. Don''t know if that actually makes sense, but it works well for me. Andreas _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>The load balancing is working great, we are doing connection tracking so I >can mark and hence prioritize interactive traffic and ACKS on the upstream, >and with ipp2p I mark p2p traffic allocating it under the non-interactive >queue. > >The problem comes when there is more than 70 users + or -, when interactive >traffic stops working at all, or it has a very VERY high latency. > >On thought occurs, which is that some P2P protocols apparently misuse the ACKs to send data: http://www.docum.org/stef.coene/qos/faq/cache/49.html Could this be the cause of some of your problems? Perhaps you should take a closer look at your ACK traffic - you could add a SFQ (or ESFQ?) to that queue? Pop some stats on it and try to find out where it is coming from and try to correlate with the traffic from that user? Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
I already did that, but thanks. The problem I think comes because I have an iplimit up to 15 parallel tcp connections for each user. The thing is, if a couple of clients open up emule,kazaa,etc... try to open lots of connections, but only 15 are allowed. Hence I thinkm, they actually produce a kind of DoS over my server, since lots of connections are trying to be stablished, but only 15 are allowed. Does that make sense? Is there a way of maybe do so, but without having this problem? I suppose the kernel treats connections as soon as they arrive, I mean in a FIFO policy. Maybe a policy there would make sense... -----Mensaje original----- De: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl] En nombre de Ed Wildgoose Enviado el: martes, 18 de mayo de 2004 12:52 Para: GoMi CC: lartc@mailman.ds9a.nl Asunto: Re: [LARTC] RV: LATENCY PROBLEMS>The load balancing is working great, we are doing connection tracking so I >can mark and hence prioritize interactive traffic and ACKS on the upstream, >and with ipp2p I mark p2p traffic allocating it under the non-interactive >queue. > >The problem comes when there is more than 70 users + or -, when interactive >traffic stops working at all, or it has a very VERY high latency. > >On thought occurs, which is that some P2P protocols apparently misuse the ACKs to send data: http://www.docum.org/stef.coene/qos/faq/cache/49.html Could this be the cause of some of your problems? Perhaps you should take a closer look at your ACK traffic - you could add a SFQ (or ESFQ?) to that queue? Pop some stats on it and try to find out where it is coming from and try to correlate with the traffic from that user? Ed W _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/