Hi everyone, first of all thanks to everyone working on shorewall; it is a great firewall. I really like it :-). The intro is a bit lengthy, but I''ve been working for a while on the problem, so I think it''s necessary, please bear with me... I downloading using BitTorrent, and wanting to be fair, I seed to at least give the same amount back that I download. Unfortunately I have a download speed of about 4Mbit/s, while my upload is only around 384kb/s, so seeding takes a long time for me. The big problem is that while I seed several files, everythings slows down significantly - web browsing gets noticably slower and ssh connections show a huge lag (often 1s or more from keypress to display back on my screen). I thought upload bandwith is clogged up, so I limited upload to 35kb/s (280kbit) in Azureus (the BitTorrent client) and started to look at traffic shaping. I setup the shorewall internal traffic shaping very close to the Wondershaper in the documentation, and putting ssh into the prioritized class 1. I even shape the upload down to 300kbit/s, just to make sure that I am really shaping it. Also, I put the bittorrent port into the lower class 3. Yet, this did not improve the situation much. SSH is still pretty unusable and web-browsing is slow. So I read up and it seems that BitTorrent might be using random ports on both sides, so I installed the ipp2p module and added rules for it to mark all p2p traffic for class 3. This did not make a difference at all from basic traffic shaping. So I''m very lost, and I''d really like to get it to work so that I can use SSH (not to mention voice) without having to stop my BitTorrent client. Any help would be greatly appreciated! I''ve attached the output of ''shorewall dump'' while I''m seeding, and also the tc files from shorewall. Any suggestions would be helpful, I don''t know what to try anymore. ~David P.S. The dump was taken while 4 files were being seeded. eth0 is connected to my cable modem. It seems that when I seed only one file, even when it''s with the full 35kb/s that I have allocated, the other connections don''t suffer much. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
(Caveat: i am not a traffic shaping expert.) David, The short answer to your question is that there''s no really good way to shape BitTorrent. The long answer is that you may be able to improve the situation somewhat by prioritising all traffic to a certain limit (e.g. 80% of your link speed), then prioritising some of your traffic higher than that. The traffic shaping documentation with a basic configuration to replace WonderShaper should give you a good start at this. Regards, Paul ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Hi Paul, On 8/20/06, Paul Gear <pgear@redlands.qld.edu.au> wrote:> (Caveat: i am not a traffic shaping expert.) > > David, > The short answer to your question is that there''s no really good way to > shape BitTorrent. The long answer is that you may be able to improveNo good way maybe, but I assumed that ipp2p would be able to recognize that traffic. I''m not using any encryption / methods to prevent shaping detection, so I don''t see why it shouldn''t work.> the situation somewhat by prioritising all traffic to a certain limit > (e.g. 80% of your link speed), then prioritising some of your traffic > higher than that. The traffic shaping documentation with a basic > configuration to replace WonderShaper should give you a good start at this.If you look at my configuration, I do do that for SSH traffic already. My problem is that it does not seem to have any effect, and I can''t figure out why. ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote:> P.S. The dump was taken while 4 files were being seeded. eth0 is > connected to my cable modem.And it''s completely useless. It covers a period of over 13 hours yet you expect us to use it to analyze what happened in the last five minutes -- we don''t have that kind of crystal ball (or I don''t, at least). But I notice that there are no send queues built up in your router -- that would tell me that if you are seeing poor interactive performance, then the queues are likely at your ISP''s end. The documentation gives useful hints about what to do about that... -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Hi Tom, On 8/20/06, Tom Eastep <teastep@shorewall.net> wrote:> David Mohr wrote: > > > P.S. The dump was taken while 4 files were being seeded. eth0 is > > connected to my cable modem. > > And it''s completely useless. It covers a period of over 13 hours yet you > expect us to use it to analyze what happened in the last five minutes -- > we don''t have that kind of crystal ball (or I don''t, at least).Well, first of all, I don''t expect anyone to do anything here - I''m just asking for help, because I can''t figure it out by myself. I don''t know how to take a better snapshot of the situation - please, let me know if I can provide a dump with more useful information. I thought it would be best to take a ''shorewall dump'' in exactly the situation where the problem occurs. I didn''t see any notes about that in the documentation.> But I notice that there are no send queues built up in your router -- > that would tell me that if you are seeing poor interactive performance, > then the queues are likely at your ISP''s end. > > The documentation gives useful hints about what to do about that...I read the documentation about traffic shaping; I would not have asked here otherwise. I know that to prevent queuing on my ISP''s end, I need to make sure that they don''t have anything to queue and I can only do by shaping below my regular maximum upload. As I mentioned in my email, I do that - I have 384kbit/s upload (verified), but I have 300kbit/s in my tcdevices. ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote:> ... > No good way maybe, but I assumed that ipp2p would be able to recognize > that traffic. I''m not using any encryption / methods to prevent > shaping detection, so I don''t see why it shouldn''t work.Check the archives of this list w.r.t. ipp2p. It is not supported, but some people have had success with it, IIRC.> ... > If you look at my configuration, I do do that for SSH traffic already. > My problem is that it does not seem to have any effect, and I can''t > figure out why.Sorry - i missed the attachments in your original post. Looking at your files, my expertise on this has just been exhausted, but i wonder whether it is what you want to be marking ipp2p as 1 instead of 3. Paul ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote: .> > Well, first of all, I don''t expect anyone to do anything here - I''m > just asking for help, because I can''t figure it out by myself. > I don''t know how to take a better snapshot of the situation - please, > let me know if I can provide a dump with more useful informationDavid, Please answer one question for me -- how did you find this mailing list? I''m not trying to be rude or sarcastic -- I really want to know because I truly do not understand how people find the mailing list without reading the instructions for submitting a problem report. The only reference to the mailing list on the Shorewall web site is (I believe) on the same page that gives complete instructions for submitting a useful problem report. Thanks, -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
On 8/20/06, Tom Eastep <teastep@shorewall.net> wrote:> David Mohr wrote: > > Well, first of all, I don''t expect anyone to do anything here - I''m > > just asking for help, because I can''t figure it out by myself. > > I don''t know how to take a better snapshot of the situation - please, > > let me know if I can provide a dump with more useful information > > David, > > Please answer one question for me -- how did you find this mailing list? > > I''m not trying to be rude or sarcastic -- I really want to know because > I truly do not understand how people find the mailing list without > reading the instructions for submitting a problem report. The only > reference to the mailing list on the Shorewall web site is (I believe) > on the same page that gives complete instructions for submitting a > useful problem report.I don''t know where to start - please forgive my impulsive response, I was a little frustrated regarding this problem. This is of course not your fault, it is just the persistance of this problem. I had all intentions of reporting this correctly (I can imagine how annoying it is to get those questions that are answered on the web), nonetheless I missed the reset somehow. I apologize for wasting your time with my previous emails. Attached is a better dump, taken right after a reset. I only took the time to verify that the problem still persisted. ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Hi Paul, On 8/20/06, Paul Gear <pgear@redlands.qld.edu.au> wrote:> David Mohr wrote: > > ... > > No good way maybe, but I assumed that ipp2p would be able to recognize > > that traffic. I''m not using any encryption / methods to prevent > > shaping detection, so I don''t see why it shouldn''t work. > > Check the archives of this list w.r.t. ipp2p. It is not supported, but > some people have had success with it, IIRC.I did not find much useful with a search (did anyone ever mention that the sf.net mailing list search kinda sucks?) I can let you know that ipp2p is not in patch-o-matic anymore, though, I cam across that message. It''s now on http://www.ipp2p.org/ . If I don''t missunderstand something, then SSH being in class 1 should work even without ipp2p - so that is not my main issue.> > If you look at my configuration, I do do that for SSH traffic already. > > My problem is that it does not seem to have any effect, and I can''t > > figure out why. > > Sorry - i missed the attachments in your original post.No problem.> Looking at your files, my expertise on this has just been exhausted, but > i wonder whether it is what you want to be marking ipp2p as 1 instead of 3.Thanks for catching that. I admit that I was confused about that. I read the docu about TEST again, and I fixed it. Do you think it correctly marks things as class 3 as follows? ==SNIP=tcrules==========RESTORE:P - - all CONTINUE:P - - all - - - !0 3:P - - ipp2p ipp2p SAVE:P - - all - - - 3 ==SNAP=============== ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote:> I don''t know where to start - please forgive my impulsive response, I > was a little frustrated regarding this problem. This is of course not > your fault, it is just the persistance of this problem. I had all > intentions of reporting this correctly (I can imagine how annoying it > is to get those questions that are answered on the web), nonetheless I > missed the reset somehow. I apologize for wasting your time with my > previous emails.And I apologize for my abruptness -- my Father has been quite ill and the stress is having an affect on me, I''m afraid.> > Attached is a better dump, taken right after a reset. I only took the > time to verify that the problem still persisted. >Look here: Chain eth0_in (1 references) pkts bytes target prot opt in out source destination 73 15498 dynamic all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID,NEW 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:67:68 7179 6447K net2fw all -- * * 0.0.0.0/0 0.0.0.0/0 Almost all of the packets that touched your firewall during the capture period were *INPUT* from the net. Have you tuned your IN-BANDWIDTH properly as described in the Traffic Control document? At least during the few seconds that the current dump was being collected, your traffic was primarily from the net to the firewall. There was no shaped output traffic at all. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
Tom Eastep wrote:> David Mohr wrote: > >> I don''t know where to start - please forgive my impulsive response, I >> was a little frustrated regarding this problem. This is of course not >> your fault, it is just the persistance of this problem. I had all >> intentions of reporting this correctly (I can imagine how annoying it >> is to get those questions that are answered on the web), nonetheless I >> missed the reset somehow. I apologize for wasting your time with my >> previous emails. > > And I apologize for my abruptness -- my Father has been quite ill and > the stress is having an affect on me, I''m afraid. > >> Attached is a better dump, taken right after a reset. I only took the >> time to verify that the problem still persisted. >> > > Look here: > > Chain eth0_in (1 references) > pkts bytes target prot opt in out source > destination > 73 15498 dynamic all -- * * 0.0.0.0/0 > 0.0.0.0/0 state INVALID,NEW > 0 0 ACCEPT udp -- * * 0.0.0.0/0 > 0.0.0.0/0 udp dpts:67:68 > 7179 6447K net2fw all -- * * 0.0.0.0/0 > 0.0.0.0/0 > > Almost all of the packets that touched your firewall during the capture > period were *INPUT* from the net. > > Have you tuned your IN-BANDWIDTH properly as described in the Traffic > Control document? At least during the few seconds that the current dump > was being collected, your traffic was primarily from the net to the > firewall. There was no shaped output traffic at all.Also -- where is your BitTorrent client running? If it is on the firewall, then you are not marking fw->net traffic at all!!! -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
On 8/20/06, Tom Eastep <teastep@shorewall.net> wrote:> David Mohr wrote: > > > I don''t know where to start - please forgive my impulsive response, I > > was a little frustrated regarding this problem. This is of course not > > your fault, it is just the persistance of this problem. I had all > > intentions of reporting this correctly (I can imagine how annoying it > > is to get those questions that are answered on the web), nonetheless I > > missed the reset somehow. I apologize for wasting your time with my > > previous emails. > > And I apologize for my abruptness -- my Father has been quite ill and > the stress is having an affect on me, I''m afraid.That''s ok. I wish your father the best, I can imagine that that stress is very overwhelming. Plus, I did do something to deserve some scolding...> > Attached is a better dump, taken right after a reset. I only took the > > time to verify that the problem still persisted. > > > > Look here: > > Chain eth0_in (1 references) > pkts bytes target prot opt in out source > destination > 73 15498 dynamic all -- * * 0.0.0.0/0 > 0.0.0.0/0 state INVALID,NEW > 0 0 ACCEPT udp -- * * 0.0.0.0/0 > 0.0.0.0/0 udp dpts:67:68 > 7179 6447K net2fw all -- * * 0.0.0.0/0 > 0.0.0.0/0 > > Almost all of the packets that touched your firewall during the capture > period were *INPUT* from the net.That''s because my upload is so much smaller than my download, and I think I was downloading the time after the reset.> Have you tuned your IN-BANDWIDTH properly as described in the Traffic > Control document? At least during the few seconds that the current dump > was being collected, your traffic was primarily from the net to the > firewall. There was no shaped output traffic at all.Yes, I tuned my in-bandwidth. Maybe not aggressively enough, but at the time of the capture, I was not downloading with my full bandwidth, but only a fraction. And the problem also occurs when there is only outgoing traffic. See attached dump, where I am only uploading. I couldn''t find a eth0_out chain, so I don''t know how to quantify this. How do you determine that there was no shaped output traffic? I''m sorry, I''m guessing it''s the qdisc/tc output, but I find it very confusing and can''t make much sense of it. Also I noted right now that my interactivity (ssh) was pretty bad when only uploading with 18.5kb/s (out of 37.5 shaping limit, so I''m assuming there was no shaping). I just don''t understand it. ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
On 8/20/06, Tom Eastep <teastep@shorewall.net> wrote:> Also -- where is your BitTorrent client running? If it is on the > firewall, then you are not marking fw->net traffic at all!!! >How dumb of me not to mention this! Yes, it is indeed running on the firewall. But why am I not marking that traffic? All entries in tcrules have 0.0.0.0/0 or ''-'' both as src and dst, so shouldn''t they apply? ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote:> ... > If I don''t missunderstand something, then SSH being in class 1 should > work even without ipp2p - so that is not my main issue.But if ipp2p and ssh are in the same shaping class, you wouldn''t expect to achieve any better a result. Also note Tom''s comment earlier about traffic originating on the firewall. Have another read of http://shorewall.net/traffic_shaping.htm and you''ll find there are some specific notes about getting traffic shaping to work on the firewall itself. Paul ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
On 8/21/06, David Mohr <damailings@mcbf.net> wrote:> On 8/20/06, Tom Eastep <teastep@shorewall.net> wrote: > > Also -- where is your BitTorrent client running? If it is on the > > firewall, then you are not marking fw->net traffic at all!!! > > > > How dumb of me not to mention this! Yes, it is indeed running on the > firewall. But why am I not marking that traffic? All entries in > tcrules have 0.0.0.0/0 or ''-'' both as src and dst, so shouldn''t they > apply?Nevermind. I need to specify $FW as the source in that case. Paul, thanks a bunch for the pointer to the docs again. It seems another time I didn''t do my homework, sorry. I will add the rules later and report what difference it makes. ~David ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
David Mohr wrote:> Hi everyone, > first of all thanks to everyone working on shorewall; it is a great > firewall. I really like it :-). > > The intro is a bit lengthy, but I''ve been working for a while on the > problem, so I think it''s necessary, please bear with me... > > I downloading using BitTorrent, and wanting to be fair, I seed to at > least give the same amount back that I download. Unfortunately I have > a download speed of about 4Mbit/s, while my upload is only around > 384kb/s, so seeding takes a long time for me. The big problem is that > while I seed several files, everythings slows down significantly - web > browsing gets noticably slower and ssh connections show a huge lag > (often 1s or more from keypress to display back on my screen). ><snip> I have run into this bittorrent problem, and from what I understand it relates to a limit on half open connections. Are you running xp sp2? Here is a writeup on it. http://blog.davidkaspar.com/archives/2005/04/windows-xp-sp2-and-event-id-4226.php It seems that this is where the problem lies, from what fiddling I have done, but, I think that once you disable the limit on xpsp2, that somewhere else down the line, something is maxing out (ie, xp->router->cable''modem''->isp router...). Anyways, when I upped my limit from 10 to 100 on xpsp2, the behavior of the slowdown changed, but again, I think that something else at my end is being a bottleneck. But, since I do not understand exactly what is going on between my tcpip stack and the inet, I cannot be sure where the problem lies. Again, if you hunt around the net in relation to "max half open connections" you can find more info about this. I am not sure where shorewall would fit into this picture, any comments shorewall people? Alex Martin http://www.rettc.com ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642