Hi all :)
I''m new to this list, as I''m new too to traffic shaping ;)
I''ve
set up an FTP server in my ADSL line and I wanted it to serve as fast
as possible as long as I don''t use my outgoing ADSL bandwidth, and
I''m currently using HTB for that (succesfully, I must add).
The problem is (when the FTP server is serving higher than its
"rate" and near to its "ceil") that protocols like SMTP or
POP-3,
when I use them as client, slow to a crawl because being short-burst
in nature never use the speed I have configured for them :(( I don''t
know if I''m missing something, but other protocols seems to work OK.
For example, when I browse the web, pages start to download slowly,
DNS queries are very slow but once the pages start dowloading, speed
is pretty good.
My setup is for a few PC connected to an ADSL router using
Ethernet cards, and I''m only shaping outgoing traffic in one box, the
one serving FTP. The other boxes are more or less inactive. My
shaping has an HTB discipline at root:
tc qdisc add dev eth0 root handle 1: htb default 110
I''ve played with r2q, because it assigns too big a quantum to
some classes and too small a quantum to others, until I noticed that
using the default r2q assigned a very big quantum to classes that I
want big quantums for and very small quantum to classes I want very
small quantums for.
After that I add a base class to be able to borrow bandwith,
althoug I''m not sure now if that''s a good idea:
tc class add dev eth0 parent 1: classid 1:1 htb rate 100Mbit ceil 100Mbit
This class has the speed and ceil of my Ethernet card (100Mbit).
Now I add two major classes, one for general LAN traffic and
other for Ethernet traffic to the ADSL router:
# Hi speed class
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 95Mbit ceil 95Mbit
tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
# Low speed class (ADSL)
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 256kbit ceil 256kbit
As you can see, bandwidth cannot be borrowed, so I think I could
get rid of 1:1. Anyway, this shouldn''t make any difference for
bursted protocols that now are slow. In the hispeed class I use 95 as
rate and ceil because being averaged values, I prefer to slow down
LAN traffic and ensure I always have a bit of unused Ethernet slots
so the low speed (ADSL) class doesn''t have to wait. Should I specify
(100M-256k)bit for rate and ceil here?
The low speed class gets 256kbit although my ADSL is capable of
upload at 300kbit. I give a bit for the other box that may use the
ADSL and runs an operating system without shaping ;) I want to make
sure that box has at least 50kbit more or less, no matter if I''m
serving FTP in my box.
For general LAN traffic I''ve chosen an SFQ queue discipline since
I sometimes use many protocols at a time and adding a bit of fairness
is desirable, although a pfifo_fast will probably work here, too.
Last, I add two classes below the ADSL class, one to shape FTP
traffic, the other class to shape the rest of traffic.
# Other ADSL traffic
tc class add dev eth0 parent 1:11 classid 1:110 htb rate 192kbit ceil 256kbit
prio 0
tc qdisc add dev eth0 parent 1:110 handle 110: sfq perturb 10
# To filter FTP traffic
tc class add dev eth0 parent 1:11 classid 1:111 htb rate 64kbit ceil 256kbit
prio 1
Here are the most important classes. They share bandwidth,
because I want the FTP server to borrow traffic if the ADSL is
otherwise unused. The FTP server gots 8Kbps and the rest of the
traffic 24kpbs.
Finally, the filters:
tc filter add dev eth0 prio 1 protocol ip parent 1:0\
u32 match ip dst 192.168.0.0/24 flowid 1:10
tc filter add dev eth0 prio 2 protocol ip parent 1:0\
u32 match ip sport 0x3000 0x3000 flowid 1:111
tc filter add dev eth0 prio 2 protocol ip parent 1:0\
u32 match ip sport 0x4000 0x4000 flowid 1:111
tc filter add dev eth0 prio 2 protocol ip parent 1:0\
u32 match ip sport 0x20 0xff flowid 1:111
I''m serving passive FTP only in ports from 0x3000 to 0x4fff, and
active FTP in port 20. The rest of LAN traffic (including FTP) is
sent to the hispeed class.
As you can see, I give more priority to general ADSL traffic, and
I''m sure such traffic is NEVER 24kbps, always much less than that,
but if someone is using my FTP server at, let''s say, 15kbps
(borrowing bandwidth and still with a backlog of 15-30 packets, I can
tell from the "tc -d -s" stats), if I try to download 300kB of email
using POP-3, the client slows to a crawl. If I browse many small
pages, the speed is slow. At the same time, browsing large pages or
downloading gcc-4.0.2 ;) gives a speed more or less similar to that I
would have without shaping and without the FTP server (well, a bit
less because the "burst effect" affects this transmissions too and
the slower pace in the ack packets makes FTP download a bit slower).
I''ve played with burst and cburst values, and increasing burst in
class 1:110 (ADSL general traffic) and its parents to 16kb helps a
bit, as helps decreasing the burst in 1:111 (FTP traffic thru ADSL),
but I still get high latency, slow downloads, etc. If I stop
borrowing bandwidth things go a bit better, but I want to share
bandwidth.
Am I doing anything wrong (probably a lot of things...)? Is slow
latency and slow "bursted protocols" a price to pay to have bandwidth
sharing and shaping? Is there any value I can tweak to make general
ADSL traffic more responsive? As you can see from the commands above,
I''ve assigned a higher priority to it to slow down latency, but...
Thanks a lot in advance. I''ve read HTB documentation and, while
it is good and informative, it''s not enough (for me) to know how to
solve this problem. I don''t want to use any other qdisc because I
need a classful one and I don''t fully understand CBQ (I understand
HTB MUCH better). Thanks again and pleased to meet you :)
Raúl Núñez de Arenas Coronado
--
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It''s my PC and I''ll cry if I want to...
DervishD wrote:> tc filter add dev eth0 ... ip sport 0x3000 0x3000 flowid 1:111 > tc filter add dev eth0 ... ip sport 0x4000 0x4000 flowid 1:111 > tc filter add dev eth0 ... ip sport 0x20 0xff flowid 1:111 > > I''m serving passive FTP only in ports from 0x3000 to 0x4fff, and > active FTP in port 20.Then you should use the following port numbers in your filters: 0x3000 0xf000 0x4000 0xf000 20 0xffff The first two of your filters were matching more ports than needed, while the latter WAS NOT MATCHING YOUR ACTIVE FTP TRAFFIC AT ALL. I suggest you read a tutorial on ip addresses and netmasks, that should cover the basis of how bitmasks work.> Is there any value I can tweak to make general ADSL traffic more > responsive?Yes, you can make another HTB class, let''s call it 1:112, for ICMP traffic (ie. ping, port unreachable...) and very small TCP packets (SYN, ACK, RST... all that stuff) and give it the highest priority. That''s a good place to put interactive SSH traffic too, if you use it: #low-latency class #remember to give sibling classes different priorities, >0 tc class add dev eth0 parent 1:11 classid 1:112 \ htb rate 1kbit ceil 256kbit prio 0 #small TCP packets, <64bytes tc filter add dev eth0 prio 2 protocol ip parent 1:0 u32 \ match ip protocol 6 0xff \ match u8 0x05 0x0f at 0 \ match u16 0x0000 0xffc0 at 2 \ flowid 1:112 #ICMP tc filter add dev eth0 prio 2 protocol ip parent 1:0 u32 \ match ip protocol 1 0xff \ flowid 1:112 #interactive SSH traffic (NOT including scp, x11 tunnels...) tc filter add dev eth0 prio 2 protocol ip parent 1:0 u32 \ match ip dport 22 0xffff \ match ip tos 0x10 0xff \ flowid 1:112 By the way, I didn''t invent all this, it''s by Bert Hubert. You should check his wondershaper script: http://lartc.org/wondershaper/ Toby -- UNIX is a lever for the intellect. -John R. Mashey
Hi Toby, and thanks for your answer :)
Excuse me for the long reply, but I wanted to put my current
settings for tc just in case. Feel free to ignore.
* Toby <tobia.conforto@linux.it> dixit:> DervishD wrote:
> > tc filter add dev eth0 ... ip sport 0x3000 0x3000 flowid 1:111
> > tc filter add dev eth0 ... ip sport 0x4000 0x4000 flowid 1:111
> > tc filter add dev eth0 ... ip sport 0x20 0xff flowid 1:111
> >
> > I''m serving passive FTP only in ports from 0x3000 to 0x4fff,
and
> > active FTP in port 20.
>
> Then you should use the following port numbers in your filters:
>
> 0x3000 0xf000
> 0x4000 0xf000
> 20 0xffff
>
> The first two of your filters were matching more ports than needed,
> while the latter WAS NOT MATCHING YOUR ACTIVE FTP TRAFFIC AT ALL.
>
> I suggest you read a tutorial on ip addresses and netmasks, that
> should cover the basis of how bitmasks work.
I know how they work, but sometimes my brain doesn''t work
correctly ;)))) The first two are a typo, in my tc setup I have masks
0xf000 and 0xf000, I don''t know why I made such mistake, because I
swear I cut''n''pasted it :??? and the third one is an error,
caused
because I was testing with ports 0x??20 to differentiate connections
(to test settings for different FTP servers) with a hand made client
that used different ports for active connections. I simply didn''t put
the mask back to 0xffff and worst, I didn''t move to 20 *decimal* and
left the 0x. I chose 0x??20 because it was easier to remeber and
fancier to read O:) than 0x??14.
Thanks for advising, because right now I don''t have active ftp
traffic and I would NEVER have spotted the errors. Thanks a lot,
really.
And I don''t understand the typo :? I''ve seen the output of
"tc
filter show dev eth0" and shows "match 30000000/f0000000 at 20".
> > Is there any value I can tweak to make general ADSL traffic more
> > responsive?
>
> Yes, you can make another HTB class, let''s call it 1:112, for ICMP
> traffic (ie. ping, port unreachable...) and very small TCP packets
> (SYN, ACK, RST... all that stuff) and give it the highest priority.
But all that traffic goes already through a higher priority
class. The "general ADSL traffic" has a higher priority (prio 0) and
"ADSL outgoing FTP traffic" has prio 1 :??? Sorry but that value
doesn''t show in what I posted, certainly I had a problem when cutting
and pasting... I had to modify what I cut because I took it from a
zsh script. Here is the real contents:
TCQA="tc qdisc add dev eth0 parent"
TCCA="tc class add dev eth0 parent"
TCFA="tc filter add dev eth0 protocol ip parent"
action "Starting traffic control on eth0"
# root qdisc, defaults to ADSL other traffic
$=TCQA root handle 1: htb default 21 r2q 1
# hispeed class (Ethernet)
$=TCCA 1: classid 1:1 htb rate 95Mbit ceil 95Mbit
$=TCQA 1:1 handle 10: sfq perturb 10
# lospeed class (ADSL)
$=TCCA 1: classid 1:2 htb rate 256kbit ceil 256kbit burst 16384 cburst 8192
# Other ADSL traffic
$=TCCA 1:2 classid 1:21 htb rate 224kbit ceil 256kbit prio 0 burst 16384
cburst 8192
$=TCQA 1:21 handle 21: sfq perturb 10
# FTP thru ADSL traffic
$=TCCA 1:2 classid 1:22 htb rate 64kbit ceil 160kbit prio 1
# Filters
$=TCFA 1:0 prio 1 u32 match ip dst 192.168.0.0/24 flowid 1:1
$=TCFA 1:0 prio 2 u32 match ip sport 0x3000 0xf000 flowid 1:22
$=TCFA 1:0 prio 2 u32 match ip sport 0x4000 0xf000 flowid 1:22
$=TCFA 1:0 prio 2 u32 match ip sport 20 0xffff flowid 1:22
I removed yesterday the old 1:1 class because I want no borrowing
between Ethernet general traffic and Ethernet ADSL traffic.
If I add this as you suggest (modifiying identifiers)
$=TCCA 1:2 classid 1:23 htb rate 1kbit ceil 256kbit prio 0
then it will have the same priority that general traffic. I don''t
undertand why it should improve responsiveness :? I''m going to test,
or course :), but I don''t understand...
> By the way, I didn''t invent all this, it''s by Bert
Hubert.
> You should check his wondershaper script: http://lartc.org/wondershaper/
I did, but if I don''t do any "borrowing" (as wondershaper
seems
to do), latency is low. I want low latency when borrowing. With the
setup I''ve posted above (that is, reducing ftp ceil to 160kbit and
raising adsl-general rate to 250kbit) there''s almost no latency, but
I would like to add a bit more of ceil to ftp traffic.
I''ll test your suggestions, which I find quite interesting, and
if I have success, I''ll tell :) Thanks for your invaluable help, but
if this works I''m afraid I won''t understand why, because by
default
all that traffic that will be matched by the new filters will go to
the fast-adsl class anyway :?
Would it be because it will go out of the queue *even before*
than general ADSL traffic? I think that''s the reason, right?
Well, I''ve finally tested your suggestion, and I''ve
noticed only
a marginal improvement in responsiveness and latency, around 10% more
or less. Anytime I increase the "ceil" of the FTP-thru-ADSL traffic
above 160kbit (that''s the value I''m currently using), the
latency
skyrockets and the download speed stinks. I no longer see any
improvement, then. With values near 160kbit, messing with "burst" and
"cburst" give as slight improvement and your suggestions give a 10%
more of download bandwidth and about the same improveness in latency
(please note that except the download bandwidth, the rest of
improvements are subjetive, they represent what I feel when using my
network).
This means that without the new class and filters I''m stuck with
160kbit max for the served FTP traffic, and with them I can raise
that limit a bit. Probably I''m making mistakes in my r2q value, the
burst and cburst values, etc. that have to be corrected, too O:)
Thanks again, Toby :) With your suggestion I''ve raised from
19kbps to almost 22kbps the FTP speed (that''s an extra 11MB per hour
and almost 300MB a day, and that''s GOOD!), while having a decent
latency.
BTW, this is what I have now:
TCQA="tc qdisc add dev eth0 parent"
TCCA="tc class add dev eth0 parent"
TCFA="tc filter add dev eth0 protocol ip parent"
action "Starting traffic control on eth0"
# root qdisc, defaults to ADSL other traffic
$=TCQA root handle 1: htb default 21 r2q 1
# hispeed class (Ethernet)
$=TCCA 1: classid 1:1 htb rate 95Mbit ceil 95Mbit
$=TCQA 1:1 handle 10: sfq perturb 10
# lospeed class (ADSL)
$=TCCA 1: classid 1:2 htb rate 256kbit ceil 256kbit burst 16384 cburst 8192
# Low latency ADSL traffic
$=TCCA 1:2 classid 1:20 htb rate 4kbit ceil 256kbit prio 0 burst 16384
cburst 8194
# Other ADSL traffic
$=TCCA 1:2 classid 1:21 htb rate 224kbit ceil 256kbit prio 1 burst 16384
cburst 8192
$=TCQA 1:21 handle 21: sfq perturb 10
# FTP thru ADSL traffic
$=TCCA 1:2 classid 1:22 htb rate 64kbit ceil 175kbit prio 2
# Filters
# Ethernet LAN traffic
$=TCFA 1:0 prio 1 u32 match ip dst 192.168.0.0/24 flowid 1:1
# Low latency traffic small TCP packets (<64 bytes)
$=TCFA 1:0 prio 2 u32 match ip protocol 6 0xff\
match u8 0x05 0x0f at 0\
match u16 0x0000 0xffc0 at 2\
flowid 1:20
# Low latency traffic, ICMP
$=TCFA 1:0 prio 2 u32 match ip protocol 1 0xff flowid 1:20
# FTP thru ADSL traffic
$=TCFA 1:0 prio 2 u32 match ip sport 0x3000 0xf000 flowid 1:22
$=TCFA 1:0 prio 2 u32 match ip sport 0x4000 0xf000 flowid 1:22
$=TCFA 1:0 prio 2 u32 match ip sport 20 0xffff flowid 1:22
Raúl Núñez de Arenas Coronado
--
Linux Registered User 88736 | http://www.dervishd.net
http://www.pleyades.net & http://www.gotesdelluna.net
It''s my PC and I''ll cry if I want to...
DervishD wrote:> If I add this as you suggest then it will have the same priority that > general traffic.It doesn''t if you lower general traffic to prio 1 and ftp traffic to prio 2, as I''m sure I said somewhere in my other email.> Would it be because it will go out of the queue *even before* than > general ADSL traffic? I think that''s the reason, right?Yes, that''s the idea. Now the only thing left to do is to lower the total rate and ceil to 16kbit below the ADSL upload cap (again, read the wondershaper documentation to understand why.) Just substitute 240kbit for 256kbit throughout the script and adjust the other rates accordingly: ... classid 1:2 htb rate 240kbit ceil 240kbit ... classid 1:20 htb rate 4kbit ceil 240kbit prio 0 ... classid 1:21 htb rate 172kbit ceil 240kbit prio 1 ... classid 1:22 htb rate 64kbit ceil 175kbit prio 2 By the way, I wouldn''t use the burst option. Toby -- UNIX is a lever for the intellect. -John R. Mashey
Hi Toby :) * Toby <tobia.conforto@linux.it> dixit:> DervishD wrote: > > If I add this as you suggest then it will have the same priority that > > general traffic. > It doesn''t if you lower general traffic to prio 1 and ftp traffic to > prio 2, as I''m sure I said somewhere in my other email.Which I''ve done ;)) You said that in the comments of the "tc" commands (you say that the priorities of the sibling classes should be lower).> > Would it be because it will go out of the queue *even before* than > > general ADSL traffic? I think that''s the reason, right? > > Yes, that''s the idea.OK, I understand now. Thanks a lot :)))> Now the only thing left to do is to lower the total rate and ceil > to 16kbit below the ADSL upload cap (again, read the wondershaper > documentation to understand why.)That''s something I learnt even when I didn''t know about wondershaper ;) My ADSL has 300kbit of upload rate, that''s why I''m using 256 and not 300. I want to leave those 44kbit for the other box. BTW, I didn''t notice that the other box was up and running (running WinDOS) is uploading traffic at almost 10kbps so it is probably filling the ADSL router buffers :( That may be the reason why I''m getting poor latency when the ADSL FTP traffic goes above 175 (in fact, I bet the traffic limit is 300kbit-otherboxkbit). Unfortunately I cannot shutdown that box by now, and probably I will need to do my shaping taking into account an upload rate from the other machine of about 12kbps or something like that. 15kbps will be better... That leaves a value nearer to 180kbit instead of 256... Well, I''ll try to make the other box upload at a lower rate if I can.> By the way, I wouldn''t use the burst option.But it gives me some improvement, why should I get rid of it? I''m not sure about why it does, but using a burst of 8192-20480 and a cburst of about the half seems to improve latency a bit. It''s just an illusion? For me it would be great if I could get rid of the burst and cburst settings, because my tc script will be easier to maintain. Thanks a lot for your help, now my tc works much better :)) Raúl Núñez de Arenas Coronado -- Linux Registered User 88736 | http://www.dervishd.net http://www.pleyades.net & http://www.gotesdelluna.net It''s my PC and I''ll cry if I want to...