LARTC - Jun 2007 - Why does scp stall on low bandwidth connections?

Hi,

I am new to tc and have been reading quite a bit on how to set it up etc.
Everything seems to be working fine, until I started scp-ing a large file
over a low bandwidth connection as part of my testing process.

Here is the setup:
my pc --- bridge running tc/htb --- rest of network

TC is filtering traffic from "my pc" and classifies it as 120kbit (see
my
script below). I then scp a 5MB file from a server in "rest of
network" to
"my pc". Everything seems to work fine and copies at a speed of around
12KB/s, which is what I would expect from a 120kbit connection. At some
stage scp stalls and eventually disconnects or I get bored and press
<Ctrl>+C. The stage at which it stalls is different every time. First it
was
at 76% of the copy progress, then at 32% of the copy progress.

For my testing purposes, there is no other traffic flowing through either
this class or any other class. My expectation was that it would copy the
entire file, just at a low speed. I expected to be able to copy a 600MB file
at 12KB/s, which would of course be very slow, but eventually arrive.

Here are the rules I specified, note that "my pc" does *not* have the
ip
address 10.0.2.42 in the test desribed above:

#eth0 qdisc
tc qdisc add dev eth0 root handle 1:0 htb default 2
tc class add dev eth0 parent 1:0 classid 1:1 htb rate 10mbit ceil 10mbit
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 120kbit ceil 120kbit
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 200kbit ceil 1mbit

#eth1 qdisc
tc qdisc add dev eth1 root handle 2:0 htb default 2
tc class add dev eth1 parent 2:1 classid 2:2 htb rate 120kbit ceil 120kbit
tc class add dev eth1 parent 2:1 classid 2:3 htb rate 200kbit ceil 1mbit

#eth0 filter
tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src
10.0.2.42 flowid 1:3

#eth1 filter
tc filter add dev eth1 parent 2:0 protocol ip prio 1 u32 match ip dst
10.0.2.42 flowid 2:3

Thank you for your comments on this situation.

Regards, /|/|arc.


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Marc wrote:
> Hi,
> 
> I am new to tc and have been reading quite a bit on how to set it up etc.
> Everything seems to be working fine, until I started scp-ing a large file
> over a low bandwidth connection as part of my testing process.
> 
> Here is the setup:
> my pc --- bridge running tc/htb --- rest of network
> 
> TC is filtering traffic from "my pc" and classifies it as 120kbit
(see my
> script below). I then scp a 5MB file from a server in "rest of
network" to
> "my pc". Everything seems to work fine and copies at a speed of
around
> 12KB/s, which is what I would expect from a 120kbit connection. At some
> stage scp stalls and eventually disconnects or I get bored and press
> <Ctrl>+C. The stage at which it stalls is different every time. First
it
> was
> at 76% of the copy progress, then at 32% of the copy progress.
> 
> For my testing purposes, there is no other traffic flowing through either
> this class or any other class. My expectation was that it would copy the
> entire file, just at a low speed. I expected to be able to copy a 600MB 
> file
> at 12KB/s, which would of course be very slow, but eventually arrive.
> 
> Here are the rules I specified, note that "my pc" does *not* have
the ip
> address 10.0.2.42 in the test desribed above:
> 
> #eth0 qdisc
> tc qdisc add dev eth0 root handle 1:0 htb default 2
> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 10mbit ceil 10mbit
> tc class add dev eth0 parent 1:1 classid 1:2 htb rate 120kbit ceil 120kbit
> tc class add dev eth0 parent 1:1 classid 1:3 htb rate 200kbit ceil 1mbit
> 
> #eth1 qdisc
> tc qdisc add dev eth1 root handle 2:0 htb default 2
> tc class add dev eth1 parent 2:1 classid 2:2 htb rate 120kbit ceil 120kbit
> tc class add dev eth1 parent 2:1 classid 2:3 htb rate 200kbit ceil 1mbit
> 
> #eth0 filter
> tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src
> 10.0.2.42 flowid 1:3
> 
> #eth1 filter
> tc filter add dev eth1 parent 2:0 protocol ip prio 1 u32 match ip dst
> 10.0.2.42 flowid 2:3
> 
> Thank you for your comments on this situation.
It''s probably because arp is being sent to 1:2 which is backlogged. Try
not using the default parameter and instead use a catch all ip tc filter
like -

tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 match u32 0 0 
flowid 1:2

You could also consider adding p/bfifos to the classes and use the limit 
parameter to make the queues shorter. At low bitrates the default 
1000pkts (picked up from the queuelen on eth) is too long.

Andy.

Hello Andy,
Is that line:
tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 match u32 0 0
flowid 1:2

not equal to:
tc qdisc add dev eth0 root handle 1:0 htb default 2

in terms of achieved results? If not, what is the difference?

Thanks,
-Nikolay

Andy Furniss wrote:
> Marc wrote:
>> Hi,
>>
>> I am new to tc and have been reading quite a bit on how to set it up
etc.
>> Everything seems to be working fine, until I started scp-ing a large
file
>> over a low bandwidth connection as part of my testing process.
>>
>> Here is the setup:
>> my pc --- bridge running tc/htb --- rest of network
>>
>> TC is filtering traffic from "my pc" and classifies it as
120kbit (see my
>> script below). I then scp a 5MB file from a server in "rest of
>> network" to
>> "my pc". Everything seems to work fine and copies at a speed
of around
>> 12KB/s, which is what I would expect from a 120kbit connection. At some
>> stage scp stalls and eventually disconnects or I get bored and press
>> <Ctrl>+C. The stage at which it stalls is different every time.
First
>> it was
>> at 76% of the copy progress, then at 32% of the copy progress.
>>
>> For my testing purposes, there is no other traffic flowing through
either
>> this class or any other class. My expectation was that it would copy
the
>> entire file, just at a low speed. I expected to be able to copy a
>> 600MB file
>> at 12KB/s, which would of course be very slow, but eventually arrive.
>>
>> Here are the rules I specified, note that "my pc" does *not*
have the ip
>> address 10.0.2.42 in the test desribed above:
>>
>> #eth0 qdisc
>> tc qdisc add dev eth0 root handle 1:0 htb default 2
>> tc class add dev eth0 parent 1:0 classid 1:1 htb rate 10mbit ceil
10mbit
>> tc class add dev eth0 parent 1:1 classid 1:2 htb rate 120kbit ceil
>> 120kbit
>> tc class add dev eth0 parent 1:1 classid 1:3 htb rate 200kbit ceil
1mbit
>>
>> #eth1 qdisc
>> tc qdisc add dev eth1 root handle 2:0 htb default 2
>> tc class add dev eth1 parent 2:1 classid 2:2 htb rate 120kbit ceil
>> 120kbit
>> tc class add dev eth1 parent 2:1 classid 2:3 htb rate 200kbit ceil
1mbit
>>
>> #eth0 filter
>> tc filter add dev eth0 parent 1:0 protocol ip prio 1 u32 match ip src
>> 10.0.2.42 flowid 1:3
>>
>> #eth1 filter
>> tc filter add dev eth1 parent 2:0 protocol ip prio 1 u32 match ip dst
>> 10.0.2.42 flowid 2:3
>>
>> Thank you for your comments on this situation.
> 
> It''s probably because arp is being sent to 1:2 which is
backlogged. Try
> not using the default parameter and instead use a catch all ip tc filter
> like -
> 
> tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 match u32 0 0
> flowid 1:2
> 
> You could also consider adding p/bfifos to the classes and use the limit
> parameter to make the queues shorter. At low bitrates the default
> 1000pkts (picked up from the queuelen on eth) is too long.
> 
> Andy.
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

The first one only recognize IP traffic, the line with default will  
match any kind of traffic.

Regards,
Andreas

Quoting Nikolay Kichukov <hijacker@oldum.net>:
> Hello Andy,
> Is that line:
> tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 match u32 0 0
> flowid 1:2
>
> not equal to:
> tc qdisc add dev eth0 root handle 1:0 htb default 2
>
> in terms of achieved results? If not, what is the difference?
>
> Thanks,
> -Nikolay

Hello Andreas,
and arp is not ip ... thanks for clarification.

Where(in which class) would all non-ip traffic go in the filter scenario?

Thanks,
-Nikolay

Andreas Unterkircher wrote:
> The first one only recognize IP traffic, the line with default will
> match any kind of traffic.
> 
> Regards,
> Andreas
> 
> Quoting Nikolay Kichukov <hijacker@oldum.net>:
> 
>> Hello Andy,
>> Is that line:
>> tc filter add dev eth0 parent 1:0 protocol ip prio 2 u32 match u32 0 0
>> flowid 1:2
>>
>> not equal to:
>> tc qdisc add dev eth0 root handle 1:0 htb default 2
>>
>> in terms of achieved results? If not, what is the difference?
>>
>> Thanks,
>> -Nikolay
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Nikolay Kichukov wrote:
> Hello Andreas,
> and arp is not ip ... thanks for clarification.
> 
> Where(in which class) would all non-ip traffic go in the filter scenario?
In the case of htb unclassified go unshaped without a default class set 
(=default 0) you do get a counter -

andy@noki:~$ /sbin/tc -s qdisc ls dev eth3
qdisc htb 1: r2q 10 default 0 direct_packets_stat 3223

In the case of HFSC unclassified get dropped - so you really need a 
default class, but not one that gets low prio IP sent to it :-)

Andy.

Hello Andy,
unshaped here means with higher priority than the rest of the classes
that have filters attached to them?

So if an arp packet is sent at the same time an ip packet is sent, the
arp packet will go first? And only then the ip packet will be matched by
the filters?

Regards,
-Nikolay

Andy Furniss wrote:
> Nikolay Kichukov wrote:
>> Hello Andreas,
>> and arp is not ip ... thanks for clarification.
>>
>> Where(in which class) would all non-ip traffic go in the filter
scenario?
> 
> In the case of htb unclassified go unshaped without a default class set
> (=default 0) you do get a counter -
> 
> andy@noki:~$ /sbin/tc -s qdisc ls dev eth3
> qdisc htb 1: r2q 10 default 0 direct_packets_stat 3223
> 
> In the case of HFSC unclassified get dropped - so you really need a
> default class, but not one that gets low prio IP sent to it :-)
> 
> Andy.

Nikolay Kichukov wrote:
> Hello Andy,
> unshaped here means with higher priority than the rest of the classes
> that have filters attached to them?
Yes it will just be passed and not be accounted for by htb (well apart 
from the counter)
> 
> So if an arp packet is sent at the same time an ip packet is sent, the
> arp packet will go first? And only then the ip packet will be matched by
> the filters?
I don''t know if two packets can arrive at the same time. The arp will 
still pass through the filters and fail to match any then just pass 
through. The ip packet may or may not pass straight through depending on 
the state of the class it gets filtered into.

Andy.

Hello Andy,
Thanks for the explanation one more time;-)

Cheers,
-Nikolay

Andy Furniss wrote:
> Nikolay Kichukov wrote:
>> Hello Andy,
>> unshaped here means with higher priority than the rest of the classes
>> that have filters attached to them?
> 
> Yes it will just be passed and not be accounted for by htb (well apart
> from the counter)
> 
>>
>> So if an arp packet is sent at the same time an ip packet is sent, the
>> arp packet will go first? And only then the ip packet will be matched
by
>> the filters?
> 
> I don''t know if two packets can arrive at the same time. The arp
will
> still pass through the filters and fail to match any then just pass
> through. The ip packet may or may not pass straight through depending on
> the state of the class it gets filtered into.
> 
> Andy.