LARTC - Mar 2004 - New IMQ device implementation supporting device EOS

Hello All

Its first time i posting to this mail list :). I have done something 
(maybe) important. I write new IMQ device driver based from Martin 
Devera and Patrick McHardy implementation with device EOS support. My 
current implementation supporting only egress trafic shaping and kernel 
2.4.25. For more details, source and examples have look at my page 
http://hyperfighter.jinak.cz/qos

Ill apreticate your feedback :)

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello Roy

Roy wrote:

 >Hello,
 >imq became realy popular now ;)
 > 
 >
Sure, same as whole GNU/Linux  :)

 >It would be good ir you wrote what is that EOS to save time for some 
people
 >on searches.
 >
 > 
 >
EOS - End of Send. Its time betwen two dequeues and packet transmits
used by qdisc for calculation. Any of current IMQ implementations do not
use it, so the calculation by qdisc may not be correct, whish causing
filling physical device fifos without control. In devices with variable
speed (WiFi, Modems) it causes overload because the physisal trafics is
out of control. It happen only on busy boxes, like my routers where is
about 70 ppl at once. In a simply test enviroment this issue will never
happen.

 >and it would be good if you posted source code directly (not inside of 
diff
 >file)
 >for review purposes
 >
 > 
 >
The whole patch contain only few new files, all other are patches of the
original kernel files. BTW Midnight Comander 4.6.0 contain very nice
diff virtual filesystem, so you can browse .diff files like a tar archive.

 >also you can look at my imq version, because it is much easer to develop
 >since is independent from kernel.
 >and does not need to patch it.
 >http://pupa.da.ru/imq/
 >
 > 
 >
I have play with with your version :) Having independet implementation
is nice idea. I play with that idea too but it was more dificult to have
stable solution in short time i need and with EOS support its quite
imposible ...

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Roy wrote:

 >this is qute intersting, and it happens on every computer not only on 
bussy
 >ones
 >but in reality I dont see any need for that, because tcp itself should 
take
 >care of this issue.
 >
Right but you lost TCP interactivity and overloaded WiFi interface is 
painfull itself  !!!

 >anyway this may solve some problems with crashes, because I suspect imq is
 >unstable because of this problem.
 >
Stability if any current implementations is issue by design. Simply 
something betwen netfilter and TCP/IP stack. I tried found it but my 
suceess was only more crashes.

 >I looked at yor code and do not understand what it does, do you 
implemented
 >that EOS for imq interface ifself?
 >this is not logical, since imq is not interface, it is placeholder to 
attach
 >shapers.
 >Or you made that your imq takes packets from kernel ant transmits them by
 >itself without returning to netfilter
 >with  dev_queue_xmit(skb)
 >this way you are very limited in what you are able to do because you can
 >hook only on the last hook.
 >
There is skb->imq_flags whis getting marked by netfilter to 
IMQ_F_ENQUEUE (part of flag is target IMQ network interface)
At "end of life in kernel" every packet coming into dev_queue_xmit. 
There also come our marked skb. When you have look at begin of the 
modified dev_queue_xmit
there is imq_flags decoder. When decoder hit our marked SKB its enqueued 
in to IMQ queue. Then we run qdisc dequeue (qdisc_run).
qdisc_run  function is a loop tat run qdisc restart until is queue 
empty, need to be schedulet (netif_schedule) or until is not set 
netif_queue_stopped atomic bit (bingo the trick is here). After dequeue 
in qdisc restart our skb passing hard_start_xmit function in IMQ driver. 
After some checkings is there stopped queue of the imq device 
(netif_queue_stopped), changet skb->imq_flags to IMQ_F_ENQUEUED and 
passed back to dev_queue_xmit. Now it will hit our real device, and at 
end of the qdisc_restart (after hard_start_xmit) is function that decode 
the skb->imq_flags and if is necesary wake_up the queue of our IMQ 
device (so we have perfect synchronization of the EOS). There is some 
protection to lock up the queue in imq_dev_xmit (imq.c). Also there are 
counted throtled skb by collision counter. I also use device watchdog to 
solve problem if skb stuck somewhere in  physical device queue and 
netif_wake_queue is not called in right time( its only protection "for 
sure" because this should happen and if this happen without protection 
behind this, whole imq should lockup and kill the trafics forewer)

 >seems you  may be unable shape ingress then. what basicaly voids all imq
 >purpose.
 >
Iam not sure. Every packet comin in in to router must come out !!! So we 
need only shape out trafics. Incoming trafics we cannot shape, only 
police. We cannot change speed and rate of incoming packets !!! have 
look at example includet in my IMQ. There i use the corret way to shape 
incoming and outgoung traffic. BTW: I have writen a TODO ....

 >unfortunately you made your driver even more invasive than old one,
 >it need to patch almost all kernel network system.
 >
Yes its also angle of view :)  I was need to solve problem and i solve 
it. Everithing there is clear. IMHO Cannot be solved by more clear 
solution. ;)


Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Roy wrote:

 >> >seems you  may be unable shape ingress then. what basicaly voids 
all imq
 >> >purpose.
 >> >
 >>Iam not sure. Every packet comin in in to router must come out !!! So
we
 >>need only shape out trafics. Incoming trafics we cannot shape, only
 >>police. We cannot change speed and rate of incoming packets !!! have
 >>look at example includet in my IMQ. There i use the corret way to shape
 >>incoming and outgoung traffic. BTW: I have writen a TODO ....
 >
 >
 >Here you are completely wrong what makes you think that you cant control
 >incoming trafic?
 >some wrong publication on internet?
 >Almost everybody is doing this and you say not possible, doesnt your isp
 >control how many packets you receive?

About what is trafics control ? And how can i control how much packet 
someone send for me ???  I can control how much can i send. Anything 
else is waste of time. EXCLUDE of application server where i want 
control traffic coming in to a specified port, eg sendmail etc. But 
combining router+traffic controler+application server in to one box is 
good idea ? No. Because of this its lower priority task for me. A quote 
from TODO:
* support for ingress queue ;)

 >as I know main imq purpose was to control ingress.
Really ? Iam not sure. Have look in to first original IMQ device driver 
writen by Martin Devera:
http://luxik.cdi.cz/~devik/qos/imq.htm

 >Ok, enough about this,
 >I would like to know how stable is your driver, if it shapes localy
 >generated trafic,
 >since it was the main problem for imq all time.
 >if it became stable I will implement your idea in my imq version too.

Its quite stable. I using it now on two busy routers in my production 
enviroment (avreage 20 NICs per router, average 70 users, 280 htb 
classes). Uptime is until kernel upgrade restart (week  ago). localy 
generated trafic is only DNS and ssh for managment and statistics 
collection (rrd graphs from Stef Coene, www.docum.org, but the data 
collections is done by ssh)  Setup scripts i usink like my example but 
with clasification of the intranet trafics (for this that much htb classes).

Thanks for the fedback.

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Jiri Fojtasek wrote:
> Hello Roy
> 
> Roy wrote:
> 
>  >Hello,
>  >imq became realy popular now ;)
>  > >
> Sure, same as whole GNU/Linux  :)
> 
>  >It would be good ir you wrote what is that EOS to save time for some 
> people
>  >on searches.
>  >
>  > >
> EOS - End of Send. Its time betwen two dequeues and packet transmits
> used by qdisc for calculation.
Is that a kernel transmit or a real "on the wire" transmit - ie can
you
keep the devices'' buffer empty or just detect when it''s full?

  Any of current IMQ implementations do not
> use it, so the calculation by qdisc may not be correct, whish causing
> filling physical device fifos without control. In devices with variable
> speed (WiFi, Modems) it causes overload because the physisal trafics is
> out of control. It happen only on busy boxes, like my routers where is
> about 70 ppl at once. In a simply test enviroment this issue will never
> happen.
> 
>  >and it would be good if you posted source code directly (not inside of
> diff
>  >file)
>  >for review purposes
>  >
>  > >
> The whole patch contain only few new files, all other are patches of the
> original kernel files. BTW Midnight Comander 4.6.0 contain very nice
> diff virtual filesystem, so you can browse .diff files like a tar archive.
> 
>  >also you can look at my imq version, because it is much easer to
develop
>  >since is independent from kernel.
>  >and does not need to patch it.
>  >http://pupa.da.ru/imq/
>  >
>  > >
> I have play with with your version :) Having independet implementation
> is nice idea. I play with that idea too but it was more dificult to have
> stable solution in short time i need and with EOS support its quite
> imposible ...
> 
> Jiri
> 
> Zkontrolovane antivirusom ClamAv
> Scanned by ClamAv - http://www.clamav.net
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Jiri Fojtasek wrote:
> Roy wrote:
> 
>  >> >seems you  may be unable shape ingress then. what basicaly
voids
> all imq
>  >> >purpose.
>  >> >
>  >>Iam not sure. Every packet comin in in to router must come out !!!
So we
>  >>need only shape out trafics. Incoming trafics we cannot shape,
only
>  >>police. We cannot change speed and rate of incoming packets !!!
have
>  >>look at example includet in my IMQ. There i use the corret way to
shape
>  >>incoming and outgoung traffic. BTW: I have writen a TODO ....
>  >
>  >
>  >Here you are completely wrong what makes you think that you cant
control
>  >incoming trafic?
>  >some wrong publication on internet?
>  >Almost everybody is doing this and you say not possible, doesnt your
isp
>  >control how many packets you receive?
> 
> About what is trafics control ? And how can i control how much packet 
> someone send for me ??? 
You can sort of control TCP from the wrong end of the bottleneck by 
dropping/queueing. It''s not perfect but is alot better than nothing. To
do better you could write an app that manipulates rwin and keeps account 
of the state of each connection so an estimate of what''s unstoppably on
the way can be used to "ease in" new connections. Better still get an 
ISP that will do it on the right end for you - some do.

Andy.

  I can control how much can i send. Anything
> else is waste of time. EXCLUDE of application server where i want 
> control traffic coming in to a specified port, eg sendmail etc. But 
> combining router+traffic controler+application server in to one box is 
> good idea ? No. Because of this its lower priority task for me. A quote 
> from TODO:
> * support for ingress queue ;)
> 
>  >as I know main imq purpose was to control ingress.
> Really ? Iam not sure. Have look in to first original IMQ device driver 
> writen by Martin Devera:
> http://luxik.cdi.cz/~devik/qos/imq.htm
> 
>  >Ok, enough about this,
>  >I would like to know how stable is your driver, if it shapes localy
>  >generated trafic,
>  >since it was the main problem for imq all time.
>  >if it became stable I will implement your idea in my imq version too.
> 
> Its quite stable. I using it now on two busy routers in my production 
> enviroment (avreage 20 NICs per router, average 70 users, 280 htb 
> classes). Uptime is until kernel upgrade restart (week  ago). localy 
> generated trafic is only DNS and ssh for managment and statistics 
> collection (rrd graphs from Stef Coene, www.docum.org, but the data 
> collections is done by ssh)  Setup scripts i usink like my example but 
> with clasification of the intranet trafics (for this that much htb 
> classes).
> 
> Thanks for the fedback.
> 
> Jiri
> 
> Zkontrolovane antivirusom ClamAv
> Scanned by ClamAv - http://www.clamav.net
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
> 

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Andy Furniss wrote:
> Jiri Fojtasek wrote:
>
>> Hello Roy
>>
>> Roy wrote:
>>
>>  >Hello,
>>  >imq became realy popular now ;)
>>  > >
>> Sure, same as whole GNU/Linux  :)
>>
>>  >It would be good ir you wrote what is that EOS to save time for 
>> some people
>>  >on searches.
>>  >
>>  > >
>> EOS - End of Send. Its time betwen two dequeues and packet transmits
>> used by qdisc for calculation.
>
>
> Is that a kernel transmit or a real "on the wire" transmit - ie
can
> you keep the devices'' buffer empty or just detect when
it''s full?
>
>  Any of current IMQ implementations do not
It leave always only one packet in the device queue. It is there until 
is not dequeued (sent to the device driver) and then is inserted another 
etc, etc

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Jiri Fojtasek wrote:
> 
> Andy Furniss wrote:
> 
>> Jiri Fojtasek wrote:
>>
>>> Hello Roy
>>>
>>> Roy wrote:
>>>
>>>  >Hello,
>>>  >imq became realy popular now ;)
>>>  > >
>>> Sure, same as whole GNU/Linux  :)
>>>
>>>  >It would be good ir you wrote what is that EOS to save time
for
>>> some people
>>>  >on searches.
>>>  >
>>>  > >
>>> EOS - End of Send. Its time betwen two dequeues and packet
transmits
>>> used by qdisc for calculation.
>>
>>
>>
>> Is that a kernel transmit or a real "on the wire" transmit -
ie can
>> you keep the devices'' buffer empty or just detect when
it''s full?
>>
>>  Any of current IMQ implementations do not
> 
> 
> It leave always only one packet in the device queue. It is there until 
> is not dequeued (sent to the device driver) and then is inserted another 
> etc, etc
I think I understand - in the case of imq -> ppp -> usb/pci -> radsl 
modem with big buffer -> phone line. imq will only let one packet wait 
in ppp queue, but there could still be a full buffer in the "real"
device.

Andy.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Andy Furniss wrote:
> Jiri Fojtasek wrote:
>>
>>
>> It leave always only one packet in the device queue. It is there until 
>> is not dequeued (sent to the device driver) and then is inserted 
>> another etc, etc
> 
> 
> I think I understand - in the case of imq -> ppp -> usb/pci ->
radsl
> modem with big buffer -> phone line. imq will only let one packet wait 
> in ppp queue, but there could still be a full buffer in the
"real" device.
I haven''t followed the discussion, but this is the way it is supposed
to
work. As IMQ can queue packets for multiple devices it is not possible
to couple the dequeue event with the "real" output device''s
dequeue
event, simply because there may be multiple. Use non-work-conserving
qdiscs in this case and make sure you don''t exceed the rate of the
individual output devices.

Regards
Patrick
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

----- Original Message ----- 
From: "Patrick McHardy" <kaber@trash.net>
To: "Andy Furniss" <andy.furniss@dsl.pipex.com>
Cc: "Jiri Fojtasek" <jiri.fojtasek@hlohovec.net>;
"LARTC"
<lartc@mailman.ds9a.nl>
Sent: Wednesday, March 31, 2004 3:46 AM
Subject: Re: [LARTC] New IMQ device implementation supporting device EOS

> Andy Furniss wrote:
> > Jiri Fojtasek wrote:
> >>
> >>
> >> It leave always only one packet in the device queue. It is
> there until
> >> is not dequeued (sent to the device driver) and then is
> inserted
> >> another etc, etc
> >
> >
> > I think I understand - in the case of imq -> ppp ->
> usb/pci -> radsl
> > modem with big buffer -> phone line. imq will only let one
> packet wait
> > in ppp queue, but there could still be a full buffer in the
> ''"''real''"'' device.
>
> I haven''t followed the discussion, but this is the way it is
supposed to
> work. As IMQ can queue packets for multiple devices it is not possible
> to couple the dequeue event with the
''"''real''"'' output
device''s dequeue
> event, simply because there may be multiple. Use non-work-conserving
> qdiscs in this case and make sure you don''t exceed the rate of the
> individual output devices.
>
> Regards
> Patrick
Even if I still not understand if all this is nesecary,
( I only suspect it can make imq stable for local trafic shaping what nome
of us was able to do)

But ths idea is posible to implement (i am not sure if what he did realy
works as supposed)
multiple devices do not harm anything because of the way how dequeuing
works.
if device refuses to accept packet it should be requeued in same place
later next packet will be dequeued which may be for other interface.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Andy Furniss wrote:
> Jiri Fojtasek wrote:
> 
>>
>> Andy Furniss wrote:
>>
>>> Jiri Fojtasek wrote:
>>>
>>>> Hello Roy
>>>>
>>>> Roy wrote:
>>>>
>>>>  >Hello,
>>>>  >imq became realy popular now ;)
>>>>  > >
>>>> Sure, same as whole GNU/Linux  :)
>>>>
>>>>  >It would be good ir you wrote what is that EOS to save
time for
>>>> some people
>>>>  >on searches.
>>>>  >
>>>>  > >
>>>> EOS - End of Send. Its time betwen two dequeues and packet
transmits
>>>> used by qdisc for calculation.
>>>
>>>
>>>
>>>
>>> Is that a kernel transmit or a real "on the wire"
transmit - ie can
>>> you keep the devices'' buffer empty or just detect when
it''s full?
>>>
>>>  Any of current IMQ implementations do not
>>
>>
>>
>> It leave always only one packet in the device queue. It is there until 
>> is not dequeued (sent to the device driver) and then is inserted 
>> another etc, etc
> 
> 
> I think I understand - in the case of imq -> ppp -> usb/pci ->
radsl
> modem with big buffer -> phone line. imq will only let one packet wait 
> in ppp queue, but there could still be a full buffer in the
"real" device.
Your problem cannot be solved IMQ and also by standart qdisc atached to 
ppp, but you can place at you ppp qdisc like tbf that will manage 
maximal rate, so the hardware buffers will not be filled quick. With 
usage with my IMQ, IMQ queue will follow the rate of the tbf qdisc 
atached to the ppp device :) I have tested this thing and working great. 
But for this  you will not need IMQ :D

Upper layer do not have anything with hardware buffer. There is not 
kernel interface to get devices state about his hardware buffer. 
Especially if you use PPP. Its software device. But device can indicate 
his busy state by netif_queue_stopped(dev). In time when 
netif_queue_stopped(dev) bit is set (hardware xmiting the packet), 
dequeueing is stopped and schedulet in to other time via softirq 
netif_schedule(dev). The same following my IMQ driver. After dequeueing 
stoping itself by netif_stop_queue() and sending the packet to real 
device queue. When the device accept this packet for delivery ( after 
dev->hard_start_xmit() ) queue of my IMQ driver is waked up by 
netif_wake_queue(). Version 1.1 of my driver have all comented in the 
source.

Jiri

Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Patrick McHardy wrote:
> I haven''t followed the discussion, but this is the way it is
supposed to
> work. As IMQ can queue packets for multiple devices it is not possible
> to couple the dequeue event with the "real" output
device''s dequeue
> event, simply because there may be multiple. Use non-work-conserving
> qdiscs in this case and make sure you don''t exceed the rate of the
> individual output devices.
> 
> Regards
> Patrick
Not posible ? With trick i did its posible ! And working very well :)

Jiri


Zkontrolovane antivirusom ClamAv
Scanned by ClamAv - http://www.clamav.net
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/