LARTC - Apr 2006 - how to do probabilistic packet loss in kernel?

Hi,

I am using iproute2 to setup fowarding, adding routes like "ip route add
192.168.1.3 via 192.168.1.2"

I was wondering where in the kernel I can insert probabilistic packet loss only
for forwarded packets?  So that for instance I can drop 5% of all forwarded
packets?

I don''t need help with the actual code, just need help finding where to
insert this code :)

Thanks!
George

-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Greetings George,

 : I am using iproute2 to setup fowarding, adding routes like "ip 
 : route add 192.168.1.3 via 192.168.1.2"
 : 
 : I was wondering where in the kernel I can insert probabilistic 
 : packet loss only for forwarded packets?  So that for instance I 
 : can drop 5% of all forwarded packets?
 : 
 : I don''t need help with the actual code, just need help finding 
 : where to insert this code :)

I believe you are looking for the netem qdisc [0].  Here''s just a 
snippet from Stephen Hemminger''s wiki page to help you imagine how 
you could use netem to introduce probabilistic packet loss.

   # tc qdisc add dev eth0 parent 1:3 handle 30: netem \
       delay 200ms 10ms distribution normal

Good luck,

-Martin

 [0] http://linux-net.osdl.org/index.php/Netem

-- 
Martin A. Brown --- http://linux-ip.net/ --- martin@linux-ip.net

Thanks Martin and Roman.

I will definately look into this.

I have a question for you though... in terms of adding loss like this, 
this will not interact with hardware layer rate control of wireless 
cards right?

For instance... dropping from 54Mbit to 11Mbit on an 802.11g card when 
loss certain loss begins occuring


Martin A. Brown wrote:
> Greetings George,
> 
>  : I am using iproute2 to setup fowarding, adding routes like "ip 
>  : route add 192.168.1.3 via 192.168.1.2"
>  : 
>  : I was wondering where in the kernel I can insert probabilistic 
>  : packet loss only for forwarded packets?  So that for instance I 
>  : can drop 5% of all forwarded packets?
>  : 
>  : I don''t need help with the actual code, just need help finding 
>  : where to insert this code :)
> 
> I believe you are looking for the netem qdisc [0].  Here''s just a 
> snippet from Stephen Hemminger''s wiki page to help you imagine how
> you could use netem to introduce probabilistic packet loss.
> 
>    # tc qdisc add dev eth0 parent 1:3 handle 30: netem \
>        delay 200ms 10ms distribution normal
> 
> Good luck,
> 
> -Martin
> 
>  [0] http://linux-net.osdl.org/index.php/Netem
>

Hello again,

 : I have a question for you though... in terms of adding loss like 
 : this, this will not interact with hardware layer rate control of 
 : wireless cards right?
 : 
 : For instance... dropping from 54Mbit to 11Mbit on an 802.11g card 
 : when loss certain loss begins occuring

Outgoing packets pass through the traffic control system (netem 
qdisc, in this case) just before being dequeued to the driver.  The 
actual behaviour of the kernel, in this case, depends on a sanely 
coded driver.  I assume a sanely coded driver, in which case this is 
what you should see when the hardware cannot accept packet for 
transmission:

   0. netem (or any other qdisc, for that matter) will operate as 
      configured (inducing loss, delaying, reordering or 
      prioritizing your outgoing packets)
   1. eventually qdisc_restart() will call hardware driver
  2A. [if success] packet is transmitted
  2B. [if failure] the hardware driver cannot handle the packet for 
      some reason (TX ring full, link failure or other problem); it 
      will propagate an error condition to higher layer
   3. qdisc_restart(), receiving such an error will cause the packet
      to be requeued [0]
   4. goto step 1

My source for this answer documents kernel 2.4, although the code in 
the networking stack seems to be fundamentally the same in this 
case.  See the DataTAG report entitled "A Map of the Networking Code 
in Linux Kernel 2.4.20" [1].  On page 19, Section 4.3.1, the authors 
refer to the function net/sched/sch_generic.c which includes 
qdisc_restart().

So, strictly speaking, there should be no interaction between your 
use of the netem qdisc and lower layer rate control (lossy 
transmissions and any compensatory mechanisms between radios).

Note!  Both of the sources for my answer are from old documentation 
(and, of course, ongoing general knowledge of the traffic control 
system).  I believe that the kernel still operates in this fashion, 
but would absolutely welcome any corrections from those who are more 
intimately familiar with the kernel and hardware perspective.

Good luck, George,

-Martin

 [0] http://qos.ittc.ku.edu/howto/node11.html
     http://qos.ittc.ku.edu/howto/
 [1] http://datatag.web.cern.ch/datatag/papers/tr-datatag-2004-1.pdf

-- 
Martin A. Brown --- http://linux-ip.net/ --- martin@linux-ip.net

On 4/17/06, George P Nychis <gnychis@cmu.edu>
wrote:
> Hi,
>
> I am using iproute2 to setup fowarding, adding routes like "ip route
add 192.168.1.3 via 192.168.1.2"
>
> I was wondering where in the kernel I can insert probabilistic packet loss
only for forwarded packets?  So that for instance I can drop 5% of all forwarded
packets?
>
Have a look at:
http://linux-net.osdl.org/index.php/Netem

--
Ian McDonald
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
Department of Computer Science
University of Waikato
New Zealand
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Okay i need a little help.

So, I have this qdisc module q_xcp.so that was compiled for tc version 2.4.7, so
I have version 2.4.7 for q_xcp.so to work properly.  I tried using q_xcp.so with
a the newest version of iproute/tc but it seems as though since version 2.4.7
qdisc_util structure has changed some and i keep getting seg faults trying to
use q_xcp.  For instance, in the newer versions of tc, i see qdisc''s
structures defined like:
struct qdisc_util netem_qdisc_util = { };

and in the same newer versions, in tc.c get_qdisc_kind, before it calls "q
= dlsym(dlh, buf);" ... it sets buf to "%s_qdisc_util"

however in the old version of tc that q_xcp was made for, it sets buf to
"%s_util"

so I guess my main question is, do you know of a version of iproute/tc that uses
this old structure that has netem support?  I tried copying the q_netem.c code
from the newer version of iproute and trying to compile it for an old version of
ip route... but its just not working out nicely at all

I''d greatly appreciate any help.

Thanks!
George

> Hello again,
> 
> : I have a question for you though... in terms of adding loss like : this,
> this will not interact with hardware layer rate control of : wireless
> cards right? : : For instance... dropping from 54Mbit to 11Mbit on an
> 802.11g card : when loss certain loss begins occuring
> 
> Outgoing packets pass through the traffic control system (netem qdisc, in
> this case) just before being dequeued to the driver.  The actual behaviour
> of the kernel, in this case, depends on a sanely coded driver.  I assume a
> sanely coded driver, in which case this is what you should see when the
> hardware cannot accept packet for transmission:
> 
> 0. netem (or any other qdisc, for that matter) will operate as configured
> (inducing loss, delaying, reordering or prioritizing your outgoing
> packets) 1. eventually qdisc_restart() will call hardware driver 2A. [if
> success] packet is transmitted 2B. [if failure] the hardware driver cannot
> handle the packet for some reason (TX ring full, link failure or other
> problem); it will propagate an error condition to higher layer 3.
> qdisc_restart(), receiving such an error will cause the packet to be
> requeued [0] 4. goto step 1
> 
> My source for this answer documents kernel 2.4, although the code in the
> networking stack seems to be fundamentally the same in this case.  See the
> DataTAG report entitled "A Map of the Networking Code in Linux Kernel
> 2.4.20" [1].  On page 19, Section 4.3.1, the authors refer to the
function
> net/sched/sch_generic.c which includes qdisc_restart().
> 
> So, strictly speaking, there should be no interaction between your use of
> the netem qdisc and lower layer rate control (lossy transmissions and any
> compensatory mechanisms between radios).
> 
> Note!  Both of the sources for my answer are from old documentation (and,
> of course, ongoing general knowledge of the traffic control system).  I
> believe that the kernel still operates in this fashion, but would
> absolutely welcome any corrections from those who are more intimately
> familiar with the kernel and hardware perspective.
> 
> Good luck, George,
> 
> -Martin
> 
> [0] http://qos.ittc.ku.edu/howto/node11.html http://qos.ittc.ku.edu/howto/
>  [1] http://datatag.web.cern.ch/datatag/papers/tr-datatag-2004-1.pdf
> 
> -- Martin A. Brown --- http://linux-ip.net/ --- martin@linux-ip.net
> 
> 

--

Sorry to spam, but just to save anyone trying to help some time, I modified
q_xcp to work with newer version of iproute2/tc now :)

So now I can use netem!

Thanks for everyones help.

- George

> Okay i need a little help.
> 
> So, I have this qdisc module q_xcp.so that was compiled for tc version
> 2.4.7, so I have version 2.4.7 for q_xcp.so to work properly.  I tried
> using q_xcp.so with a the newest version of iproute/tc but it seems as
> though since version 2.4.7 qdisc_util structure has changed some and i
> keep getting seg faults trying to use q_xcp.  For instance, in the newer
> versions of tc, i see qdisc''s structures defined like: struct
qdisc_util
> netem_qdisc_util = { };
> 
> and in the same newer versions, in tc.c get_qdisc_kind, before it calls
> "q = dlsym(dlh, buf);" ... it sets buf to
"%s_qdisc_util"
> 
> however in the old version of tc that q_xcp was made for, it sets buf to
> "%s_util"
> 
> so I guess my main question is, do you know of a version of iproute/tc
> that uses this old structure that has netem support?  I tried copying the
> q_netem.c code from the newer version of iproute and trying to compile it
> for an old version of ip route... but its just not working out nicely at
> all
> 
> I''d greatly appreciate any help.
> 
> Thanks! George
> 
> 
>> Hello again,
>> 
>> : I have a question for you though... in terms of adding loss like :
>> this, this will not interact with hardware layer rate control of :
>> wireless cards right? : : For instance... dropping from 54Mbit to
11Mbit
>> on an 802.11g card : when loss certain loss begins occuring
>> 
>> Outgoing packets pass through the traffic control system (netem qdisc,
>> in this case) just before being dequeued to the driver.  The actual
>> behaviour of the kernel, in this case, depends on a sanely coded
driver.
>> I assume a sanely coded driver, in which case this is what you should
>> see when the hardware cannot accept packet for transmission:
>> 
>> 0. netem (or any other qdisc, for that matter) will operate as
>> configured (inducing loss, delaying, reordering or prioritizing your
>> outgoing packets) 1. eventually qdisc_restart() will call hardware
>> driver 2A. [if success] packet is transmitted 2B. [if failure] the
>> hardware driver cannot handle the packet for some reason (TX ring full,
>> link failure or other problem); it will propagate an error condition to
>> higher layer 3. qdisc_restart(), receiving such an error will cause the
>> packet to be requeued [0] 4. goto step 1
>> 
>> My source for this answer documents kernel 2.4, although the code in
>> the networking stack seems to be fundamentally the same in this case.
>> See the DataTAG report entitled "A Map of the Networking Code in
Linux
>> Kernel 2.4.20" [1].  On page 19, Section 4.3.1, the authors refer
to the
>> function net/sched/sch_generic.c which includes qdisc_restart().
>> 
>> So, strictly speaking, there should be no interaction between your use
>> of the netem qdisc and lower layer rate control (lossy transmissions
and
>> any compensatory mechanisms between radios).
>> 
>> Note!  Both of the sources for my answer are from old documentation
>> (and, of course, ongoing general knowledge of the traffic control
>> system).  I believe that the kernel still operates in this fashion, but
>> would absolutely welcome any corrections from those who are more
>> intimately familiar with the kernel and hardware perspective.
>> 
>> Good luck, George,
>> 
>> -Martin
>> 
>> [0] http://qos.ittc.ku.edu/howto/node11.html
>> http://qos.ittc.ku.edu/howto/ [1]
>> http://datatag.web.cern.ch/datatag/papers/tr-datatag-2004-1.pdf
>> 
>> -- Martin A. Brown --- http://linux-ip.net/ --- martin@linux-ip.net
>> 
>> 
> 
> 
> --
> 
> _______________________________________________ LARTC mailing list 
> LARTC@mailman.ds9a.nl 
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 
> 

--

Hey Tim,

Thanks for your constant help.  I managed to port q_xcp.c to work in modern
versions of tc.  I am using the 2.4.32 kernel and compiled in netem support into
the kernel.  I then compiled iproute2-2.6.11 including tc that came with it,
which installed q_xcp.so to /usr/lib/tc and q_netem.so to /usr/lib/tc

I then try:
tc qdisc change dev ath0 root netem loss .1%

and get:
RTNETLINK answers: Invalid argument

I also tried:
tc qdisc change dev eth0 root netem duplicate 1%

and get the same exact Invalid argument response...

Any ideas here?

Thanks!
George

> It sounds like you should simply port your research q_xcp.c to work in the
> more modern versions of tc.  That port should be straightforward once you
> understand how the netem_qdisc_util works.
> 
> To make debugging easier, you can always link q_xcp.o into the binary at
> build time by adding the appropriate thing to the make file, and compile
> the whole thing with -g so that it is easier to debug.
> 
> -Tim Shepard shep@alum.mit.edu
> 
> 

--

Hey Martin,

I was able to do it with netem and its working great now.

I''ve actually moved on to another challenge, I would like to drop 
packets at the hardware level such as to see rate control.

Because when netem drops a packet, TCP responds, however the lower level 
card will not interact because it never sees the loss.

What I want to do is somehow cause the card to send a corrupted packet 
based on a probability, or not send the packet but make it think that it 
did.

I''m using madwifi and I''ve found in the code where it does
rate control
and sends out the data, so i''m hoping to make this happen, but having 
troubles!

So if anyone else has any ideas on how to get rate control interactive 
packet loss, i''d love it.

- George


Martin A. Brown wrote:
> Hello George,
> 
> Unfortunately, I cannot answer your most recent question.  I''m 
> hoping that Stephen Hemminger can answer your question.  He is 
> subscribed to the LARTC list, is also the author of netem and 
> seems to be a smart cookie.
> 
> Good luck,
> 
> -Martin
> 
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 4/20/06, George Nychis <gnychis@cmu.edu> wrote:
> Hey Martin,
>
> I was able to do it with netem and its working great now.
>
> I''ve actually moved on to another challenge, I would like to drop
> packets at the hardware level such as to see rate control.
>
Have a look at:
http://linux-net.osdl.org/index.php/Netem#Rate_control

Works well for me...

Ian
--
Ian McDonald
Web: http://wand.net.nz/~iam4
Blog: http://imcdnzl.blogspot.com
WAND Network Research Group
Department of Computer Science
University of Waikato
New Zealand
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html