thr3ads.net - crossbow discuss - [crossbow-discuss] Flows aren''t respecting my limits [Nov 2009]

If this information is useful, please help other people find it:
Share via:

Cesar Delgado

2009-Nov-23 19:27 UTC

[crossbow-discuss] Flows aren''t respecting my limits

I''m setting up a server to go to a hosting site where I have a 1Mbps
pipe.  From what I read I know I can''t set the limit to this as the
lowest setting is ~1.2Mbps and this is something that''s getting worked
on in Crossbow2.  I am seeing some strange behavior.

FIrst I have a question on flowadm''s show-usage command.  When I try to
run show-prop with the name of a flow I get an error.  The flow exists.  What am
I doing wrong?

root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow
flowadm: invalid flow: ''(null)''


Ok, now for my problem.  I have the following setting:

root at myhost:~# flowadm show-flowprop http-flow
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
http-flow    maxbw               1.228      --             1228k 
http-flow    priority        medium         --             medium 

I ran a test hitting the webserver and I see this:

root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
http-flow    01:32:22      01:32:42      1512     2571     0.001 Mbps
ssh-flow     01:32:42      01:33:02      1818     3578     0.002 Mbps
http-flow    01:33:02      01:33:22      66917    3165136      1.292 Mbp
ssh-flow     01:33:02      01:33:22      3618     5344     0.003 Mbps
http-flow    01:33:22      01:33:42      117947   5713018      2.332 Mbp
ssh-flow     01:33:22      01:33:42      4182     3020     0.002 Mbps
http-flow    01:33:42      01:34:02      118998   5685520      2.321 Mbp
ssh-flow     01:33:42      01:34:02      11616    9924     0.008 Mbps
http-flow    01:34:02      01:34:22      117084   5725664      2.337 Mbp
http-flow    01:34:22      01:34:42      119130   5725168      2.337 Mbp
http-flow    01:34:42      01:35:02      114180   5725168      2.335 Mbp
http-flow    01:35:02      01:35:22      109230   5725664      2.333 Mbp
http-flow    01:35:22      01:35:42      116160   5725168      2.336 Mbp
http-flow    01:35:42      01:36:02      119262   5725168      2.337 Mbp
http-flow    01:36:02      01:36:22      119196   5725664      2.337 Mbp
http-flow    01:36:22      01:36:42      117216   5725168      2.336 Mbp
http-flow    01:36:42      01:37:02      119394   5722636      2.336 Mbp
http-flow    01:37:02      01:37:22      119526   5725168      2.337 Mbp
http-flow    01:37:22      01:37:42      119460   5725168      2.337 Mbp
http-flow    01:37:42      01:38:02      119460   5725664      2.338 Mbp
http-flow    01:38:02      01:38:22      119724   5725168      2.337 Mbp
http-flow    01:38:22      01:38:42      119724   5725168      2.337 Mbp
http-flow    01:38:42      01:39:02      119130   5722636      2.336 Mbp
http-flow    01:39:02      01:39:22      118866   5725168      2.337 Mbp
http-flow    01:39:22      01:39:42      116490   5725664      2.336 Mbp
http-flow    01:39:42      01:40:02      119790   5725168      2.337 Mbp
http-flow    01:40:02      01:40:22      117678   5725168      2.337 Mbp
http-flow    01:40:22      01:40:42      118668   5725664      2.337 Mbp
http-flow    01:40:42      01:41:02      117414   5725168      2.337 Mbp
http-flow    01:41:02      01:41:22      119790   5725168      2.337 Mbp
http-flow    01:41:22      01:41:42      119813   5720510      2.336 Mbp
http-flow    01:41:42      01:42:02      119394   5725664      2.338 Mbp
http-flow    01:42:02      01:42:22      119724   5722272      2.336 Mbp
http-flow    01:42:22      01:42:42      119526   5725664      2.338 Mbp
http-flow    01:42:42      01:43:02      119196   5722140      2.336 Mbp
http-flow    01:43:02      01:43:22      119394   5725664      2.338 Mbp
http-flow    01:43:22      01:43:42      119658   5725168      2.337 Mbp
http-flow    01:43:42      01:44:02      119064   5725168      2.337 Mbp
http-flow    01:44:02      01:44:22      113256   5676668      2.315 Mbp
ssh-flow     01:44:02      01:44:22      18414    49646    0.027 Mbps
http-flow    01:44:22      01:44:42      118206   5725664      2.337 Mbp
http-flow    01:44:42      01:45:02      117282   5722140      2.335 Mbp
ssh-flow     01:44:42      01:45:02      4698     3544     0.003 Mbps
http-flow    01:45:02      01:45:22      118536   5688284      2.322 Mbp
ssh-flow     01:45:02      01:45:22      4092     3198     0.002 Mbps
http-flow    01:45:22      01:45:42      119130   5725168      2.337 Mbp
ssh-flow     01:45:22      01:45:42      1980     1478     0.001 Mbps

That''s above the flow''s maxbw parameter.  After that I tried
to change the  maxbw of the link with dladm and that brought the bandwidth down
but still not down to 1.2 Mbps.

root at myhost:~# dladm show-linkprop -p maxbw e1000g0
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g0      maxbw           rw       1.228      --             -- 

root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
http-flow    01:46:02      01:46:22      119394   5725168      2.337 Mbp
ssh-flow     01:46:02      01:46:22      4680     2980     0.003 Mbps
http-flow    01:46:22      01:46:42      94314    4520316      1.845 Mbp

Any ideas or is there a subtlety that I''m missing and the behavior is
correct?

Thanks for the help.

-Cesar

venugopal iyer

2009-Nov-24 17:29 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Hi, Cesar:

On Mon, 23 Nov 2009, Cesar Delgado wrote:
> I''m setting up a server to go to a hosting site where I have a
1Mbps pipe.  From what I read I know I can''t set the limit to this as
the lowest setting is ~1.2Mbps and this is something that''s getting
worked on in Crossbow2.  I am seeing some strange behavior.
>
> FIrst I have a question on flowadm''s show-usage command.  When I
try to run show-prop with the name of a flow I get an error.  The flow exists. 
What am I doing wrong?
>
> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow
> flowadm: invalid flow: ''(null)''
This is a bug, I have submitted

6904427 flowadm show-usage doesn''t work with a flow name

>
>
> Ok, now for my problem.  I have the following setting:
>
> root at myhost:~# flowadm show-flowprop http-flow
> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
> http-flow    maxbw               1.228      --             1228k
> http-flow    priority        medium         --             medium
>
> I ran a test hitting the webserver and I see this:
I have the following flow

# flowadm show-flow 
FLOW        LINK        IPADDR                   PROTO  LPORT   RPORT   DSFLD
tcp-flow    <link>      --                       tcp    --      --      --


# flowadm show-flowprop tcp-flow
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
tcp-flow     maxbw               1.228      --             1228K 
tcp-flow     priority        --             --             ?

When I send TCP traffic (am using a traffic generator - netperf, to
this machine from a peer) for about 2 mins.

On the peer the traffic generator (sender) says I am capped to about
1.14 Mbps.

Size   Size    Size     Time     Throughput 
bytes  bytes   bytes    secs.    10^6bits/sec

  49152  49152  49152    120.49      1.14

Now, when I try show-usage during the traffic flow on
the machine with the above flow in place (receiver), I am seeing:

# flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
FLOW         START         END           RBYTES   OBYTES   BANDWIDTH
tcp-flow     08:51:48      08:52:08      3428658  107802       1.414 Mbp
tcp-flow     08:52:08      08:52:28      3431198  107802       1.415 Mbp
tcp-flow     08:52:28      08:52:48      3434614  107888       1.417 Mbp
tcp-flow     08:52:48      08:53:08      3443298  107802       1.420 Mbp
tcp-flow     08:53:08      08:53:28      3444324  107802       1.420 Mbp
tcp-flow     08:53:28      08:53:48      1376806  43576    0.568 Mbps

...

I think the difference you see is likely to be because of the time
period when the stats are written to the file (the bandwidth is 
computed for every 20 seconds period which might not be exactly in
sync with the bandwidth enforcement period in the kernel) and also
could be because of rounding up etc. But, if you look at the  entire
duration, it averages to about the configured limit (in the above
example, I think it is about 1.275 Mbps for the 2 min duration)

BTW, setting a maxbw for a link (dladm) doesn''t really impact the
flow as the bandwidth for both are independent.

thanks,

-venu










>
> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
> http-flow    01:32:22      01:32:42      1512     2571     0.001 Mbps
> ssh-flow     01:32:42      01:33:02      1818     3578     0.002 Mbps
> http-flow    01:33:02      01:33:22      66917    3165136      1.292 Mbp
> ssh-flow     01:33:02      01:33:22      3618     5344     0.003 Mbps
> http-flow    01:33:22      01:33:42      117947   5713018      2.332 Mbp
> ssh-flow     01:33:22      01:33:42      4182     3020     0.002 Mbps
> http-flow    01:33:42      01:34:02      118998   5685520      2.321 Mbp
> ssh-flow     01:33:42      01:34:02      11616    9924     0.008 Mbps
> http-flow    01:34:02      01:34:22      117084   5725664      2.337 Mbp
> http-flow    01:34:22      01:34:42      119130   5725168      2.337 Mbp
> http-flow    01:34:42      01:35:02      114180   5725168      2.335 Mbp
> http-flow    01:35:02      01:35:22      109230   5725664      2.333 Mbp
> http-flow    01:35:22      01:35:42      116160   5725168      2.336 Mbp
> http-flow    01:35:42      01:36:02      119262   5725168      2.337 Mbp
> http-flow    01:36:02      01:36:22      119196   5725664      2.337 Mbp
> http-flow    01:36:22      01:36:42      117216   5725168      2.336 Mbp
> http-flow    01:36:42      01:37:02      119394   5722636      2.336 Mbp
> http-flow    01:37:02      01:37:22      119526   5725168      2.337 Mbp
> http-flow    01:37:22      01:37:42      119460   5725168      2.337 Mbp
> http-flow    01:37:42      01:38:02      119460   5725664      2.338 Mbp
> http-flow    01:38:02      01:38:22      119724   5725168      2.337 Mbp
> http-flow    01:38:22      01:38:42      119724   5725168      2.337 Mbp
> http-flow    01:38:42      01:39:02      119130   5722636      2.336 Mbp
> http-flow    01:39:02      01:39:22      118866   5725168      2.337 Mbp
> http-flow    01:39:22      01:39:42      116490   5725664      2.336 Mbp
> http-flow    01:39:42      01:40:02      119790   5725168      2.337 Mbp
> http-flow    01:40:02      01:40:22      117678   5725168      2.337 Mbp
> http-flow    01:40:22      01:40:42      118668   5725664      2.337 Mbp
> http-flow    01:40:42      01:41:02      117414   5725168      2.337 Mbp
> http-flow    01:41:02      01:41:22      119790   5725168      2.337 Mbp
> http-flow    01:41:22      01:41:42      119813   5720510      2.336 Mbp
> http-flow    01:41:42      01:42:02      119394   5725664      2.338 Mbp
> http-flow    01:42:02      01:42:22      119724   5722272      2.336 Mbp
> http-flow    01:42:22      01:42:42      119526   5725664      2.338 Mbp
> http-flow    01:42:42      01:43:02      119196   5722140      2.336 Mbp
> http-flow    01:43:02      01:43:22      119394   5725664      2.338 Mbp
> http-flow    01:43:22      01:43:42      119658   5725168      2.337 Mbp
> http-flow    01:43:42      01:44:02      119064   5725168      2.337 Mbp
> http-flow    01:44:02      01:44:22      113256   5676668      2.315 Mbp
> ssh-flow     01:44:02      01:44:22      18414    49646    0.027 Mbps
> http-flow    01:44:22      01:44:42      118206   5725664      2.337 Mbp
> http-flow    01:44:42      01:45:02      117282   5722140      2.335 Mbp
> ssh-flow     01:44:42      01:45:02      4698     3544     0.003 Mbps
> http-flow    01:45:02      01:45:22      118536   5688284      2.322 Mbp
> ssh-flow     01:45:02      01:45:22      4092     3198     0.002 Mbps
> http-flow    01:45:22      01:45:42      119130   5725168      2.337 Mbp
> ssh-flow     01:45:22      01:45:42      1980     1478     0.001 Mbps
>
> That''s above the flow''s maxbw parameter.  After that I
tried to change the  maxbw of the link with dladm and that brought the bandwidth
down but still not down to 1.2 Mbps.
>
> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
> LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
> e1000g0      maxbw           rw       1.228      --             --
>
> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
> http-flow    01:46:02      01:46:22      119394   5725168      2.337 Mbp
> ssh-flow     01:46:02      01:46:22      4680     2980     0.003 Mbps
> http-flow    01:46:22      01:46:42      94314    4520316      1.845 Mbp
>
> Any ideas or is there a subtlety that I''m missing and the behavior
is correct?
>
> Thanks for the help.
>
> -Cesar
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

Cesar Delgado

2009-Nov-24 18:12 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Venugopal,

I''m sorry if these sounds like basic questions.  I really appreciate
the patience and the help.  Replies in-line.

On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
> 
> 
> Hi, Cesar:
> 
> On Mon, 23 Nov 2009, Cesar Delgado wrote:
> 
>> I''m setting up a server to go to a hosting site where I have a
1Mbps pipe.  From what I read I know I can''t set the limit to this as
the lowest setting is ~1.2Mbps and this is something that''s getting
worked on in Crossbow2.  I am seeing some strange behavior.
>> 
>> FIrst I have a question on flowadm''s show-usage command.  When
I try to run show-prop with the name of a flow I get an error.  The flow exists.
What am I doing wrong?
>> 
>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow
>> flowadm: invalid flow: ''(null)''
> 
> This is a bug, I have submitted
> 
> 6904427 flowadm show-usage doesn''t work with a flow name
Thanks for submitting that.  I haven''t been able to find a link to the
bugtracker for Crossbow.  Could you please send me the URL?
> 
> 
>> 
>> 
>> Ok, now for my problem.  I have the following setting:
>> 
>> root at myhost:~# flowadm show-flowprop http-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> http-flow    maxbw               1.228      --             1228k
>> http-flow    priority        medium         --             medium
>> 
>> I ran a test hitting the webserver and I see this:
> 
> I have the following flow
> 
> # flowadm show-flow FLOW        LINK        IPADDR                   PROTO 
LPORT   RPORT   DSFLD
> tcp-flow    <link>      --                       tcp    --      --   
--
> 
> 
> # flowadm show-flowprop tcp-flow
> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
> tcp-flow     maxbw               1.228      --             1228K tcp-flow  
priority        --             --             ?
> 
> When I send TCP traffic (am using a traffic generator - netperf, to
> this machine from a peer) for about 2 mins.
> 
> On the peer the traffic generator (sender) says I am capped to about
> 1.14 Mbps.
> 
> Size   Size    Size     Time     Throughput bytes  bytes   bytes    secs.  
10^6bits/sec
> 
> 49152  49152  49152    120.49      1.14
> 
> Now, when I try show-usage during the traffic flow on
> the machine with the above flow in place (receiver), I am seeing:
> 
> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
> FLOW         START         END           RBYTES   OBYTES   BANDWIDTH
> tcp-flow     08:51:48      08:52:08      3428658  107802       1.414 Mbp
> tcp-flow     08:52:08      08:52:28      3431198  107802       1.415 Mbp
> tcp-flow     08:52:28      08:52:48      3434614  107888       1.417 Mbp
> tcp-flow     08:52:48      08:53:08      3443298  107802       1.420 Mbp
> tcp-flow     08:53:08      08:53:28      3444324  107802       1.420 Mbp
> tcp-flow     08:53:28      08:53:48      1376806  43576    0.568 Mbps
> 
> ...
> 
> I think the difference you see is likely to be because of the time
> period when the stats are written to the file (the bandwidth is computed
for every 20 seconds period which might not be exactly in
> sync with the bandwidth enforcement period in the kernel) and also
> could be because of rounding up etc. But, if you look at the  entire
> duration, it averages to about the configured limit (in the above
> example, I think it is about 1.275 Mbps for the 2 min duration)
The way I''m testing it is setting up Apache and then moving down a file
with `wget`.  The use case for this machine is an Apache based app that serves
large files to customers.  That is why I think a `wget` is more telling of
"real" performance than netperf.  I''m running the test again
and on the client side I am seeing usage over the maxbw limit I have set. 
`wget` is reporting about 2Mbps transfer rate which is much closer to what I was
seeing in the show-usage statistics.

[cdelgado at Bluegene tmp]$ wget sol/myfile.dat
--10:01:30--  http://sol/myfile.dat
Resolving sol... 192.168.69.104
Connecting to sol|192.168.69.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M)
Saving to: `myfile.dat''

 5% [==>                                                      ] 55,530,974  
267K/s  eta 60m 44s

> 
> BTW, setting a maxbw for a link (dladm) doesn''t really impact the
> flow as the bandwidth for both are independent.
Thank you for this clarification but I still don''t understand how I can
be seeing ~2Mbps transfer if both the link and the flow are both capped at
1.2Mbps.
 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>> 
>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>> http-flow    01:32:22      01:32:42      1512     2571     0.001 Mbps
>> ssh-flow     01:32:42      01:33:02      1818     3578     0.002 Mbps
>> http-flow    01:33:02      01:33:22      66917    3165136      1.292
Mbp
>> ssh-flow     01:33:02      01:33:22      3618     5344     0.003 Mbps
>> http-flow    01:33:22      01:33:42      117947   5713018      2.332
Mbp
>> ssh-flow     01:33:22      01:33:42      4182     3020     0.002 Mbps
>> http-flow    01:33:42      01:34:02      118998   5685520      2.321
Mbp
>> ssh-flow     01:33:42      01:34:02      11616    9924     0.008 Mbps
>> http-flow    01:34:02      01:34:22      117084   5725664      2.337
Mbp
>> http-flow    01:34:22      01:34:42      119130   5725168      2.337
Mbp
>> http-flow    01:34:42      01:35:02      114180   5725168      2.335
Mbp
>> http-flow    01:35:02      01:35:22      109230   5725664      2.333
Mbp
>> http-flow    01:35:22      01:35:42      116160   5725168      2.336
Mbp
>> http-flow    01:35:42      01:36:02      119262   5725168      2.337
Mbp
>> http-flow    01:36:02      01:36:22      119196   5725664      2.337
Mbp
>> http-flow    01:36:22      01:36:42      117216   5725168      2.336
Mbp
>> http-flow    01:36:42      01:37:02      119394   5722636      2.336
Mbp
>> http-flow    01:37:02      01:37:22      119526   5725168      2.337
Mbp
>> http-flow    01:37:22      01:37:42      119460   5725168      2.337
Mbp
>> http-flow    01:37:42      01:38:02      119460   5725664      2.338
Mbp
>> http-flow    01:38:02      01:38:22      119724   5725168      2.337
Mbp
>> http-flow    01:38:22      01:38:42      119724   5725168      2.337
Mbp
>> http-flow    01:38:42      01:39:02      119130   5722636      2.336
Mbp
>> http-flow    01:39:02      01:39:22      118866   5725168      2.337
Mbp
>> http-flow    01:39:22      01:39:42      116490   5725664      2.336
Mbp
>> http-flow    01:39:42      01:40:02      119790   5725168      2.337
Mbp
>> http-flow    01:40:02      01:40:22      117678   5725168      2.337
Mbp
>> http-flow    01:40:22      01:40:42      118668   5725664      2.337
Mbp
>> http-flow    01:40:42      01:41:02      117414   5725168      2.337
Mbp
>> http-flow    01:41:02      01:41:22      119790   5725168      2.337
Mbp
>> http-flow    01:41:22      01:41:42      119813   5720510      2.336
Mbp
>> http-flow    01:41:42      01:42:02      119394   5725664      2.338
Mbp
>> http-flow    01:42:02      01:42:22      119724   5722272      2.336
Mbp
>> http-flow    01:42:22      01:42:42      119526   5725664      2.338
Mbp
>> http-flow    01:42:42      01:43:02      119196   5722140      2.336
Mbp
>> http-flow    01:43:02      01:43:22      119394   5725664      2.338
Mbp
>> http-flow    01:43:22      01:43:42      119658   5725168      2.337
Mbp
>> http-flow    01:43:42      01:44:02      119064   5725168      2.337
Mbp
>> http-flow    01:44:02      01:44:22      113256   5676668      2.315
Mbp
>> ssh-flow     01:44:02      01:44:22      18414    49646    0.027 Mbps
>> http-flow    01:44:22      01:44:42      118206   5725664      2.337
Mbp
>> http-flow    01:44:42      01:45:02      117282   5722140      2.335
Mbp
>> ssh-flow     01:44:42      01:45:02      4698     3544     0.003 Mbps
>> http-flow    01:45:02      01:45:22      118536   5688284      2.322
Mbp
>> ssh-flow     01:45:02      01:45:22      4092     3198     0.002 Mbps
>> http-flow    01:45:22      01:45:42      119130   5725168      2.337
Mbp
>> ssh-flow     01:45:22      01:45:42      1980     1478     0.001 Mbps
>> 
>> That''s above the flow''s maxbw parameter.  After that
I tried to change the  maxbw of the link with dladm and that brought the
bandwidth down but still not down to 1.2 Mbps.
>> 
>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
>> LINK         PROPERTY        PERM VALUE          DEFAULT       
POSSIBLE
>> e1000g0      maxbw           rw       1.228      --             --
>> 
>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>> http-flow    01:46:02      01:46:22      119394   5725168      2.337
Mbp
>> ssh-flow     01:46:02      01:46:22      4680     2980     0.003 Mbps
>> http-flow    01:46:22      01:46:42      94314    4520316      1.845
Mbp
>> 
>> Any ideas or is there a subtlety that I''m missing and the
behavior is correct?
>> 
>> Thanks for the help.
>> 
>> -Cesar
>> 
>> _______________________________________________
>> crossbow-discuss mailing list
>> crossbow-discuss at opensolaris.org
>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>

venugopal iyer

2009-Nov-25 17:49 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Hi, Cesar:

On Tue, 24 Nov 2009, Cesar Delgado wrote:
> Venugopal,
>
> I''m sorry if these sounds like basic questions.  I really
appreciate the patience and the help.  Replies in-line.
>
> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>
>>
>>
>> Hi, Cesar:
>>
>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>
>>> I''m setting up a server to go to a hosting site where I
have a 1Mbps pipe.  From what I read I know I can''t set the limit to
this as the lowest setting is ~1.2Mbps and this is something that''s
getting worked on in Crossbow2.  I am seeing some strange behavior.
>>>
>>> FIrst I have a question on flowadm''s show-usage command. 
When I try to run show-prop with the name of a flow I get an error.  The flow
exists.  What am I doing wrong?
>>>
>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow
>>> flowadm: invalid flow: ''(null)''
>>
>> This is a bug, I have submitted
>>
>> 6904427 flowadm show-usage doesn''t work with a flow name
>
> Thanks for submitting that.  I haven''t been able to find a link to
the bugtracker for Crossbow.  Could you please send me the URL?
I think it should show up on
http://bugs.opensolaris.org/bugdatabase/index.jsp 
soon.
>>
>>
>>>
>>>
>>> Ok, now for my problem.  I have the following setting:
>>>
>>> root at myhost:~# flowadm show-flowprop http-flow
>>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>>> http-flow    maxbw               1.228      --             1228k
>>> http-flow    priority        medium         --             medium
>>>
>>> I ran a test hitting the webserver and I see this:
>>
>> I have the following flow
>>
>> # flowadm show-flow FLOW        LINK        IPADDR                  
PROTO  LPORT   RPORT   DSFLD
>> tcp-flow    <link>      --                       tcp    --     
--      --
>>
>>
>> # flowadm show-flowprop tcp-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> tcp-flow     maxbw               1.228      --             1228K
tcp-flow     priority        --             --             ?
>>
>> When I send TCP traffic (am using a traffic generator - netperf, to
>> this machine from a peer) for about 2 mins.
>>
>> On the peer the traffic generator (sender) says I am capped to about
>> 1.14 Mbps.
>>
>> Size   Size    Size     Time     Throughput bytes  bytes   bytes   
secs.    10^6bits/sec
>>
>> 49152  49152  49152    120.49      1.14
>>
>> Now, when I try show-usage during the traffic flow on
>> the machine with the above flow in place (receiver), I am seeing:
>>
>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
>> FLOW         START         END           RBYTES   OBYTES   BANDWIDTH
>> tcp-flow     08:51:48      08:52:08      3428658  107802       1.414
Mbp
>> tcp-flow     08:52:08      08:52:28      3431198  107802       1.415
Mbp
>> tcp-flow     08:52:28      08:52:48      3434614  107888       1.417
Mbp
>> tcp-flow     08:52:48      08:53:08      3443298  107802       1.420
Mbp
>> tcp-flow     08:53:08      08:53:28      3444324  107802       1.420
Mbp
>> tcp-flow     08:53:28      08:53:48      1376806  43576    0.568 Mbps
>>
>> ...
>>
>> I think the difference you see is likely to be because of the time
>> period when the stats are written to the file (the bandwidth is
computed for every 20 seconds period which might not be exactly in
>> sync with the bandwidth enforcement period in the kernel) and also
>> could be because of rounding up etc. But, if you look at the  entire
>> duration, it averages to about the configured limit (in the above
>> example, I think it is about 1.275 Mbps for the 2 min duration)
>
> The way I''m testing it is setting up Apache and then moving down a
file with `wget`.  The use case for this machine is an Apache based app that
serves large files to customers.  That is why I think a `wget` is more telling
of "real" performance than netperf.  I''m running the test
again and on the client side I am seeing usage over the maxbw limit I have set. 
`wget` is reporting about 2Mbps transfer rate which is much closer to what I was
seeing in the show-usage statistics.
>
> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
> --10:01:30--  http://sol/myfile.dat
> Resolving sol... 192.168.69.104
> Connecting to sol|192.168.69.104|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 1048576000 (1000M)
> Saving to: `myfile.dat''
>
> 5% [==>                                                      ]
55,530,974   267K/s  eta 60m 44s
>
>
>>
>> BTW, setting a maxbw for a link (dladm) doesn''t really impact
the
>> flow as the bandwidth for both are independent.
>
> Thank you for this clarification but I still don''t understand how
I can be seeing ~2Mbps transfer if both the link and the flow are both capped at
1.2Mbps.
>
Can you try with a higher bandwidth, say 100 Mbps and see what the results
are when compared to wget''s output?

Also, another way of manually checking would be to do a
 	# kstat -c flow -n http-flow

before and after the wget run and see how many bytes (rbytes) the
kernel has seen for that flow (assuming there isn''t any other traffic
going over the flow), and then determine the bandwidth (you might need
to get the duration of the wget run pretty close to get the
right bandwdith value).

-venu
>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>>
>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>> http-flow    01:32:22      01:32:42      1512     2571     0.001
Mbps
>>> ssh-flow     01:32:42      01:33:02      1818     3578     0.002
Mbps
>>> http-flow    01:33:02      01:33:22      66917    3165136     
1.292 Mbp
>>> ssh-flow     01:33:02      01:33:22      3618     5344     0.003
Mbps
>>> http-flow    01:33:22      01:33:42      117947   5713018     
2.332 Mbp
>>> ssh-flow     01:33:22      01:33:42      4182     3020     0.002
Mbps
>>> http-flow    01:33:42      01:34:02      118998   5685520     
2.321 Mbp
>>> ssh-flow     01:33:42      01:34:02      11616    9924     0.008
Mbps
>>> http-flow    01:34:02      01:34:22      117084   5725664     
2.337 Mbp
>>> http-flow    01:34:22      01:34:42      119130   5725168     
2.337 Mbp
>>> http-flow    01:34:42      01:35:02      114180   5725168     
2.335 Mbp
>>> http-flow    01:35:02      01:35:22      109230   5725664     
2.333 Mbp
>>> http-flow    01:35:22      01:35:42      116160   5725168     
2.336 Mbp
>>> http-flow    01:35:42      01:36:02      119262   5725168     
2.337 Mbp
>>> http-flow    01:36:02      01:36:22      119196   5725664     
2.337 Mbp
>>> http-flow    01:36:22      01:36:42      117216   5725168     
2.336 Mbp
>>> http-flow    01:36:42      01:37:02      119394   5722636     
2.336 Mbp
>>> http-flow    01:37:02      01:37:22      119526   5725168     
2.337 Mbp
>>> http-flow    01:37:22      01:37:42      119460   5725168     
2.337 Mbp
>>> http-flow    01:37:42      01:38:02      119460   5725664     
2.338 Mbp
>>> http-flow    01:38:02      01:38:22      119724   5725168     
2.337 Mbp
>>> http-flow    01:38:22      01:38:42      119724   5725168     
2.337 Mbp
>>> http-flow    01:38:42      01:39:02      119130   5722636     
2.336 Mbp
>>> http-flow    01:39:02      01:39:22      118866   5725168     
2.337 Mbp
>>> http-flow    01:39:22      01:39:42      116490   5725664     
2.336 Mbp
>>> http-flow    01:39:42      01:40:02      119790   5725168     
2.337 Mbp
>>> http-flow    01:40:02      01:40:22      117678   5725168     
2.337 Mbp
>>> http-flow    01:40:22      01:40:42      118668   5725664     
2.337 Mbp
>>> http-flow    01:40:42      01:41:02      117414   5725168     
2.337 Mbp
>>> http-flow    01:41:02      01:41:22      119790   5725168     
2.337 Mbp
>>> http-flow    01:41:22      01:41:42      119813   5720510     
2.336 Mbp
>>> http-flow    01:41:42      01:42:02      119394   5725664     
2.338 Mbp
>>> http-flow    01:42:02      01:42:22      119724   5722272     
2.336 Mbp
>>> http-flow    01:42:22      01:42:42      119526   5725664     
2.338 Mbp
>>> http-flow    01:42:42      01:43:02      119196   5722140     
2.336 Mbp
>>> http-flow    01:43:02      01:43:22      119394   5725664     
2.338 Mbp
>>> http-flow    01:43:22      01:43:42      119658   5725168     
2.337 Mbp
>>> http-flow    01:43:42      01:44:02      119064   5725168     
2.337 Mbp
>>> http-flow    01:44:02      01:44:22      113256   5676668     
2.315 Mbp
>>> ssh-flow     01:44:02      01:44:22      18414    49646    0.027
Mbps
>>> http-flow    01:44:22      01:44:42      118206   5725664     
2.337 Mbp
>>> http-flow    01:44:42      01:45:02      117282   5722140     
2.335 Mbp
>>> ssh-flow     01:44:42      01:45:02      4698     3544     0.003
Mbps
>>> http-flow    01:45:02      01:45:22      118536   5688284     
2.322 Mbp
>>> ssh-flow     01:45:02      01:45:22      4092     3198     0.002
Mbps
>>> http-flow    01:45:22      01:45:42      119130   5725168     
2.337 Mbp
>>> ssh-flow     01:45:22      01:45:42      1980     1478     0.001
Mbps
>>>
>>> That''s above the flow''s maxbw parameter.  After
that I tried to change the  maxbw of the link with dladm and that brought the
bandwidth down but still not down to 1.2 Mbps.
>>>
>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
>>> LINK         PROPERTY        PERM VALUE          DEFAULT       
POSSIBLE
>>> e1000g0      maxbw           rw       1.228      --             --
>>>
>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>> http-flow    01:46:02      01:46:22      119394   5725168     
2.337 Mbp
>>> ssh-flow     01:46:02      01:46:22      4680     2980     0.003
Mbps
>>> http-flow    01:46:22      01:46:42      94314    4520316     
1.845 Mbp
>>>
>>> Any ideas or is there a subtlety that I''m missing and the
behavior is correct?
>>>
>>> Thanks for the help.
>>>
>>> -Cesar
>>>
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>
>
>

Cesar Delgado

2009-Nov-25 21:49 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Venu,

On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
> 
> Hi, Cesar:
> 
> On Tue, 24 Nov 2009, Cesar Delgado wrote:
> 
>> Venugopal,
>> 
>> I''m sorry if these sounds like basic questions.  I really
appreciate the patience and the help.  Replies in-line.
>> 
>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>> 
>>> 
>>> 
>>> Hi, Cesar:
>>> 
>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>> 
>>>> I''m setting up a server to go to a hosting site where
I have a 1Mbps pipe.  From what I read I know I can''t set the limit to
this as the lowest setting is ~1.2Mbps and this is something that''s
getting worked on in Crossbow2.  I am seeing some strange behavior.
>>>> 
>>>> FIrst I have a question on flowadm''s show-usage
command.  When I try to run show-prop with the name of a flow I get an error. 
The flow exists.  What am I doing wrong?
>>>> 
>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log
http-flow
>>>> flowadm: invalid flow: ''(null)''
>>> 
>>> This is a bug, I have submitted
>>> 
>>> 6904427 flowadm show-usage doesn''t work with a flow name
>> 
>> Thanks for submitting that.  I haven''t been able to find a
link to the bugtracker for Crossbow.  Could you please send me the URL?
> 
> I think it should show up on
> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
> 
>>> 
>>> 
>>>> 
>>>> 
>>>> Ok, now for my problem.  I have the following setting:
>>>> 
>>>> root at myhost:~# flowadm show-flowprop http-flow
>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>> http-flow    maxbw               1.228      --            
1228k
>>>> http-flow    priority        medium         --            
medium
>>>> 
>>>> I ran a test hitting the webserver and I see this:
>>> 
>>> I have the following flow
>>> 
>>> # flowadm show-flow FLOW        LINK        IPADDR                 
PROTO  LPORT   RPORT   DSFLD
>>> tcp-flow    <link>      --                       tcp    --   
--      --
>>> 
>>> 
>>> # flowadm show-flowprop tcp-flow
>>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>>> tcp-flow     maxbw               1.228      --             1228K
tcp-flow     priority        --             --             ?
>>> 
>>> When I send TCP traffic (am using a traffic generator - netperf, to
>>> this machine from a peer) for about 2 mins.
>>> 
>>> On the peer the traffic generator (sender) says I am capped to
about
>>> 1.14 Mbps.
>>> 
>>> Size   Size    Size     Time     Throughput bytes  bytes   bytes   
secs.    10^6bits/sec
>>> 
>>> 49152  49152  49152    120.49      1.14
>>> 
>>> Now, when I try show-usage during the traffic flow on
>>> the machine with the above flow in place (receiver), I am seeing:
>>> 
>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
>>> FLOW         START         END           RBYTES   OBYTES  
BANDWIDTH
>>> tcp-flow     08:51:48      08:52:08      3428658  107802      
1.414 Mbp
>>> tcp-flow     08:52:08      08:52:28      3431198  107802      
1.415 Mbp
>>> tcp-flow     08:52:28      08:52:48      3434614  107888      
1.417 Mbp
>>> tcp-flow     08:52:48      08:53:08      3443298  107802      
1.420 Mbp
>>> tcp-flow     08:53:08      08:53:28      3444324  107802      
1.420 Mbp
>>> tcp-flow     08:53:28      08:53:48      1376806  43576    0.568
Mbps
>>> 
>>> ...
>>> 
>>> I think the difference you see is likely to be because of the time
>>> period when the stats are written to the file (the bandwidth is
computed for every 20 seconds period which might not be exactly in
>>> sync with the bandwidth enforcement period in the kernel) and also
>>> could be because of rounding up etc. But, if you look at the 
entire
>>> duration, it averages to about the configured limit (in the above
>>> example, I think it is about 1.275 Mbps for the 2 min duration)
>> 
>> The way I''m testing it is setting up Apache and then moving
down a file with `wget`.  The use case for this machine is an Apache based app
that serves large files to customers.  That is why I think a `wget` is more
telling of "real" performance than netperf.  I''m running the
test again and on the client side I am seeing usage over the maxbw limit I have
set.  `wget` is reporting about 2Mbps transfer rate which is much closer to what
I was seeing in the show-usage statistics.
>> 
> 
>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>> --10:01:30--  http://sol/myfile.dat
>> Resolving sol... 192.168.69.104
>> Connecting to sol|192.168.69.104|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 1048576000 (1000M)
>> Saving to: `myfile.dat''
>> 
>> 5% [==>                                                      ]
55,530,974   267K/s  eta 60m 44s
>> 
>> 
>>> 
>>> BTW, setting a maxbw for a link (dladm) doesn''t really
impact the
>>> flow as the bandwidth for both are independent.
>> 
>> Thank you for this clarification but I still don''t understand
how I can be seeing ~2Mbps transfer if both the link and the flow are both
capped at 1.2Mbps.
>> 
> 
> Can you try with a higher bandwidth, say 100 Mbps and see what the results
> are when compared to wget''s output?
> 
> Also, another way of manually checking would be to do a
> 	# kstat -c flow -n http-flow
> 
> before and after the wget run and see how many bytes (rbytes) the
> kernel has seen for that flow (assuming there isn''t any other
traffic
> going over the flow), and then determine the bandwidth (you might need
> to get the duration of the wget run pretty close to get the
> right bandwdith value).
> 
> -venu
I changed the flow to be 100Mbps.

root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
http-flow    maxbw             100          --             100 

I also removed the maxbw=1.228 I had set on the link.  When I changed this value
on the link I lost all network to the machine.  Had to go in and manually reboot
it.  The network came up fine.

I created two files one before wget and one after with no other network traffic
except the ssh session.  I created a 1 GB file to transfer.  The output is a bit
confusing to me.

root at myhost:/tmp# ls http-flow-*
http-flow-1.txt  http-flow-2.txt
root at myhost:/tmp# cat http-flow-*
module: unix                            instance: 0     
name:   http-flow                       class:    flow
	crtime                          188662.418688816
	ierrors                         0
	ipackets                        4680
	obytes                          182520974
	oerrors                         0
	opackets                        127521
	rbytes                          308138
	snaptime                        189431.882363589

module: unix                            instance: 0     
name:   http-flow                       class:    flow
	crtime                          188662.418688816
	ierrors                         0
	ipackets                        31262
	obytes                          1281785975
	oerrors                         0
	opackets                        895533
	rbytes                          2062678
	snaptime                        189595.122347726

So this means the network only passed ~1.6 MB of data?

The wget command was telling me it was getting ~80Mbps which is under the
threshold of the flow.

[cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
--13:21:49--  http://sol/myfile.dat
Resolving sol... 192.168.69.104
Connecting to sol|192.168.69.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M)
Saving to: `myfile.dat''

100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s

13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000]


real	1m33.701s
user	0m0.899s
sys	0m23.874s

Maybe a value of 100Mbps is too high for this machine.  I might try with 50Mbps
to see what wget says.

-Cesar

>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> 
>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>>> http-flow    01:32:22      01:32:42      1512     2571    
0.001 Mbps
>>>> ssh-flow     01:32:42      01:33:02      1818     3578    
0.002 Mbps
>>>> http-flow    01:33:02      01:33:22      66917    3165136     
1.292 Mbp
>>>> ssh-flow     01:33:02      01:33:22      3618     5344    
0.003 Mbps
>>>> http-flow    01:33:22      01:33:42      117947   5713018     
2.332 Mbp
>>>> ssh-flow     01:33:22      01:33:42      4182     3020    
0.002 Mbps
>>>> http-flow    01:33:42      01:34:02      118998   5685520     
2.321 Mbp
>>>> ssh-flow     01:33:42      01:34:02      11616    9924    
0.008 Mbps
>>>> http-flow    01:34:02      01:34:22      117084   5725664     
2.337 Mbp
>>>> http-flow    01:34:22      01:34:42      119130   5725168     
2.337 Mbp
>>>> http-flow    01:34:42      01:35:02      114180   5725168     
2.335 Mbp
>>>> http-flow    01:35:02      01:35:22      109230   5725664     
2.333 Mbp
>>>> http-flow    01:35:22      01:35:42      116160   5725168     
2.336 Mbp
>>>> http-flow    01:35:42      01:36:02      119262   5725168     
2.337 Mbp
>>>> http-flow    01:36:02      01:36:22      119196   5725664     
2.337 Mbp
>>>> http-flow    01:36:22      01:36:42      117216   5725168     
2.336 Mbp
>>>> http-flow    01:36:42      01:37:02      119394   5722636     
2.336 Mbp
>>>> http-flow    01:37:02      01:37:22      119526   5725168     
2.337 Mbp
>>>> http-flow    01:37:22      01:37:42      119460   5725168     
2.337 Mbp
>>>> http-flow    01:37:42      01:38:02      119460   5725664     
2.338 Mbp
>>>> http-flow    01:38:02      01:38:22      119724   5725168     
2.337 Mbp
>>>> http-flow    01:38:22      01:38:42      119724   5725168     
2.337 Mbp
>>>> http-flow    01:38:42      01:39:02      119130   5722636     
2.336 Mbp
>>>> http-flow    01:39:02      01:39:22      118866   5725168     
2.337 Mbp
>>>> http-flow    01:39:22      01:39:42      116490   5725664     
2.336 Mbp
>>>> http-flow    01:39:42      01:40:02      119790   5725168     
2.337 Mbp
>>>> http-flow    01:40:02      01:40:22      117678   5725168     
2.337 Mbp
>>>> http-flow    01:40:22      01:40:42      118668   5725664     
2.337 Mbp
>>>> http-flow    01:40:42      01:41:02      117414   5725168     
2.337 Mbp
>>>> http-flow    01:41:02      01:41:22      119790   5725168     
2.337 Mbp
>>>> http-flow    01:41:22      01:41:42      119813   5720510     
2.336 Mbp
>>>> http-flow    01:41:42      01:42:02      119394   5725664     
2.338 Mbp
>>>> http-flow    01:42:02      01:42:22      119724   5722272     
2.336 Mbp
>>>> http-flow    01:42:22      01:42:42      119526   5725664     
2.338 Mbp
>>>> http-flow    01:42:42      01:43:02      119196   5722140     
2.336 Mbp
>>>> http-flow    01:43:02      01:43:22      119394   5725664     
2.338 Mbp
>>>> http-flow    01:43:22      01:43:42      119658   5725168     
2.337 Mbp
>>>> http-flow    01:43:42      01:44:02      119064   5725168     
2.337 Mbp
>>>> http-flow    01:44:02      01:44:22      113256   5676668     
2.315 Mbp
>>>> ssh-flow     01:44:02      01:44:22      18414    49646   
0.027 Mbps
>>>> http-flow    01:44:22      01:44:42      118206   5725664     
2.337 Mbp
>>>> http-flow    01:44:42      01:45:02      117282   5722140     
2.335 Mbp
>>>> ssh-flow     01:44:42      01:45:02      4698     3544    
0.003 Mbps
>>>> http-flow    01:45:02      01:45:22      118536   5688284     
2.322 Mbp
>>>> ssh-flow     01:45:02      01:45:22      4092     3198    
0.002 Mbps
>>>> http-flow    01:45:22      01:45:42      119130   5725168     
2.337 Mbp
>>>> ssh-flow     01:45:22      01:45:42      1980     1478    
0.001 Mbps
>>>> 
>>>> That''s above the flow''s maxbw parameter. 
After that I tried to change the  maxbw of the link with dladm and that brought
the bandwidth down but still not down to 1.2 Mbps.
>>>> 
>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
>>>> LINK         PROPERTY        PERM VALUE          DEFAULT       
POSSIBLE
>>>> e1000g0      maxbw           rw       1.228      --            
--
>>>> 
>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e
11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>>> http-flow    01:46:02      01:46:22      119394   5725168     
2.337 Mbp
>>>> ssh-flow     01:46:02      01:46:22      4680     2980    
0.003 Mbps
>>>> http-flow    01:46:22      01:46:42      94314    4520316     
1.845 Mbp
>>>> 
>>>> Any ideas or is there a subtlety that I''m missing and
the behavior is correct?
>>>> 
>>>> Thanks for the help.
>>>> 
>>>> -Cesar
>>>> 
>>>> _______________________________________________
>>>> crossbow-discuss mailing list
>>>> crossbow-discuss at opensolaris.org
>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>> 
>> 
>>

venugopal iyer

2009-Nov-25 22:05 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

On Wed, 25 Nov 2009, Cesar Delgado wrote:
> Venu,
>
> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>
>>
>> Hi, Cesar:
>>
>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>
>>> Venugopal,
>>>
>>> I''m sorry if these sounds like basic questions.  I really
appreciate the patience and the help.  Replies in-line.
>>>
>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>
>>>>
>>>>
>>>> Hi, Cesar:
>>>>
>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>
>>>>> I''m setting up a server to go to a hosting site
where I have a 1Mbps pipe.  From what I read I know I can''t set the
limit to this as the lowest setting is ~1.2Mbps and this is something
that''s getting worked on in Crossbow2.  I am seeing some strange
behavior.
>>>>>
>>>>> FIrst I have a question on flowadm''s show-usage
command.  When I try to run show-prop with the name of a flow I get an error. 
The flow exists.  What am I doing wrong?
>>>>>
>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log
http-flow
>>>>> flowadm: invalid flow: ''(null)''
>>>>
>>>> This is a bug, I have submitted
>>>>
>>>> 6904427 flowadm show-usage doesn''t work with a flow
name
>>>
>>> Thanks for submitting that.  I haven''t been able to find a
link to the bugtracker for Crossbow.  Could you please send me the URL?
>>
>> I think it should show up on
>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> Ok, now for my problem.  I have the following setting:
>>>>>
>>>>> root at myhost:~# flowadm show-flowprop http-flow
>>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>>> http-flow    maxbw               1.228      --            
1228k
>>>>> http-flow    priority        medium         --            
medium
>>>>>
>>>>> I ran a test hitting the webserver and I see this:
>>>>
>>>> I have the following flow
>>>>
>>>> # flowadm show-flow FLOW        LINK        IPADDR             
PROTO  LPORT   RPORT   DSFLD
>>>> tcp-flow    <link>      --                       tcp   
--      --      --
>>>>
>>>>
>>>> # flowadm show-flowprop tcp-flow
>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>> tcp-flow     maxbw               1.228      --            
1228K tcp-flow     priority        --             --             ?
>>>>
>>>> When I send TCP traffic (am using a traffic generator -
netperf, to
>>>> this machine from a peer) for about 2 mins.
>>>>
>>>> On the peer the traffic generator (sender) says I am capped to
about
>>>> 1.14 Mbps.
>>>>
>>>> Size   Size    Size     Time     Throughput bytes  bytes  
bytes    secs.    10^6bits/sec
>>>>
>>>> 49152  49152  49152    120.49      1.14
>>>>
>>>> Now, when I try show-usage during the traffic flow on
>>>> the machine with the above flow in place (receiver), I am
seeing:
>>>>
>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
>>>> FLOW         START         END           RBYTES   OBYTES  
BANDWIDTH
>>>> tcp-flow     08:51:48      08:52:08      3428658  107802      
1.414 Mbp
>>>> tcp-flow     08:52:08      08:52:28      3431198  107802      
1.415 Mbp
>>>> tcp-flow     08:52:28      08:52:48      3434614  107888      
1.417 Mbp
>>>> tcp-flow     08:52:48      08:53:08      3443298  107802      
1.420 Mbp
>>>> tcp-flow     08:53:08      08:53:28      3444324  107802      
1.420 Mbp
>>>> tcp-flow     08:53:28      08:53:48      1376806  43576   
0.568 Mbps
>>>>
>>>> ...
>>>>
>>>> I think the difference you see is likely to be because of the
time
>>>> period when the stats are written to the file (the bandwidth is
computed for every 20 seconds period which might not be exactly in
>>>> sync with the bandwidth enforcement period in the kernel) and
also
>>>> could be because of rounding up etc. But, if you look at the 
entire
>>>> duration, it averages to about the configured limit (in the
above
>>>> example, I think it is about 1.275 Mbps for the 2 min duration)
>>>
>>> The way I''m testing it is setting up Apache and then
moving down a file with `wget`.  The use case for this machine is an Apache
based app that serves large files to customers.  That is why I think a `wget` is
more telling of "real" performance than netperf.  I''m running
the test again and on the client side I am seeing usage over the maxbw limit I
have set.  `wget` is reporting about 2Mbps transfer rate which is much closer to
what I was seeing in the show-usage statistics.
>>>
>>
>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>> --10:01:30--  http://sol/myfile.dat
>>> Resolving sol... 192.168.69.104
>>> Connecting to sol|192.168.69.104|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 1048576000 (1000M)
>>> Saving to: `myfile.dat''
>>>
>>> 5% [==>                                                      ]
55,530,974   267K/s  eta 60m 44s
>>>
>>>
>>>>
>>>> BTW, setting a maxbw for a link (dladm) doesn''t really
impact the
>>>> flow as the bandwidth for both are independent.
>>>
>>> Thank you for this clarification but I still don''t
understand how I can be seeing ~2Mbps transfer if both the link and the flow are
both capped at 1.2Mbps.
>>>
>>
>> Can you try with a higher bandwidth, say 100 Mbps and see what the
results
>> are when compared to wget''s output?
>>
>> Also, another way of manually checking would be to do a
>> 	# kstat -c flow -n http-flow
>>
>> before and after the wget run and see how many bytes (rbytes) the
>> kernel has seen for that flow (assuming there isn''t any other
traffic
>> going over the flow), and then determine the bandwidth (you might need
>> to get the duration of the wget run pretty close to get the
>> right bandwdith value).
>>
>> -venu
>
> I changed the flow to be 100Mbps.
>
> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
> http-flow    maxbw             100          --             100
>
> I also removed the maxbw=1.228 I had set on the link.  When I changed this
value on the link I lost all network to the machine.  Had to go in and manually
reboot it.  The network came up fine.
>
> I created two files one before wget and one after with no other network
traffic except the ssh session.  I created a 1 GB file to transfer.  The output
is a bit confusing to me.
>
> root at myhost:/tmp# ls http-flow-*
> http-flow-1.txt  http-flow-2.txt
> root at myhost:/tmp# cat http-flow-*
> module: unix                            instance: 0
> name:   http-flow                       class:    flow
> 	crtime                          188662.418688816
> 	ierrors                         0
> 	ipackets                        4680
> 	obytes                          182520974
> 	oerrors                         0
> 	opackets                        127521
> 	rbytes                          308138
> 	snaptime                        189431.882363589
>
> module: unix                            instance: 0
> name:   http-flow                       class:    flow
> 	crtime                          188662.418688816
> 	ierrors                         0
> 	ipackets                        31262
> 	obytes                          1281785975
> 	oerrors                         0
> 	opackets                        895533
> 	rbytes                          2062678
> 	snaptime                        189595.122347726
>
> So this means the network only passed ~1.6 MB of data?
is this on the outbound side or inbound?

looks like it has a total of 8794120008 bits in the outbound side 
(obytes) and 14036320 bits on the inbound (rbytes). What is the duration
93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.

>
> The wget command was telling me it was getting ~80Mbps which is under the
threshold of the flow.
>
> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
> --13:21:49--  http://sol/myfile.dat
> Resolving sol... 192.168.69.104
> Connecting to sol|192.168.69.104|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 1048576000 (1000M)
> Saving to: `myfile.dat''
>
> 100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>
> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000]
>
>
> real	1m33.701s
> user	0m0.899s
> sys	0m23.874s
>
> Maybe a value of 100Mbps is too high for this machine.  I might try with
50Mbps to see what wget says.
>
OK,

-venu
> -Cesar
>
>
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22
-e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>>>> http-flow    01:32:22      01:32:42      1512     2571    
0.001 Mbps
>>>>> ssh-flow     01:32:42      01:33:02      1818     3578    
0.002 Mbps
>>>>> http-flow    01:33:02      01:33:22      66917    3165136  
1.292 Mbp
>>>>> ssh-flow     01:33:02      01:33:22      3618     5344    
0.003 Mbps
>>>>> http-flow    01:33:22      01:33:42      117947   5713018  
2.332 Mbp
>>>>> ssh-flow     01:33:22      01:33:42      4182     3020    
0.002 Mbps
>>>>> http-flow    01:33:42      01:34:02      118998   5685520  
2.321 Mbp
>>>>> ssh-flow     01:33:42      01:34:02      11616    9924    
0.008 Mbps
>>>>> http-flow    01:34:02      01:34:22      117084   5725664  
2.337 Mbp
>>>>> http-flow    01:34:22      01:34:42      119130   5725168  
2.337 Mbp
>>>>> http-flow    01:34:42      01:35:02      114180   5725168  
2.335 Mbp
>>>>> http-flow    01:35:02      01:35:22      109230   5725664  
2.333 Mbp
>>>>> http-flow    01:35:22      01:35:42      116160   5725168  
2.336 Mbp
>>>>> http-flow    01:35:42      01:36:02      119262   5725168  
2.337 Mbp
>>>>> http-flow    01:36:02      01:36:22      119196   5725664  
2.337 Mbp
>>>>> http-flow    01:36:22      01:36:42      117216   5725168  
2.336 Mbp
>>>>> http-flow    01:36:42      01:37:02      119394   5722636  
2.336 Mbp
>>>>> http-flow    01:37:02      01:37:22      119526   5725168  
2.337 Mbp
>>>>> http-flow    01:37:22      01:37:42      119460   5725168  
2.337 Mbp
>>>>> http-flow    01:37:42      01:38:02      119460   5725664  
2.338 Mbp
>>>>> http-flow    01:38:02      01:38:22      119724   5725168  
2.337 Mbp
>>>>> http-flow    01:38:22      01:38:42      119724   5725168  
2.337 Mbp
>>>>> http-flow    01:38:42      01:39:02      119130   5722636  
2.336 Mbp
>>>>> http-flow    01:39:02      01:39:22      118866   5725168  
2.337 Mbp
>>>>> http-flow    01:39:22      01:39:42      116490   5725664  
2.336 Mbp
>>>>> http-flow    01:39:42      01:40:02      119790   5725168  
2.337 Mbp
>>>>> http-flow    01:40:02      01:40:22      117678   5725168  
2.337 Mbp
>>>>> http-flow    01:40:22      01:40:42      118668   5725664  
2.337 Mbp
>>>>> http-flow    01:40:42      01:41:02      117414   5725168  
2.337 Mbp
>>>>> http-flow    01:41:02      01:41:22      119790   5725168  
2.337 Mbp
>>>>> http-flow    01:41:22      01:41:42      119813   5720510  
2.336 Mbp
>>>>> http-flow    01:41:42      01:42:02      119394   5725664  
2.338 Mbp
>>>>> http-flow    01:42:02      01:42:22      119724   5722272  
2.336 Mbp
>>>>> http-flow    01:42:22      01:42:42      119526   5725664  
2.338 Mbp
>>>>> http-flow    01:42:42      01:43:02      119196   5722140  
2.336 Mbp
>>>>> http-flow    01:43:02      01:43:22      119394   5725664  
2.338 Mbp
>>>>> http-flow    01:43:22      01:43:42      119658   5725168  
2.337 Mbp
>>>>> http-flow    01:43:42      01:44:02      119064   5725168  
2.337 Mbp
>>>>> http-flow    01:44:02      01:44:22      113256   5676668  
2.315 Mbp
>>>>> ssh-flow     01:44:02      01:44:22      18414    49646   
0.027 Mbps
>>>>> http-flow    01:44:22      01:44:42      118206   5725664  
2.337 Mbp
>>>>> http-flow    01:44:42      01:45:02      117282   5722140  
2.335 Mbp
>>>>> ssh-flow     01:44:42      01:45:02      4698     3544    
0.003 Mbps
>>>>> http-flow    01:45:02      01:45:22      118536   5688284  
2.322 Mbp
>>>>> ssh-flow     01:45:02      01:45:22      4092     3198    
0.002 Mbps
>>>>> http-flow    01:45:22      01:45:42      119130   5725168  
2.337 Mbp
>>>>> ssh-flow     01:45:22      01:45:42      1980     1478    
0.001 Mbps
>>>>>
>>>>> That''s above the flow''s maxbw parameter. 
After that I tried to change the  maxbw of the link with dladm and that brought
the bandwidth down but still not down to 1.2 Mbps.
>>>>>
>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
>>>>> LINK         PROPERTY        PERM VALUE          DEFAULT   
POSSIBLE
>>>>> e1000g0      maxbw           rw       1.228      --        
--
>>>>>
>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02
-e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW"
>>>>> http-flow    01:46:02      01:46:22      119394   5725168  
2.337 Mbp
>>>>> ssh-flow     01:46:02      01:46:22      4680     2980    
0.003 Mbps
>>>>> http-flow    01:46:22      01:46:42      94314    4520316  
1.845 Mbp
>>>>>
>>>>> Any ideas or is there a subtlety that I''m missing
and the behavior is correct?
>>>>>
>>>>> Thanks for the help.
>>>>>
>>>>> -Cesar
>>>>>
>>>>> _______________________________________________
>>>>> crossbow-discuss mailing list
>>>>> crossbow-discuss at opensolaris.org
>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>
>>>
>>>
>
>

Cesar Delgado

2009-Nov-25 23:31 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
> 
> 
> 
> On Wed, 25 Nov 2009, Cesar Delgado wrote:
> 
>> Venu,
>> 
>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>> 
>>> 
>>> Hi, Cesar:
>>> 
>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>> 
>>>> Venugopal,
>>>> 
>>>> I''m sorry if these sounds like basic questions.  I
really appreciate the patience and the help.  Replies in-line.
>>>> 
>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> Hi, Cesar:
>>>>> 
>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>> 
>>>>>> I''m setting up a server to go to a hosting
site where I have a 1Mbps pipe.  From what I read I know I can''t set
the limit to this as the lowest setting is ~1.2Mbps and this is something
that''s getting worked on in Crossbow2.  I am seeing some strange
behavior.
>>>>>> 
>>>>>> FIrst I have a question on flowadm''s
show-usage command.  When I try to run show-prop with the name of a flow I get
an error.  The flow exists.  What am I doing wrong?
>>>>>> 
>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>> flowadm: invalid flow: ''(null)''
>>>>> 
>>>>> This is a bug, I have submitted
>>>>> 
>>>>> 6904427 flowadm show-usage doesn''t work with a
flow name
>>>> 
>>>> Thanks for submitting that.  I haven''t been able to
find a link to the bugtracker for Crossbow.  Could you please send me the URL?
>>> 
>>> I think it should show up on
>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Ok, now for my problem.  I have the following setting:
>>>>>> 
>>>>>> root at myhost:~# flowadm show-flowprop http-flow
>>>>>> FLOW         PROPERTY        VALUE          DEFAULT    
POSSIBLE
>>>>>> http-flow    maxbw               1.228      --         
1228k
>>>>>> http-flow    priority        medium         --         
medium
>>>>>> 
>>>>>> I ran a test hitting the webserver and I see this:
>>>>> 
>>>>> I have the following flow
>>>>> 
>>>>> # flowadm show-flow FLOW        LINK        IPADDR         
PROTO  LPORT   RPORT   DSFLD
>>>>> tcp-flow    <link>      --                       tcp 
--      --      --
>>>>> 
>>>>> 
>>>>> # flowadm show-flowprop tcp-flow
>>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>>> tcp-flow     maxbw               1.228      --            
1228K tcp-flow     priority        --             --             ?
>>>>> 
>>>>> When I send TCP traffic (am using a traffic generator -
netperf, to
>>>>> this machine from a peer) for about 2 mins.
>>>>> 
>>>>> On the peer the traffic generator (sender) says I am capped
to about
>>>>> 1.14 Mbps.
>>>>> 
>>>>> Size   Size    Size     Time     Throughput bytes  bytes  
bytes    secs.    10^6bits/sec
>>>>> 
>>>>> 49152  49152  49152    120.49      1.14
>>>>> 
>>>>> Now, when I try show-usage during the traffic flow on
>>>>> the machine with the above flow in place (receiver), I am
seeing:
>>>>> 
>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
>>>>> FLOW         START         END           RBYTES   OBYTES  
BANDWIDTH
>>>>> tcp-flow     08:51:48      08:52:08      3428658  107802   
1.414 Mbp
>>>>> tcp-flow     08:52:08      08:52:28      3431198  107802   
1.415 Mbp
>>>>> tcp-flow     08:52:28      08:52:48      3434614  107888   
1.417 Mbp
>>>>> tcp-flow     08:52:48      08:53:08      3443298  107802   
1.420 Mbp
>>>>> tcp-flow     08:53:08      08:53:28      3444324  107802   
1.420 Mbp
>>>>> tcp-flow     08:53:28      08:53:48      1376806  43576   
0.568 Mbps
>>>>> 
>>>>> ...
>>>>> 
>>>>> I think the difference you see is likely to be because of
the time
>>>>> period when the stats are written to the file (the
bandwidth is computed for every 20 seconds period which might not be exactly in
>>>>> sync with the bandwidth enforcement period in the kernel)
and also
>>>>> could be because of rounding up etc. But, if you look at
the  entire
>>>>> duration, it averages to about the configured limit (in the
above
>>>>> example, I think it is about 1.275 Mbps for the 2 min
duration)
>>>> 
>>>> The way I''m testing it is setting up Apache and then
moving down a file with `wget`.  The use case for this machine is an Apache
based app that serves large files to customers.  That is why I think a `wget` is
more telling of "real" performance than netperf.  I''m running
the test again and on the client side I am seeing usage over the maxbw limit I
have set.  `wget` is reporting about 2Mbps transfer rate which is much closer to
what I was seeing in the show-usage statistics.
>>>> 
>>> 
>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>> --10:01:30--  http://sol/myfile.dat
>>>> Resolving sol... 192.168.69.104
>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>> HTTP request sent, awaiting response... 200 OK
>>>> Length: 1048576000 (1000M)
>>>> Saving to: `myfile.dat''
>>>> 
>>>> 5% [==>                                                     
] 55,530,974   267K/s  eta 60m 44s
>>>> 
>>>> 
>>>>> 
>>>>> BTW, setting a maxbw for a link (dladm) doesn''t
really impact the
>>>>> flow as the bandwidth for both are independent.
>>>> 
>>>> Thank you for this clarification but I still don''t
understand how I can be seeing ~2Mbps transfer if both the link and the flow are
both capped at 1.2Mbps.
>>>> 
>>> 
>>> Can you try with a higher bandwidth, say 100 Mbps and see what the
results
>>> are when compared to wget''s output?
>>> 
>>> Also, another way of manually checking would be to do a
>>> 	# kstat -c flow -n http-flow
>>> 
>>> before and after the wget run and see how many bytes (rbytes) the
>>> kernel has seen for that flow (assuming there isn''t any
other traffic
>>> going over the flow), and then determine the bandwidth (you might
need
>>> to get the duration of the wget run pretty close to get the
>>> right bandwdith value).
>>> 
>>> -venu
>> 
>> I changed the flow to be 100Mbps.
>> 
>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> http-flow    maxbw             100          --             100
>> 
>> I also removed the maxbw=1.228 I had set on the link.  When I changed
this value on the link I lost all network to the machine.  Had to go in and
manually reboot it.  The network came up fine.
>> 
>> I created two files one before wget and one after with no other network
traffic except the ssh session.  I created a 1 GB file to transfer.  The output
is a bit confusing to me.
>> 
>> root at myhost:/tmp# ls http-flow-*
>> http-flow-1.txt  http-flow-2.txt
>> root at myhost:/tmp# cat http-flow-*
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        4680
>> 	obytes                          182520974
>> 	oerrors                         0
>> 	opackets                        127521
>> 	rbytes                          308138
>> 	snaptime                        189431.882363589
>> 
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        31262
>> 	obytes                          1281785975
>> 	oerrors                         0
>> 	opackets                        895533
>> 	rbytes                          2062678
>> 	snaptime                        189595.122347726
>> 
>> So this means the network only passed ~1.6 MB of data?
> 
> is this on the outbound side or inbound?
> 
> looks like it has a total of 8794120008 bits in the outbound side (obytes)
and 14036320 bits on the inbound (rbytes). What is the duration
> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.
> 
Sorry, outbound.  I was looking at the wrong row.  With the flow set at 50
I''m getting getting ~37 Mbps.  In my mind the flow is not respecting
the value I''m giving it.  With the maxbw set to 100 I could get ~80
Mbps and now I''m only seeing ~37Mbps.   Could it be something in the
way Crossbow is averaging the actually bits and bytes going down the pipe or is
it me not understanding the system?

root at myhost:/tmp# flowadm show-flowprop http-flow
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
http-flow    maxbw              50          --             50 
http-flow    priority        medium         --             medium 

root at myhost:/tmp# cat http-flow-*
module: unix                            instance: 0     
name:   http-flow                       class:    flow
	crtime                          188662.418688816
	ierrors                         0
	ipackets                        32346
	obytes                          1300129135
	oerrors                         0
	opackets                        908351
	rbytes                          2133486
	snaptime                        191261.607310645

module: unix                            instance: 0     
name:   http-flow                       class:    flow
	crtime                          188662.418688816
	ierrors                         0
	ipackets                        85361
	obytes                          2399401106
	oerrors                         0
	opackets                        1676468
	rbytes                          5632604
	snaptime                        192621.873203347


[cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
--13:52:20--  http://sol/myfile.dat
Resolving sol... 192.168.69.104
Connecting to sol|192.168.69.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M)
Saving to: `myfile.dat''

100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s

13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000]


real	3m44.364s
user	0m1.042s
sys	0m24.065s


> 
>> 
>> The wget command was telling me it was getting ~80Mbps which is under
the threshold of the flow.
>> 
>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>> --13:21:49--  http://sol/myfile.dat
>> Resolving sol... 192.168.69.104
>> Connecting to sol|192.168.69.104|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 1048576000 (1000M)
>> Saving to: `myfile.dat''
>> 
>> 100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>> 
>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>> 
>> 
>> real	1m33.701s
>> user	0m0.899s
>> sys	0m23.874s
>> 
>> Maybe a value of 100Mbps is too high for this machine.  I might try
with 50Mbps to see what wget says.
>> 
> 
> OK,
> 
> -venu
> 
>> -Cesar
>> 
>> 
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> 
>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>> http-flow    01:32:22      01:32:42      1512     2571 
0.001 Mbps
>>>>>> ssh-flow     01:32:42      01:33:02      1818     3578 
0.002 Mbps
>>>>>> http-flow    01:33:02      01:33:22      66917   
3165136      1.292 Mbp
>>>>>> ssh-flow     01:33:02      01:33:22      3618     5344 
0.003 Mbps
>>>>>> http-flow    01:33:22      01:33:42      117947  
5713018      2.332 Mbp
>>>>>> ssh-flow     01:33:22      01:33:42      4182     3020 
0.002 Mbps
>>>>>> http-flow    01:33:42      01:34:02      118998  
5685520      2.321 Mbp
>>>>>> ssh-flow     01:33:42      01:34:02      11616    9924 
0.008 Mbps
>>>>>> http-flow    01:34:02      01:34:22      117084  
5725664      2.337 Mbp
>>>>>> http-flow    01:34:22      01:34:42      119130  
5725168      2.337 Mbp
>>>>>> http-flow    01:34:42      01:35:02      114180  
5725168      2.335 Mbp
>>>>>> http-flow    01:35:02      01:35:22      109230  
5725664      2.333 Mbp
>>>>>> http-flow    01:35:22      01:35:42      116160  
5725168      2.336 Mbp
>>>>>> http-flow    01:35:42      01:36:02      119262  
5725168      2.337 Mbp
>>>>>> http-flow    01:36:02      01:36:22      119196  
5725664      2.337 Mbp
>>>>>> http-flow    01:36:22      01:36:42      117216  
5725168      2.336 Mbp
>>>>>> http-flow    01:36:42      01:37:02      119394  
5722636      2.336 Mbp
>>>>>> http-flow    01:37:02      01:37:22      119526  
5725168      2.337 Mbp
>>>>>> http-flow    01:37:22      01:37:42      119460  
5725168      2.337 Mbp
>>>>>> http-flow    01:37:42      01:38:02      119460  
5725664      2.338 Mbp
>>>>>> http-flow    01:38:02      01:38:22      119724  
5725168      2.337 Mbp
>>>>>> http-flow    01:38:22      01:38:42      119724  
5725168      2.337 Mbp
>>>>>> http-flow    01:38:42      01:39:02      119130  
5722636      2.336 Mbp
>>>>>> http-flow    01:39:02      01:39:22      118866  
5725168      2.337 Mbp
>>>>>> http-flow    01:39:22      01:39:42      116490  
5725664      2.336 Mbp
>>>>>> http-flow    01:39:42      01:40:02      119790  
5725168      2.337 Mbp
>>>>>> http-flow    01:40:02      01:40:22      117678  
5725168      2.337 Mbp
>>>>>> http-flow    01:40:22      01:40:42      118668  
5725664      2.337 Mbp
>>>>>> http-flow    01:40:42      01:41:02      117414  
5725168      2.337 Mbp
>>>>>> http-flow    01:41:02      01:41:22      119790  
5725168      2.337 Mbp
>>>>>> http-flow    01:41:22      01:41:42      119813  
5720510      2.336 Mbp
>>>>>> http-flow    01:41:42      01:42:02      119394  
5725664      2.338 Mbp
>>>>>> http-flow    01:42:02      01:42:22      119724  
5722272      2.336 Mbp
>>>>>> http-flow    01:42:22      01:42:42      119526  
5725664      2.338 Mbp
>>>>>> http-flow    01:42:42      01:43:02      119196  
5722140      2.336 Mbp
>>>>>> http-flow    01:43:02      01:43:22      119394  
5725664      2.338 Mbp
>>>>>> http-flow    01:43:22      01:43:42      119658  
5725168      2.337 Mbp
>>>>>> http-flow    01:43:42      01:44:02      119064  
5725168      2.337 Mbp
>>>>>> http-flow    01:44:02      01:44:22      113256  
5676668      2.315 Mbp
>>>>>> ssh-flow     01:44:02      01:44:22      18414    49646
0.027 Mbps
>>>>>> http-flow    01:44:22      01:44:42      118206  
5725664      2.337 Mbp
>>>>>> http-flow    01:44:42      01:45:02      117282  
5722140      2.335 Mbp
>>>>>> ssh-flow     01:44:42      01:45:02      4698     3544 
0.003 Mbps
>>>>>> http-flow    01:45:02      01:45:22      118536  
5688284      2.322 Mbp
>>>>>> ssh-flow     01:45:02      01:45:22      4092     3198 
0.002 Mbps
>>>>>> http-flow    01:45:22      01:45:42      119130  
5725168      2.337 Mbp
>>>>>> ssh-flow     01:45:22      01:45:42      1980     1478 
0.001 Mbps
>>>>>> 
>>>>>> That''s above the flow''s maxbw
parameter.  After that I tried to change the  maxbw of the link with dladm and
that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>> 
>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0
>>>>>> LINK         PROPERTY        PERM VALUE         
DEFAULT        POSSIBLE
>>>>>> e1000g0      maxbw           rw       1.228      --    
--
>>>>>> 
>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>> http-flow    01:46:02      01:46:22      119394  
5725168      2.337 Mbp
>>>>>> ssh-flow     01:46:02      01:46:22      4680     2980 
0.003 Mbps
>>>>>> http-flow    01:46:22      01:46:42      94314   
4520316      1.845 Mbp
>>>>>> 
>>>>>> Any ideas or is there a subtlety that I''m
missing and the behavior is correct?
>>>>>> 
>>>>>> Thanks for the help.
>>>>>> 
>>>>>> -Cesar
>>>>>> 
>>>>>> _______________________________________________
>>>>>> crossbow-discuss mailing list
>>>>>> crossbow-discuss at opensolaris.org
>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>> 
>>>> 
>>>> 
>> 
>>

venugopal iyer

2009-Nov-25 23:39 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Hi, Cesar:

On Wed, 25 Nov 2009, Cesar Delgado wrote:
>
> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>
>>
>>
>>
>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>
>>> Venu,
>>>
>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>
>>>>
>>>> Hi, Cesar:
>>>>
>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>
>>>>> Venugopal,
>>>>>
>>>>> I''m sorry if these sounds like basic questions.  I
really appreciate the patience and the help.  Replies in-line.
>>>>>
>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi, Cesar:
>>>>>>
>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>
>>>>>>> I''m setting up a server to go to a hosting
site where I have a 1Mbps pipe.  From what I read I know I can''t set
the limit to this as the lowest setting is ~1.2Mbps and this is something
that''s getting worked on in Crossbow2.  I am seeing some strange
behavior.
>>>>>>>
>>>>>>> FIrst I have a question on flowadm''s
show-usage command.  When I try to run show-prop with the name of a flow I get
an error.  The flow exists.  What am I doing wrong?
>>>>>>>
>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>> flowadm: invalid flow: ''(null)''
>>>>>>
>>>>>> This is a bug, I have submitted
>>>>>>
>>>>>> 6904427 flowadm show-usage doesn''t work with a
flow name
>>>>>
>>>>> Thanks for submitting that.  I haven''t been able
to find a link to the bugtracker for Crossbow.  Could you please send me the
URL?
>>>>
>>>> I think it should show up on
>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Ok, now for my problem.  I have the following
setting:
>>>>>>>
>>>>>>> root at myhost:~# flowadm show-flowprop http-flow
>>>>>>> FLOW         PROPERTY        VALUE          DEFAULT
POSSIBLE
>>>>>>> http-flow    maxbw               1.228      --     
1228k
>>>>>>> http-flow    priority        medium         --     
medium
>>>>>>>
>>>>>>> I ran a test hitting the webserver and I see this:
>>>>>>
>>>>>> I have the following flow
>>>>>>
>>>>>> # flowadm show-flow FLOW        LINK        IPADDR     
PROTO  LPORT   RPORT   DSFLD
>>>>>> tcp-flow    <link>      --                      
tcp    --      --      --
>>>>>>
>>>>>>
>>>>>> # flowadm show-flowprop tcp-flow
>>>>>> FLOW         PROPERTY        VALUE          DEFAULT    
POSSIBLE
>>>>>> tcp-flow     maxbw               1.228      --         
1228K tcp-flow     priority        --             --             ?
>>>>>>
>>>>>> When I send TCP traffic (am using a traffic generator -
netperf, to
>>>>>> this machine from a peer) for about 2 mins.
>>>>>>
>>>>>> On the peer the traffic generator (sender) says I am
capped to about
>>>>>> 1.14 Mbps.
>>>>>>
>>>>>> Size   Size    Size     Time     Throughput bytes 
bytes   bytes    secs.    10^6bits/sec
>>>>>>
>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>
>>>>>> Now, when I try show-usage during the traffic flow on
>>>>>> the machine with the above flow in place (receiver), I
am seeing:
>>>>>>
>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow
>>>>>> FLOW         START         END           RBYTES  
OBYTES   BANDWIDTH
>>>>>> tcp-flow     08:51:48      08:52:08      3428658 
107802       1.414 Mbp
>>>>>> tcp-flow     08:52:08      08:52:28      3431198 
107802       1.415 Mbp
>>>>>> tcp-flow     08:52:28      08:52:48      3434614 
107888       1.417 Mbp
>>>>>> tcp-flow     08:52:48      08:53:08      3443298 
107802       1.420 Mbp
>>>>>> tcp-flow     08:53:08      08:53:28      3444324 
107802       1.420 Mbp
>>>>>> tcp-flow     08:53:28      08:53:48      1376806  43576
0.568 Mbps
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> I think the difference you see is likely to be because
of the time
>>>>>> period when the stats are written to the file (the
bandwidth is computed for every 20 seconds period which might not be exactly in
>>>>>> sync with the bandwidth enforcement period in the
kernel) and also
>>>>>> could be because of rounding up etc. But, if you look
at the  entire
>>>>>> duration, it averages to about the configured limit (in
the above
>>>>>> example, I think it is about 1.275 Mbps for the 2 min
duration)
>>>>>
>>>>> The way I''m testing it is setting up Apache and
then moving down a file with `wget`.  The use case for this machine is an Apache
based app that serves large files to customers.  That is why I think a `wget` is
more telling of "real" performance than netperf.  I''m running
the test again and on the client side I am seeing usage over the maxbw limit I
have set.  `wget` is reporting about 2Mbps transfer rate which is much closer to
what I was seeing in the show-usage statistics.
>>>>>
>>>>
>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>> --10:01:30--  http://sol/myfile.dat
>>>>> Resolving sol... 192.168.69.104
>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>> HTTP request sent, awaiting response... 200 OK
>>>>> Length: 1048576000 (1000M)
>>>>> Saving to: `myfile.dat''
>>>>>
>>>>> 5% [==>                                                 
] 55,530,974   267K/s  eta 60m 44s
>>>>>
>>>>>
>>>>>>
>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>> flow as the bandwidth for both are independent.
>>>>>
>>>>> Thank you for this clarification but I still don''t
understand how I can be seeing ~2Mbps transfer if both the link and the flow are
both capped at 1.2Mbps.
>>>>>
>>>>
>>>> Can you try with a higher bandwidth, say 100 Mbps and see what
the results
>>>> are when compared to wget''s output?
>>>>
>>>> Also, another way of manually checking would be to do a
>>>> 	# kstat -c flow -n http-flow
>>>>
>>>> before and after the wget run and see how many bytes (rbytes)
the
>>>> kernel has seen for that flow (assuming there isn''t
any other traffic
>>>> going over the flow), and then determine the bandwidth (you
might need
>>>> to get the duration of the wget run pretty close to get the
>>>> right bandwdith value).
>>>>
>>>> -venu
>>>
>>> I changed the flow to be 100Mbps.
>>>
>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
>>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>>> http-flow    maxbw             100          --             100
>>>
>>> I also removed the maxbw=1.228 I had set on the link.  When I
changed this value on the link I lost all network to the machine.  Had to go in
and manually reboot it.  The network came up fine.
>>>
>>> I created two files one before wget and one after with no other
network traffic except the ssh session.  I created a 1 GB file to transfer.  The
output is a bit confusing to me.
>>>
>>> root at myhost:/tmp# ls http-flow-*
>>> http-flow-1.txt  http-flow-2.txt
>>> root at myhost:/tmp# cat http-flow-*
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        4680
>>> 	obytes                          182520974
>>> 	oerrors                         0
>>> 	opackets                        127521
>>> 	rbytes                          308138
>>> 	snaptime                        189431.882363589
>>>
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        31262
>>> 	obytes                          1281785975
>>> 	oerrors                         0
>>> 	opackets                        895533
>>> 	rbytes                          2062678
>>> 	snaptime                        189595.122347726
>>>
>>> So this means the network only passed ~1.6 MB of data?
>>
>> is this on the outbound side or inbound?
>>
>> looks like it has a total of 8794120008 bits in the outbound side
(obytes) and 14036320 bits on the inbound (rbytes). What is the duration
>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.
>>
>
> Sorry, outbound.  I was looking at the wrong row.  With the flow set at 50
I''m getting getting ~37 Mbps.  In my mind the flow is not respecting
the value I''m giving it.  With the maxbw set to 100 I could get ~80
Mbps and now I''m only seeing ~37Mbps.   Could it be something in the
way Crossbow is averaging the actually bits and bytes going down the pipe or is
it me not understanding the system?
For TCP you might get a little less than the set bandwidth as TCP
would do flow control too (i.e if for bandwidth reasons ACKs come back
slower or packets get dropped, TCP will react and will slow down). Also,
note that the current property (maxbw) is a *limit* which means we will
ensure that the flow will not get *more* than what you have asked for.
Going forwards we are looking at adding guarantee where in the flow would
get what it asked for.

-venu
>
> root at myhost:/tmp# flowadm show-flowprop http-flow
> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
> http-flow    maxbw              50          --             50
> http-flow    priority        medium         --             medium
>
> root at myhost:/tmp# cat http-flow-*
> module: unix                            instance: 0
> name:   http-flow                       class:    flow
> 	crtime                          188662.418688816
> 	ierrors                         0
> 	ipackets                        32346
> 	obytes                          1300129135
> 	oerrors                         0
> 	opackets                        908351
> 	rbytes                          2133486
> 	snaptime                        191261.607310645
>
> module: unix                            instance: 0
> name:   http-flow                       class:    flow
> 	crtime                          188662.418688816
> 	ierrors                         0
> 	ipackets                        85361
> 	obytes                          2399401106
> 	oerrors                         0
> 	opackets                        1676468
> 	rbytes                          5632604
> 	snaptime                        192621.873203347
>
>
> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
> --13:52:20--  http://sol/myfile.dat
> Resolving sol... 192.168.69.104
> Connecting to sol|192.168.69.104|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 1048576000 (1000M)
> Saving to: `myfile.dat''
>
> 100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>
> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000]
>
>
> real	3m44.364s
> user	0m1.042s
> sys	0m24.065s
>
>
>
>>
>>>
>>> The wget command was telling me it was getting ~80Mbps which is
under the threshold of the flow.
>>>
>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>> --13:21:49--  http://sol/myfile.dat
>>> Resolving sol... 192.168.69.104
>>> Connecting to sol|192.168.69.104|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 1048576000 (1000M)
>>> Saving to: `myfile.dat''
>>>
>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>
>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>
>>>
>>> real	1m33.701s
>>> user	0m0.899s
>>> sys	0m23.874s
>>>
>>> Maybe a value of 100Mbps is too high for this machine.  I might try
with 50Mbps to see what wget says.
>>>
>>
>> OK,
>>
>> -venu
>>
>>> -Cesar
>>>
>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>> http-flow    01:32:22      01:32:42      1512    
2571     0.001 Mbps
>>>>>>> ssh-flow     01:32:42      01:33:02      1818    
3578     0.002 Mbps
>>>>>>> http-flow    01:33:02      01:33:22      66917   
3165136      1.292 Mbp
>>>>>>> ssh-flow     01:33:02      01:33:22      3618    
5344     0.003 Mbps
>>>>>>> http-flow    01:33:22      01:33:42      117947  
5713018      2.332 Mbp
>>>>>>> ssh-flow     01:33:22      01:33:42      4182    
3020     0.002 Mbps
>>>>>>> http-flow    01:33:42      01:34:02      118998  
5685520      2.321 Mbp
>>>>>>> ssh-flow     01:33:42      01:34:02      11616   
9924     0.008 Mbps
>>>>>>> http-flow    01:34:02      01:34:22      117084  
5725664      2.337 Mbp
>>>>>>> http-flow    01:34:22      01:34:42      119130  
5725168      2.337 Mbp
>>>>>>> http-flow    01:34:42      01:35:02      114180  
5725168      2.335 Mbp
>>>>>>> http-flow    01:35:02      01:35:22      109230  
5725664      2.333 Mbp
>>>>>>> http-flow    01:35:22      01:35:42      116160  
5725168      2.336 Mbp
>>>>>>> http-flow    01:35:42      01:36:02      119262  
5725168      2.337 Mbp
>>>>>>> http-flow    01:36:02      01:36:22      119196  
5725664      2.337 Mbp
>>>>>>> http-flow    01:36:22      01:36:42      117216  
5725168      2.336 Mbp
>>>>>>> http-flow    01:36:42      01:37:02      119394  
5722636      2.336 Mbp
>>>>>>> http-flow    01:37:02      01:37:22      119526  
5725168      2.337 Mbp
>>>>>>> http-flow    01:37:22      01:37:42      119460  
5725168      2.337 Mbp
>>>>>>> http-flow    01:37:42      01:38:02      119460  
5725664      2.338 Mbp
>>>>>>> http-flow    01:38:02      01:38:22      119724  
5725168      2.337 Mbp
>>>>>>> http-flow    01:38:22      01:38:42      119724  
5725168      2.337 Mbp
>>>>>>> http-flow    01:38:42      01:39:02      119130  
5722636      2.336 Mbp
>>>>>>> http-flow    01:39:02      01:39:22      118866  
5725168      2.337 Mbp
>>>>>>> http-flow    01:39:22      01:39:42      116490  
5725664      2.336 Mbp
>>>>>>> http-flow    01:39:42      01:40:02      119790  
5725168      2.337 Mbp
>>>>>>> http-flow    01:40:02      01:40:22      117678  
5725168      2.337 Mbp
>>>>>>> http-flow    01:40:22      01:40:42      118668  
5725664      2.337 Mbp
>>>>>>> http-flow    01:40:42      01:41:02      117414  
5725168      2.337 Mbp
>>>>>>> http-flow    01:41:02      01:41:22      119790  
5725168      2.337 Mbp
>>>>>>> http-flow    01:41:22      01:41:42      119813  
5720510      2.336 Mbp
>>>>>>> http-flow    01:41:42      01:42:02      119394  
5725664      2.338 Mbp
>>>>>>> http-flow    01:42:02      01:42:22      119724  
5722272      2.336 Mbp
>>>>>>> http-flow    01:42:22      01:42:42      119526  
5725664      2.338 Mbp
>>>>>>> http-flow    01:42:42      01:43:02      119196  
5722140      2.336 Mbp
>>>>>>> http-flow    01:43:02      01:43:22      119394  
5725664      2.338 Mbp
>>>>>>> http-flow    01:43:22      01:43:42      119658  
5725168      2.337 Mbp
>>>>>>> http-flow    01:43:42      01:44:02      119064  
5725168      2.337 Mbp
>>>>>>> http-flow    01:44:02      01:44:22      113256  
5676668      2.315 Mbp
>>>>>>> ssh-flow     01:44:02      01:44:22      18414   
49646    0.027 Mbps
>>>>>>> http-flow    01:44:22      01:44:42      118206  
5725664      2.337 Mbp
>>>>>>> http-flow    01:44:42      01:45:02      117282  
5722140      2.335 Mbp
>>>>>>> ssh-flow     01:44:42      01:45:02      4698    
3544     0.003 Mbps
>>>>>>> http-flow    01:45:02      01:45:22      118536  
5688284      2.322 Mbp
>>>>>>> ssh-flow     01:45:02      01:45:22      4092    
3198     0.002 Mbps
>>>>>>> http-flow    01:45:22      01:45:42      119130  
5725168      2.337 Mbp
>>>>>>> ssh-flow     01:45:22      01:45:42      1980    
1478     0.001 Mbps
>>>>>>>
>>>>>>> That''s above the flow''s maxbw
parameter.  After that I tried to change the  maxbw of the link with dladm and
that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>>>
>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw
e1000g0
>>>>>>> LINK         PROPERTY        PERM VALUE         
DEFAULT        POSSIBLE
>>>>>>> e1000g0      maxbw           rw       1.228      --
--
>>>>>>>
>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>> http-flow    01:46:02      01:46:22      119394  
5725168      2.337 Mbp
>>>>>>> ssh-flow     01:46:02      01:46:22      4680    
2980     0.003 Mbps
>>>>>>> http-flow    01:46:22      01:46:42      94314   
4520316      1.845 Mbp
>>>>>>>
>>>>>>> Any ideas or is there a subtlety that I''m
missing and the behavior is correct?
>>>>>>>
>>>>>>> Thanks for the help.
>>>>>>>
>>>>>>> -Cesar
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> crossbow-discuss mailing list
>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

Cesar Delgado

2009-Nov-27 04:41 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
> 
> Hi, Cesar:
> 
> On Wed, 25 Nov 2009, Cesar Delgado wrote:
> 
>> 
>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>> 
>>> 
>>> 
>>> 
>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>> 
>>>> Venu,
>>>> 
>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>> 
>>>>> 
>>>>> Hi, Cesar:
>>>>> 
>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>> 
>>>>>> Venugopal,
>>>>>> 
>>>>>> I''m sorry if these sounds like basic
questions.  I really appreciate the patience and the help.  Replies in-line.
>>>>>> 
>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi, Cesar:
>>>>>>> 
>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>> 
>>>>>>>> I''m setting up a server to go to a
hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>> 
>>>>>>>> FIrst I have a question on flowadm''s
show-usage command.  When I try to run show-prop with the name of a flow I get
an error.  The flow exists.  What am I doing wrong?
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>> 
>>>>>>> This is a bug, I have submitted
>>>>>>> 
>>>>>>> 6904427 flowadm show-usage doesn''t work
with a flow name
>>>>>> 
>>>>>> Thanks for submitting that.  I haven''t been
able to find a link to the bugtracker for Crossbow.  Could you please send me
the URL?
>>>>> 
>>>>> I think it should show up on
>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Ok, now for my problem.  I have the following
setting:
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-flowprop
http-flow
>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>> http-flow    maxbw               1.228      -- 
1228k
>>>>>>>> http-flow    priority        medium         -- 
medium
>>>>>>>> 
>>>>>>>> I ran a test hitting the webserver and I see
this:
>>>>>>> 
>>>>>>> I have the following flow
>>>>>>> 
>>>>>>> # flowadm show-flow FLOW        LINK        IPADDR 
PROTO  LPORT   RPORT   DSFLD
>>>>>>> tcp-flow    <link>      --                   
tcp    --      --      --
>>>>>>> 
>>>>>>> 
>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>> FLOW         PROPERTY        VALUE          DEFAULT
POSSIBLE
>>>>>>> tcp-flow     maxbw               1.228      --     
1228K tcp-flow     priority        --             --             ?
>>>>>>> 
>>>>>>> When I send TCP traffic (am using a traffic
generator - netperf, to
>>>>>>> this machine from a peer) for about 2 mins.
>>>>>>> 
>>>>>>> On the peer the traffic generator (sender) says I
am capped to about
>>>>>>> 1.14 Mbps.
>>>>>>> 
>>>>>>> Size   Size    Size     Time     Throughput bytes 
bytes   bytes    secs.    10^6bits/sec
>>>>>>> 
>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>> 
>>>>>>> Now, when I try show-usage during the traffic flow
on
>>>>>>> the machine with the above flow in place
(receiver), I am seeing:
>>>>>>> 
>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>> FLOW         START         END           RBYTES  
OBYTES   BANDWIDTH
>>>>>>> tcp-flow     08:51:48      08:52:08      3428658 
107802       1.414 Mbp
>>>>>>> tcp-flow     08:52:08      08:52:28      3431198 
107802       1.415 Mbp
>>>>>>> tcp-flow     08:52:28      08:52:48      3434614 
107888       1.417 Mbp
>>>>>>> tcp-flow     08:52:48      08:53:08      3443298 
107802       1.420 Mbp
>>>>>>> tcp-flow     08:53:08      08:53:28      3444324 
107802       1.420 Mbp
>>>>>>> tcp-flow     08:53:28      08:53:48      1376806 
43576    0.568 Mbps
>>>>>>> 
>>>>>>> ...
>>>>>>> 
>>>>>>> I think the difference you see is likely to be
because of the time
>>>>>>> period when the stats are written to the file (the
bandwidth is computed for every 20 seconds period which might not be exactly in
>>>>>>> sync with the bandwidth enforcement period in the
kernel) and also
>>>>>>> could be because of rounding up etc. But, if you
look at the  entire
>>>>>>> duration, it averages to about the configured limit
(in the above
>>>>>>> example, I think it is about 1.275 Mbps for the 2
min duration)
>>>>>> 
>>>>>> The way I''m testing it is setting up Apache
and then moving down a file with `wget`.  The use case for this machine is an
Apache based app that serves large files to customers.  That is why I think a
`wget` is more telling of "real" performance than netperf. 
I''m running the test again and on the client side I am seeing usage
over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer rate
which is much closer to what I was seeing in the show-usage statistics.
>>>>>> 
>>>>> 
>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>> Resolving sol... 192.168.69.104
>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>> Length: 1048576000 (1000M)
>>>>>> Saving to: `myfile.dat''
>>>>>> 
>>>>>> 5% [==>                                             
] 55,530,974   267K/s  eta 60m 44s
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>> flow as the bandwidth for both are independent.
>>>>>> 
>>>>>> Thank you for this clarification but I still
don''t understand how I can be seeing ~2Mbps transfer if both the link
and the flow are both capped at 1.2Mbps.
>>>>>> 
>>>>> 
>>>>> Can you try with a higher bandwidth, say 100 Mbps and see
what the results
>>>>> are when compared to wget''s output?
>>>>> 
>>>>> Also, another way of manually checking would be to do a
>>>>> 	# kstat -c flow -n http-flow
>>>>> 
>>>>> before and after the wget run and see how many bytes
(rbytes) the
>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>> going over the flow), and then determine the bandwidth (you
might need
>>>>> to get the duration of the wget run pretty close to get the
>>>>> right bandwdith value).
>>>>> 
>>>>> -venu
>>>> 
>>>> I changed the flow to be 100Mbps.
>>>> 
>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>> http-flow    maxbw             100          --             100
>>>> 
>>>> I also removed the maxbw=1.228 I had set on the link.  When I
changed this value on the link I lost all network to the machine.  Had to go in
and manually reboot it.  The network came up fine.
>>>> 
>>>> I created two files one before wget and one after with no other
network traffic except the ssh session.  I created a 1 GB file to transfer.  The
output is a bit confusing to me.
>>>> 
>>>> root at myhost:/tmp# ls http-flow-*
>>>> http-flow-1.txt  http-flow-2.txt
>>>> root at myhost:/tmp# cat http-flow-*
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        4680
>>>> 	obytes                          182520974
>>>> 	oerrors                         0
>>>> 	opackets                        127521
>>>> 	rbytes                          308138
>>>> 	snaptime                        189431.882363589
>>>> 
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        31262
>>>> 	obytes                          1281785975
>>>> 	oerrors                         0
>>>> 	opackets                        895533
>>>> 	rbytes                          2062678
>>>> 	snaptime                        189595.122347726
>>>> 
>>>> So this means the network only passed ~1.6 MB of data?
>>> 
>>> is this on the outbound side or inbound?
>>> 
>>> looks like it has a total of 8794120008 bits in the outbound side
(obytes) and 14036320 bits on the inbound (rbytes). What is the duration
>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.
>>> 
>> 
>> Sorry, outbound.  I was looking at the wrong row.  With the flow set at
50 I''m getting getting ~37 Mbps.  In my mind the flow is not respecting
the value I''m giving it.  With the maxbw set to 100 I could get ~80
Mbps and now I''m only seeing ~37Mbps.   Could it be something in the
way Crossbow is averaging the actually bits and bytes going down the pipe or is
it me not understanding the system?
> 
> For TCP you might get a little less than the set bandwidth as TCP
> would do flow control too (i.e if for bandwidth reasons ACKs come back
> slower or packets get dropped, TCP will react and will slow down). Also,
> note that the current property (maxbw) is a *limit* which means we will
> ensure that the flow will not get *more* than what you have asked for.
> Going forwards we are looking at adding guarantee where in the flow would
> get what it asked for.
> 
> -venu
Good to know.  And it''s true about TCP, for some reason I though it
wouldn''t happen with maxbw but I do unserstand now t hat the entire TVP
connection is being affected by the parameter, including all the extra TCP stuff
that''s needed like the handshake.  I don''t need t set
guarantees at the moment so this doesn''t worry me much.  I just want to
cap the usage so I don''t go over my 1Mbit network quota.  Like overages
in cellphones, network overages are expensive.

So now back to my original problem.  In the new test I''m seeing
~2.1Mbps, that''s almost double the maxbw value.  If maxbw is a limit as
you say in the previous reply, why am I seeing more than 1.2Mbps when I set
maxbw to 1228k?  Could it be a corner case being that it''s the smallest
value possible?

cdelgado at myhost:~$ tail /tmp/http-flow-*
==> /tmp/http-flow-1.txt <=name:   http-flow                       class: 
flow
        crtime                          188662.418688816
        ierrors                         0
        ipackets                        101961
        obytes                          2446610048
        oerrors                         0
        opackets                        1709453
        rbytes                          6727462
        snaptime                        198359.572083418


==> /tmp/http-flow-2.txt <=name:   http-flow                       class: 
flow
        crtime                          188662.418688816
        ierrors                         0
        ipackets                        485927
        obytes                          3545874452
        oerrors                         0
        opackets                        2477456
        rbytes                          32069346
        snaptime                        202630.597287968


[cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
--15:50:35--  http://sol/myfile.dat
Resolving sol... 192.168.69.104
Connecting to sol|192.168.69.104|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1048576000 (1000M)
Saving to: `myfile.dat''

100%[==========================================================>]
1,048,576,000  267K/s   in 64m 8s

16:54:43 (266 KB/s) - `myfile.dat'' saved [1048576000/1048576000]


real    64m7.855s
user    0m4.982s
sys     0m32.007s

> 
>> 
>> root at myhost:/tmp# flowadm show-flowprop http-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> http-flow    maxbw              50          --             50
>> http-flow    priority        medium         --             medium
>> 
>> root at myhost:/tmp# cat http-flow-*
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        32346
>> 	obytes                          1300129135
>> 	oerrors                         0
>> 	opackets                        908351
>> 	rbytes                          2133486
>> 	snaptime                        191261.607310645
>> 
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        85361
>> 	obytes                          2399401106
>> 	oerrors                         0
>> 	opackets                        1676468
>> 	rbytes                          5632604
>> 	snaptime                        192621.873203347
>> 
>> 
>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>> --13:52:20--  http://sol/myfile.dat
>> Resolving sol... 192.168.69.104
>> Connecting to sol|192.168.69.104|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 1048576000 (1000M)
>> Saving to: `myfile.dat''
>> 
>> 100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>> 
>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>> 
>> 
>> real	3m44.364s
>> user	0m1.042s
>> sys	0m24.065s
>> 
>> 
>> 
>>> 
>>>> 
>>>> The wget command was telling me it was getting ~80Mbps which is
under the threshold of the flow.
>>>> 
>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>> --13:21:49--  http://sol/myfile.dat
>>>> Resolving sol... 192.168.69.104
>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>> HTTP request sent, awaiting response... 200 OK
>>>> Length: 1048576000 (1000M)
>>>> Saving to: `myfile.dat''
>>>> 
>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>> 
>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>> 
>>>> 
>>>> real	1m33.701s
>>>> user	0m0.899s
>>>> sys	0m23.874s
>>>> 
>>>> Maybe a value of 100Mbps is too high for this machine.  I might
try with 50Mbps to see what wget says.
>>>> 
>>> 
>>> OK,
>>> 
>>> -venu
>>> 
>>>> -Cesar
>>>> 
>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>> http-flow    01:32:22      01:32:42      1512  
2571     0.001 Mbps
>>>>>>>> ssh-flow     01:32:42      01:33:02      1818  
3578     0.002 Mbps
>>>>>>>> http-flow    01:33:02      01:33:22      66917 
3165136      1.292 Mbp
>>>>>>>> ssh-flow     01:33:02      01:33:22      3618  
5344     0.003 Mbps
>>>>>>>> http-flow    01:33:22      01:33:42      117947
5713018      2.332 Mbp
>>>>>>>> ssh-flow     01:33:22      01:33:42      4182  
3020     0.002 Mbps
>>>>>>>> http-flow    01:33:42      01:34:02      118998
5685520      2.321 Mbp
>>>>>>>> ssh-flow     01:33:42      01:34:02      11616 
9924     0.008 Mbps
>>>>>>>> http-flow    01:34:02      01:34:22      117084
5725664      2.337 Mbp
>>>>>>>> http-flow    01:34:22      01:34:42      119130
5725168      2.337 Mbp
>>>>>>>> http-flow    01:34:42      01:35:02      114180
5725168      2.335 Mbp
>>>>>>>> http-flow    01:35:02      01:35:22      109230
5725664      2.333 Mbp
>>>>>>>> http-flow    01:35:22      01:35:42      116160
5725168      2.336 Mbp
>>>>>>>> http-flow    01:35:42      01:36:02      119262
5725168      2.337 Mbp
>>>>>>>> http-flow    01:36:02      01:36:22      119196
5725664      2.337 Mbp
>>>>>>>> http-flow    01:36:22      01:36:42      117216
5725168      2.336 Mbp
>>>>>>>> http-flow    01:36:42      01:37:02      119394
5722636      2.336 Mbp
>>>>>>>> http-flow    01:37:02      01:37:22      119526
5725168      2.337 Mbp
>>>>>>>> http-flow    01:37:22      01:37:42      119460
5725168      2.337 Mbp
>>>>>>>> http-flow    01:37:42      01:38:02      119460
5725664      2.338 Mbp
>>>>>>>> http-flow    01:38:02      01:38:22      119724
5725168      2.337 Mbp
>>>>>>>> http-flow    01:38:22      01:38:42      119724
5725168      2.337 Mbp
>>>>>>>> http-flow    01:38:42      01:39:02      119130
5722636      2.336 Mbp
>>>>>>>> http-flow    01:39:02      01:39:22      118866
5725168      2.337 Mbp
>>>>>>>> http-flow    01:39:22      01:39:42      116490
5725664      2.336 Mbp
>>>>>>>> http-flow    01:39:42      01:40:02      119790
5725168      2.337 Mbp
>>>>>>>> http-flow    01:40:02      01:40:22      117678
5725168      2.337 Mbp
>>>>>>>> http-flow    01:40:22      01:40:42      118668
5725664      2.337 Mbp
>>>>>>>> http-flow    01:40:42      01:41:02      117414
5725168      2.337 Mbp
>>>>>>>> http-flow    01:41:02      01:41:22      119790
5725168      2.337 Mbp
>>>>>>>> http-flow    01:41:22      01:41:42      119813
5720510      2.336 Mbp
>>>>>>>> http-flow    01:41:42      01:42:02      119394
5725664      2.338 Mbp
>>>>>>>> http-flow    01:42:02      01:42:22      119724
5722272      2.336 Mbp
>>>>>>>> http-flow    01:42:22      01:42:42      119526
5725664      2.338 Mbp
>>>>>>>> http-flow    01:42:42      01:43:02      119196
5722140      2.336 Mbp
>>>>>>>> http-flow    01:43:02      01:43:22      119394
5725664      2.338 Mbp
>>>>>>>> http-flow    01:43:22      01:43:42      119658
5725168      2.337 Mbp
>>>>>>>> http-flow    01:43:42      01:44:02      119064
5725168      2.337 Mbp
>>>>>>>> http-flow    01:44:02      01:44:22      113256
5676668      2.315 Mbp
>>>>>>>> ssh-flow     01:44:02      01:44:22      18414 
49646    0.027 Mbps
>>>>>>>> http-flow    01:44:22      01:44:42      118206
5725664      2.337 Mbp
>>>>>>>> http-flow    01:44:42      01:45:02      117282
5722140      2.335 Mbp
>>>>>>>> ssh-flow     01:44:42      01:45:02      4698  
3544     0.003 Mbps
>>>>>>>> http-flow    01:45:02      01:45:22      118536
5688284      2.322 Mbp
>>>>>>>> ssh-flow     01:45:02      01:45:22      4092  
3198     0.002 Mbps
>>>>>>>> http-flow    01:45:22      01:45:42      119130
5725168      2.337 Mbp
>>>>>>>> ssh-flow     01:45:22      01:45:42      1980  
1478     0.001 Mbps
>>>>>>>> 
>>>>>>>> That''s above the flow''s maxbw
parameter.  After that I tried to change the  maxbw of the link with dladm and
that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>>>> 
>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw
e1000g0
>>>>>>>> LINK         PROPERTY        PERM VALUE        
DEFAULT        POSSIBLE
>>>>>>>> e1000g0      maxbw           rw       1.228    
--             --
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>> http-flow    01:46:02      01:46:22      119394
5725168      2.337 Mbp
>>>>>>>> ssh-flow     01:46:02      01:46:22      4680  
2980     0.003 Mbps
>>>>>>>> http-flow    01:46:22      01:46:42      94314 
4520316      1.845 Mbp
>>>>>>>> 
>>>>>>>> Any ideas or is there a subtlety that
I''m missing and the behavior is correct?
>>>>>>>> 
>>>>>>>> Thanks for the help.
>>>>>>>> 
>>>>>>>> -Cesar
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> crossbow-discuss mailing list
>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>>

venugopal iyer

2009-Dec-01 01:26 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Cesar:


On Thu, 26 Nov 2009, Cesar Delgado wrote:
>
>
> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
>
>>
>> Hi, Cesar:
>>
>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>
>>>
>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>>>
>>>>> Venu,
>>>>>
>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>>>
>>>>>>
>>>>>> Hi, Cesar:
>>>>>>
>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>>>
>>>>>>> Venugopal,
>>>>>>>
>>>>>>> I''m sorry if these sounds like basic
questions.  I really appreciate the patience and the help.  Replies in-line.
>>>>>>>
>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi, Cesar:
>>>>>>>>
>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>>>
>>>>>>>>> I''m setting up a server to go to a
hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>>>
>>>>>>>>> FIrst I have a question on
flowadm''s show-usage command.  When I try to run show-prop with the
name of a flow I get an error.  The flow exists.  What am I doing wrong?
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>>>
>>>>>>>> This is a bug, I have submitted
>>>>>>>>
>>>>>>>> 6904427 flowadm show-usage doesn''t
work with a flow name
>>>>>>>
>>>>>>> Thanks for submitting that.  I haven''t
been able to find a link to the bugtracker for Crossbow.  Could you please send
me the URL?
>>>>>>
>>>>>> I think it should show up on
>>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, now for my problem.  I have the
following setting:
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-flowprop
http-flow
>>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>>> http-flow    maxbw               1.228     
--             1228k
>>>>>>>>> http-flow    priority        medium        
--             medium
>>>>>>>>>
>>>>>>>>> I ran a test hitting the webserver and I
see this:
>>>>>>>>
>>>>>>>> I have the following flow
>>>>>>>>
>>>>>>>> # flowadm show-flow FLOW        LINK       
IPADDR                   PROTO  LPORT   RPORT   DSFLD
>>>>>>>> tcp-flow    <link>      --               
tcp    --      --      --
>>>>>>>>
>>>>>>>>
>>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>> tcp-flow     maxbw               1.228      -- 
1228K tcp-flow     priority        --             --             ?
>>>>>>>>
>>>>>>>> When I send TCP traffic (am using a traffic
generator - netperf, to
>>>>>>>> this machine from a peer) for about 2 mins.
>>>>>>>>
>>>>>>>> On the peer the traffic generator (sender) says
I am capped to about
>>>>>>>> 1.14 Mbps.
>>>>>>>>
>>>>>>>> Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
>>>>>>>>
>>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>>>
>>>>>>>> Now, when I try show-usage during the traffic
flow on
>>>>>>>> the machine with the above flow in place
(receiver), I am seeing:
>>>>>>>>
>>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>>> FLOW         START         END           RBYTES
OBYTES   BANDWIDTH
>>>>>>>> tcp-flow     08:51:48      08:52:08     
3428658  107802       1.414 Mbp
>>>>>>>> tcp-flow     08:52:08      08:52:28     
3431198  107802       1.415 Mbp
>>>>>>>> tcp-flow     08:52:28      08:52:48     
3434614  107888       1.417 Mbp
>>>>>>>> tcp-flow     08:52:48      08:53:08     
3443298  107802       1.420 Mbp
>>>>>>>> tcp-flow     08:53:08      08:53:28     
3444324  107802       1.420 Mbp
>>>>>>>> tcp-flow     08:53:28      08:53:48     
1376806  43576    0.568 Mbps
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> I think the difference you see is likely to be
because of the time
>>>>>>>> period when the stats are written to the file
(the bandwidth is computed for every 20 seconds period which might not be
exactly in
>>>>>>>> sync with the bandwidth enforcement period in
the kernel) and also
>>>>>>>> could be because of rounding up etc. But, if
you look at the  entire
>>>>>>>> duration, it averages to about the configured
limit (in the above
>>>>>>>> example, I think it is about 1.275 Mbps for the
2 min duration)
>>>>>>>
>>>>>>> The way I''m testing it is setting up
Apache and then moving down a file with `wget`.  The use case for this machine
is an Apache based app that serves large files to customers.  That is why I
think a `wget` is more telling of "real" performance than netperf. 
I''m running the test again and on the client side I am seeing usage
over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer rate
which is much closer to what I was seeing in the show-usage statistics.
>>>>>>>
>>>>>>
>>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>>> Resolving sol... 192.168.69.104
>>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>>> Length: 1048576000 (1000M)
>>>>>>> Saving to: `myfile.dat''
>>>>>>>
>>>>>>> 5% [==>                                         
] 55,530,974   267K/s  eta 60m 44s
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>>> flow as the bandwidth for both are independent.
>>>>>>>
>>>>>>> Thank you for this clarification but I still
don''t understand how I can be seeing ~2Mbps transfer if both the link
and the flow are both capped at 1.2Mbps.
>>>>>>>
>>>>>>
>>>>>> Can you try with a higher bandwidth, say 100 Mbps and
see what the results
>>>>>> are when compared to wget''s output?
>>>>>>
>>>>>> Also, another way of manually checking would be to do a
>>>>>> 	# kstat -c flow -n http-flow
>>>>>>
>>>>>> before and after the wget run and see how many bytes
(rbytes) the
>>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>>> going over the flow), and then determine the bandwidth
(you might need
>>>>>> to get the duration of the wget run pretty close to get
the
>>>>>> right bandwdith value).
>>>>>>
>>>>>> -venu
>>>>>
>>>>> I changed the flow to be 100Mbps.
>>>>>
>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw
http-flow
>>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>>> http-flow    maxbw             100          --            
100
>>>>>
>>>>> I also removed the maxbw=1.228 I had set on the link.  When
I changed this value on the link I lost all network to the machine.  Had to go
in and manually reboot it.  The network came up fine.
>>>>>
>>>>> I created two files one before wget and one after with no
other network traffic except the ssh session.  I created a 1 GB file to
transfer.  The output is a bit confusing to me.
>>>>>
>>>>> root at myhost:/tmp# ls http-flow-*
>>>>> http-flow-1.txt  http-flow-2.txt
>>>>> root at myhost:/tmp# cat http-flow-*
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        4680
>>>>> 	obytes                          182520974
>>>>> 	oerrors                         0
>>>>> 	opackets                        127521
>>>>> 	rbytes                          308138
>>>>> 	snaptime                        189431.882363589
>>>>>
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        31262
>>>>> 	obytes                          1281785975
>>>>> 	oerrors                         0
>>>>> 	opackets                        895533
>>>>> 	rbytes                          2062678
>>>>> 	snaptime                        189595.122347726
>>>>>
>>>>> So this means the network only passed ~1.6 MB of data?
>>>>
>>>> is this on the outbound side or inbound?
>>>>
>>>> looks like it has a total of 8794120008 bits in the outbound
side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration
>>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150
Kbps.
>>>>
>>>
>>> Sorry, outbound.  I was looking at the wrong row.  With the flow
set at 50 I''m getting getting ~37 Mbps.  In my mind the flow is not
respecting the value I''m giving it.  With the maxbw set to 100 I could
get ~80 Mbps and now I''m only seeing ~37Mbps.   Could it be something
in the way Crossbow is averaging the actually bits and bytes going down the pipe
or is it me not understanding the system?
>>
>> For TCP you might get a little less than the set bandwidth as TCP
>> would do flow control too (i.e if for bandwidth reasons ACKs come back
>> slower or packets get dropped, TCP will react and will slow down).
Also,
>> note that the current property (maxbw) is a *limit* which means we will
>> ensure that the flow will not get *more* than what you have asked for.
>> Going forwards we are looking at adding guarantee where in the flow
would
>> get what it asked for.
>>
>> -venu
>
> Good to know.  And it''s true about TCP, for some reason I though
it wouldn''t happen with maxbw but I do unserstand now t hat the entire
TVP connection is being affected by the parameter, including all the extra TCP
stuff that''s needed like the handshake.  I don''t need t set
guarantees at the moment so this doesn''t worry me much.  I just want to
cap the usage so I don''t go over my 1Mbit network quota.  Like overages
in cellphones, network overages are expensive.
>
> So now back to my original problem.  In the new test I''m seeing
~2.1Mbps, that''s almost double the maxbw value.  If maxbw is a limit as
you say in the previous reply, why am I seeing more than 1.2Mbps when I set
maxbw to 1228k?  Could it be a corner case being that it''s the smallest
value possible?
Looking at the output below, I suspect so. Let me try it at my end and
check this..  will let you know.

-venu
>
> cdelgado at myhost:~$ tail /tmp/http-flow-*
> ==> /tmp/http-flow-1.txt <=> name:   http-flow                    
class:    flow
>        crtime                          188662.418688816
>        ierrors                         0
>        ipackets                        101961
>        obytes                          2446610048
>        oerrors                         0
>        opackets                        1709453
>        rbytes                          6727462
>        snaptime                        198359.572083418
>
>
> ==> /tmp/http-flow-2.txt <=> name:   http-flow                    
class:    flow
>        crtime                          188662.418688816
>        ierrors                         0
>        ipackets                        485927
>        obytes                          3545874452
>        oerrors                         0
>        opackets                        2477456
>        rbytes                          32069346
>        snaptime                        202630.597287968
>
>
> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
> --15:50:35--  http://sol/myfile.dat
> Resolving sol... 192.168.69.104
> Connecting to sol|192.168.69.104|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 1048576000 (1000M)
> Saving to: `myfile.dat''
>
> 100%[==========================================================>]
1,048,576,000  267K/s   in 64m 8s
>
> 16:54:43 (266 KB/s) - `myfile.dat'' saved [1048576000/1048576000]
>
>
> real    64m7.855s
> user    0m4.982s
> sys     0m32.007s
>
>
>>
>>>
>>> root at myhost:/tmp# flowadm show-flowprop http-flow
>>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>>> http-flow    maxbw              50          --             50
>>> http-flow    priority        medium         --             medium
>>>
>>> root at myhost:/tmp# cat http-flow-*
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        32346
>>> 	obytes                          1300129135
>>> 	oerrors                         0
>>> 	opackets                        908351
>>> 	rbytes                          2133486
>>> 	snaptime                        191261.607310645
>>>
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        85361
>>> 	obytes                          2399401106
>>> 	oerrors                         0
>>> 	opackets                        1676468
>>> 	rbytes                          5632604
>>> 	snaptime                        192621.873203347
>>>
>>>
>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>> --13:52:20--  http://sol/myfile.dat
>>> Resolving sol... 192.168.69.104
>>> Connecting to sol|192.168.69.104|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 1048576000 (1000M)
>>> Saving to: `myfile.dat''
>>>
>>>
100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>>>
>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>
>>>
>>> real	3m44.364s
>>> user	0m1.042s
>>> sys	0m24.065s
>>>
>>>
>>>
>>>>
>>>>>
>>>>> The wget command was telling me it was getting ~80Mbps
which is under the threshold of the flow.
>>>>>
>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>>> --13:21:49--  http://sol/myfile.dat
>>>>> Resolving sol... 192.168.69.104
>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>> HTTP request sent, awaiting response... 200 OK
>>>>> Length: 1048576000 (1000M)
>>>>> Saving to: `myfile.dat''
>>>>>
>>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>>>
>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>>>
>>>>>
>>>>> real	1m33.701s
>>>>> user	0m0.899s
>>>>> sys	0m23.874s
>>>>>
>>>>> Maybe a value of 100Mbps is too high for this machine.  I
might try with 50Mbps to see what wget says.
>>>>>
>>>>
>>>> OK,
>>>>
>>>> -venu
>>>>
>>>>> -Cesar
>>>>>
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>> http-flow    01:32:22      01:32:42     
1512     2571     0.001 Mbps
>>>>>>>>> ssh-flow     01:32:42      01:33:02     
1818     3578     0.002 Mbps
>>>>>>>>> http-flow    01:33:02      01:33:22     
66917    3165136      1.292 Mbp
>>>>>>>>> ssh-flow     01:33:02      01:33:22     
3618     5344     0.003 Mbps
>>>>>>>>> http-flow    01:33:22      01:33:42     
117947   5713018      2.332 Mbp
>>>>>>>>> ssh-flow     01:33:22      01:33:42     
4182     3020     0.002 Mbps
>>>>>>>>> http-flow    01:33:42      01:34:02     
118998   5685520      2.321 Mbp
>>>>>>>>> ssh-flow     01:33:42      01:34:02     
11616    9924     0.008 Mbps
>>>>>>>>> http-flow    01:34:02      01:34:22     
117084   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:34:22      01:34:42     
119130   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:34:42      01:35:02     
114180   5725168      2.335 Mbp
>>>>>>>>> http-flow    01:35:02      01:35:22     
109230   5725664      2.333 Mbp
>>>>>>>>> http-flow    01:35:22      01:35:42     
116160   5725168      2.336 Mbp
>>>>>>>>> http-flow    01:35:42      01:36:02     
119262   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:36:02      01:36:22     
119196   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:36:22      01:36:42     
117216   5725168      2.336 Mbp
>>>>>>>>> http-flow    01:36:42      01:37:02     
119394   5722636      2.336 Mbp
>>>>>>>>> http-flow    01:37:02      01:37:22     
119526   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:37:22      01:37:42     
119460   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:37:42      01:38:02     
119460   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:38:02      01:38:22     
119724   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:38:22      01:38:42     
119724   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:38:42      01:39:02     
119130   5722636      2.336 Mbp
>>>>>>>>> http-flow    01:39:02      01:39:22     
118866   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:39:22      01:39:42     
116490   5725664      2.336 Mbp
>>>>>>>>> http-flow    01:39:42      01:40:02     
119790   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:40:02      01:40:22     
117678   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:40:22      01:40:42     
118668   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:40:42      01:41:02     
117414   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:41:02      01:41:22     
119790   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:41:22      01:41:42     
119813   5720510      2.336 Mbp
>>>>>>>>> http-flow    01:41:42      01:42:02     
119394   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:42:02      01:42:22     
119724   5722272      2.336 Mbp
>>>>>>>>> http-flow    01:42:22      01:42:42     
119526   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:42:42      01:43:02     
119196   5722140      2.336 Mbp
>>>>>>>>> http-flow    01:43:02      01:43:22     
119394   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:43:22      01:43:42     
119658   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:43:42      01:44:02     
119064   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:44:02      01:44:22     
113256   5676668      2.315 Mbp
>>>>>>>>> ssh-flow     01:44:02      01:44:22     
18414    49646    0.027 Mbps
>>>>>>>>> http-flow    01:44:22      01:44:42     
118206   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:44:42      01:45:02     
117282   5722140      2.335 Mbp
>>>>>>>>> ssh-flow     01:44:42      01:45:02     
4698     3544     0.003 Mbps
>>>>>>>>> http-flow    01:45:02      01:45:22     
118536   5688284      2.322 Mbp
>>>>>>>>> ssh-flow     01:45:02      01:45:22     
4092     3198     0.002 Mbps
>>>>>>>>> http-flow    01:45:22      01:45:42     
119130   5725168      2.337 Mbp
>>>>>>>>> ssh-flow     01:45:22      01:45:42     
1980     1478     0.001 Mbps
>>>>>>>>>
>>>>>>>>> That''s above the flow''s
maxbw parameter.  After that I tried to change the  maxbw of the link with dladm
and that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>>>>>
>>>>>>>>> root at myhost:~# dladm show-linkprop -p
maxbw e1000g0
>>>>>>>>> LINK         PROPERTY        PERM VALUE    
DEFAULT        POSSIBLE
>>>>>>>>> e1000g0      maxbw           rw       1.228
--             --
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>> http-flow    01:46:02      01:46:22     
119394   5725168      2.337 Mbp
>>>>>>>>> ssh-flow     01:46:02      01:46:22     
4680     2980     0.003 Mbps
>>>>>>>>> http-flow    01:46:22      01:46:42     
94314    4520316      1.845 Mbp
>>>>>>>>>
>>>>>>>>> Any ideas or is there a subtlety that
I''m missing and the behavior is correct?
>>>>>>>>>
>>>>>>>>> Thanks for the help.
>>>>>>>>>
>>>>>>>>> -Cesar
>>>>>>>>>
>>>>>>>>>
_______________________________________________
>>>>>>>>> crossbow-discuss mailing list
>>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

Cesar Delgado

2009-Dec-15 04:39 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Venu,

Any new information on this?  Is it a bug?

-Cesar

On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
> 
> Hi, Cesar:
> 
> On Wed, 25 Nov 2009, Cesar Delgado wrote:
> 
>> 
>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>> 
>>> 
>>> 
>>> 
>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>> 
>>>> Venu,
>>>> 
>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>> 
>>>>> 
>>>>> Hi, Cesar:
>>>>> 
>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>> 
>>>>>> Venugopal,
>>>>>> 
>>>>>> I''m sorry if these sounds like basic
questions.  I really appreciate the patience and the help.  Replies in-line.
>>>>>> 
>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Hi, Cesar:
>>>>>>> 
>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>> 
>>>>>>>> I''m setting up a server to go to a
hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>> 
>>>>>>>> FIrst I have a question on flowadm''s
show-usage command.  When I try to run show-prop with the name of a flow I get
an error.  The flow exists.  What am I doing wrong?
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>> 
>>>>>>> This is a bug, I have submitted
>>>>>>> 
>>>>>>> 6904427 flowadm show-usage doesn''t work
with a flow name
>>>>>> 
>>>>>> Thanks for submitting that.  I haven''t been
able to find a link to the bugtracker for Crossbow.  Could you please send me
the URL?
>>>>> 
>>>>> I think it should show up on
>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Ok, now for my problem.  I have the following
setting:
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-flowprop
http-flow
>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>> http-flow    maxbw               1.228      -- 
1228k
>>>>>>>> http-flow    priority        medium         -- 
medium
>>>>>>>> 
>>>>>>>> I ran a test hitting the webserver and I see
this:
>>>>>>> 
>>>>>>> I have the following flow
>>>>>>> 
>>>>>>> # flowadm show-flow FLOW        LINK        IPADDR 
PROTO  LPORT   RPORT   DSFLD
>>>>>>> tcp-flow    <link>      --                   
tcp    --      --      --
>>>>>>> 
>>>>>>> 
>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>> FLOW         PROPERTY        VALUE          DEFAULT
POSSIBLE
>>>>>>> tcp-flow     maxbw               1.228      --     
1228K tcp-flow     priority        --             --             ?
>>>>>>> 
>>>>>>> When I send TCP traffic (am using a traffic
generator - netperf, to
>>>>>>> this machine from a peer) for about 2 mins.
>>>>>>> 
>>>>>>> On the peer the traffic generator (sender) says I
am capped to about
>>>>>>> 1.14 Mbps.
>>>>>>> 
>>>>>>> Size   Size    Size     Time     Throughput bytes 
bytes   bytes    secs.    10^6bits/sec
>>>>>>> 
>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>> 
>>>>>>> Now, when I try show-usage during the traffic flow
on
>>>>>>> the machine with the above flow in place
(receiver), I am seeing:
>>>>>>> 
>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>> FLOW         START         END           RBYTES  
OBYTES   BANDWIDTH
>>>>>>> tcp-flow     08:51:48      08:52:08      3428658 
107802       1.414 Mbp
>>>>>>> tcp-flow     08:52:08      08:52:28      3431198 
107802       1.415 Mbp
>>>>>>> tcp-flow     08:52:28      08:52:48      3434614 
107888       1.417 Mbp
>>>>>>> tcp-flow     08:52:48      08:53:08      3443298 
107802       1.420 Mbp
>>>>>>> tcp-flow     08:53:08      08:53:28      3444324 
107802       1.420 Mbp
>>>>>>> tcp-flow     08:53:28      08:53:48      1376806 
43576    0.568 Mbps
>>>>>>> 
>>>>>>> ...
>>>>>>> 
>>>>>>> I think the difference you see is likely to be
because of the time
>>>>>>> period when the stats are written to the file (the
bandwidth is computed for every 20 seconds period which might not be exactly in
>>>>>>> sync with the bandwidth enforcement period in the
kernel) and also
>>>>>>> could be because of rounding up etc. But, if you
look at the  entire
>>>>>>> duration, it averages to about the configured limit
(in the above
>>>>>>> example, I think it is about 1.275 Mbps for the 2
min duration)
>>>>>> 
>>>>>> The way I''m testing it is setting up Apache
and then moving down a file with `wget`.  The use case for this machine is an
Apache based app that serves large files to customers.  That is why I think a
`wget` is more telling of "real" performance than netperf. 
I''m running the test again and on the client side I am seeing usage
over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer rate
which is much closer to what I was seeing in the show-usage statistics.
>>>>>> 
>>>>> 
>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>> Resolving sol... 192.168.69.104
>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>> Length: 1048576000 (1000M)
>>>>>> Saving to: `myfile.dat''
>>>>>> 
>>>>>> 5% [==>                                             
] 55,530,974   267K/s  eta 60m 44s
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>> flow as the bandwidth for both are independent.
>>>>>> 
>>>>>> Thank you for this clarification but I still
don''t understand how I can be seeing ~2Mbps transfer if both the link
and the flow are both capped at 1.2Mbps.
>>>>>> 
>>>>> 
>>>>> Can you try with a higher bandwidth, say 100 Mbps and see
what the results
>>>>> are when compared to wget''s output?
>>>>> 
>>>>> Also, another way of manually checking would be to do a
>>>>> 	# kstat -c flow -n http-flow
>>>>> 
>>>>> before and after the wget run and see how many bytes
(rbytes) the
>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>> going over the flow), and then determine the bandwidth (you
might need
>>>>> to get the duration of the wget run pretty close to get the
>>>>> right bandwdith value).
>>>>> 
>>>>> -venu
>>>> 
>>>> I changed the flow to be 100Mbps.
>>>> 
>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow
>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>> http-flow    maxbw             100          --             100
>>>> 
>>>> I also removed the maxbw=1.228 I had set on the link.  When I
changed this value on the link I lost all network to the machine.  Had to go in
and manually reboot it.  The network came up fine.
>>>> 
>>>> I created two files one before wget and one after with no other
network traffic except the ssh session.  I created a 1 GB file to transfer.  The
output is a bit confusing to me.
>>>> 
>>>> root at myhost:/tmp# ls http-flow-*
>>>> http-flow-1.txt  http-flow-2.txt
>>>> root at myhost:/tmp# cat http-flow-*
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        4680
>>>> 	obytes                          182520974
>>>> 	oerrors                         0
>>>> 	opackets                        127521
>>>> 	rbytes                          308138
>>>> 	snaptime                        189431.882363589
>>>> 
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        31262
>>>> 	obytes                          1281785975
>>>> 	oerrors                         0
>>>> 	opackets                        895533
>>>> 	rbytes                          2062678
>>>> 	snaptime                        189595.122347726
>>>> 
>>>> So this means the network only passed ~1.6 MB of data?
>>> 
>>> is this on the outbound side or inbound?
>>> 
>>> looks like it has a total of 8794120008 bits in the outbound side
(obytes) and 14036320 bits on the inbound (rbytes). What is the duration
>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.
>>> 
>> 
>> Sorry, outbound.  I was looking at the wrong row.  With the flow set at
50 I''m getting getting ~37 Mbps.  In my mind the flow is not respecting
the value I''m giving it.  With the maxbw set to 100 I could get ~80
Mbps and now I''m only seeing ~37Mbps.   Could it be something in the
way Crossbow is averaging the actually bits and bytes going down the pipe or is
it me not understanding the system?
> 
> For TCP you might get a little less than the set bandwidth as TCP
> would do flow control too (i.e if for bandwidth reasons ACKs come back
> slower or packets get dropped, TCP will react and will slow down). Also,
> note that the current property (maxbw) is a *limit* which means we will
> ensure that the flow will not get *more* than what you have asked for.
> Going forwards we are looking at adding guarantee where in the flow would
> get what it asked for.
> 
> -venu
> 
>> 
>> root at myhost:/tmp# flowadm show-flowprop http-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> http-flow    maxbw              50          --             50
>> http-flow    priority        medium         --             medium
>> 
>> root at myhost:/tmp# cat http-flow-*
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        32346
>> 	obytes                          1300129135
>> 	oerrors                         0
>> 	opackets                        908351
>> 	rbytes                          2133486
>> 	snaptime                        191261.607310645
>> 
>> module: unix                            instance: 0
>> name:   http-flow                       class:    flow
>> 	crtime                          188662.418688816
>> 	ierrors                         0
>> 	ipackets                        85361
>> 	obytes                          2399401106
>> 	oerrors                         0
>> 	opackets                        1676468
>> 	rbytes                          5632604
>> 	snaptime                        192621.873203347
>> 
>> 
>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>> --13:52:20--  http://sol/myfile.dat
>> Resolving sol... 192.168.69.104
>> Connecting to sol|192.168.69.104|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 1048576000 (1000M)
>> Saving to: `myfile.dat''
>> 
>> 100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>> 
>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>> 
>> 
>> real	3m44.364s
>> user	0m1.042s
>> sys	0m24.065s
>> 
>> 
>> 
>>> 
>>>> 
>>>> The wget command was telling me it was getting ~80Mbps which is
under the threshold of the flow.
>>>> 
>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>> --13:21:49--  http://sol/myfile.dat
>>>> Resolving sol... 192.168.69.104
>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>> HTTP request sent, awaiting response... 200 OK
>>>> Length: 1048576000 (1000M)
>>>> Saving to: `myfile.dat''
>>>> 
>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>> 
>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>> 
>>>> 
>>>> real	1m33.701s
>>>> user	0m0.899s
>>>> sys	0m23.874s
>>>> 
>>>> Maybe a value of 100Mbps is too high for this machine.  I might
try with 50Mbps to see what wget says.
>>>> 
>>> 
>>> OK,
>>> 
>>> -venu
>>> 
>>>> -Cesar
>>>> 
>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>> http-flow    01:32:22      01:32:42      1512  
2571     0.001 Mbps
>>>>>>>> ssh-flow     01:32:42      01:33:02      1818  
3578     0.002 Mbps
>>>>>>>> http-flow    01:33:02      01:33:22      66917 
3165136      1.292 Mbp
>>>>>>>> ssh-flow     01:33:02      01:33:22      3618  
5344     0.003 Mbps
>>>>>>>> http-flow    01:33:22      01:33:42      117947
5713018      2.332 Mbp
>>>>>>>> ssh-flow     01:33:22      01:33:42      4182  
3020     0.002 Mbps
>>>>>>>> http-flow    01:33:42      01:34:02      118998
5685520      2.321 Mbp
>>>>>>>> ssh-flow     01:33:42      01:34:02      11616 
9924     0.008 Mbps
>>>>>>>> http-flow    01:34:02      01:34:22      117084
5725664      2.337 Mbp
>>>>>>>> http-flow    01:34:22      01:34:42      119130
5725168      2.337 Mbp
>>>>>>>> http-flow    01:34:42      01:35:02      114180
5725168      2.335 Mbp
>>>>>>>> http-flow    01:35:02      01:35:22      109230
5725664      2.333 Mbp
>>>>>>>> http-flow    01:35:22      01:35:42      116160
5725168      2.336 Mbp
>>>>>>>> http-flow    01:35:42      01:36:02      119262
5725168      2.337 Mbp
>>>>>>>> http-flow    01:36:02      01:36:22      119196
5725664      2.337 Mbp
>>>>>>>> http-flow    01:36:22      01:36:42      117216
5725168      2.336 Mbp
>>>>>>>> http-flow    01:36:42      01:37:02      119394
5722636      2.336 Mbp
>>>>>>>> http-flow    01:37:02      01:37:22      119526
5725168      2.337 Mbp
>>>>>>>> http-flow    01:37:22      01:37:42      119460
5725168      2.337 Mbp
>>>>>>>> http-flow    01:37:42      01:38:02      119460
5725664      2.338 Mbp
>>>>>>>> http-flow    01:38:02      01:38:22      119724
5725168      2.337 Mbp
>>>>>>>> http-flow    01:38:22      01:38:42      119724
5725168      2.337 Mbp
>>>>>>>> http-flow    01:38:42      01:39:02      119130
5722636      2.336 Mbp
>>>>>>>> http-flow    01:39:02      01:39:22      118866
5725168      2.337 Mbp
>>>>>>>> http-flow    01:39:22      01:39:42      116490
5725664      2.336 Mbp
>>>>>>>> http-flow    01:39:42      01:40:02      119790
5725168      2.337 Mbp
>>>>>>>> http-flow    01:40:02      01:40:22      117678
5725168      2.337 Mbp
>>>>>>>> http-flow    01:40:22      01:40:42      118668
5725664      2.337 Mbp
>>>>>>>> http-flow    01:40:42      01:41:02      117414
5725168      2.337 Mbp
>>>>>>>> http-flow    01:41:02      01:41:22      119790
5725168      2.337 Mbp
>>>>>>>> http-flow    01:41:22      01:41:42      119813
5720510      2.336 Mbp
>>>>>>>> http-flow    01:41:42      01:42:02      119394
5725664      2.338 Mbp
>>>>>>>> http-flow    01:42:02      01:42:22      119724
5722272      2.336 Mbp
>>>>>>>> http-flow    01:42:22      01:42:42      119526
5725664      2.338 Mbp
>>>>>>>> http-flow    01:42:42      01:43:02      119196
5722140      2.336 Mbp
>>>>>>>> http-flow    01:43:02      01:43:22      119394
5725664      2.338 Mbp
>>>>>>>> http-flow    01:43:22      01:43:42      119658
5725168      2.337 Mbp
>>>>>>>> http-flow    01:43:42      01:44:02      119064
5725168      2.337 Mbp
>>>>>>>> http-flow    01:44:02      01:44:22      113256
5676668      2.315 Mbp
>>>>>>>> ssh-flow     01:44:02      01:44:22      18414 
49646    0.027 Mbps
>>>>>>>> http-flow    01:44:22      01:44:42      118206
5725664      2.337 Mbp
>>>>>>>> http-flow    01:44:42      01:45:02      117282
5722140      2.335 Mbp
>>>>>>>> ssh-flow     01:44:42      01:45:02      4698  
3544     0.003 Mbps
>>>>>>>> http-flow    01:45:02      01:45:22      118536
5688284      2.322 Mbp
>>>>>>>> ssh-flow     01:45:02      01:45:22      4092  
3198     0.002 Mbps
>>>>>>>> http-flow    01:45:22      01:45:42      119130
5725168      2.337 Mbp
>>>>>>>> ssh-flow     01:45:22      01:45:42      1980  
1478     0.001 Mbps
>>>>>>>> 
>>>>>>>> That''s above the flow''s maxbw
parameter.  After that I tried to change the  maxbw of the link with dladm and
that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>>>> 
>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw
e1000g0
>>>>>>>> LINK         PROPERTY        PERM VALUE        
DEFAULT        POSSIBLE
>>>>>>>> e1000g0      maxbw           rw       1.228    
--             --
>>>>>>>> 
>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>> http-flow    01:46:02      01:46:22      119394
5725168      2.337 Mbp
>>>>>>>> ssh-flow     01:46:02      01:46:22      4680  
2980     0.003 Mbps
>>>>>>>> http-flow    01:46:22      01:46:42      94314 
4520316      1.845 Mbp
>>>>>>>> 
>>>>>>>> Any ideas or is there a subtlety that
I''m missing and the behavior is correct?
>>>>>>>> 
>>>>>>>> Thanks for the help.
>>>>>>>> 
>>>>>>>> -Cesar
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> crossbow-discuss mailing list
>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>>

venugopal iyer

2009-Dec-16 01:30 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Sorry, Cesar, for the delayed response.

I tried setting up your scenario on a recent version
(build 130) and see that the bandwidth is close to what I am
setting.

# flowadm show-flowprop tcp-flow
FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
tcp-flow     maxbw               1.200      --             1200K 
tcp-flow     priority        --             --             ?


# wget 10.0.0.2/myfile.dat
--2009-12-15 15:47:24--  http://10.0.0.2/myfile.dat
Connecting to 10.0.0.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 46104240 (44M) [text/plain]
Saving to: `myfile.dat.4''

100%[======================================>] 46,104,240   160K/s   in 4m 42s

2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved
[46104240/46104240]

160 KB/s  is 1.280 Mbps.

I see this consistently.


The kstat does give me slightly higher about 1.3 Mbps.

FYI, without the flow (with limt) in place, I see

# wget 10.0.0.2/myfile.dat
--2009-12-15 15:47:12--  http://10.0.0.2/myfile.dat
Connecting to 10.0.0.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 46104240 (44M) [text/plain]
Saving to: `myfile.dat.3''

100%[======================================>] 46,104,240  78.2M/s   in 0.6s

2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved
[46104240/46104240]


What kind of hardware are you using?

-venu


On Mon, 14 Dec 2009, Cesar Delgado wrote:
> Venu,
>
> Any new information on this?  Is it a bug?
>
> -Cesar
>
> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
>
>>
>> Hi, Cesar:
>>
>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>
>>>
>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>>>
>>>>> Venu,
>>>>>
>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>>>
>>>>>>
>>>>>> Hi, Cesar:
>>>>>>
>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>>>
>>>>>>> Venugopal,
>>>>>>>
>>>>>>> I''m sorry if these sounds like basic
questions.  I really appreciate the patience and the help.  Replies in-line.
>>>>>>>
>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi, Cesar:
>>>>>>>>
>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>>>
>>>>>>>>> I''m setting up a server to go to a
hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>>>
>>>>>>>>> FIrst I have a question on
flowadm''s show-usage command.  When I try to run show-prop with the
name of a flow I get an error.  The flow exists.  What am I doing wrong?
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>>>
>>>>>>>> This is a bug, I have submitted
>>>>>>>>
>>>>>>>> 6904427 flowadm show-usage doesn''t
work with a flow name
>>>>>>>
>>>>>>> Thanks for submitting that.  I haven''t
been able to find a link to the bugtracker for Crossbow.  Could you please send
me the URL?
>>>>>>
>>>>>> I think it should show up on
>>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok, now for my problem.  I have the
following setting:
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-flowprop
http-flow
>>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>>> http-flow    maxbw               1.228     
--             1228k
>>>>>>>>> http-flow    priority        medium        
--             medium
>>>>>>>>>
>>>>>>>>> I ran a test hitting the webserver and I
see this:
>>>>>>>>
>>>>>>>> I have the following flow
>>>>>>>>
>>>>>>>> # flowadm show-flow FLOW        LINK       
IPADDR                   PROTO  LPORT   RPORT   DSFLD
>>>>>>>> tcp-flow    <link>      --               
tcp    --      --      --
>>>>>>>>
>>>>>>>>
>>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>> tcp-flow     maxbw               1.228      -- 
1228K tcp-flow     priority        --             --             ?
>>>>>>>>
>>>>>>>> When I send TCP traffic (am using a traffic
generator - netperf, to
>>>>>>>> this machine from a peer) for about 2 mins.
>>>>>>>>
>>>>>>>> On the peer the traffic generator (sender) says
I am capped to about
>>>>>>>> 1.14 Mbps.
>>>>>>>>
>>>>>>>> Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
>>>>>>>>
>>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>>>
>>>>>>>> Now, when I try show-usage during the traffic
flow on
>>>>>>>> the machine with the above flow in place
(receiver), I am seeing:
>>>>>>>>
>>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>>> FLOW         START         END           RBYTES
OBYTES   BANDWIDTH
>>>>>>>> tcp-flow     08:51:48      08:52:08     
3428658  107802       1.414 Mbp
>>>>>>>> tcp-flow     08:52:08      08:52:28     
3431198  107802       1.415 Mbp
>>>>>>>> tcp-flow     08:52:28      08:52:48     
3434614  107888       1.417 Mbp
>>>>>>>> tcp-flow     08:52:48      08:53:08     
3443298  107802       1.420 Mbp
>>>>>>>> tcp-flow     08:53:08      08:53:28     
3444324  107802       1.420 Mbp
>>>>>>>> tcp-flow     08:53:28      08:53:48     
1376806  43576    0.568 Mbps
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> I think the difference you see is likely to be
because of the time
>>>>>>>> period when the stats are written to the file
(the bandwidth is computed for every 20 seconds period which might not be
exactly in
>>>>>>>> sync with the bandwidth enforcement period in
the kernel) and also
>>>>>>>> could be because of rounding up etc. But, if
you look at the  entire
>>>>>>>> duration, it averages to about the configured
limit (in the above
>>>>>>>> example, I think it is about 1.275 Mbps for the
2 min duration)
>>>>>>>
>>>>>>> The way I''m testing it is setting up
Apache and then moving down a file with `wget`.  The use case for this machine
is an Apache based app that serves large files to customers.  That is why I
think a `wget` is more telling of "real" performance than netperf. 
I''m running the test again and on the client side I am seeing usage
over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer rate
which is much closer to what I was seeing in the show-usage statistics.
>>>>>>>
>>>>>>
>>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>>> Resolving sol... 192.168.69.104
>>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>>> Length: 1048576000 (1000M)
>>>>>>> Saving to: `myfile.dat''
>>>>>>>
>>>>>>> 5% [==>                                         
] 55,530,974   267K/s  eta 60m 44s
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>>> flow as the bandwidth for both are independent.
>>>>>>>
>>>>>>> Thank you for this clarification but I still
don''t understand how I can be seeing ~2Mbps transfer if both the link
and the flow are both capped at 1.2Mbps.
>>>>>>>
>>>>>>
>>>>>> Can you try with a higher bandwidth, say 100 Mbps and
see what the results
>>>>>> are when compared to wget''s output?
>>>>>>
>>>>>> Also, another way of manually checking would be to do a
>>>>>> 	# kstat -c flow -n http-flow
>>>>>>
>>>>>> before and after the wget run and see how many bytes
(rbytes) the
>>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>>> going over the flow), and then determine the bandwidth
(you might need
>>>>>> to get the duration of the wget run pretty close to get
the
>>>>>> right bandwdith value).
>>>>>>
>>>>>> -venu
>>>>>
>>>>> I changed the flow to be 100Mbps.
>>>>>
>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw
http-flow
>>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>>> http-flow    maxbw             100          --            
100
>>>>>
>>>>> I also removed the maxbw=1.228 I had set on the link.  When
I changed this value on the link I lost all network to the machine.  Had to go
in and manually reboot it.  The network came up fine.
>>>>>
>>>>> I created two files one before wget and one after with no
other network traffic except the ssh session.  I created a 1 GB file to
transfer.  The output is a bit confusing to me.
>>>>>
>>>>> root at myhost:/tmp# ls http-flow-*
>>>>> http-flow-1.txt  http-flow-2.txt
>>>>> root at myhost:/tmp# cat http-flow-*
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        4680
>>>>> 	obytes                          182520974
>>>>> 	oerrors                         0
>>>>> 	opackets                        127521
>>>>> 	rbytes                          308138
>>>>> 	snaptime                        189431.882363589
>>>>>
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        31262
>>>>> 	obytes                          1281785975
>>>>> 	oerrors                         0
>>>>> 	opackets                        895533
>>>>> 	rbytes                          2062678
>>>>> 	snaptime                        189595.122347726
>>>>>
>>>>> So this means the network only passed ~1.6 MB of data?
>>>>
>>>> is this on the outbound side or inbound?
>>>>
>>>> looks like it has a total of 8794120008 bits in the outbound
side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration
>>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150
Kbps.
>>>>
>>>
>>> Sorry, outbound.  I was looking at the wrong row.  With the flow
set at 50 I''m getting getting ~37 Mbps.  In my mind the flow is not
respecting the value I''m giving it.  With the maxbw set to 100 I could
get ~80 Mbps and now I''m only seeing ~37Mbps.   Could it be something
in the way Crossbow is averaging the actually bits and bytes going down the pipe
or is it me not understanding the system?
>>
>> For TCP you might get a little less than the set bandwidth as TCP
>> would do flow control too (i.e if for bandwidth reasons ACKs come back
>> slower or packets get dropped, TCP will react and will slow down).
Also,
>> note that the current property (maxbw) is a *limit* which means we will
>> ensure that the flow will not get *more* than what you have asked for.
>> Going forwards we are looking at adding guarantee where in the flow
would
>> get what it asked for.
>>
>> -venu
>>
>>>
>>> root at myhost:/tmp# flowadm show-flowprop http-flow
>>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>>> http-flow    maxbw              50          --             50
>>> http-flow    priority        medium         --             medium
>>>
>>> root at myhost:/tmp# cat http-flow-*
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        32346
>>> 	obytes                          1300129135
>>> 	oerrors                         0
>>> 	opackets                        908351
>>> 	rbytes                          2133486
>>> 	snaptime                        191261.607310645
>>>
>>> module: unix                            instance: 0
>>> name:   http-flow                       class:    flow
>>> 	crtime                          188662.418688816
>>> 	ierrors                         0
>>> 	ipackets                        85361
>>> 	obytes                          2399401106
>>> 	oerrors                         0
>>> 	opackets                        1676468
>>> 	rbytes                          5632604
>>> 	snaptime                        192621.873203347
>>>
>>>
>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>> --13:52:20--  http://sol/myfile.dat
>>> Resolving sol... 192.168.69.104
>>> Connecting to sol|192.168.69.104|:80... connected.
>>> HTTP request sent, awaiting response... 200 OK
>>> Length: 1048576000 (1000M)
>>> Saving to: `myfile.dat''
>>>
>>>
100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>>>
>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>
>>>
>>> real	3m44.364s
>>> user	0m1.042s
>>> sys	0m24.065s
>>>
>>>
>>>
>>>>
>>>>>
>>>>> The wget command was telling me it was getting ~80Mbps
which is under the threshold of the flow.
>>>>>
>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>>> --13:21:49--  http://sol/myfile.dat
>>>>> Resolving sol... 192.168.69.104
>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>> HTTP request sent, awaiting response... 200 OK
>>>>> Length: 1048576000 (1000M)
>>>>> Saving to: `myfile.dat''
>>>>>
>>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>>>
>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>>>
>>>>>
>>>>> real	1m33.701s
>>>>> user	0m0.899s
>>>>> sys	0m23.874s
>>>>>
>>>>> Maybe a value of 100Mbps is too high for this machine.  I
might try with 50Mbps to see what wget says.
>>>>>
>>>>
>>>> OK,
>>>>
>>>> -venu
>>>>
>>>>> -Cesar
>>>>>
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>> http-flow    01:32:22      01:32:42     
1512     2571     0.001 Mbps
>>>>>>>>> ssh-flow     01:32:42      01:33:02     
1818     3578     0.002 Mbps
>>>>>>>>> http-flow    01:33:02      01:33:22     
66917    3165136      1.292 Mbp
>>>>>>>>> ssh-flow     01:33:02      01:33:22     
3618     5344     0.003 Mbps
>>>>>>>>> http-flow    01:33:22      01:33:42     
117947   5713018      2.332 Mbp
>>>>>>>>> ssh-flow     01:33:22      01:33:42     
4182     3020     0.002 Mbps
>>>>>>>>> http-flow    01:33:42      01:34:02     
118998   5685520      2.321 Mbp
>>>>>>>>> ssh-flow     01:33:42      01:34:02     
11616    9924     0.008 Mbps
>>>>>>>>> http-flow    01:34:02      01:34:22     
117084   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:34:22      01:34:42     
119130   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:34:42      01:35:02     
114180   5725168      2.335 Mbp
>>>>>>>>> http-flow    01:35:02      01:35:22     
109230   5725664      2.333 Mbp
>>>>>>>>> http-flow    01:35:22      01:35:42     
116160   5725168      2.336 Mbp
>>>>>>>>> http-flow    01:35:42      01:36:02     
119262   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:36:02      01:36:22     
119196   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:36:22      01:36:42     
117216   5725168      2.336 Mbp
>>>>>>>>> http-flow    01:36:42      01:37:02     
119394   5722636      2.336 Mbp
>>>>>>>>> http-flow    01:37:02      01:37:22     
119526   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:37:22      01:37:42     
119460   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:37:42      01:38:02     
119460   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:38:02      01:38:22     
119724   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:38:22      01:38:42     
119724   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:38:42      01:39:02     
119130   5722636      2.336 Mbp
>>>>>>>>> http-flow    01:39:02      01:39:22     
118866   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:39:22      01:39:42     
116490   5725664      2.336 Mbp
>>>>>>>>> http-flow    01:39:42      01:40:02     
119790   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:40:02      01:40:22     
117678   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:40:22      01:40:42     
118668   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:40:42      01:41:02     
117414   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:41:02      01:41:22     
119790   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:41:22      01:41:42     
119813   5720510      2.336 Mbp
>>>>>>>>> http-flow    01:41:42      01:42:02     
119394   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:42:02      01:42:22     
119724   5722272      2.336 Mbp
>>>>>>>>> http-flow    01:42:22      01:42:42     
119526   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:42:42      01:43:02     
119196   5722140      2.336 Mbp
>>>>>>>>> http-flow    01:43:02      01:43:22     
119394   5725664      2.338 Mbp
>>>>>>>>> http-flow    01:43:22      01:43:42     
119658   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:43:42      01:44:02     
119064   5725168      2.337 Mbp
>>>>>>>>> http-flow    01:44:02      01:44:22     
113256   5676668      2.315 Mbp
>>>>>>>>> ssh-flow     01:44:02      01:44:22     
18414    49646    0.027 Mbps
>>>>>>>>> http-flow    01:44:22      01:44:42     
118206   5725664      2.337 Mbp
>>>>>>>>> http-flow    01:44:42      01:45:02     
117282   5722140      2.335 Mbp
>>>>>>>>> ssh-flow     01:44:42      01:45:02     
4698     3544     0.003 Mbps
>>>>>>>>> http-flow    01:45:02      01:45:22     
118536   5688284      2.322 Mbp
>>>>>>>>> ssh-flow     01:45:02      01:45:22     
4092     3198     0.002 Mbps
>>>>>>>>> http-flow    01:45:22      01:45:42     
119130   5725168      2.337 Mbp
>>>>>>>>> ssh-flow     01:45:22      01:45:42     
1980     1478     0.001 Mbps
>>>>>>>>>
>>>>>>>>> That''s above the flow''s
maxbw parameter.  After that I tried to change the  maxbw of the link with dladm
and that brought the bandwidth down but still not down to 1.2 Mbps.
>>>>>>>>>
>>>>>>>>> root at myhost:~# dladm show-linkprop -p
maxbw e1000g0
>>>>>>>>> LINK         PROPERTY        PERM VALUE    
DEFAULT        POSSIBLE
>>>>>>>>> e1000g0      maxbw           rw       1.228
--             --
>>>>>>>>>
>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>> http-flow    01:46:02      01:46:22     
119394   5725168      2.337 Mbp
>>>>>>>>> ssh-flow     01:46:02      01:46:22     
4680     2980     0.003 Mbps
>>>>>>>>> http-flow    01:46:22      01:46:42     
94314    4520316      1.845 Mbp
>>>>>>>>>
>>>>>>>>> Any ideas or is there a subtlety that
I''m missing and the behavior is correct?
>>>>>>>>>
>>>>>>>>> Thanks for the help.
>>>>>>>>>
>>>>>>>>> -Cesar
>>>>>>>>>
>>>>>>>>>
_______________________________________________
>>>>>>>>> crossbow-discuss mailing list
>>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

Cesar Delgado

2009-Dec-16 06:26 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Venu,

The machine is just a little Intel PC that was given to me.  It''s the
only PC I have that passed the hardware test.  No drivers were needed to run
OpenSolaris on the box.  What''s the best way for me to send you the
specs of the box?  I''ll also try using your settings on my box to see
what happens.

Thank you very much,

-Cesar

On Dec 15, 2009, at 5:30 PM, venugopal iyer wrote:
> 
> Sorry, Cesar, for the delayed response.
> 
> I tried setting up your scenario on a recent version
> (build 130) and see that the bandwidth is close to what I am
> setting.
> 
> # flowadm show-flowprop tcp-flow
> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
> tcp-flow     maxbw               1.200      --             1200K tcp-flow  
priority        --             --             ?
> 
> 
> # wget 10.0.0.2/myfile.dat
> --2009-12-15 15:47:24--  http://10.0.0.2/myfile.dat
> Connecting to 10.0.0.2:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 46104240 (44M) [text/plain]
> Saving to: `myfile.dat.4''
> 
> 100%[======================================>] 46,104,240   160K/s   in
4m 42s
> 
> 2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved
[46104240/46104240]
> 
> 160 KB/s  is 1.280 Mbps.
> 
> I see this consistently.
> 
> 
> The kstat does give me slightly higher about 1.3 Mbps.
> 
> FYI, without the flow (with limt) in place, I see
> 
> # wget 10.0.0.2/myfile.dat
> --2009-12-15 15:47:12--  http://10.0.0.2/myfile.dat
> Connecting to 10.0.0.2:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 46104240 (44M) [text/plain]
> Saving to: `myfile.dat.3''
> 
> 100%[======================================>] 46,104,240  78.2M/s   in
0.6s
> 
> 2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved
[46104240/46104240]
> 
> 
> What kind of hardware are you using?
> 
> -venu
> 
> 
> On Mon, 14 Dec 2009, Cesar Delgado wrote:
> 
>> Venu,
>> 
>> Any new information on this?  Is it a bug?
>> 
>> -Cesar
>> 
>> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
>> 
>>> 
>>> Hi, Cesar:
>>> 
>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>> 
>>>> 
>>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>>>> 
>>>>>> Venu,
>>>>>> 
>>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>>>> 
>>>>>>> 
>>>>>>> Hi, Cesar:
>>>>>>> 
>>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>>>> 
>>>>>>>> Venugopal,
>>>>>>>> 
>>>>>>>> I''m sorry if these sounds like basic
questions.  I really appreciate the patience and the help.  Replies in-line.
>>>>>>>> 
>>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer
wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi, Cesar:
>>>>>>>>> 
>>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote:
>>>>>>>>> 
>>>>>>>>>> I''m setting up a server to go
to a hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>>>> 
>>>>>>>>>> FIrst I have a question on
flowadm''s show-usage command.  When I try to run show-prop with the
name of a flow I get an error.  The flow exists.  What am I doing wrong?
>>>>>>>>>> 
>>>>>>>>>> root at myhost:~# flowadm show-usage -f
/var/log/net.log http-flow
>>>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>>>> 
>>>>>>>>> This is a bug, I have submitted
>>>>>>>>> 
>>>>>>>>> 6904427 flowadm show-usage doesn''t
work with a flow name
>>>>>>>> 
>>>>>>>> Thanks for submitting that.  I haven''t
been able to find a link to the bugtracker for Crossbow.  Could you please send
me the URL?
>>>>>>> 
>>>>>>> I think it should show up on
>>>>>>> http://bugs.opensolaris.org/bugdatabase/index.jsp
soon.
>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Ok, now for my problem.  I have the
following setting:
>>>>>>>>>> 
>>>>>>>>>> root at myhost:~# flowadm show-flowprop
http-flow
>>>>>>>>>> FLOW         PROPERTY        VALUE     
DEFAULT        POSSIBLE
>>>>>>>>>> http-flow    maxbw               1.228 
--             1228k
>>>>>>>>>> http-flow    priority        medium    
--             medium
>>>>>>>>>> 
>>>>>>>>>> I ran a test hitting the webserver and
I see this:
>>>>>>>>> 
>>>>>>>>> I have the following flow
>>>>>>>>> 
>>>>>>>>> # flowadm show-flow FLOW        LINK       
IPADDR                   PROTO  LPORT   RPORT   DSFLD
>>>>>>>>> tcp-flow    <link>      --           
tcp    --      --      --
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>>>> FLOW         PROPERTY        VALUE         
DEFAULT        POSSIBLE
>>>>>>>>> tcp-flow     maxbw               1.228     
--             1228K tcp-flow     priority        --             --            
?
>>>>>>>>> 
>>>>>>>>> When I send TCP traffic (am using a traffic
generator - netperf, to
>>>>>>>>> this machine from a peer) for about 2 mins.
>>>>>>>>> 
>>>>>>>>> On the peer the traffic generator (sender)
says I am capped to about
>>>>>>>>> 1.14 Mbps.
>>>>>>>>> 
>>>>>>>>> Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
>>>>>>>>> 
>>>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>>>> 
>>>>>>>>> Now, when I try show-usage during the
traffic flow on
>>>>>>>>> the machine with the above flow in place
(receiver), I am seeing:
>>>>>>>>> 
>>>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>>>> FLOW         START         END          
RBYTES   OBYTES   BANDWIDTH
>>>>>>>>> tcp-flow     08:51:48      08:52:08     
3428658  107802       1.414 Mbp
>>>>>>>>> tcp-flow     08:52:08      08:52:28     
3431198  107802       1.415 Mbp
>>>>>>>>> tcp-flow     08:52:28      08:52:48     
3434614  107888       1.417 Mbp
>>>>>>>>> tcp-flow     08:52:48      08:53:08     
3443298  107802       1.420 Mbp
>>>>>>>>> tcp-flow     08:53:08      08:53:28     
3444324  107802       1.420 Mbp
>>>>>>>>> tcp-flow     08:53:28      08:53:48     
1376806  43576    0.568 Mbps
>>>>>>>>> 
>>>>>>>>> ...
>>>>>>>>> 
>>>>>>>>> I think the difference you see is likely to
be because of the time
>>>>>>>>> period when the stats are written to the
file (the bandwidth is computed for every 20 seconds period which might not be
exactly in
>>>>>>>>> sync with the bandwidth enforcement period
in the kernel) and also
>>>>>>>>> could be because of rounding up etc. But,
if you look at the  entire
>>>>>>>>> duration, it averages to about the
configured limit (in the above
>>>>>>>>> example, I think it is about 1.275 Mbps for
the 2 min duration)
>>>>>>>> 
>>>>>>>> The way I''m testing it is setting up
Apache and then moving down a file with `wget`.  The use case for this machine
is an Apache based app that serves large files to customers.  That is why I
think a `wget` is more telling of "real" performance than netperf. 
I''m running the test again and on the client side I am seeing usage
over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer rate
which is much closer to what I was seeing in the show-usage statistics.
>>>>>>>> 
>>>>>>> 
>>>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat
>>>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>>>> Resolving sol... 192.168.69.104
>>>>>>>> Connecting to sol|192.168.69.104|:80...
connected.
>>>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>>>> Length: 1048576000 (1000M)
>>>>>>>> Saving to: `myfile.dat''
>>>>>>>> 
>>>>>>>> 5% [==>                                     
] 55,530,974   267K/s  eta 60m 44s
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>>>> flow as the bandwidth for both are
independent.
>>>>>>>> 
>>>>>>>> Thank you for this clarification but I still
don''t understand how I can be seeing ~2Mbps transfer if both the link
and the flow are both capped at 1.2Mbps.
>>>>>>>> 
>>>>>>> 
>>>>>>> Can you try with a higher bandwidth, say 100 Mbps
and see what the results
>>>>>>> are when compared to wget''s output?
>>>>>>> 
>>>>>>> Also, another way of manually checking would be to
do a
>>>>>>> 	# kstat -c flow -n http-flow
>>>>>>> 
>>>>>>> before and after the wget run and see how many
bytes (rbytes) the
>>>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>>>> going over the flow), and then determine the
bandwidth (you might need
>>>>>>> to get the duration of the wget run pretty close to
get the
>>>>>>> right bandwdith value).
>>>>>>> 
>>>>>>> -venu
>>>>>> 
>>>>>> I changed the flow to be 100Mbps.
>>>>>> 
>>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw
http-flow
>>>>>> FLOW         PROPERTY        VALUE          DEFAULT    
POSSIBLE
>>>>>> http-flow    maxbw             100          --         
100
>>>>>> 
>>>>>> I also removed the maxbw=1.228 I had set on the link. 
When I changed this value on the link I lost all network to the machine.  Had to
go in and manually reboot it.  The network came up fine.
>>>>>> 
>>>>>> I created two files one before wget and one after with
no other network traffic except the ssh session.  I created a 1 GB file to
transfer.  The output is a bit confusing to me.
>>>>>> 
>>>>>> root at myhost:/tmp# ls http-flow-*
>>>>>> http-flow-1.txt  http-flow-2.txt
>>>>>> root at myhost:/tmp# cat http-flow-*
>>>>>> module: unix                            instance: 0
>>>>>> name:   http-flow                       class:    flow
>>>>>> 	crtime                          188662.418688816
>>>>>> 	ierrors                         0
>>>>>> 	ipackets                        4680
>>>>>> 	obytes                          182520974
>>>>>> 	oerrors                         0
>>>>>> 	opackets                        127521
>>>>>> 	rbytes                          308138
>>>>>> 	snaptime                        189431.882363589
>>>>>> 
>>>>>> module: unix                            instance: 0
>>>>>> name:   http-flow                       class:    flow
>>>>>> 	crtime                          188662.418688816
>>>>>> 	ierrors                         0
>>>>>> 	ipackets                        31262
>>>>>> 	obytes                          1281785975
>>>>>> 	oerrors                         0
>>>>>> 	opackets                        895533
>>>>>> 	rbytes                          2062678
>>>>>> 	snaptime                        189595.122347726
>>>>>> 
>>>>>> So this means the network only passed ~1.6 MB of data?
>>>>> 
>>>>> is this on the outbound side or inbound?
>>>>> 
>>>>> looks like it has a total of 8794120008 bits in the
outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the
duration
>>>>> 93 seconds? So the outbound would be ~94 Mbps and inbound
150 Kbps.
>>>>> 
>>>> 
>>>> Sorry, outbound.  I was looking at the wrong row.  With the
flow set at 50 I''m getting getting ~37 Mbps.  In my mind the flow is
not respecting the value I''m giving it.  With the maxbw set to 100 I
could get ~80 Mbps and now I''m only seeing ~37Mbps.   Could it be
something in the way Crossbow is averaging the actually bits and bytes going
down the pipe or is it me not understanding the system?
>>> 
>>> For TCP you might get a little less than the set bandwidth as TCP
>>> would do flow control too (i.e if for bandwidth reasons ACKs come
back
>>> slower or packets get dropped, TCP will react and will slow down).
Also,
>>> note that the current property (maxbw) is a *limit* which means we
will
>>> ensure that the flow will not get *more* than what you have asked
for.
>>> Going forwards we are looking at adding guarantee where in the flow
would
>>> get what it asked for.
>>> 
>>> -venu
>>> 
>>>> 
>>>> root at myhost:/tmp# flowadm show-flowprop http-flow
>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>> http-flow    maxbw              50          --             50
>>>> http-flow    priority        medium         --            
medium
>>>> 
>>>> root at myhost:/tmp# cat http-flow-*
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        32346
>>>> 	obytes                          1300129135
>>>> 	oerrors                         0
>>>> 	opackets                        908351
>>>> 	rbytes                          2133486
>>>> 	snaptime                        191261.607310645
>>>> 
>>>> module: unix                            instance: 0
>>>> name:   http-flow                       class:    flow
>>>> 	crtime                          188662.418688816
>>>> 	ierrors                         0
>>>> 	ipackets                        85361
>>>> 	obytes                          2399401106
>>>> 	oerrors                         0
>>>> 	opackets                        1676468
>>>> 	rbytes                          5632604
>>>> 	snaptime                        192621.873203347
>>>> 
>>>> 
>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>> --13:52:20--  http://sol/myfile.dat
>>>> Resolving sol... 192.168.69.104
>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>> HTTP request sent, awaiting response... 200 OK
>>>> Length: 1048576000 (1000M)
>>>> Saving to: `myfile.dat''
>>>> 
>>>>
100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>>>> 
>>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>> 
>>>> 
>>>> real	3m44.364s
>>>> user	0m1.042s
>>>> sys	0m24.065s
>>>> 
>>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> The wget command was telling me it was getting ~80Mbps
which is under the threshold of the flow.
>>>>>> 
>>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>>>> --13:21:49--  http://sol/myfile.dat
>>>>>> Resolving sol... 192.168.69.104
>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>> Length: 1048576000 (1000M)
>>>>>> Saving to: `myfile.dat''
>>>>>> 
>>>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>>>> 
>>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>>>> 
>>>>>> 
>>>>>> real	1m33.701s
>>>>>> user	0m0.899s
>>>>>> sys	0m23.874s
>>>>>> 
>>>>>> Maybe a value of 100Mbps is too high for this machine. 
I might try with 50Mbps to see what wget says.
>>>>>> 
>>>>> 
>>>>> OK,
>>>>> 
>>>>> -venu
>>>>> 
>>>>>> -Cesar
>>>>>> 
>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>>> http-flow    01:32:22      01:32:42    
1512     2571     0.001 Mbps
>>>>>>>>>> ssh-flow     01:32:42      01:33:02    
1818     3578     0.002 Mbps
>>>>>>>>>> http-flow    01:33:02      01:33:22    
66917    3165136      1.292 Mbp
>>>>>>>>>> ssh-flow     01:33:02      01:33:22    
3618     5344     0.003 Mbps
>>>>>>>>>> http-flow    01:33:22      01:33:42    
117947   5713018      2.332 Mbp
>>>>>>>>>> ssh-flow     01:33:22      01:33:42    
4182     3020     0.002 Mbps
>>>>>>>>>> http-flow    01:33:42      01:34:02    
118998   5685520      2.321 Mbp
>>>>>>>>>> ssh-flow     01:33:42      01:34:02    
11616    9924     0.008 Mbps
>>>>>>>>>> http-flow    01:34:02      01:34:22    
117084   5725664      2.337 Mbp
>>>>>>>>>> http-flow    01:34:22      01:34:42    
119130   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:34:42      01:35:02    
114180   5725168      2.335 Mbp
>>>>>>>>>> http-flow    01:35:02      01:35:22    
109230   5725664      2.333 Mbp
>>>>>>>>>> http-flow    01:35:22      01:35:42    
116160   5725168      2.336 Mbp
>>>>>>>>>> http-flow    01:35:42      01:36:02    
119262   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:36:02      01:36:22    
119196   5725664      2.337 Mbp
>>>>>>>>>> http-flow    01:36:22      01:36:42    
117216   5725168      2.336 Mbp
>>>>>>>>>> http-flow    01:36:42      01:37:02    
119394   5722636      2.336 Mbp
>>>>>>>>>> http-flow    01:37:02      01:37:22    
119526   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:37:22      01:37:42    
119460   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:37:42      01:38:02    
119460   5725664      2.338 Mbp
>>>>>>>>>> http-flow    01:38:02      01:38:22    
119724   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:38:22      01:38:42    
119724   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:38:42      01:39:02    
119130   5722636      2.336 Mbp
>>>>>>>>>> http-flow    01:39:02      01:39:22    
118866   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:39:22      01:39:42    
116490   5725664      2.336 Mbp
>>>>>>>>>> http-flow    01:39:42      01:40:02    
119790   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:40:02      01:40:22    
117678   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:40:22      01:40:42    
118668   5725664      2.337 Mbp
>>>>>>>>>> http-flow    01:40:42      01:41:02    
117414   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:41:02      01:41:22    
119790   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:41:22      01:41:42    
119813   5720510      2.336 Mbp
>>>>>>>>>> http-flow    01:41:42      01:42:02    
119394   5725664      2.338 Mbp
>>>>>>>>>> http-flow    01:42:02      01:42:22    
119724   5722272      2.336 Mbp
>>>>>>>>>> http-flow    01:42:22      01:42:42    
119526   5725664      2.338 Mbp
>>>>>>>>>> http-flow    01:42:42      01:43:02    
119196   5722140      2.336 Mbp
>>>>>>>>>> http-flow    01:43:02      01:43:22    
119394   5725664      2.338 Mbp
>>>>>>>>>> http-flow    01:43:22      01:43:42    
119658   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:43:42      01:44:02    
119064   5725168      2.337 Mbp
>>>>>>>>>> http-flow    01:44:02      01:44:22    
113256   5676668      2.315 Mbp
>>>>>>>>>> ssh-flow     01:44:02      01:44:22    
18414    49646    0.027 Mbps
>>>>>>>>>> http-flow    01:44:22      01:44:42    
118206   5725664      2.337 Mbp
>>>>>>>>>> http-flow    01:44:42      01:45:02    
117282   5722140      2.335 Mbp
>>>>>>>>>> ssh-flow     01:44:42      01:45:02    
4698     3544     0.003 Mbps
>>>>>>>>>> http-flow    01:45:02      01:45:22    
118536   5688284      2.322 Mbp
>>>>>>>>>> ssh-flow     01:45:02      01:45:22    
4092     3198     0.002 Mbps
>>>>>>>>>> http-flow    01:45:22      01:45:42    
119130   5725168      2.337 Mbp
>>>>>>>>>> ssh-flow     01:45:22      01:45:42    
1980     1478     0.001 Mbps
>>>>>>>>>> 
>>>>>>>>>> That''s above the
flow''s maxbw parameter.  After that I tried to change the  maxbw of the
link with dladm and that brought the bandwidth down but still not down to 1.2
Mbps.
>>>>>>>>>> 
>>>>>>>>>> root at myhost:~# dladm show-linkprop
-p maxbw e1000g0
>>>>>>>>>> LINK         PROPERTY        PERM VALUE
DEFAULT        POSSIBLE
>>>>>>>>>> e1000g0      maxbw           rw      
1.228      --             --
>>>>>>>>>> 
>>>>>>>>>> root at myhost:~# flowadm show-usage -s
11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0
Mbps\|^FLOW"
>>>>>>>>>> http-flow    01:46:02      01:46:22    
119394   5725168      2.337 Mbp
>>>>>>>>>> ssh-flow     01:46:02      01:46:22    
4680     2980     0.003 Mbps
>>>>>>>>>> http-flow    01:46:22      01:46:42    
94314    4520316      1.845 Mbp
>>>>>>>>>> 
>>>>>>>>>> Any ideas or is there a subtlety that
I''m missing and the behavior is correct?
>>>>>>>>>> 
>>>>>>>>>> Thanks for the help.
>>>>>>>>>> 
>>>>>>>>>> -Cesar
>>>>>>>>>> 
>>>>>>>>>>
_______________________________________________
>>>>>>>>>> crossbow-discuss mailing list
>>>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
> 
>

venugopal iyer

2009-Dec-16 17:47 UTC

head link

[crossbow-discuss] Flows aren''t respecting my limits

Hi, Ceasr:

On Tue, 15 Dec 2009, Cesar Delgado wrote:
> Venu,
>
> The machine is just a little Intel PC that was given to me.  It''s
the only PC I have that passed the hardware test.  No drivers were needed to run
OpenSolaris on the box.  What''s the best way for me to send you the
specs of the box?>
I think that is fine, I was just curious. It might not have much to
to do with the hardware (info. on the nic on the system might be
useful - I am using bge card - Broadcom), given we are testing very
low bandwidth (and seeing higher than expected throughput).  One
thing that may be different on my setup is that both the machines
were connected back to back, i.e. this (wget) was the only traffic
on the network.

I am assuming you are running Opensolaris 2009.06, right? and the
normal MTU (1500) sized packets, right?

-venu
>I''ll also try using your settings on my box to see what happens.
>
> Thank you very much,
>
> -Cesar
>
> On Dec 15, 2009, at 5:30 PM, venugopal iyer wrote:
>
>>
>> Sorry, Cesar, for the delayed response.
>>
>> I tried setting up your scenario on a recent version
>> (build 130) and see that the bandwidth is close to what I am
>> setting.
>>
>> # flowadm show-flowprop tcp-flow
>> FLOW         PROPERTY        VALUE          DEFAULT        POSSIBLE
>> tcp-flow     maxbw               1.200      --             1200K
tcp-flow     priority        --             --             ?
>>
>>
>> # wget 10.0.0.2/myfile.dat
>> --2009-12-15 15:47:24--  http://10.0.0.2/myfile.dat
>> Connecting to 10.0.0.2:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 46104240 (44M) [text/plain]
>> Saving to: `myfile.dat.4''
>>
>> 100%[======================================>] 46,104,240   160K/s  
in 4m 42s
>>
>> 2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved
[46104240/46104240]
>>
>> 160 KB/s  is 1.280 Mbps.
>>
>> I see this consistently.
>>
>>
>> The kstat does give me slightly higher about 1.3 Mbps.
>>
>> FYI, without the flow (with limt) in place, I see
>>
>> # wget 10.0.0.2/myfile.dat
>> --2009-12-15 15:47:12--  http://10.0.0.2/myfile.dat
>> Connecting to 10.0.0.2:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 46104240 (44M) [text/plain]
>> Saving to: `myfile.dat.3''
>>
>> 100%[======================================>] 46,104,240  78.2M/s  
in 0.6s
>>
>> 2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved
[46104240/46104240]
>>
>>
>> What kind of hardware are you using?
>>
>> -venu
>>
>>
>> On Mon, 14 Dec 2009, Cesar Delgado wrote:
>>
>>> Venu,
>>>
>>> Any new information on this?  Is it a bug?
>>>
>>> -Cesar
>>>
>>> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:
>>>
>>>>
>>>> Hi, Cesar:
>>>>
>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>>>
>>>>>
>>>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote:
>>>>>>
>>>>>>> Venu,
>>>>>>>
>>>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi, Cesar:
>>>>>>>>
>>>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote:
>>>>>>>>
>>>>>>>>> Venugopal,
>>>>>>>>>
>>>>>>>>> I''m sorry if these sounds like
basic questions.  I really appreciate the patience and the help.  Replies
in-line.
>>>>>>>>>
>>>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer
wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, Cesar:
>>>>>>>>>>
>>>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado
wrote:
>>>>>>>>>>
>>>>>>>>>>> I''m setting up a server to
go to a hosting site where I have a 1Mbps pipe.  From what I read I know I
can''t set the limit to this as the lowest setting is ~1.2Mbps and this
is something that''s getting worked on in Crossbow2.  I am seeing some
strange behavior.
>>>>>>>>>>>
>>>>>>>>>>> FIrst I have a question on
flowadm''s show-usage command.  When I try to run show-prop with the
name of a flow I get an error.  The flow exists.  What am I doing wrong?
>>>>>>>>>>>
>>>>>>>>>>> root at myhost:~# flowadm
show-usage -f /var/log/net.log http-flow
>>>>>>>>>>> flowadm: invalid flow:
''(null)''
>>>>>>>>>>
>>>>>>>>>> This is a bug, I have submitted
>>>>>>>>>>
>>>>>>>>>> 6904427 flowadm show-usage
doesn''t work with a flow name
>>>>>>>>>
>>>>>>>>> Thanks for submitting that.  I
haven''t been able to find a link to the bugtracker for Crossbow.  Could
you please send me the URL?
>>>>>>>>
>>>>>>>> I think it should show up on
>>>>>>>>
http://bugs.opensolaris.org/bugdatabase/index.jsp soon.
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ok, now for my problem.  I have the
following setting:
>>>>>>>>>>>
>>>>>>>>>>> root at myhost:~# flowadm
show-flowprop http-flow
>>>>>>>>>>> FLOW         PROPERTY        VALUE 
DEFAULT        POSSIBLE
>>>>>>>>>>> http-flow    maxbw              
1.228      --             1228k
>>>>>>>>>>> http-flow    priority        medium
--             medium
>>>>>>>>>>>
>>>>>>>>>>> I ran a test hitting the webserver
and I see this:
>>>>>>>>>>
>>>>>>>>>> I have the following flow
>>>>>>>>>>
>>>>>>>>>> # flowadm show-flow FLOW        LINK   
IPADDR                   PROTO  LPORT   RPORT   DSFLD
>>>>>>>>>> tcp-flow    <link>      --       
tcp    --      --      --
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> # flowadm show-flowprop tcp-flow
>>>>>>>>>> FLOW         PROPERTY        VALUE     
DEFAULT        POSSIBLE
>>>>>>>>>> tcp-flow     maxbw               1.228 
--             1228K tcp-flow     priority        --             --            
?
>>>>>>>>>>
>>>>>>>>>> When I send TCP traffic (am using a
traffic generator - netperf, to
>>>>>>>>>> this machine from a peer) for about 2
mins.
>>>>>>>>>>
>>>>>>>>>> On the peer the traffic generator
(sender) says I am capped to about
>>>>>>>>>> 1.14 Mbps.
>>>>>>>>>>
>>>>>>>>>> Size   Size    Size     Time    
Throughput bytes  bytes   bytes    secs.    10^6bits/sec
>>>>>>>>>>
>>>>>>>>>> 49152  49152  49152    120.49      1.14
>>>>>>>>>>
>>>>>>>>>> Now, when I try show-usage during the
traffic flow on
>>>>>>>>>> the machine with the above flow in
place (receiver), I am seeing:
>>>>>>>>>>
>>>>>>>>>> # flowadm show-usage -s 11/24/2009 -f
/var/tmp/tcpflow
>>>>>>>>>> FLOW         START         END         
RBYTES   OBYTES   BANDWIDTH
>>>>>>>>>> tcp-flow     08:51:48      08:52:08    
3428658  107802       1.414 Mbp
>>>>>>>>>> tcp-flow     08:52:08      08:52:28    
3431198  107802       1.415 Mbp
>>>>>>>>>> tcp-flow     08:52:28      08:52:48    
3434614  107888       1.417 Mbp
>>>>>>>>>> tcp-flow     08:52:48      08:53:08    
3443298  107802       1.420 Mbp
>>>>>>>>>> tcp-flow     08:53:08      08:53:28    
3444324  107802       1.420 Mbp
>>>>>>>>>> tcp-flow     08:53:28      08:53:48    
1376806  43576    0.568 Mbps
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>> I think the difference you see is
likely to be because of the time
>>>>>>>>>> period when the stats are written to
the file (the bandwidth is computed for every 20 seconds period which might not
be exactly in
>>>>>>>>>> sync with the bandwidth enforcement
period in the kernel) and also
>>>>>>>>>> could be because of rounding up etc.
But, if you look at the  entire
>>>>>>>>>> duration, it averages to about the
configured limit (in the above
>>>>>>>>>> example, I think it is about 1.275 Mbps
for the 2 min duration)
>>>>>>>>>
>>>>>>>>> The way I''m testing it is setting
up Apache and then moving down a file with `wget`.  The use case for this
machine is an Apache based app that serves large files to customers.  That is
why I think a `wget` is more telling of "real" performance than
netperf.  I''m running the test again and on the client side I am seeing
usage over the maxbw limit I have set.  `wget` is reporting about 2Mbps transfer
rate which is much closer to what I was seeing in the show-usage statistics.
>>>>>>>>>
>>>>>>>>
>>>>>>>>> [cdelgado at Bluegene tmp]$ wget
sol/myfile.dat
>>>>>>>>> --10:01:30--  http://sol/myfile.dat
>>>>>>>>> Resolving sol... 192.168.69.104
>>>>>>>>> Connecting to sol|192.168.69.104|:80...
connected.
>>>>>>>>> HTTP request sent, awaiting response... 200
OK
>>>>>>>>> Length: 1048576000 (1000M)
>>>>>>>>> Saving to: `myfile.dat''
>>>>>>>>>
>>>>>>>>> 5% [==>                                 
] 55,530,974   267K/s  eta 60m 44s
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> BTW, setting a maxbw for a link (dladm)
doesn''t really impact the
>>>>>>>>>> flow as the bandwidth for both are
independent.
>>>>>>>>>
>>>>>>>>> Thank you for this clarification but I
still don''t understand how I can be seeing ~2Mbps transfer if both the
link and the flow are both capped at 1.2Mbps.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Can you try with a higher bandwidth, say 100
Mbps and see what the results
>>>>>>>> are when compared to wget''s output?
>>>>>>>>
>>>>>>>> Also, another way of manually checking would be
to do a
>>>>>>>> 	# kstat -c flow -n http-flow
>>>>>>>>
>>>>>>>> before and after the wget run and see how many
bytes (rbytes) the
>>>>>>>> kernel has seen for that flow (assuming there
isn''t any other traffic
>>>>>>>> going over the flow), and then determine the
bandwidth (you might need
>>>>>>>> to get the duration of the wget run pretty
close to get the
>>>>>>>> right bandwdith value).
>>>>>>>>
>>>>>>>> -venu
>>>>>>>
>>>>>>> I changed the flow to be 100Mbps.
>>>>>>>
>>>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw
http-flow
>>>>>>> FLOW         PROPERTY        VALUE          DEFAULT
POSSIBLE
>>>>>>> http-flow    maxbw             100          --     
100
>>>>>>>
>>>>>>> I also removed the maxbw=1.228 I had set on the
link.  When I changed this value on the link I lost all network to the machine. 
Had to go in and manually reboot it.  The network came up fine.
>>>>>>>
>>>>>>> I created two files one before wget and one after
with no other network traffic except the ssh session.  I created a 1 GB file to
transfer.  The output is a bit confusing to me.
>>>>>>>
>>>>>>> root at myhost:/tmp# ls http-flow-*
>>>>>>> http-flow-1.txt  http-flow-2.txt
>>>>>>> root at myhost:/tmp# cat http-flow-*
>>>>>>> module: unix                            instance: 0
>>>>>>> name:   http-flow                       class:   
flow
>>>>>>> 	crtime                          188662.418688816
>>>>>>> 	ierrors                         0
>>>>>>> 	ipackets                        4680
>>>>>>> 	obytes                          182520974
>>>>>>> 	oerrors                         0
>>>>>>> 	opackets                        127521
>>>>>>> 	rbytes                          308138
>>>>>>> 	snaptime                        189431.882363589
>>>>>>>
>>>>>>> module: unix                            instance: 0
>>>>>>> name:   http-flow                       class:   
flow
>>>>>>> 	crtime                          188662.418688816
>>>>>>> 	ierrors                         0
>>>>>>> 	ipackets                        31262
>>>>>>> 	obytes                          1281785975
>>>>>>> 	oerrors                         0
>>>>>>> 	opackets                        895533
>>>>>>> 	rbytes                          2062678
>>>>>>> 	snaptime                        189595.122347726
>>>>>>>
>>>>>>> So this means the network only passed ~1.6 MB of
data?
>>>>>>
>>>>>> is this on the outbound side or inbound?
>>>>>>
>>>>>> looks like it has a total of 8794120008 bits in the
outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the
duration
>>>>>> 93 seconds? So the outbound would be ~94 Mbps and
inbound 150 Kbps.
>>>>>>
>>>>>
>>>>> Sorry, outbound.  I was looking at the wrong row.  With the
flow set at 50 I''m getting getting ~37 Mbps.  In my mind the flow is
not respecting the value I''m giving it.  With the maxbw set to 100 I
could get ~80 Mbps and now I''m only seeing ~37Mbps.   Could it be
something in the way Crossbow is averaging the actually bits and bytes going
down the pipe or is it me not understanding the system?
>>>>
>>>> For TCP you might get a little less than the set bandwidth as
TCP
>>>> would do flow control too (i.e if for bandwidth reasons ACKs
come back
>>>> slower or packets get dropped, TCP will react and will slow
down). Also,
>>>> note that the current property (maxbw) is a *limit* which means
we will
>>>> ensure that the flow will not get *more* than what you have
asked for.
>>>> Going forwards we are looking at adding guarantee where in the
flow would
>>>> get what it asked for.
>>>>
>>>> -venu
>>>>
>>>>>
>>>>> root at myhost:/tmp# flowadm show-flowprop http-flow
>>>>> FLOW         PROPERTY        VALUE          DEFAULT       
POSSIBLE
>>>>> http-flow    maxbw              50          --            
50
>>>>> http-flow    priority        medium         --            
medium
>>>>>
>>>>> root at myhost:/tmp# cat http-flow-*
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        32346
>>>>> 	obytes                          1300129135
>>>>> 	oerrors                         0
>>>>> 	opackets                        908351
>>>>> 	rbytes                          2133486
>>>>> 	snaptime                        191261.607310645
>>>>>
>>>>> module: unix                            instance: 0
>>>>> name:   http-flow                       class:    flow
>>>>> 	crtime                          188662.418688816
>>>>> 	ierrors                         0
>>>>> 	ipackets                        85361
>>>>> 	obytes                          2399401106
>>>>> 	oerrors                         0
>>>>> 	opackets                        1676468
>>>>> 	rbytes                          5632604
>>>>> 	snaptime                        192621.873203347
>>>>>
>>>>>
>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat
>>>>> --13:52:20--  http://sol/myfile.dat
>>>>> Resolving sol... 192.168.69.104
>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>> HTTP request sent, awaiting response... 200 OK
>>>>> Length: 1048576000 (1000M)
>>>>> Saving to: `myfile.dat''
>>>>>
>>>>>
100%[==========================================================>]
1,048,576,000 4.06M/s   in 3m 44s
>>>>>
>>>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>>>
>>>>>
>>>>> real	3m44.364s
>>>>> user	0m1.042s
>>>>> sys	0m24.065s
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> The wget command was telling me it was getting
~80Mbps which is under the threshold of the flow.
>>>>>>>
>>>>>>> [cdelgado at Bluegene tmp]$ time wget
sol/myfile.dat
>>>>>>> --13:21:49--  http://sol/myfile.dat
>>>>>>> Resolving sol... 192.168.69.104
>>>>>>> Connecting to sol|192.168.69.104|:80... connected.
>>>>>>> HTTP request sent, awaiting response... 200 OK
>>>>>>> Length: 1048576000 (1000M)
>>>>>>> Saving to: `myfile.dat''
>>>>>>>
>>>>>>>
100%[==========================================================>]
1,048,576,000 10.5M/s   in 93s
>>>>>>>
>>>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved
[1048576000/1048576000]
>>>>>>>
>>>>>>>
>>>>>>> real	1m33.701s
>>>>>>> user	0m0.899s
>>>>>>> sys	0m23.874s
>>>>>>>
>>>>>>> Maybe a value of 100Mbps is too high for this
machine.  I might try with 50Mbps to see what wget says.
>>>>>>>
>>>>>>
>>>>>> OK,
>>>>>>
>>>>>> -venu
>>>>>>
>>>>>>> -Cesar
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> root at myhost:~# flowadm
show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log |
grep -v "0 Mbps\|^FLOW"
>>>>>>>>>>> http-flow    01:32:22      01:32:42
1512     2571     0.001 Mbps
>>>>>>>>>>> ssh-flow     01:32:42      01:33:02
1818     3578     0.002 Mbps
>>>>>>>>>>> http-flow    01:33:02      01:33:22
66917    3165136      1.292 Mbp
>>>>>>>>>>> ssh-flow     01:33:02      01:33:22
3618     5344     0.003 Mbps
>>>>>>>>>>> http-flow    01:33:22      01:33:42
117947   5713018      2.332 Mbp
>>>>>>>>>>> ssh-flow     01:33:22      01:33:42
4182     3020     0.002 Mbps
>>>>>>>>>>> http-flow    01:33:42      01:34:02
118998   5685520      2.321 Mbp
>>>>>>>>>>> ssh-flow     01:33:42      01:34:02
11616    9924     0.008 Mbps
>>>>>>>>>>> http-flow    01:34:02      01:34:22
117084   5725664      2.337 Mbp
>>>>>>>>>>> http-flow    01:34:22      01:34:42
119130   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:34:42      01:35:02
114180   5725168      2.335 Mbp
>>>>>>>>>>> http-flow    01:35:02      01:35:22
109230   5725664      2.333 Mbp
>>>>>>>>>>> http-flow    01:35:22      01:35:42
116160   5725168      2.336 Mbp
>>>>>>>>>>> http-flow    01:35:42      01:36:02
119262   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:36:02      01:36:22
119196   5725664      2.337 Mbp
>>>>>>>>>>> http-flow    01:36:22      01:36:42
117216   5725168      2.336 Mbp
>>>>>>>>>>> http-flow    01:36:42      01:37:02
119394   5722636      2.336 Mbp
>>>>>>>>>>> http-flow    01:37:02      01:37:22
119526   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:37:22      01:37:42
119460   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:37:42      01:38:02
119460   5725664      2.338 Mbp
>>>>>>>>>>> http-flow    01:38:02      01:38:22
119724   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:38:22      01:38:42
119724   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:38:42      01:39:02
119130   5722636      2.336 Mbp
>>>>>>>>>>> http-flow    01:39:02      01:39:22
118866   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:39:22      01:39:42
116490   5725664      2.336 Mbp
>>>>>>>>>>> http-flow    01:39:42      01:40:02
119790   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:40:02      01:40:22
117678   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:40:22      01:40:42
118668   5725664      2.337 Mbp
>>>>>>>>>>> http-flow    01:40:42      01:41:02
117414   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:41:02      01:41:22
119790   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:41:22      01:41:42
119813   5720510      2.336 Mbp
>>>>>>>>>>> http-flow    01:41:42      01:42:02
119394   5725664      2.338 Mbp
>>>>>>>>>>> http-flow    01:42:02      01:42:22
119724   5722272      2.336 Mbp
>>>>>>>>>>> http-flow    01:42:22      01:42:42
119526   5725664      2.338 Mbp
>>>>>>>>>>> http-flow    01:42:42      01:43:02
119196   5722140      2.336 Mbp
>>>>>>>>>>> http-flow    01:43:02      01:43:22
119394   5725664      2.338 Mbp
>>>>>>>>>>> http-flow    01:43:22      01:43:42
119658   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:43:42      01:44:02
119064   5725168      2.337 Mbp
>>>>>>>>>>> http-flow    01:44:02      01:44:22
113256   5676668      2.315 Mbp
>>>>>>>>>>> ssh-flow     01:44:02      01:44:22
18414    49646    0.027 Mbps
>>>>>>>>>>> http-flow    01:44:22      01:44:42
118206   5725664      2.337 Mbp
>>>>>>>>>>> http-flow    01:44:42      01:45:02
117282   5722140      2.335 Mbp
>>>>>>>>>>> ssh-flow     01:44:42      01:45:02
4698     3544     0.003 Mbps
>>>>>>>>>>> http-flow    01:45:02      01:45:22
118536   5688284      2.322 Mbp
>>>>>>>>>>> ssh-flow     01:45:02      01:45:22
4092     3198     0.002 Mbps
>>>>>>>>>>> http-flow    01:45:22      01:45:42
119130   5725168      2.337 Mbp
>>>>>>>>>>> ssh-flow     01:45:22      01:45:42
1980     1478     0.001 Mbps
>>>>>>>>>>>
>>>>>>>>>>> That''s above the
flow''s maxbw parameter.  After that I tried to change the  maxbw of the
link with dladm and that brought the bandwidth down but still not down to 1.2
Mbps.
>>>>>>>>>>>
>>>>>>>>>>> root at myhost:~# dladm
show-linkprop -p maxbw e1000g0
>>>>>>>>>>> LINK         PROPERTY        PERM
VALUE          DEFAULT        POSSIBLE
>>>>>>>>>>> e1000g0      maxbw           rw    
1.228      --             --
>>>>>>>>>>>
>>>>>>>>>>> root at myhost:~# flowadm
show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log |
grep -v "0 Mbps\|^FLOW"
>>>>>>>>>>> http-flow    01:46:02      01:46:22
119394   5725168      2.337 Mbp
>>>>>>>>>>> ssh-flow     01:46:02      01:46:22
4680     2980     0.003 Mbps
>>>>>>>>>>> http-flow    01:46:22      01:46:42
94314    4520316      1.845 Mbp
>>>>>>>>>>>
>>>>>>>>>>> Any ideas or is there a subtlety
that I''m missing and the behavior is correct?
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the help.
>>>>>>>>>>>
>>>>>>>>>>> -Cesar
>>>>>>>>>>>
>>>>>>>>>>>
_______________________________________________
>>>>>>>>>>> crossbow-discuss mailing list
>>>>>>>>>>> crossbow-discuss at opensolaris.org
>>>>>>>>>>>
http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>
>

crossbow discuss - Nov 2009 - Flows aren't respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits

[crossbow-discuss] Flows aren''t respecting my limits