I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow flowadm: invalid flow: ''(null)'' Ok, now for my problem. I have the following setting: root at myhost:~# flowadm show-flowprop http-flow FLOW PROPERTY VALUE DEFAULT POSSIBLE http-flow maxbw 1.228 -- 1228k http-flow priority medium -- medium I ran a test hitting the webserver and I see this: root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. root at myhost:~# dladm show-linkprop -p maxbw e1000g0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE e1000g0 maxbw rw 1.228 -- -- root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp Any ideas or is there a subtlety that I''m missing and the behavior is correct? Thanks for the help. -Cesar
venugopal iyer
2009-Nov-24 17:29 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Hi, Cesar: On Mon, 23 Nov 2009, Cesar Delgado wrote:> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. > > FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? > > root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow > flowadm: invalid flow: ''(null)''This is a bug, I have submitted 6904427 flowadm show-usage doesn''t work with a flow name> > > Ok, now for my problem. I have the following setting: > > root at myhost:~# flowadm show-flowprop http-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > http-flow maxbw 1.228 -- 1228k > http-flow priority medium -- medium > > I ran a test hitting the webserver and I see this:I have the following flow # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD tcp-flow <link> -- tcp -- -- -- # flowadm show-flowprop tcp-flow FLOW PROPERTY VALUE DEFAULT POSSIBLE tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? When I send TCP traffic (am using a traffic generator - netperf, to this machine from a peer) for about 2 mins. On the peer the traffic generator (sender) says I am capped to about 1.14 Mbps. Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 49152 49152 49152 120.49 1.14 Now, when I try show-usage during the traffic flow on the machine with the above flow in place (receiver), I am seeing: # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow FLOW START END RBYTES OBYTES BANDWIDTH tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps ... I think the difference you see is likely to be because of the time period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in sync with the bandwidth enforcement period in the kernel) and also could be because of rounding up etc. But, if you look at the entire duration, it averages to about the configured limit (in the above example, I think it is about 1.275 Mbps for the 2 min duration) BTW, setting a maxbw for a link (dladm) doesn''t really impact the flow as the bandwidth for both are independent. thanks, -venu> > root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" > http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps > ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps > http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp > ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps > http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp > ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps > http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp > ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps > http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp > http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp > http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp > http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp > http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp > http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp > http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp > http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp > http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp > http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp > http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp > http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp > http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp > http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp > http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp > http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp > http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp > http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp > http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp > http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp > http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp > http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp > http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp > http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp > http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp > http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp > http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp > http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp > http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp > http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp > http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp > ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps > http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp > http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp > ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps > http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp > ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps > http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp > ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps > > That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. > > root at myhost:~# dladm show-linkprop -p maxbw e1000g0 > LINK PROPERTY PERM VALUE DEFAULT POSSIBLE > e1000g0 maxbw rw 1.228 -- -- > > root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" > http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp > ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps > http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp > > Any ideas or is there a subtlety that I''m missing and the behavior is correct? > > Thanks for the help. > > -Cesar > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
Venugopal, I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote:> > > Hi, Cesar: > > On Mon, 23 Nov 2009, Cesar Delgado wrote: > >> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >> >> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >> >> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >> flowadm: invalid flow: ''(null)'' > > This is a bug, I have submitted > > 6904427 flowadm show-usage doesn''t work with a flow nameThanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL?> > >> >> >> Ok, now for my problem. I have the following setting: >> >> root at myhost:~# flowadm show-flowprop http-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> http-flow maxbw 1.228 -- 1228k >> http-flow priority medium -- medium >> >> I ran a test hitting the webserver and I see this: > > I have the following flow > > # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD > tcp-flow <link> -- tcp -- -- -- > > > # flowadm show-flowprop tcp-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? > > When I send TCP traffic (am using a traffic generator - netperf, to > this machine from a peer) for about 2 mins. > > On the peer the traffic generator (sender) says I am capped to about > 1.14 Mbps. > > Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec > > 49152 49152 49152 120.49 1.14 > > Now, when I try show-usage during the traffic flow on > the machine with the above flow in place (receiver), I am seeing: > > # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow > FLOW START END RBYTES OBYTES BANDWIDTH > tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp > tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp > tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp > tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp > tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp > tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps > > ... > > I think the difference you see is likely to be because of the time > period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in > sync with the bandwidth enforcement period in the kernel) and also > could be because of rounding up etc. But, if you look at the entire > duration, it averages to about the configured limit (in the above > example, I think it is about 1.275 Mbps for the 2 min duration)The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. [cdelgado at Bluegene tmp]$ wget sol/myfile.dat --10:01:30-- sol/myfile.dat Resolving sol... 192.168.69.104 Connecting to sol|192.168.69.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) Saving to: `myfile.dat'' 5% [==> ] 55,530,974 267K/s eta 60m 44s> > BTW, setting a maxbw for a link (dladm) doesn''t really impact the > flow as the bandwidth for both are independent.Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps.> > > > > > > > > > > >> >> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >> >> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >> >> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >> e1000g0 maxbw rw 1.228 -- -- >> >> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >> >> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >> >> Thanks for the help. >> >> -Cesar >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>
venugopal iyer
2009-Nov-25 17:49 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Hi, Cesar: On Tue, 24 Nov 2009, Cesar Delgado wrote:> Venugopal, > > I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. > > On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: > >> >> >> Hi, Cesar: >> >> On Mon, 23 Nov 2009, Cesar Delgado wrote: >> >>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>> >>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>> >>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>> flowadm: invalid flow: ''(null)'' >> >> This is a bug, I have submitted >> >> 6904427 flowadm show-usage doesn''t work with a flow name > > Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL?I think it should show up on bugs.opensolaris.org/bugdatabase/index.jsp soon.>> >> >>> >>> >>> Ok, now for my problem. I have the following setting: >>> >>> root at myhost:~# flowadm show-flowprop http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 1.228 -- 1228k >>> http-flow priority medium -- medium >>> >>> I ran a test hitting the webserver and I see this: >> >> I have the following flow >> >> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >> tcp-flow <link> -- tcp -- -- -- >> >> >> # flowadm show-flowprop tcp-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >> >> When I send TCP traffic (am using a traffic generator - netperf, to >> this machine from a peer) for about 2 mins. >> >> On the peer the traffic generator (sender) says I am capped to about >> 1.14 Mbps. >> >> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >> >> 49152 49152 49152 120.49 1.14 >> >> Now, when I try show-usage during the traffic flow on >> the machine with the above flow in place (receiver), I am seeing: >> >> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >> FLOW START END RBYTES OBYTES BANDWIDTH >> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >> >> ... >> >> I think the difference you see is likely to be because of the time >> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >> sync with the bandwidth enforcement period in the kernel) and also >> could be because of rounding up etc. But, if you look at the entire >> duration, it averages to about the configured limit (in the above >> example, I think it is about 1.275 Mbps for the 2 min duration) > > The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat > --10:01:30-- sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat'' > > 5% [==> ] 55,530,974 267K/s eta 60m 44s > > >> >> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >> flow as the bandwidth for both are independent. > > Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >Can you try with a higher bandwidth, say 100 Mbps and see what the results are when compared to wget''s output? Also, another way of manually checking would be to do a # kstat -c flow -n http-flow before and after the wget run and see how many bytes (rbytes) the kernel has seen for that flow (assuming there isn''t any other traffic going over the flow), and then determine the bandwidth (you might need to get the duration of the wget run pretty close to get the right bandwdith value). -venu> >> >> >> >> >> >> >> >> >> >> >> >>> >>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>> >>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>> >>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>> e1000g0 maxbw rw 1.228 -- -- >>> >>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>> >>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>> >>> Thanks for the help. >>> >>> -Cesar >>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>> > >
Venu, On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote:> > Hi, Cesar: > > On Tue, 24 Nov 2009, Cesar Delgado wrote: > >> Venugopal, >> >> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >> >> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >> >>> >>> >>> Hi, Cesar: >>> >>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>> >>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>> >>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>> >>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>> flowadm: invalid flow: ''(null)'' >>> >>> This is a bug, I have submitted >>> >>> 6904427 flowadm show-usage doesn''t work with a flow name >> >> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? > > I think it should show up on > bugs.opensolaris.org/bugdatabase/index.jsp soon. > >>> >>> >>>> >>>> >>>> Ok, now for my problem. I have the following setting: >>>> >>>> root at myhost:~# flowadm show-flowprop http-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> http-flow maxbw 1.228 -- 1228k >>>> http-flow priority medium -- medium >>>> >>>> I ran a test hitting the webserver and I see this: >>> >>> I have the following flow >>> >>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>> tcp-flow <link> -- tcp -- -- -- >>> >>> >>> # flowadm show-flowprop tcp-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>> >>> When I send TCP traffic (am using a traffic generator - netperf, to >>> this machine from a peer) for about 2 mins. >>> >>> On the peer the traffic generator (sender) says I am capped to about >>> 1.14 Mbps. >>> >>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>> >>> 49152 49152 49152 120.49 1.14 >>> >>> Now, when I try show-usage during the traffic flow on >>> the machine with the above flow in place (receiver), I am seeing: >>> >>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>> FLOW START END RBYTES OBYTES BANDWIDTH >>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>> >>> ... >>> >>> I think the difference you see is likely to be because of the time >>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>> sync with the bandwidth enforcement period in the kernel) and also >>> could be because of rounding up etc. But, if you look at the entire >>> duration, it averages to about the configured limit (in the above >>> example, I think it is about 1.275 Mbps for the 2 min duration) >> >> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >> > >> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >> --10:01:30-- sol/myfile.dat >> Resolving sol... 192.168.69.104 >> Connecting to sol|192.168.69.104|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 1048576000 (1000M) >> Saving to: `myfile.dat'' >> >> 5% [==> ] 55,530,974 267K/s eta 60m 44s >> >> >>> >>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>> flow as the bandwidth for both are independent. >> >> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >> > > Can you try with a higher bandwidth, say 100 Mbps and see what the results > are when compared to wget''s output? > > Also, another way of manually checking would be to do a > # kstat -c flow -n http-flow > > before and after the wget run and see how many bytes (rbytes) the > kernel has seen for that flow (assuming there isn''t any other traffic > going over the flow), and then determine the bandwidth (you might need > to get the duration of the wget run pretty close to get the > right bandwdith value). > > -venuI changed the flow to be 100Mbps. root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow FLOW PROPERTY VALUE DEFAULT POSSIBLE http-flow maxbw 100 -- 100 I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. root at myhost:/tmp# ls http-flow-* http-flow-1.txt http-flow-2.txt root at myhost:/tmp# cat http-flow-* module: unix instance: 0 name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 4680 obytes 182520974 oerrors 0 opackets 127521 rbytes 308138 snaptime 189431.882363589 module: unix instance: 0 name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 31262 obytes 1281785975 oerrors 0 opackets 895533 rbytes 2062678 snaptime 189595.122347726 So this means the network only passed ~1.6 MB of data? The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat --13:21:49-- sol/myfile.dat Resolving sol... 192.168.69.104 Connecting to sol|192.168.69.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) Saving to: `myfile.dat'' 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] real 1m33.701s user 0m0.899s sys 0m23.874s Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. -Cesar>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>> >>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>> >>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>> >>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>> e1000g0 maxbw rw 1.228 -- -- >>>> >>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>> >>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>> >>>> Thanks for the help. >>>> >>>> -Cesar >>>> >>>> _______________________________________________ >>>> crossbow-discuss mailing list >>>> crossbow-discuss at opensolaris.org >>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>> >> >>
venugopal iyer
2009-Nov-25 22:05 UTC
[crossbow-discuss] Flows aren''t respecting my limits
On Wed, 25 Nov 2009, Cesar Delgado wrote:> Venu, > > On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: > >> >> Hi, Cesar: >> >> On Tue, 24 Nov 2009, Cesar Delgado wrote: >> >>> Venugopal, >>> >>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>> >>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>> >>>> >>>> >>>> Hi, Cesar: >>>> >>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>> >>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>> >>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>> >>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>> flowadm: invalid flow: ''(null)'' >>>> >>>> This is a bug, I have submitted >>>> >>>> 6904427 flowadm show-usage doesn''t work with a flow name >>> >>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >> >> I think it should show up on >> bugs.opensolaris.org/bugdatabase/index.jsp soon. >> >>>> >>>> >>>>> >>>>> >>>>> Ok, now for my problem. I have the following setting: >>>>> >>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> http-flow maxbw 1.228 -- 1228k >>>>> http-flow priority medium -- medium >>>>> >>>>> I ran a test hitting the webserver and I see this: >>>> >>>> I have the following flow >>>> >>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>> tcp-flow <link> -- tcp -- -- -- >>>> >>>> >>>> # flowadm show-flowprop tcp-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>> >>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>> this machine from a peer) for about 2 mins. >>>> >>>> On the peer the traffic generator (sender) says I am capped to about >>>> 1.14 Mbps. >>>> >>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>> >>>> 49152 49152 49152 120.49 1.14 >>>> >>>> Now, when I try show-usage during the traffic flow on >>>> the machine with the above flow in place (receiver), I am seeing: >>>> >>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>> >>>> ... >>>> >>>> I think the difference you see is likely to be because of the time >>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>> sync with the bandwidth enforcement period in the kernel) and also >>>> could be because of rounding up etc. But, if you look at the entire >>>> duration, it averages to about the configured limit (in the above >>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>> >>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>> >> >>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>> --10:01:30-- sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat'' >>> >>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>> >>> >>>> >>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>> flow as the bandwidth for both are independent. >>> >>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>> >> >> Can you try with a higher bandwidth, say 100 Mbps and see what the results >> are when compared to wget''s output? >> >> Also, another way of manually checking would be to do a >> # kstat -c flow -n http-flow >> >> before and after the wget run and see how many bytes (rbytes) the >> kernel has seen for that flow (assuming there isn''t any other traffic >> going over the flow), and then determine the bandwidth (you might need >> to get the duration of the wget run pretty close to get the >> right bandwdith value). >> >> -venu > > I changed the flow to be 100Mbps. > > root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > http-flow maxbw 100 -- 100 > > I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. > > I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. > > root at myhost:/tmp# ls http-flow-* > http-flow-1.txt http-flow-2.txt > root at myhost:/tmp# cat http-flow-* > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 4680 > obytes 182520974 > oerrors 0 > opackets 127521 > rbytes 308138 > snaptime 189431.882363589 > > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 31262 > obytes 1281785975 > oerrors 0 > opackets 895533 > rbytes 2062678 > snaptime 189595.122347726 > > So this means the network only passed ~1.6 MB of data?is this on the outbound side or inbound? looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps.> > The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. > > [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat > --13:21:49-- sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat'' > > 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s > > 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] > > > real 1m33.701s > user 0m0.899s > sys 0m23.874s > > Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >OK, -venu> -Cesar > > >>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> >>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>> >>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>> >>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>> e1000g0 maxbw rw 1.228 -- -- >>>>> >>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>> >>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>> >>>>> Thanks for the help. >>>>> >>>>> -Cesar >>>>> >>>>> _______________________________________________ >>>>> crossbow-discuss mailing list >>>>> crossbow-discuss at opensolaris.org >>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>> >>> >>> > >
On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote:> > > > On Wed, 25 Nov 2009, Cesar Delgado wrote: > >> Venu, >> >> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >> >>> >>> Hi, Cesar: >>> >>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>> >>>> Venugopal, >>>> >>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>> >>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>> >>>>> >>>>> >>>>> Hi, Cesar: >>>>> >>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>> >>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>> >>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>> >>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>> flowadm: invalid flow: ''(null)'' >>>>> >>>>> This is a bug, I have submitted >>>>> >>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>> >>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>> >>> I think it should show up on >>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> Ok, now for my problem. I have the following setting: >>>>>> >>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>> http-flow maxbw 1.228 -- 1228k >>>>>> http-flow priority medium -- medium >>>>>> >>>>>> I ran a test hitting the webserver and I see this: >>>>> >>>>> I have the following flow >>>>> >>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>> tcp-flow <link> -- tcp -- -- -- >>>>> >>>>> >>>>> # flowadm show-flowprop tcp-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>> >>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>> this machine from a peer) for about 2 mins. >>>>> >>>>> On the peer the traffic generator (sender) says I am capped to about >>>>> 1.14 Mbps. >>>>> >>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>> >>>>> 49152 49152 49152 120.49 1.14 >>>>> >>>>> Now, when I try show-usage during the traffic flow on >>>>> the machine with the above flow in place (receiver), I am seeing: >>>>> >>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>> >>>>> ... >>>>> >>>>> I think the difference you see is likely to be because of the time >>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>> could be because of rounding up etc. But, if you look at the entire >>>>> duration, it averages to about the configured limit (in the above >>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>> >>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>> >>> >>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>> --10:01:30-- sol/myfile.dat >>>> Resolving sol... 192.168.69.104 >>>> Connecting to sol|192.168.69.104|:80... connected. >>>> HTTP request sent, awaiting response... 200 OK >>>> Length: 1048576000 (1000M) >>>> Saving to: `myfile.dat'' >>>> >>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>> >>>> >>>>> >>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>> flow as the bandwidth for both are independent. >>>> >>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>> >>> >>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>> are when compared to wget''s output? >>> >>> Also, another way of manually checking would be to do a >>> # kstat -c flow -n http-flow >>> >>> before and after the wget run and see how many bytes (rbytes) the >>> kernel has seen for that flow (assuming there isn''t any other traffic >>> going over the flow), and then determine the bandwidth (you might need >>> to get the duration of the wget run pretty close to get the >>> right bandwdith value). >>> >>> -venu >> >> I changed the flow to be 100Mbps. >> >> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> http-flow maxbw 100 -- 100 >> >> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >> >> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >> >> root at myhost:/tmp# ls http-flow-* >> http-flow-1.txt http-flow-2.txt >> root at myhost:/tmp# cat http-flow-* >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 4680 >> obytes 182520974 >> oerrors 0 >> opackets 127521 >> rbytes 308138 >> snaptime 189431.882363589 >> >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 31262 >> obytes 1281785975 >> oerrors 0 >> opackets 895533 >> rbytes 2062678 >> snaptime 189595.122347726 >> >> So this means the network only passed ~1.6 MB of data? > > is this on the outbound side or inbound? > > looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration > 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? root at myhost:/tmp# flowadm show-flowprop http-flow FLOW PROPERTY VALUE DEFAULT POSSIBLE http-flow maxbw 50 -- 50 http-flow priority medium -- medium root at myhost:/tmp# cat http-flow-* module: unix instance: 0 name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 32346 obytes 1300129135 oerrors 0 opackets 908351 rbytes 2133486 snaptime 191261.607310645 module: unix instance: 0 name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 85361 obytes 2399401106 oerrors 0 opackets 1676468 rbytes 5632604 snaptime 192621.873203347 [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat --13:52:20-- sol/myfile.dat Resolving sol... 192.168.69.104 Connecting to sol|192.168.69.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) Saving to: `myfile.dat'' 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] real 3m44.364s user 0m1.042s sys 0m24.065s> >> >> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >> >> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >> --13:21:49-- sol/myfile.dat >> Resolving sol... 192.168.69.104 >> Connecting to sol|192.168.69.104|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 1048576000 (1000M) >> Saving to: `myfile.dat'' >> >> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >> >> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >> >> >> real 1m33.701s >> user 0m0.899s >> sys 0m23.874s >> >> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >> > > OK, > > -venu > >> -Cesar >> >> >>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>> >>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>> >>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>> >>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>> >>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>> >>>>>> Thanks for the help. >>>>>> >>>>>> -Cesar >>>>>> >>>>>> _______________________________________________ >>>>>> crossbow-discuss mailing list >>>>>> crossbow-discuss at opensolaris.org >>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>> >>>> >>>> >> >>
venugopal iyer
2009-Nov-25 23:39 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Hi, Cesar: On Wed, 25 Nov 2009, Cesar Delgado wrote:> > On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: > >> >> >> >> On Wed, 25 Nov 2009, Cesar Delgado wrote: >> >>> Venu, >>> >>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>> >>>> >>>> Hi, Cesar: >>>> >>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>> >>>>> Venugopal, >>>>> >>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>> >>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> >>>>>> Hi, Cesar: >>>>>> >>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>> >>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>> flowadm: invalid flow: ''(null)'' >>>>>> >>>>>> This is a bug, I have submitted >>>>>> >>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>> >>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>> >>>> I think it should show up on >>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>> >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Ok, now for my problem. I have the following setting: >>>>>>> >>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>> http-flow priority medium -- medium >>>>>>> >>>>>>> I ran a test hitting the webserver and I see this: >>>>>> >>>>>> I have the following flow >>>>>> >>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>> >>>>>> >>>>>> # flowadm show-flowprop tcp-flow >>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>> >>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>> this machine from a peer) for about 2 mins. >>>>>> >>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>> 1.14 Mbps. >>>>>> >>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>> >>>>>> 49152 49152 49152 120.49 1.14 >>>>>> >>>>>> Now, when I try show-usage during the traffic flow on >>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>> >>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>> >>>>>> ... >>>>>> >>>>>> I think the difference you see is likely to be because of the time >>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>> duration, it averages to about the configured limit (in the above >>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>> >>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>> >>>> >>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>> --10:01:30-- sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat'' >>>>> >>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>> >>>>> >>>>>> >>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>> flow as the bandwidth for both are independent. >>>>> >>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>> >>>> >>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>> are when compared to wget''s output? >>>> >>>> Also, another way of manually checking would be to do a >>>> # kstat -c flow -n http-flow >>>> >>>> before and after the wget run and see how many bytes (rbytes) the >>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>> going over the flow), and then determine the bandwidth (you might need >>>> to get the duration of the wget run pretty close to get the >>>> right bandwdith value). >>>> >>>> -venu >>> >>> I changed the flow to be 100Mbps. >>> >>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 100 -- 100 >>> >>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>> >>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>> >>> root at myhost:/tmp# ls http-flow-* >>> http-flow-1.txt http-flow-2.txt >>> root at myhost:/tmp# cat http-flow-* >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 4680 >>> obytes 182520974 >>> oerrors 0 >>> opackets 127521 >>> rbytes 308138 >>> snaptime 189431.882363589 >>> >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 31262 >>> obytes 1281785975 >>> oerrors 0 >>> opackets 895533 >>> rbytes 2062678 >>> snaptime 189595.122347726 >>> >>> So this means the network only passed ~1.6 MB of data? >> >> is this on the outbound side or inbound? >> >> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >> > > Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system?For TCP you might get a little less than the set bandwidth as TCP would do flow control too (i.e if for bandwidth reasons ACKs come back slower or packets get dropped, TCP will react and will slow down). Also, note that the current property (maxbw) is a *limit* which means we will ensure that the flow will not get *more* than what you have asked for. Going forwards we are looking at adding guarantee where in the flow would get what it asked for. -venu> > root at myhost:/tmp# flowadm show-flowprop http-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > http-flow maxbw 50 -- 50 > http-flow priority medium -- medium > > root at myhost:/tmp# cat http-flow-* > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 32346 > obytes 1300129135 > oerrors 0 > opackets 908351 > rbytes 2133486 > snaptime 191261.607310645 > > module: unix instance: 0 > name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 85361 > obytes 2399401106 > oerrors 0 > opackets 1676468 > rbytes 5632604 > snaptime 192621.873203347 > > > [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat > --13:52:20-- sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat'' > > 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s > > 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] > > > real 3m44.364s > user 0m1.042s > sys 0m24.065s > > > >> >>> >>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>> >>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>> --13:21:49-- sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat'' >>> >>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>> >>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>> >>> >>> real 1m33.701s >>> user 0m0.899s >>> sys 0m23.874s >>> >>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>> >> >> OK, >> >> -venu >> >>> -Cesar >>> >>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>> >>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>> >>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>> >>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>> >>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>> >>>>>>> Thanks for the help. >>>>>>> >>>>>>> -Cesar >>>>>>> >>>>>>> _______________________________________________ >>>>>>> crossbow-discuss mailing list >>>>>>> crossbow-discuss at opensolaris.org >>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>> >>>>> >>>>> >>> >>> > >
On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:> > Hi, Cesar: > > On Wed, 25 Nov 2009, Cesar Delgado wrote: > >> >> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >> >>> >>> >>> >>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>> >>>> Venu, >>>> >>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>> >>>>> >>>>> Hi, Cesar: >>>>> >>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>> >>>>>> Venugopal, >>>>>> >>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>> >>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, Cesar: >>>>>>> >>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>> >>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>> >>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>> >>>>>>> This is a bug, I have submitted >>>>>>> >>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>> >>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>> >>>>> I think it should show up on >>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>> http-flow priority medium -- medium >>>>>>>> >>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>> >>>>>>> I have the following flow >>>>>>> >>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>> >>>>>>> >>>>>>> # flowadm show-flowprop tcp-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>> >>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>> this machine from a peer) for about 2 mins. >>>>>>> >>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>> 1.14 Mbps. >>>>>>> >>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>> >>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>> >>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>> >>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> I think the difference you see is likely to be because of the time >>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>> duration, it averages to about the configured limit (in the above >>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>> >>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>> >>>>> >>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>> --10:01:30-- sol/myfile.dat >>>>>> Resolving sol... 192.168.69.104 >>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>> HTTP request sent, awaiting response... 200 OK >>>>>> Length: 1048576000 (1000M) >>>>>> Saving to: `myfile.dat'' >>>>>> >>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>> >>>>>> >>>>>>> >>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>> flow as the bandwidth for both are independent. >>>>>> >>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>> >>>>> >>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>> are when compared to wget''s output? >>>>> >>>>> Also, another way of manually checking would be to do a >>>>> # kstat -c flow -n http-flow >>>>> >>>>> before and after the wget run and see how many bytes (rbytes) the >>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>> going over the flow), and then determine the bandwidth (you might need >>>>> to get the duration of the wget run pretty close to get the >>>>> right bandwdith value). >>>>> >>>>> -venu >>>> >>>> I changed the flow to be 100Mbps. >>>> >>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> http-flow maxbw 100 -- 100 >>>> >>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>> >>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>> >>>> root at myhost:/tmp# ls http-flow-* >>>> http-flow-1.txt http-flow-2.txt >>>> root at myhost:/tmp# cat http-flow-* >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 4680 >>>> obytes 182520974 >>>> oerrors 0 >>>> opackets 127521 >>>> rbytes 308138 >>>> snaptime 189431.882363589 >>>> >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 31262 >>>> obytes 1281785975 >>>> oerrors 0 >>>> opackets 895533 >>>> rbytes 2062678 >>>> snaptime 189595.122347726 >>>> >>>> So this means the network only passed ~1.6 MB of data? >>> >>> is this on the outbound side or inbound? >>> >>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>> >> >> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? > > For TCP you might get a little less than the set bandwidth as TCP > would do flow control too (i.e if for bandwidth reasons ACKs come back > slower or packets get dropped, TCP will react and will slow down). Also, > note that the current property (maxbw) is a *limit* which means we will > ensure that the flow will not get *more* than what you have asked for. > Going forwards we are looking at adding guarantee where in the flow would > get what it asked for. > > -venuGood to know. And it''s true about TCP, for some reason I though it wouldn''t happen with maxbw but I do unserstand now t hat the entire TVP connection is being affected by the parameter, including all the extra TCP stuff that''s needed like the handshake. I don''t need t set guarantees at the moment so this doesn''t worry me much. I just want to cap the usage so I don''t go over my 1Mbit network quota. Like overages in cellphones, network overages are expensive. So now back to my original problem. In the new test I''m seeing ~2.1Mbps, that''s almost double the maxbw value. If maxbw is a limit as you say in the previous reply, why am I seeing more than 1.2Mbps when I set maxbw to 1228k? Could it be a corner case being that it''s the smallest value possible? cdelgado at myhost:~$ tail /tmp/http-flow-* ==> /tmp/http-flow-1.txt <=name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 101961 obytes 2446610048 oerrors 0 opackets 1709453 rbytes 6727462 snaptime 198359.572083418 ==> /tmp/http-flow-2.txt <=name: http-flow class: flow crtime 188662.418688816 ierrors 0 ipackets 485927 obytes 3545874452 oerrors 0 opackets 2477456 rbytes 32069346 snaptime 202630.597287968 [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat --15:50:35-- sol/myfile.dat Resolving sol... 192.168.69.104 Connecting to sol|192.168.69.104|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1048576000 (1000M) Saving to: `myfile.dat'' 100%[==========================================================>] 1,048,576,000 267K/s in 64m 8s 16:54:43 (266 KB/s) - `myfile.dat'' saved [1048576000/1048576000] real 64m7.855s user 0m4.982s sys 0m32.007s> >> >> root at myhost:/tmp# flowadm show-flowprop http-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> http-flow maxbw 50 -- 50 >> http-flow priority medium -- medium >> >> root at myhost:/tmp# cat http-flow-* >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 32346 >> obytes 1300129135 >> oerrors 0 >> opackets 908351 >> rbytes 2133486 >> snaptime 191261.607310645 >> >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 85361 >> obytes 2399401106 >> oerrors 0 >> opackets 1676468 >> rbytes 5632604 >> snaptime 192621.873203347 >> >> >> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >> --13:52:20-- sol/myfile.dat >> Resolving sol... 192.168.69.104 >> Connecting to sol|192.168.69.104|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 1048576000 (1000M) >> Saving to: `myfile.dat'' >> >> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >> >> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >> >> >> real 3m44.364s >> user 0m1.042s >> sys 0m24.065s >> >> >> >>> >>>> >>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>> >>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>> --13:21:49-- sol/myfile.dat >>>> Resolving sol... 192.168.69.104 >>>> Connecting to sol|192.168.69.104|:80... connected. >>>> HTTP request sent, awaiting response... 200 OK >>>> Length: 1048576000 (1000M) >>>> Saving to: `myfile.dat'' >>>> >>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>> >>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>> >>>> >>>> real 1m33.701s >>>> user 0m0.899s >>>> sys 0m23.874s >>>> >>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>> >>> >>> OK, >>> >>> -venu >>> >>>> -Cesar >>>> >>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>> >>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>> >>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>> >>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>> >>>>>>>> Thanks for the help. >>>>>>>> >>>>>>>> -Cesar >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> crossbow-discuss mailing list >>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
venugopal iyer
2009-Dec-01 01:26 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Cesar: On Thu, 26 Nov 2009, Cesar Delgado wrote:> > > On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: > >> >> Hi, Cesar: >> >> On Wed, 25 Nov 2009, Cesar Delgado wrote: >> >>> >>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >>> >>>> >>>> >>>> >>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>> >>>>> Venu, >>>>> >>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> Hi, Cesar: >>>>>> >>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> Venugopal, >>>>>>> >>>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>>> >>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, Cesar: >>>>>>>> >>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>>> >>>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>>> >>>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>>> >>>>>>>> This is a bug, I have submitted >>>>>>>> >>>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>>> >>>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>>> >>>>>> I think it should show up on >>>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>>> http-flow priority medium -- medium >>>>>>>>> >>>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>>> >>>>>>>> I have the following flow >>>>>>>> >>>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>>> >>>>>>>> >>>>>>>> # flowadm show-flowprop tcp-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>>> >>>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>>> this machine from a peer) for about 2 mins. >>>>>>>> >>>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>>> 1.14 Mbps. >>>>>>>> >>>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>>> >>>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>>> >>>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>>> >>>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> I think the difference you see is likely to be because of the time >>>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>>> duration, it averages to about the configured limit (in the above >>>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>>> >>>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>>> >>>>>> >>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>>> --10:01:30-- sol/myfile.dat >>>>>>> Resolving sol... 192.168.69.104 >>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>> Length: 1048576000 (1000M) >>>>>>> Saving to: `myfile.dat'' >>>>>>> >>>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>>> flow as the bandwidth for both are independent. >>>>>>> >>>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>>> >>>>>> >>>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>>> are when compared to wget''s output? >>>>>> >>>>>> Also, another way of manually checking would be to do a >>>>>> # kstat -c flow -n http-flow >>>>>> >>>>>> before and after the wget run and see how many bytes (rbytes) the >>>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>>> going over the flow), and then determine the bandwidth (you might need >>>>>> to get the duration of the wget run pretty close to get the >>>>>> right bandwdith value). >>>>>> >>>>>> -venu >>>>> >>>>> I changed the flow to be 100Mbps. >>>>> >>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> http-flow maxbw 100 -- 100 >>>>> >>>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>>> >>>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>>> >>>>> root at myhost:/tmp# ls http-flow-* >>>>> http-flow-1.txt http-flow-2.txt >>>>> root at myhost:/tmp# cat http-flow-* >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 4680 >>>>> obytes 182520974 >>>>> oerrors 0 >>>>> opackets 127521 >>>>> rbytes 308138 >>>>> snaptime 189431.882363589 >>>>> >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 31262 >>>>> obytes 1281785975 >>>>> oerrors 0 >>>>> opackets 895533 >>>>> rbytes 2062678 >>>>> snaptime 189595.122347726 >>>>> >>>>> So this means the network only passed ~1.6 MB of data? >>>> >>>> is this on the outbound side or inbound? >>>> >>>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>>> >>> >>> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? >> >> For TCP you might get a little less than the set bandwidth as TCP >> would do flow control too (i.e if for bandwidth reasons ACKs come back >> slower or packets get dropped, TCP will react and will slow down). Also, >> note that the current property (maxbw) is a *limit* which means we will >> ensure that the flow will not get *more* than what you have asked for. >> Going forwards we are looking at adding guarantee where in the flow would >> get what it asked for. >> >> -venu > > Good to know. And it''s true about TCP, for some reason I though it wouldn''t happen with maxbw but I do unserstand now t hat the entire TVP connection is being affected by the parameter, including all the extra TCP stuff that''s needed like the handshake. I don''t need t set guarantees at the moment so this doesn''t worry me much. I just want to cap the usage so I don''t go over my 1Mbit network quota. Like overages in cellphones, network overages are expensive. > > So now back to my original problem. In the new test I''m seeing ~2.1Mbps, that''s almost double the maxbw value. If maxbw is a limit as you say in the previous reply, why am I seeing more than 1.2Mbps when I set maxbw to 1228k? Could it be a corner case being that it''s the smallest value possible?Looking at the output below, I suspect so. Let me try it at my end and check this.. will let you know. -venu> > cdelgado at myhost:~$ tail /tmp/http-flow-* > ==> /tmp/http-flow-1.txt <=> name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 101961 > obytes 2446610048 > oerrors 0 > opackets 1709453 > rbytes 6727462 > snaptime 198359.572083418 > > > ==> /tmp/http-flow-2.txt <=> name: http-flow class: flow > crtime 188662.418688816 > ierrors 0 > ipackets 485927 > obytes 3545874452 > oerrors 0 > opackets 2477456 > rbytes 32069346 > snaptime 202630.597287968 > > > [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat > --15:50:35-- sol/myfile.dat > Resolving sol... 192.168.69.104 > Connecting to sol|192.168.69.104|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 1048576000 (1000M) > Saving to: `myfile.dat'' > > 100%[==========================================================>] 1,048,576,000 267K/s in 64m 8s > > 16:54:43 (266 KB/s) - `myfile.dat'' saved [1048576000/1048576000] > > > real 64m7.855s > user 0m4.982s > sys 0m32.007s > > >> >>> >>> root at myhost:/tmp# flowadm show-flowprop http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 50 -- 50 >>> http-flow priority medium -- medium >>> >>> root at myhost:/tmp# cat http-flow-* >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 32346 >>> obytes 1300129135 >>> oerrors 0 >>> opackets 908351 >>> rbytes 2133486 >>> snaptime 191261.607310645 >>> >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 85361 >>> obytes 2399401106 >>> oerrors 0 >>> opackets 1676468 >>> rbytes 5632604 >>> snaptime 192621.873203347 >>> >>> >>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>> --13:52:20-- sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat'' >>> >>> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >>> >>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>> >>> >>> real 3m44.364s >>> user 0m1.042s >>> sys 0m24.065s >>> >>> >>> >>>> >>>>> >>>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>>> >>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>> --13:21:49-- sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat'' >>>>> >>>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>>> >>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>>> >>>>> >>>>> real 1m33.701s >>>>> user 0m0.899s >>>>> sys 0m23.874s >>>>> >>>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>>> >>>> >>>> OK, >>>> >>>> -venu >>>> >>>>> -Cesar >>>>> >>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>>> >>>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>>> >>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>>> >>>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>>> >>>>>>>>> Thanks for the help. >>>>>>>>> >>>>>>>>> -Cesar >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> crossbow-discuss mailing list >>>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> > >
Venu, Any new information on this? Is it a bug? -Cesar On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote:> > Hi, Cesar: > > On Wed, 25 Nov 2009, Cesar Delgado wrote: > >> >> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >> >>> >>> >>> >>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>> >>>> Venu, >>>> >>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>> >>>>> >>>>> Hi, Cesar: >>>>> >>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>> >>>>>> Venugopal, >>>>>> >>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>> >>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, Cesar: >>>>>>> >>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>> >>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>> >>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>> >>>>>>> This is a bug, I have submitted >>>>>>> >>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>> >>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>> >>>>> I think it should show up on >>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>> http-flow priority medium -- medium >>>>>>>> >>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>> >>>>>>> I have the following flow >>>>>>> >>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>> >>>>>>> >>>>>>> # flowadm show-flowprop tcp-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>> >>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>> this machine from a peer) for about 2 mins. >>>>>>> >>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>> 1.14 Mbps. >>>>>>> >>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>> >>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>> >>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>> >>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>> >>>>>>> ... >>>>>>> >>>>>>> I think the difference you see is likely to be because of the time >>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>> duration, it averages to about the configured limit (in the above >>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>> >>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>> >>>>> >>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>> --10:01:30-- sol/myfile.dat >>>>>> Resolving sol... 192.168.69.104 >>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>> HTTP request sent, awaiting response... 200 OK >>>>>> Length: 1048576000 (1000M) >>>>>> Saving to: `myfile.dat'' >>>>>> >>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>> >>>>>> >>>>>>> >>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>> flow as the bandwidth for both are independent. >>>>>> >>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>> >>>>> >>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>> are when compared to wget''s output? >>>>> >>>>> Also, another way of manually checking would be to do a >>>>> # kstat -c flow -n http-flow >>>>> >>>>> before and after the wget run and see how many bytes (rbytes) the >>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>> going over the flow), and then determine the bandwidth (you might need >>>>> to get the duration of the wget run pretty close to get the >>>>> right bandwdith value). >>>>> >>>>> -venu >>>> >>>> I changed the flow to be 100Mbps. >>>> >>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> http-flow maxbw 100 -- 100 >>>> >>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>> >>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>> >>>> root at myhost:/tmp# ls http-flow-* >>>> http-flow-1.txt http-flow-2.txt >>>> root at myhost:/tmp# cat http-flow-* >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 4680 >>>> obytes 182520974 >>>> oerrors 0 >>>> opackets 127521 >>>> rbytes 308138 >>>> snaptime 189431.882363589 >>>> >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 31262 >>>> obytes 1281785975 >>>> oerrors 0 >>>> opackets 895533 >>>> rbytes 2062678 >>>> snaptime 189595.122347726 >>>> >>>> So this means the network only passed ~1.6 MB of data? >>> >>> is this on the outbound side or inbound? >>> >>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>> >> >> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? > > For TCP you might get a little less than the set bandwidth as TCP > would do flow control too (i.e if for bandwidth reasons ACKs come back > slower or packets get dropped, TCP will react and will slow down). Also, > note that the current property (maxbw) is a *limit* which means we will > ensure that the flow will not get *more* than what you have asked for. > Going forwards we are looking at adding guarantee where in the flow would > get what it asked for. > > -venu > >> >> root at myhost:/tmp# flowadm show-flowprop http-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> http-flow maxbw 50 -- 50 >> http-flow priority medium -- medium >> >> root at myhost:/tmp# cat http-flow-* >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 32346 >> obytes 1300129135 >> oerrors 0 >> opackets 908351 >> rbytes 2133486 >> snaptime 191261.607310645 >> >> module: unix instance: 0 >> name: http-flow class: flow >> crtime 188662.418688816 >> ierrors 0 >> ipackets 85361 >> obytes 2399401106 >> oerrors 0 >> opackets 1676468 >> rbytes 5632604 >> snaptime 192621.873203347 >> >> >> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >> --13:52:20-- sol/myfile.dat >> Resolving sol... 192.168.69.104 >> Connecting to sol|192.168.69.104|:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 1048576000 (1000M) >> Saving to: `myfile.dat'' >> >> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >> >> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >> >> >> real 3m44.364s >> user 0m1.042s >> sys 0m24.065s >> >> >> >>> >>>> >>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>> >>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>> --13:21:49-- sol/myfile.dat >>>> Resolving sol... 192.168.69.104 >>>> Connecting to sol|192.168.69.104|:80... connected. >>>> HTTP request sent, awaiting response... 200 OK >>>> Length: 1048576000 (1000M) >>>> Saving to: `myfile.dat'' >>>> >>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>> >>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>> >>>> >>>> real 1m33.701s >>>> user 0m0.899s >>>> sys 0m23.874s >>>> >>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>> >>> >>> OK, >>> >>> -venu >>> >>>> -Cesar >>>> >>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>> >>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>> >>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>> >>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>> >>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>> >>>>>>>> Thanks for the help. >>>>>>>> >>>>>>>> -Cesar >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> crossbow-discuss mailing list >>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
venugopal iyer
2009-Dec-16 01:30 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Sorry, Cesar, for the delayed response. I tried setting up your scenario on a recent version (build 130) and see that the bandwidth is close to what I am setting. # flowadm show-flowprop tcp-flow FLOW PROPERTY VALUE DEFAULT POSSIBLE tcp-flow maxbw 1.200 -- 1200K tcp-flow priority -- -- ? # wget 10.0.0.2/myfile.dat --2009-12-15 15:47:24-- 10.0.0.2/myfile.dat Connecting to 10.0.0.2:80... connected. HTTP request sent, awaiting response... 200 OK Length: 46104240 (44M) [text/plain] Saving to: `myfile.dat.4'' 100%[======================================>] 46,104,240 160K/s in 4m 42s 2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved [46104240/46104240] 160 KB/s is 1.280 Mbps. I see this consistently. The kstat does give me slightly higher about 1.3 Mbps. FYI, without the flow (with limt) in place, I see # wget 10.0.0.2/myfile.dat --2009-12-15 15:47:12-- 10.0.0.2/myfile.dat Connecting to 10.0.0.2:80... connected. HTTP request sent, awaiting response... 200 OK Length: 46104240 (44M) [text/plain] Saving to: `myfile.dat.3'' 100%[======================================>] 46,104,240 78.2M/s in 0.6s 2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved [46104240/46104240] What kind of hardware are you using? -venu On Mon, 14 Dec 2009, Cesar Delgado wrote:> Venu, > > Any new information on this? Is it a bug? > > -Cesar > > On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: > >> >> Hi, Cesar: >> >> On Wed, 25 Nov 2009, Cesar Delgado wrote: >> >>> >>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >>> >>>> >>>> >>>> >>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>> >>>>> Venu, >>>>> >>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> Hi, Cesar: >>>>>> >>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> Venugopal, >>>>>>> >>>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>>> >>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, Cesar: >>>>>>>> >>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>>> >>>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>>> >>>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>>> >>>>>>>> This is a bug, I have submitted >>>>>>>> >>>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>>> >>>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>>> >>>>>> I think it should show up on >>>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>>> http-flow priority medium -- medium >>>>>>>>> >>>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>>> >>>>>>>> I have the following flow >>>>>>>> >>>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>>> >>>>>>>> >>>>>>>> # flowadm show-flowprop tcp-flow >>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>>> >>>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>>> this machine from a peer) for about 2 mins. >>>>>>>> >>>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>>> 1.14 Mbps. >>>>>>>> >>>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>>> >>>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>>> >>>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>>> >>>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>>> >>>>>>>> ... >>>>>>>> >>>>>>>> I think the difference you see is likely to be because of the time >>>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>>> duration, it averages to about the configured limit (in the above >>>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>>> >>>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>>> >>>>>> >>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>>> --10:01:30-- sol/myfile.dat >>>>>>> Resolving sol... 192.168.69.104 >>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>> Length: 1048576000 (1000M) >>>>>>> Saving to: `myfile.dat'' >>>>>>> >>>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>>> flow as the bandwidth for both are independent. >>>>>>> >>>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>>> >>>>>> >>>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>>> are when compared to wget''s output? >>>>>> >>>>>> Also, another way of manually checking would be to do a >>>>>> # kstat -c flow -n http-flow >>>>>> >>>>>> before and after the wget run and see how many bytes (rbytes) the >>>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>>> going over the flow), and then determine the bandwidth (you might need >>>>>> to get the duration of the wget run pretty close to get the >>>>>> right bandwdith value). >>>>>> >>>>>> -venu >>>>> >>>>> I changed the flow to be 100Mbps. >>>>> >>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> http-flow maxbw 100 -- 100 >>>>> >>>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>>> >>>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>>> >>>>> root at myhost:/tmp# ls http-flow-* >>>>> http-flow-1.txt http-flow-2.txt >>>>> root at myhost:/tmp# cat http-flow-* >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 4680 >>>>> obytes 182520974 >>>>> oerrors 0 >>>>> opackets 127521 >>>>> rbytes 308138 >>>>> snaptime 189431.882363589 >>>>> >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 31262 >>>>> obytes 1281785975 >>>>> oerrors 0 >>>>> opackets 895533 >>>>> rbytes 2062678 >>>>> snaptime 189595.122347726 >>>>> >>>>> So this means the network only passed ~1.6 MB of data? >>>> >>>> is this on the outbound side or inbound? >>>> >>>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>>> >>> >>> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? >> >> For TCP you might get a little less than the set bandwidth as TCP >> would do flow control too (i.e if for bandwidth reasons ACKs come back >> slower or packets get dropped, TCP will react and will slow down). Also, >> note that the current property (maxbw) is a *limit* which means we will >> ensure that the flow will not get *more* than what you have asked for. >> Going forwards we are looking at adding guarantee where in the flow would >> get what it asked for. >> >> -venu >> >>> >>> root at myhost:/tmp# flowadm show-flowprop http-flow >>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>> http-flow maxbw 50 -- 50 >>> http-flow priority medium -- medium >>> >>> root at myhost:/tmp# cat http-flow-* >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 32346 >>> obytes 1300129135 >>> oerrors 0 >>> opackets 908351 >>> rbytes 2133486 >>> snaptime 191261.607310645 >>> >>> module: unix instance: 0 >>> name: http-flow class: flow >>> crtime 188662.418688816 >>> ierrors 0 >>> ipackets 85361 >>> obytes 2399401106 >>> oerrors 0 >>> opackets 1676468 >>> rbytes 5632604 >>> snaptime 192621.873203347 >>> >>> >>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>> --13:52:20-- sol/myfile.dat >>> Resolving sol... 192.168.69.104 >>> Connecting to sol|192.168.69.104|:80... connected. >>> HTTP request sent, awaiting response... 200 OK >>> Length: 1048576000 (1000M) >>> Saving to: `myfile.dat'' >>> >>> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >>> >>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>> >>> >>> real 3m44.364s >>> user 0m1.042s >>> sys 0m24.065s >>> >>> >>> >>>> >>>>> >>>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>>> >>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>> --13:21:49-- sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat'' >>>>> >>>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>>> >>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>>> >>>>> >>>>> real 1m33.701s >>>>> user 0m0.899s >>>>> sys 0m23.874s >>>>> >>>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>>> >>>> >>>> OK, >>>> >>>> -venu >>>> >>>>> -Cesar >>>>> >>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>>> >>>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>>> >>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>>> >>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>>> >>>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>>> >>>>>>>>> Thanks for the help. >>>>>>>>> >>>>>>>>> -Cesar >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> crossbow-discuss mailing list >>>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> > >
Venu, The machine is just a little Intel PC that was given to me. It''s the only PC I have that passed the hardware test. No drivers were needed to run OpenSolaris on the box. What''s the best way for me to send you the specs of the box? I''ll also try using your settings on my box to see what happens. Thank you very much, -Cesar On Dec 15, 2009, at 5:30 PM, venugopal iyer wrote:> > Sorry, Cesar, for the delayed response. > > I tried setting up your scenario on a recent version > (build 130) and see that the bandwidth is close to what I am > setting. > > # flowadm show-flowprop tcp-flow > FLOW PROPERTY VALUE DEFAULT POSSIBLE > tcp-flow maxbw 1.200 -- 1200K tcp-flow priority -- -- ? > > > # wget 10.0.0.2/myfile.dat > --2009-12-15 15:47:24-- 10.0.0.2/myfile.dat > Connecting to 10.0.0.2:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 46104240 (44M) [text/plain] > Saving to: `myfile.dat.4'' > > 100%[======================================>] 46,104,240 160K/s in 4m 42s > > 2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved [46104240/46104240] > > 160 KB/s is 1.280 Mbps. > > I see this consistently. > > > The kstat does give me slightly higher about 1.3 Mbps. > > FYI, without the flow (with limt) in place, I see > > # wget 10.0.0.2/myfile.dat > --2009-12-15 15:47:12-- 10.0.0.2/myfile.dat > Connecting to 10.0.0.2:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 46104240 (44M) [text/plain] > Saving to: `myfile.dat.3'' > > 100%[======================================>] 46,104,240 78.2M/s in 0.6s > > 2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved [46104240/46104240] > > > What kind of hardware are you using? > > -venu > > > On Mon, 14 Dec 2009, Cesar Delgado wrote: > >> Venu, >> >> Any new information on this? Is it a bug? >> >> -Cesar >> >> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: >> >>> >>> Hi, Cesar: >>> >>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>> >>>> >>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >>>> >>>>> >>>>> >>>>> >>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>>> >>>>>> Venu, >>>>>> >>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>>>> >>>>>>> >>>>>>> Hi, Cesar: >>>>>>> >>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>>>> >>>>>>>> Venugopal, >>>>>>>> >>>>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>>>> >>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, Cesar: >>>>>>>>> >>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>>>> >>>>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>>>> >>>>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>>>> >>>>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>>>> >>>>>>>>> This is a bug, I have submitted >>>>>>>>> >>>>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>>>> >>>>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>>>> >>>>>>> I think it should show up on >>>>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>>>> >>>>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>>>> http-flow priority medium -- medium >>>>>>>>>> >>>>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>>>> >>>>>>>>> I have the following flow >>>>>>>>> >>>>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>>>> >>>>>>>>> >>>>>>>>> # flowadm show-flowprop tcp-flow >>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>>>> >>>>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>>>> this machine from a peer) for about 2 mins. >>>>>>>>> >>>>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>>>> 1.14 Mbps. >>>>>>>>> >>>>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>>>> >>>>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>>>> >>>>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>>>> >>>>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>>>> >>>>>>>>> ... >>>>>>>>> >>>>>>>>> I think the difference you see is likely to be because of the time >>>>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>>>> duration, it averages to about the configured limit (in the above >>>>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>>>> >>>>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>>>> >>>>>>> >>>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>>>> --10:01:30-- sol/myfile.dat >>>>>>>> Resolving sol... 192.168.69.104 >>>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>>> Length: 1048576000 (1000M) >>>>>>>> Saving to: `myfile.dat'' >>>>>>>> >>>>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>>>> flow as the bandwidth for both are independent. >>>>>>>> >>>>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>>>> >>>>>>> >>>>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>>>> are when compared to wget''s output? >>>>>>> >>>>>>> Also, another way of manually checking would be to do a >>>>>>> # kstat -c flow -n http-flow >>>>>>> >>>>>>> before and after the wget run and see how many bytes (rbytes) the >>>>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>>>> going over the flow), and then determine the bandwidth (you might need >>>>>>> to get the duration of the wget run pretty close to get the >>>>>>> right bandwdith value). >>>>>>> >>>>>>> -venu >>>>>> >>>>>> I changed the flow to be 100Mbps. >>>>>> >>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>> http-flow maxbw 100 -- 100 >>>>>> >>>>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>>>> >>>>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>>>> >>>>>> root at myhost:/tmp# ls http-flow-* >>>>>> http-flow-1.txt http-flow-2.txt >>>>>> root at myhost:/tmp# cat http-flow-* >>>>>> module: unix instance: 0 >>>>>> name: http-flow class: flow >>>>>> crtime 188662.418688816 >>>>>> ierrors 0 >>>>>> ipackets 4680 >>>>>> obytes 182520974 >>>>>> oerrors 0 >>>>>> opackets 127521 >>>>>> rbytes 308138 >>>>>> snaptime 189431.882363589 >>>>>> >>>>>> module: unix instance: 0 >>>>>> name: http-flow class: flow >>>>>> crtime 188662.418688816 >>>>>> ierrors 0 >>>>>> ipackets 31262 >>>>>> obytes 1281785975 >>>>>> oerrors 0 >>>>>> opackets 895533 >>>>>> rbytes 2062678 >>>>>> snaptime 189595.122347726 >>>>>> >>>>>> So this means the network only passed ~1.6 MB of data? >>>>> >>>>> is this on the outbound side or inbound? >>>>> >>>>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>>>> >>>> >>>> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? >>> >>> For TCP you might get a little less than the set bandwidth as TCP >>> would do flow control too (i.e if for bandwidth reasons ACKs come back >>> slower or packets get dropped, TCP will react and will slow down). Also, >>> note that the current property (maxbw) is a *limit* which means we will >>> ensure that the flow will not get *more* than what you have asked for. >>> Going forwards we are looking at adding guarantee where in the flow would >>> get what it asked for. >>> >>> -venu >>> >>>> >>>> root at myhost:/tmp# flowadm show-flowprop http-flow >>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>> http-flow maxbw 50 -- 50 >>>> http-flow priority medium -- medium >>>> >>>> root at myhost:/tmp# cat http-flow-* >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 32346 >>>> obytes 1300129135 >>>> oerrors 0 >>>> opackets 908351 >>>> rbytes 2133486 >>>> snaptime 191261.607310645 >>>> >>>> module: unix instance: 0 >>>> name: http-flow class: flow >>>> crtime 188662.418688816 >>>> ierrors 0 >>>> ipackets 85361 >>>> obytes 2399401106 >>>> oerrors 0 >>>> opackets 1676468 >>>> rbytes 5632604 >>>> snaptime 192621.873203347 >>>> >>>> >>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>> --13:52:20-- sol/myfile.dat >>>> Resolving sol... 192.168.69.104 >>>> Connecting to sol|192.168.69.104|:80... connected. >>>> HTTP request sent, awaiting response... 200 OK >>>> Length: 1048576000 (1000M) >>>> Saving to: `myfile.dat'' >>>> >>>> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >>>> >>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>> >>>> >>>> real 3m44.364s >>>> user 0m1.042s >>>> sys 0m24.065s >>>> >>>> >>>> >>>>> >>>>>> >>>>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>>>> >>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>>> --13:21:49-- sol/myfile.dat >>>>>> Resolving sol... 192.168.69.104 >>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>> HTTP request sent, awaiting response... 200 OK >>>>>> Length: 1048576000 (1000M) >>>>>> Saving to: `myfile.dat'' >>>>>> >>>>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>>>> >>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>>>> >>>>>> >>>>>> real 1m33.701s >>>>>> user 0m0.899s >>>>>> sys 0m23.874s >>>>>> >>>>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>>>> >>>>> >>>>> OK, >>>>> >>>>> -venu >>>>> >>>>>> -Cesar >>>>>> >>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>>>> >>>>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>>>> >>>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>>>> >>>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>>>> >>>>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>>>> >>>>>>>>>> Thanks for the help. >>>>>>>>>> >>>>>>>>>> -Cesar >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> crossbow-discuss mailing list >>>>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >> > >
venugopal iyer
2009-Dec-16 17:47 UTC
[crossbow-discuss] Flows aren''t respecting my limits
Hi, Ceasr: On Tue, 15 Dec 2009, Cesar Delgado wrote:> Venu, > > The machine is just a little Intel PC that was given to me. It''s the only PC I have that passed the hardware test. No drivers were needed to run OpenSolaris on the box. What''s the best way for me to send you the specs of the box?>I think that is fine, I was just curious. It might not have much to to do with the hardware (info. on the nic on the system might be useful - I am using bge card - Broadcom), given we are testing very low bandwidth (and seeing higher than expected throughput). One thing that may be different on my setup is that both the machines were connected back to back, i.e. this (wget) was the only traffic on the network. I am assuming you are running Opensolaris 2009.06, right? and the normal MTU (1500) sized packets, right? -venu>I''ll also try using your settings on my box to see what happens. > > Thank you very much, > > -Cesar > > On Dec 15, 2009, at 5:30 PM, venugopal iyer wrote: > >> >> Sorry, Cesar, for the delayed response. >> >> I tried setting up your scenario on a recent version >> (build 130) and see that the bandwidth is close to what I am >> setting. >> >> # flowadm show-flowprop tcp-flow >> FLOW PROPERTY VALUE DEFAULT POSSIBLE >> tcp-flow maxbw 1.200 -- 1200K tcp-flow priority -- -- ? >> >> >> # wget 10.0.0.2/myfile.dat >> --2009-12-15 15:47:24-- 10.0.0.2/myfile.dat >> Connecting to 10.0.0.2:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 46104240 (44M) [text/plain] >> Saving to: `myfile.dat.4'' >> >> 100%[======================================>] 46,104,240 160K/s in 4m 42s >> >> 2009-12-15 15:52:05 (160 KB/s) - `myfile.dat.4'' saved [46104240/46104240] >> >> 160 KB/s is 1.280 Mbps. >> >> I see this consistently. >> >> >> The kstat does give me slightly higher about 1.3 Mbps. >> >> FYI, without the flow (with limt) in place, I see >> >> # wget 10.0.0.2/myfile.dat >> --2009-12-15 15:47:12-- 10.0.0.2/myfile.dat >> Connecting to 10.0.0.2:80... connected. >> HTTP request sent, awaiting response... 200 OK >> Length: 46104240 (44M) [text/plain] >> Saving to: `myfile.dat.3'' >> >> 100%[======================================>] 46,104,240 78.2M/s in 0.6s >> >> 2009-12-15 15:47:13 (78.2 MB/s) - `myfile.dat.3'' saved [46104240/46104240] >> >> >> What kind of hardware are you using? >> >> -venu >> >> >> On Mon, 14 Dec 2009, Cesar Delgado wrote: >> >>> Venu, >>> >>> Any new information on this? Is it a bug? >>> >>> -Cesar >>> >>> On Nov 25, 2009, at 3:39 PM, venugopal iyer wrote: >>> >>>> >>>> Hi, Cesar: >>>> >>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>> >>>>> >>>>> On Nov 25, 2009, at 2:05 PM, venugopal iyer wrote: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, 25 Nov 2009, Cesar Delgado wrote: >>>>>> >>>>>>> Venu, >>>>>>> >>>>>>> On Nov 25, 2009, at 9:49 AM, venugopal iyer wrote: >>>>>>> >>>>>>>> >>>>>>>> Hi, Cesar: >>>>>>>> >>>>>>>> On Tue, 24 Nov 2009, Cesar Delgado wrote: >>>>>>>> >>>>>>>>> Venugopal, >>>>>>>>> >>>>>>>>> I''m sorry if these sounds like basic questions. I really appreciate the patience and the help. Replies in-line. >>>>>>>>> >>>>>>>>> On Nov 24, 2009, at 9:29 AM, venugopal iyer wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, Cesar: >>>>>>>>>> >>>>>>>>>> On Mon, 23 Nov 2009, Cesar Delgado wrote: >>>>>>>>>> >>>>>>>>>>> I''m setting up a server to go to a hosting site where I have a 1Mbps pipe. From what I read I know I can''t set the limit to this as the lowest setting is ~1.2Mbps and this is something that''s getting worked on in Crossbow2. I am seeing some strange behavior. >>>>>>>>>>> >>>>>>>>>>> FIrst I have a question on flowadm''s show-usage command. When I try to run show-prop with the name of a flow I get an error. The flow exists. What am I doing wrong? >>>>>>>>>>> >>>>>>>>>>> root at myhost:~# flowadm show-usage -f /var/log/net.log http-flow >>>>>>>>>>> flowadm: invalid flow: ''(null)'' >>>>>>>>>> >>>>>>>>>> This is a bug, I have submitted >>>>>>>>>> >>>>>>>>>> 6904427 flowadm show-usage doesn''t work with a flow name >>>>>>>>> >>>>>>>>> Thanks for submitting that. I haven''t been able to find a link to the bugtracker for Crossbow. Could you please send me the URL? >>>>>>>> >>>>>>>> I think it should show up on >>>>>>>> bugs.opensolaris.org/bugdatabase/index.jsp soon. >>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ok, now for my problem. I have the following setting: >>>>>>>>>>> >>>>>>>>>>> root at myhost:~# flowadm show-flowprop http-flow >>>>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>>>> http-flow maxbw 1.228 -- 1228k >>>>>>>>>>> http-flow priority medium -- medium >>>>>>>>>>> >>>>>>>>>>> I ran a test hitting the webserver and I see this: >>>>>>>>>> >>>>>>>>>> I have the following flow >>>>>>>>>> >>>>>>>>>> # flowadm show-flow FLOW LINK IPADDR PROTO LPORT RPORT DSFLD >>>>>>>>>> tcp-flow <link> -- tcp -- -- -- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> # flowadm show-flowprop tcp-flow >>>>>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>>>>> tcp-flow maxbw 1.228 -- 1228K tcp-flow priority -- -- ? >>>>>>>>>> >>>>>>>>>> When I send TCP traffic (am using a traffic generator - netperf, to >>>>>>>>>> this machine from a peer) for about 2 mins. >>>>>>>>>> >>>>>>>>>> On the peer the traffic generator (sender) says I am capped to about >>>>>>>>>> 1.14 Mbps. >>>>>>>>>> >>>>>>>>>> Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec >>>>>>>>>> >>>>>>>>>> 49152 49152 49152 120.49 1.14 >>>>>>>>>> >>>>>>>>>> Now, when I try show-usage during the traffic flow on >>>>>>>>>> the machine with the above flow in place (receiver), I am seeing: >>>>>>>>>> >>>>>>>>>> # flowadm show-usage -s 11/24/2009 -f /var/tmp/tcpflow >>>>>>>>>> FLOW START END RBYTES OBYTES BANDWIDTH >>>>>>>>>> tcp-flow 08:51:48 08:52:08 3428658 107802 1.414 Mbp >>>>>>>>>> tcp-flow 08:52:08 08:52:28 3431198 107802 1.415 Mbp >>>>>>>>>> tcp-flow 08:52:28 08:52:48 3434614 107888 1.417 Mbp >>>>>>>>>> tcp-flow 08:52:48 08:53:08 3443298 107802 1.420 Mbp >>>>>>>>>> tcp-flow 08:53:08 08:53:28 3444324 107802 1.420 Mbp >>>>>>>>>> tcp-flow 08:53:28 08:53:48 1376806 43576 0.568 Mbps >>>>>>>>>> >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> I think the difference you see is likely to be because of the time >>>>>>>>>> period when the stats are written to the file (the bandwidth is computed for every 20 seconds period which might not be exactly in >>>>>>>>>> sync with the bandwidth enforcement period in the kernel) and also >>>>>>>>>> could be because of rounding up etc. But, if you look at the entire >>>>>>>>>> duration, it averages to about the configured limit (in the above >>>>>>>>>> example, I think it is about 1.275 Mbps for the 2 min duration) >>>>>>>>> >>>>>>>>> The way I''m testing it is setting up Apache and then moving down a file with `wget`. The use case for this machine is an Apache based app that serves large files to customers. That is why I think a `wget` is more telling of "real" performance than netperf. I''m running the test again and on the client side I am seeing usage over the maxbw limit I have set. `wget` is reporting about 2Mbps transfer rate which is much closer to what I was seeing in the show-usage statistics. >>>>>>>>> >>>>>>>> >>>>>>>>> [cdelgado at Bluegene tmp]$ wget sol/myfile.dat >>>>>>>>> --10:01:30-- sol/myfile.dat >>>>>>>>> Resolving sol... 192.168.69.104 >>>>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>>>> Length: 1048576000 (1000M) >>>>>>>>> Saving to: `myfile.dat'' >>>>>>>>> >>>>>>>>> 5% [==> ] 55,530,974 267K/s eta 60m 44s >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> BTW, setting a maxbw for a link (dladm) doesn''t really impact the >>>>>>>>>> flow as the bandwidth for both are independent. >>>>>>>>> >>>>>>>>> Thank you for this clarification but I still don''t understand how I can be seeing ~2Mbps transfer if both the link and the flow are both capped at 1.2Mbps. >>>>>>>>> >>>>>>>> >>>>>>>> Can you try with a higher bandwidth, say 100 Mbps and see what the results >>>>>>>> are when compared to wget''s output? >>>>>>>> >>>>>>>> Also, another way of manually checking would be to do a >>>>>>>> # kstat -c flow -n http-flow >>>>>>>> >>>>>>>> before and after the wget run and see how many bytes (rbytes) the >>>>>>>> kernel has seen for that flow (assuming there isn''t any other traffic >>>>>>>> going over the flow), and then determine the bandwidth (you might need >>>>>>>> to get the duration of the wget run pretty close to get the >>>>>>>> right bandwdith value). >>>>>>>> >>>>>>>> -venu >>>>>>> >>>>>>> I changed the flow to be 100Mbps. >>>>>>> >>>>>>> root at myhost:/tmp# flowadm show-flowprop -p maxbw http-flow >>>>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>>>> http-flow maxbw 100 -- 100 >>>>>>> >>>>>>> I also removed the maxbw=1.228 I had set on the link. When I changed this value on the link I lost all network to the machine. Had to go in and manually reboot it. The network came up fine. >>>>>>> >>>>>>> I created two files one before wget and one after with no other network traffic except the ssh session. I created a 1 GB file to transfer. The output is a bit confusing to me. >>>>>>> >>>>>>> root at myhost:/tmp# ls http-flow-* >>>>>>> http-flow-1.txt http-flow-2.txt >>>>>>> root at myhost:/tmp# cat http-flow-* >>>>>>> module: unix instance: 0 >>>>>>> name: http-flow class: flow >>>>>>> crtime 188662.418688816 >>>>>>> ierrors 0 >>>>>>> ipackets 4680 >>>>>>> obytes 182520974 >>>>>>> oerrors 0 >>>>>>> opackets 127521 >>>>>>> rbytes 308138 >>>>>>> snaptime 189431.882363589 >>>>>>> >>>>>>> module: unix instance: 0 >>>>>>> name: http-flow class: flow >>>>>>> crtime 188662.418688816 >>>>>>> ierrors 0 >>>>>>> ipackets 31262 >>>>>>> obytes 1281785975 >>>>>>> oerrors 0 >>>>>>> opackets 895533 >>>>>>> rbytes 2062678 >>>>>>> snaptime 189595.122347726 >>>>>>> >>>>>>> So this means the network only passed ~1.6 MB of data? >>>>>> >>>>>> is this on the outbound side or inbound? >>>>>> >>>>>> looks like it has a total of 8794120008 bits in the outbound side (obytes) and 14036320 bits on the inbound (rbytes). What is the duration >>>>>> 93 seconds? So the outbound would be ~94 Mbps and inbound 150 Kbps. >>>>>> >>>>> >>>>> Sorry, outbound. I was looking at the wrong row. With the flow set at 50 I''m getting getting ~37 Mbps. In my mind the flow is not respecting the value I''m giving it. With the maxbw set to 100 I could get ~80 Mbps and now I''m only seeing ~37Mbps. Could it be something in the way Crossbow is averaging the actually bits and bytes going down the pipe or is it me not understanding the system? >>>> >>>> For TCP you might get a little less than the set bandwidth as TCP >>>> would do flow control too (i.e if for bandwidth reasons ACKs come back >>>> slower or packets get dropped, TCP will react and will slow down). Also, >>>> note that the current property (maxbw) is a *limit* which means we will >>>> ensure that the flow will not get *more* than what you have asked for. >>>> Going forwards we are looking at adding guarantee where in the flow would >>>> get what it asked for. >>>> >>>> -venu >>>> >>>>> >>>>> root at myhost:/tmp# flowadm show-flowprop http-flow >>>>> FLOW PROPERTY VALUE DEFAULT POSSIBLE >>>>> http-flow maxbw 50 -- 50 >>>>> http-flow priority medium -- medium >>>>> >>>>> root at myhost:/tmp# cat http-flow-* >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 32346 >>>>> obytes 1300129135 >>>>> oerrors 0 >>>>> opackets 908351 >>>>> rbytes 2133486 >>>>> snaptime 191261.607310645 >>>>> >>>>> module: unix instance: 0 >>>>> name: http-flow class: flow >>>>> crtime 188662.418688816 >>>>> ierrors 0 >>>>> ipackets 85361 >>>>> obytes 2399401106 >>>>> oerrors 0 >>>>> opackets 1676468 >>>>> rbytes 5632604 >>>>> snaptime 192621.873203347 >>>>> >>>>> >>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>> --13:52:20-- sol/myfile.dat >>>>> Resolving sol... 192.168.69.104 >>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>> HTTP request sent, awaiting response... 200 OK >>>>> Length: 1048576000 (1000M) >>>>> Saving to: `myfile.dat'' >>>>> >>>>> 100%[==========================================================>] 1,048,576,000 4.06M/s in 3m 44s >>>>> >>>>> 13:56:04 (4.46 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>>> >>>>> >>>>> real 3m44.364s >>>>> user 0m1.042s >>>>> sys 0m24.065s >>>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> The wget command was telling me it was getting ~80Mbps which is under the threshold of the flow. >>>>>>> >>>>>>> [cdelgado at Bluegene tmp]$ time wget sol/myfile.dat >>>>>>> --13:21:49-- sol/myfile.dat >>>>>>> Resolving sol... 192.168.69.104 >>>>>>> Connecting to sol|192.168.69.104|:80... connected. >>>>>>> HTTP request sent, awaiting response... 200 OK >>>>>>> Length: 1048576000 (1000M) >>>>>>> Saving to: `myfile.dat'' >>>>>>> >>>>>>> 100%[==========================================================>] 1,048,576,000 10.5M/s in 93s >>>>>>> >>>>>>> 13:23:23 (10.7 MB/s) - `myfile.dat'' saved [1048576000/1048576000] >>>>>>> >>>>>>> >>>>>>> real 1m33.701s >>>>>>> user 0m0.899s >>>>>>> sys 0m23.874s >>>>>>> >>>>>>> Maybe a value of 100Mbps is too high for this machine. I might try with 50Mbps to see what wget says. >>>>>>> >>>>>> >>>>>> OK, >>>>>> >>>>>> -venu >>>>>> >>>>>>> -Cesar >>>>>>> >>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:32:22 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>>>> http-flow 01:32:22 01:32:42 1512 2571 0.001 Mbps >>>>>>>>>>> ssh-flow 01:32:42 01:33:02 1818 3578 0.002 Mbps >>>>>>>>>>> http-flow 01:33:02 01:33:22 66917 3165136 1.292 Mbp >>>>>>>>>>> ssh-flow 01:33:02 01:33:22 3618 5344 0.003 Mbps >>>>>>>>>>> http-flow 01:33:22 01:33:42 117947 5713018 2.332 Mbp >>>>>>>>>>> ssh-flow 01:33:22 01:33:42 4182 3020 0.002 Mbps >>>>>>>>>>> http-flow 01:33:42 01:34:02 118998 5685520 2.321 Mbp >>>>>>>>>>> ssh-flow 01:33:42 01:34:02 11616 9924 0.008 Mbps >>>>>>>>>>> http-flow 01:34:02 01:34:22 117084 5725664 2.337 Mbp >>>>>>>>>>> http-flow 01:34:22 01:34:42 119130 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:34:42 01:35:02 114180 5725168 2.335 Mbp >>>>>>>>>>> http-flow 01:35:02 01:35:22 109230 5725664 2.333 Mbp >>>>>>>>>>> http-flow 01:35:22 01:35:42 116160 5725168 2.336 Mbp >>>>>>>>>>> http-flow 01:35:42 01:36:02 119262 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:36:02 01:36:22 119196 5725664 2.337 Mbp >>>>>>>>>>> http-flow 01:36:22 01:36:42 117216 5725168 2.336 Mbp >>>>>>>>>>> http-flow 01:36:42 01:37:02 119394 5722636 2.336 Mbp >>>>>>>>>>> http-flow 01:37:02 01:37:22 119526 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:37:22 01:37:42 119460 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:37:42 01:38:02 119460 5725664 2.338 Mbp >>>>>>>>>>> http-flow 01:38:02 01:38:22 119724 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:38:22 01:38:42 119724 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:38:42 01:39:02 119130 5722636 2.336 Mbp >>>>>>>>>>> http-flow 01:39:02 01:39:22 118866 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:39:22 01:39:42 116490 5725664 2.336 Mbp >>>>>>>>>>> http-flow 01:39:42 01:40:02 119790 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:40:02 01:40:22 117678 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:40:22 01:40:42 118668 5725664 2.337 Mbp >>>>>>>>>>> http-flow 01:40:42 01:41:02 117414 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:41:02 01:41:22 119790 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:41:22 01:41:42 119813 5720510 2.336 Mbp >>>>>>>>>>> http-flow 01:41:42 01:42:02 119394 5725664 2.338 Mbp >>>>>>>>>>> http-flow 01:42:02 01:42:22 119724 5722272 2.336 Mbp >>>>>>>>>>> http-flow 01:42:22 01:42:42 119526 5725664 2.338 Mbp >>>>>>>>>>> http-flow 01:42:42 01:43:02 119196 5722140 2.336 Mbp >>>>>>>>>>> http-flow 01:43:02 01:43:22 119394 5725664 2.338 Mbp >>>>>>>>>>> http-flow 01:43:22 01:43:42 119658 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:43:42 01:44:02 119064 5725168 2.337 Mbp >>>>>>>>>>> http-flow 01:44:02 01:44:22 113256 5676668 2.315 Mbp >>>>>>>>>>> ssh-flow 01:44:02 01:44:22 18414 49646 0.027 Mbps >>>>>>>>>>> http-flow 01:44:22 01:44:42 118206 5725664 2.337 Mbp >>>>>>>>>>> http-flow 01:44:42 01:45:02 117282 5722140 2.335 Mbp >>>>>>>>>>> ssh-flow 01:44:42 01:45:02 4698 3544 0.003 Mbps >>>>>>>>>>> http-flow 01:45:02 01:45:22 118536 5688284 2.322 Mbp >>>>>>>>>>> ssh-flow 01:45:02 01:45:22 4092 3198 0.002 Mbps >>>>>>>>>>> http-flow 01:45:22 01:45:42 119130 5725168 2.337 Mbp >>>>>>>>>>> ssh-flow 01:45:22 01:45:42 1980 1478 0.001 Mbps >>>>>>>>>>> >>>>>>>>>>> That''s above the flow''s maxbw parameter. After that I tried to change the maxbw of the link with dladm and that brought the bandwidth down but still not down to 1.2 Mbps. >>>>>>>>>>> >>>>>>>>>>> root at myhost:~# dladm show-linkprop -p maxbw e1000g0 >>>>>>>>>>> LINK PROPERTY PERM VALUE DEFAULT POSSIBLE >>>>>>>>>>> e1000g0 maxbw rw 1.228 -- -- >>>>>>>>>>> >>>>>>>>>>> root at myhost:~# flowadm show-usage -s 11/23/2009,01:46:02 -e 11/23/2009,01:46:22 -f /var/log/net.log | grep -v "0 Mbps\|^FLOW" >>>>>>>>>>> http-flow 01:46:02 01:46:22 119394 5725168 2.337 Mbp >>>>>>>>>>> ssh-flow 01:46:02 01:46:22 4680 2980 0.003 Mbps >>>>>>>>>>> http-flow 01:46:22 01:46:42 94314 4520316 1.845 Mbp >>>>>>>>>>> >>>>>>>>>>> Any ideas or is there a subtlety that I''m missing and the behavior is correct? >>>>>>>>>>> >>>>>>>>>>> Thanks for the help. >>>>>>>>>>> >>>>>>>>>>> -Cesar >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> crossbow-discuss mailing list >>>>>>>>>>> crossbow-discuss at opensolaris.org >>>>>>>>>>> mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >> >> > >