crossbow discuss - Jan 2008 - Does capping network bandwidth work in Crossbow?

Hi,

I am trying to set bandwidth caps for two zones. I created two vnics and
assigned them to two zones ''exclusive''ly. Then, I used the
following
commands to set the bandwidth limits

#dladm modify-vnic -b 100 1
#dladm modify-vnic -b 900 2

and I see that they are set correctly.

# dladm show-vnic
        device  speed
vnic1   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:bd  (fixed)  bw limit:
100 Mbps    cpus: 0
vnic2   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:be  (fixed)  bw limit:
900 Mbps    cpus: 0

I am measuring the bandwidth used by a zone, by calculating tx + rx bytes as
seen by ''kstat -n vnic?''. Is this the right way to measure it?

I use httperf to connect to the webservers running in the two zones, and I
always see the bandwidth used by the zones exceeding the limits often
reaching 400Mbps (my workload is very network heavy).

I tried using the -L option as specified in the dladm man page
(http://dlc.sun.com/osol/netvirt/downloads/20070214/dladm.1m.txt) to cap
bandwidth, but I get an error.

# dladm modify-vnic -b 100 -L 1
dladm: unrecognized option ''-L''

I am not sure whether this is the right way to cap bandwidth. Another way to
do it, I believe, is to use flowadm and setup flows with limited bandwidth.
I thought that probably uses more protocol processing than necessary, since
I just want to cap the full vnic bandwidth.

Any ideas?

Pradeep

Hi, Pradeep:

On Thu, 17 Jan 2008, Pradeep Padala wrote:
> Hi,
>
> I am trying to set bandwidth caps for two zones. I created two vnics and
> assigned them to two zones ''exclusive''ly. Then, I used
the following
> commands to set the bandwidth limits
>
> #dladm modify-vnic -b 100 1
> #dladm modify-vnic -b 900 2
>
> and I see that they are set correctly.
>
> # dladm show-vnic
>        device  speed
> vnic1   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:bd  (fixed)  bw limit:
> 100 Mbps    cpus: 0
> vnic2   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:be  (fixed)  bw limit:
> 900 Mbps    cpus: 0
That''s right.
>
> I am measuring the bandwidth used by a zone, by calculating tx + rx bytes
as
> seen by ''kstat -n vnic?''. Is this the right way to
measure it?
I suspect that may  work. I think you might have earlier bits that
did bandwidth regulation in squeues (in IP). So, unless the squeue always
polls the vnic to get the no. of pkts per the set limit, the numbers above
might not give you the right info. E.g packet that comes via an interrupt
will update the vnic counters and be sent upstream to the appropriate
squeue, which will enforce the limit. (This logic has changed and we will be
pushing out the latest bits on OpenSolaris shortly, possibly in the next
week. With the latest bits the above method - kstat - should be fine; see
http://www.opensolaris.org/os/project/crossbow/Design_softringset.txt
for some info. about bandwidth control et. al.)

>
> I use httperf to connect to the webservers running in the two zones, and I
> always see the bandwidth used by the zones exceeding the limits often
> reaching 400Mbps (my workload is very network heavy).
>
> I tried using the -L option as specified in the dladm man page
> (http://dlc.sun.com/osol/netvirt/downloads/20070214/dladm.1m.txt) to cap
> bandwidth, but I get an error.
>
> # dladm modify-vnic -b 100 -L 1
> dladm: unrecognized option ''-L''
-L is not needed for dladm.
>
> I am not sure whether this is the right way to cap bandwidth. Another way
to
> do it, I believe, is to use flowadm and setup flows with limited bandwidth.
> I thought that probably uses more protocol processing than necessary, since
> I just want to cap the full vnic bandwidth.
>
Yes, you should just be able to set the vnic limit, if that''s
sufficient.

> Any ideas?
Until the new bits are made available: I haven''t used httperf,  but you
could try netperf and look at its results just to check if the limit is
being enforced.


-venu
>
> Pradeep
>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

Thanks for the clarifications Venu. I was measuring bandwidth by checking
''kstat -n'' and as you mentioned, it''s probably not
accurate.

Using ''netstat -k'', I see that the bandwidth is capped
correctly. The only
problem I have now is to get the output of ''netstat -k'' in
plain text,
instead of the curses based output. My python wrapper runs with out a
terminal, as I mentioned to you in the other e-mail.

Please let me know when the new bits are available.

Thanks again!
Pradeep 
> -----Original Message-----
> From: venugopal iyer [mailto:venu at sun.com] 
> Sent: Friday, January 18, 2008 12:12 PM
> To: Pradeep Padala
> Cc: crossbow-discuss at opensolaris.org
> Subject: Re: [crossbow-discuss] Does capping network 
> bandwidth work in Crossbow?
> 
> 
> Hi, Pradeep:
> 
> On Thu, 17 Jan 2008, Pradeep Padala wrote:
> 
> > Hi,
> >
> > I am trying to set bandwidth caps for two zones. I created 
> two vnics 
> > and assigned them to two zones ''exclusive''ly. Then,
I used the
> > following commands to set the bandwidth limits
> >
> > #dladm modify-vnic -b 100 1
> > #dladm modify-vnic -b 900 2
> >
> > and I see that they are set correctly.
> >
> > # dladm show-vnic
> >        device  speed
> > vnic1   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:bd  
> (fixed)  bw limit:
> > 100 Mbps    cpus: 0
> > vnic2   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:be  
> (fixed)  bw limit:
> > 900 Mbps    cpus: 0
> 
> That''s right.
> 
> >
> > I am measuring the bandwidth used by a zone, by calculating tx + rx 
> > bytes as seen by ''kstat -n vnic?''. Is this the right
way to
> measure it?
> 
> I suspect that may  work. I think you might have earlier bits 
> that did bandwidth regulation in squeues (in IP). So, unless 
> the squeue always polls the vnic to get the no. of pkts per 
> the set limit, the numbers above might not give you the right 
> info. E.g packet that comes via an interrupt will update the 
> vnic counters and be sent upstream to the appropriate squeue, 
> which will enforce the limit. (This logic has changed and we 
> will be pushing out the latest bits on OpenSolaris shortly, 
> possibly in the next week. With the latest bits the above 
> method - kstat - should be fine; see 
> http://www.opensolaris.org/os/project/crossbow/Design_softringset.txt
> for some info. about bandwidth control et. al.)
> 
> 
> >
> > I use httperf to connect to the webservers running in the 
> two zones, 
> > and I always see the bandwidth used by the zones exceeding 
> the limits 
> > often reaching 400Mbps (my workload is very network heavy).
> >
> > I tried using the -L option as specified in the dladm man page
> > 
> (http://dlc.sun.com/osol/netvirt/downloads/20070214/dladm.1m.txt) to 
> > cap bandwidth, but I get an error.
> >
> > # dladm modify-vnic -b 100 -L 1
> > dladm: unrecognized option ''-L''
> 
> -L is not needed for dladm.
> 
> >
> > I am not sure whether this is the right way to cap 
> bandwidth. Another 
> > way to do it, I believe, is to use flowadm and setup flows 
> with limited bandwidth.
> > I thought that probably uses more protocol processing than 
> necessary, 
> > since I just want to cap the full vnic bandwidth.
> >
> 
> Yes, you should just be able to set the vnic limit, if that''s 
> sufficient.
> 
> 
> > Any ideas?
> 
> Until the new bits are made available: I haven''t used 
> httperf,  but you could try netperf and look at its results 
> just to check if the limit is being enforced.
> 
> 
> -venu
> >
> > Pradeep
> >
> > _______________________________________________
> > crossbow-discuss mailing list
> > crossbow-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
> >
>

Pradeep:

On Fri, 18 Jan 2008, Pradeep Padala wrote:
> Thanks for the clarifications Venu. I was measuring bandwidth by checking
> ''kstat -n'' and as you mentioned, it''s probably
not accurate.
>
> Using ''netstat -k'', I see that the bandwidth is capped
correctly.
OK, while the stats may not be accurate, but the output from netstat
should not be very different from that calculated using kstat. netstat
gets the kstat values for vnic1, gets the increment over the last time you
read (via netstat) and gives the average input/output for that time duration.
Perhaps you are not computing the values from kstat -n correctly?
I believe netstat uses the following logic:

e.g for input:
 	input kbps = diff in (rbytes64 in kb) since prev read / time since
 		     last read;

similarly for output.
> The only
> problem I have now is to get the output of ''netstat -k''
in plain text,
> instead of the curses based output. My python wrapper runs with out a
> terminal, as I mentioned to you in the other e-mail.
Could you check that you are using the kstat info. correctly in you
computation. Then, I guess, this should not be an issue.
>
> Please let me know when the new bits are available.
OK,

-venu
>
> Thanks again!
> Pradeep
>
>> -----Original Message-----
>> From: venugopal iyer [mailto:venu at sun.com]
>> Sent: Friday, January 18, 2008 12:12 PM
>> To: Pradeep Padala
>> Cc: crossbow-discuss at opensolaris.org
>> Subject: Re: [crossbow-discuss] Does capping network
>> bandwidth work in Crossbow?
>>
>>
>> Hi, Pradeep:
>>
>> On Thu, 17 Jan 2008, Pradeep Padala wrote:
>>
>>> Hi,
>>>
>>> I am trying to set bandwidth caps for two zones. I created
>> two vnics
>>> and assigned them to two zones ''exclusive''ly.
Then, I used the
>>> following commands to set the bandwidth limits
>>>
>>> #dladm modify-vnic -b 100 1
>>> #dladm modify-vnic -b 900 2
>>>
>>> and I see that they are set correctly.
>>>
>>> # dladm show-vnic
>>>        device  speed
>>> vnic1   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:bd
>> (fixed)  bw limit:
>>> 100 Mbps    cpus: 0
>>> vnic2   e1000g0 1000  Mbps      mac: 0:14:4f:8d:a9:be
>> (fixed)  bw limit:
>>> 900 Mbps    cpus: 0
>>
>> That''s right.
>>
>>>
>>> I am measuring the bandwidth used by a zone, by calculating tx + rx
>>> bytes as seen by ''kstat -n vnic?''. Is this the
right way to
>> measure it?
>>
>> I suspect that may  work. I think you might have earlier bits
>> that did bandwidth regulation in squeues (in IP). So, unless
>> the squeue always polls the vnic to get the no. of pkts per
>> the set limit, the numbers above might not give you the right
>> info. E.g packet that comes via an interrupt will update the
>> vnic counters and be sent upstream to the appropriate squeue,
>> which will enforce the limit. (This logic has changed and we
>> will be pushing out the latest bits on OpenSolaris shortly,
>> possibly in the next week. With the latest bits the above
>> method - kstat - should be fine; see
>> http://www.opensolaris.org/os/project/crossbow/Design_softringset.txt
>> for some info. about bandwidth control et. al.)
>>
>>
>>>
>>> I use httperf to connect to the webservers running in the
>> two zones,
>>> and I always see the bandwidth used by the zones exceeding
>> the limits
>>> often reaching 400Mbps (my workload is very network heavy).
>>>
>>> I tried using the -L option as specified in the dladm man page
>>>
>> (http://dlc.sun.com/osol/netvirt/downloads/20070214/dladm.1m.txt) to
>>> cap bandwidth, but I get an error.
>>>
>>> # dladm modify-vnic -b 100 -L 1
>>> dladm: unrecognized option ''-L''
>>
>> -L is not needed for dladm.
>>
>>>
>>> I am not sure whether this is the right way to cap
>> bandwidth. Another
>>> way to do it, I believe, is to use flowadm and setup flows
>> with limited bandwidth.
>>> I thought that probably uses more protocol processing than
>> necessary,
>>> since I just want to cap the full vnic bandwidth.
>>>
>>
>> Yes, you should just be able to set the vnic limit, if that''s
>> sufficient.
>>
>>
>>> Any ideas?
>>
>> Until the new bits are made available: I haven''t used
>> httperf,  but you could try netperf and look at its results
>> just to check if the limit is being enforced.
>>
>>
>> -venu
>>>
>>> Pradeep
>>>
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>
>>
>
>

> > Thanks for the clarifications Venu. I was measuring bandwidth by 
> > checking ''kstat -n'' and as you mentioned,
it''s probably not
> accurate.
> >
> > Using ''netstat -k'', I see that the bandwidth is
capped correctly.
> 
> OK, while the stats may not be accurate, but the output from 
> netstat should not be very different from that calculated 
> using kstat. netstat gets the kstat values for vnic1, gets 
> the increment over the last time you read (via netstat) and 
> gives the average input/output for that time duration.
> Perhaps you are not computing the values from kstat -n correctly?
> I believe netstat uses the following logic:
> 
> e.g for input:
>  	input kbps = diff in (rbytes64 in kb) since prev read / 
> time since
>  		     last read;
> 
> similarly for output.
That''s exactly how I am measuring it. My python code looks like this

timestamp = time.time()
output = commands.getoutput(''kstat -n vnic%d | grep bytes'' %
get_coid(coname)).split(''\n'')
obytes = float(re.split(''\s+'', output[0])[2])
rbytes = float(re.split(''\s+'', output[1])[2])
return rbytes + obytes

One question. The output of ''kstat -n vnic?'' looks like this
(I just
rebooted, so most numbers are zero)

module: unix                            instance: 0     
name:   vnic1                           class:    flow
        crtime                          109.436840067
        ierrors                         0
        ipackets                        268
        obytes                          0
        oerrors                         0
        opackets                        0
        rbytes                          24488
        snaptime                        520.551380521

module: vnic                            instance: 1     
name:   vnic1                           class:    net
        brdcstrcv                       1977
        brdcstxmt                       0
        collisions                      0
        crtime                          484.32586088
        ierrors                         2361
        ifspeed                         200000000
        ipackets                        268
        ipackets64                      268
        multircv                        0
        multixmt                        0
        norcvbuf                        0
        noxmtbuf                        0
        obytes                          0
        obytes64                        0
        oerrors                         0
        opackets                        0
        opackets64                      0
        rbytes                          24488
        rbytes64                        24488
        snaptime                        520.552493729
        unknowns                        0

As you can see, there are a set of stats for ''flow'' and for
''net''. Which
ones should I be using? So far, I am taking the first set of stats.

Pradeep

Pradeep Padala wrote:
>>> Thanks for the clarifications Venu. I was measuring bandwidth by 
>>> checking ''kstat -n'' and as you mentioned,
it''s probably not
>>>       
>> accurate.
>>     
>>> Using ''netstat -k'', I see that the bandwidth is
capped correctly.
>>>       
>> OK, while the stats may not be accurate, but the output from 
>> netstat should not be very different from that calculated 
>> using kstat. netstat gets the kstat values for vnic1, gets 
>> the increment over the last time you read (via netstat) and 
>> gives the average input/output for that time duration.
>> Perhaps you are not computing the values from kstat -n correctly?
>> I believe netstat uses the following logic:
>>
>> e.g for input:
>>  	input kbps = diff in (rbytes64 in kb) since prev read / 
>> time since
>>  		     last read;
>>
>> similarly for output.
>>     
>
> That''s exactly how I am measuring it. My python code looks like
this
>
> timestamp = time.time()
> output = commands.getoutput(''kstat -n vnic%d | grep
bytes'' %
> get_coid(coname)).split(''\n'')
> obytes = float(re.split(''\s+'', output[0])[2])
> rbytes = float(re.split(''\s+'', output[1])[2])
> return rbytes + obytes
>
> One question. The output of ''kstat -n vnic?'' looks like
this (I just
> rebooted, so most numbers are zero)
>
> module: unix                            instance: 0     
> name:   vnic1                           class:    flow
>         crtime                          109.436840067
>         ierrors                         0
>         ipackets                        268
>         obytes                          0
>         oerrors                         0
>         opackets                        0
>         rbytes                          24488
>         snaptime                        520.551380521
>
> module: vnic                            instance: 1     
> name:   vnic1                           class:    net
>         brdcstrcv                       1977
>         brdcstxmt                       0
>         collisions                      0
>         crtime                          484.32586088
>         ierrors                         2361
>         ifspeed                         200000000
>         ipackets                        268
>         ipackets64                      268
>         multircv                        0
>         multixmt                        0
>         norcvbuf                        0
>         noxmtbuf                        0
>         obytes                          0
>         obytes64                        0
>         oerrors                         0
>         opackets                        0
>         opackets64                      0
>         rbytes                          24488
>         rbytes64                        24488
>         snaptime                        520.552493729
>         unknowns                        0
>
> As you can see, there are a set of stats for ''flow'' and
for ''net''. Which
> ones should I be using? So far, I am taking the first set of stats.
>   
I''d use the second set of stats but at least these 2 look the same.

Everything that is displayed in netstat -k is derived from kstats.
If you want to avoid the curses stuff, you could try a form of
dladm show-vnic with -s (for stats) and -i (for interval).

-Mike

> I''d use the second set of stats but at least these 2 look the
same.
> 
> Everything that is displayed in netstat -k is derived from kstats.
> If you want to avoid the curses stuff, you could try a form 
> of dladm show-vnic with -s (for stats) and -i (for interval).
Thanks, that seems like a simpler solution.

But, I am still getting consumptions far exceeding the bw caps. I also see
netstat -k reporting utlization exceeding 100% at times. I checked my code
as well, and I don''t see any problems with the measurement 

To see what''s happening, I ran a wget on a 1G file. When I set the bw
cap to
a certain number, it works fine. Say to 200Mbps, and I download the file,
it''s limited to 200Mbps, no problem. But, if I change it in the middle
to
say 300Mbps, no cap is being enforced after I execute the ''dladm
modify-vnic'' command. In my SpecWeb experiment, I am changing bandwidth
every 10 secs. Is that too fast? How long does it take for the (new) cap to
get enforced?

This may or may not be related to a problem we saw with ''tc''
on Linux. We
found similar problem with capping in tc and figured out that we had to set
the right MTU and burst size for 1Gbps networks.

Pradeep

>> I''d use the second set of stats but at least these 2 look the
same.
>>
>> Everything that is displayed in netstat -k is derived from kstats.
>> If you want to avoid the curses stuff, you could try a form 
>> of dladm show-vnic with -s (for stats) and -i (for interval).
> 
> Thanks, that seems like a simpler solution.
> 
> But, I am still getting consumptions far exceeding the bw caps. I also see
> netstat -k reporting utlization exceeding 100% at times. I checked my code
> as well, and I don''t see any problems with the measurement 
> 
> To see what''s happening, I ran a wget on a 1G file. When I set the
bw cap to
> a certain number, it works fine. Say to 200Mbps, and I download the file,
> it''s limited to 200Mbps, no problem. But, if I change it in the
middle to
> say 300Mbps, no cap is being enforced after I execute the ''dladm
> modify-vnic'' command. In my SpecWeb experiment, I am changing
bandwidth
> every 10 secs. Is that too fast? How long does it take for the (new) cap to
> get enforced?
> 
> This may or may not be related to a problem we saw with
''tc'' on Linux. We
> found similar problem with capping in tc and figured out that we had to set
> the right MTU and burst size for 1Gbps networks.
Any update on this? Is this a real problem or Am I using the tools 
incorrectly? For our control experiments, correct setting of bandwidth 
cap is crucial.

Let me know how I can make capping work correctly.

Thanks,
Pradeep