Hi,
I''d like to understand how priorities set on VNICs operating over a
single NIC relate to priorities set on the flows associated with those
VNICs. Namely:
1. Suppose I have multiple VNICs with multiple flows per VNIC, operating
over a single physical NIC. If I specify some priority for a flow, is
that priority relative to the priorities of all other flows across all
VNICs or only those flows that belong to the same VNIC? For example, if
I have two VNICs - vnic1 and vnic2 - and two flows - flow1-1 and flow1-2
- defined on vnic1 and three flows - flow2-1, flow2-2, flow2-3 - defined
on vnic2, on a fully loaded system, with the priority set to the same
value on all VNICs and flows, will the bandwidth be distributed as:
OBJECT BANDWIDTH FRACTION
vnic1 50%
flow1-1 25%
flow1-2 25%
vnic2 50%
flow2-1 16.7%
flow2-2 16.7%
flow2-3 16.7%
or as:
OBJECT BANDWIDTH FRACTION
vnic1 40%
flow1-1 20%
flow1-2 20%
vnic2 60%
flow2-1 20%
flow2-2 20%
flow2-3 20%
I expect the former.
http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says:
"When you create a flow, you can allocate bandwidth to it and then
give the flow a relative priority among all flows on the interface."
but I''d like to confirm whether that applies to VNICs operating over
the
same physical interface.
2. How do I access the "default flow" of a VNIC? The diagram on the
http://opensolaris.org/os/project/crossbow/ page shows a flow named
"default flow", but in my experiments with flowadm I haven''t
seen such a
flow. Does the default flow exist? Can I set the priority or show
statistics for it?
3. What if I don''t set the priority on a VNIC or flow? The man pages
say
the default is "high" for interfaces and "medium" for flows,
but if I
don''t set the priority on e.g. a VNIC and do "dladm
show-linkprop" I see
"--" in the VALUE column and "High" in the DEFAULT column,
while some
other properties have equal values in both columns. Does "--" means
the
default priority or no priority?
snv101a, BFU Oct 31 2008.
Thank you,
Vasiliy
Vasiliy Baranov wrote:> 1. Suppose I have multiple VNICs with multiple flows per VNIC, > operating over a single physical NIC. If I specify some priority for a > flow, is that priority relative to the priorities of all other flows > across all VNICs or only those flows that belong to the same VNIC? For > example, if I have two VNICs - vnic1 and vnic2 - and two flows - > flow1-1 and flow1-2 - defined on vnic1 and three flows - flow2-1, > flow2-2, flow2-3 - defined on vnic2, on a fully loaded system, with > the priority set to the same value on all VNICs and flows, will the > bandwidth be distributed as: > > OBJECT BANDWIDTH FRACTION > vnic1 50% > flow1-1 25% > flow1-2 25% > vnic2 50% > flow2-1 16.7% > flow2-2 16.7% > flow2-3 16.7% > > or as: > > OBJECT BANDWIDTH FRACTION > vnic1 40% > flow1-1 20% > flow1-2 20% > vnic2 60% > flow2-1 20% > flow2-2 20% > flow2-3 20% > > I expect the former. > http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says: > > "When you create a flow, you can allocate bandwidth to it and then > give the flow a relative priority among all flows on the interface." > > but I''d like to confirm whether that applies to VNICs operating over > the same physical interface.I studied some more documentation and got even more confused. This thread http://markmail.org/message/m4mgdpgfixjqfzyz#query:+page:1+mid:xtip474mo76vqzwu+state:results seems to imply that NIC and flow priorities actually map to priorities of threads processing the corresponding stacks. Now it is unclear to me how deterministic the effect of priority on bandwidth distribution across competing stacks is. Say if I have two VNICs with the same priority and on both VNICs try to consume more than the total available bandwidth, will the bandwidth be distributed 50%/50% between the VNICs? Adding flows and flow priorities back here, when I have multiple VNICs with multiple flows per VNIC, is the distribution of bandwidth between VNICs determined solely by the priorities of those VNICs? Do flow priorities affect distribution of bandwidth only within respective VNICs or also between VNICs? Am I asking stupid questions? Thanks, Vasiliy
Vasiliy Baranov wrote:> Vasiliy Baranov wrote: >> 1. Suppose I have multiple VNICs with multiple flows per VNIC, >> operating over a single physical NIC. If I specify some priority for a >> flow, is that priority relative to the priorities of all other flows >> across all VNICs or only those flows that belong to the same VNIC? For >> example, if I have two VNICs - vnic1 and vnic2 - and two flows - >> flow1-1 and flow1-2 - defined on vnic1 and three flows - flow2-1, >> flow2-2, flow2-3 - defined on vnic2, on a fully loaded system, with >> the priority set to the same value on all VNICs and flows, will the >> bandwidth be distributed as: >> >> OBJECT BANDWIDTH FRACTION >> vnic1 50% >> flow1-1 25% >> flow1-2 25% >> vnic2 50% >> flow2-1 16.7% >> flow2-2 16.7% >> flow2-3 16.7% >> >> or as: >> >> OBJECT BANDWIDTH FRACTION >> vnic1 40% >> flow1-1 20% >> flow1-2 20% >> vnic2 60% >> flow2-1 20% >> flow2-2 20% >> flow2-3 20% >> >> I expect the former. >> http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says: >> >> "When you create a flow, you can allocate bandwidth to it and then >> give the flow a relative priority among all flows on the interface." >> >> but I''d like to confirm whether that applies to VNICs operating over >> the same physical interface. > > I studied some more documentation and got even more confused. This > thread > http://markmail.org/message/m4mgdpgfixjqfzyz#query:+page:1+mid:xtip474mo76vqzwu+state:results > seems to imply that NIC and flow priorities actually map to priorities > of threads processing the corresponding stacks. Now it is unclear to me > how deterministic the effect of priority on bandwidth distribution > across competing stacks is. Say if I have two VNICs with the same > priority and on both VNICs try to consume more than the total available > bandwidth, will the bandwidth be distributed 50%/50% between the VNICs? > > Adding flows and flow priorities back here, when I have multiple VNICs > with multiple flows per VNIC, is the distribution of bandwidth between > VNICs determined solely by the priorities of those VNICs? Do flow > priorities affect distribution of bandwidth only within respective VNICs > or also between VNICs? > > Am I asking stupid questions?No, you are asking pretty reasonable questions so let me try and answer them. The VNICs are currently the key. They along with the primary (what you used to see has e1000g, nxge, ixgbe, etc) are all links in crossbow world. The primary link has the default or primary mac address but other than that its no different (by default it gets more Rx rings as well). Now links are what maintain fairness across all links. So in principal, if the links have a Rx/Tx ring assigned to them and they get a separate CPUs (either there were enough or administrator specified CPUs via dladm -C), the priorities have no role to play. The fairness should be there by default and if you want to share (or partition) bandwidth based on policy, then specify the bandwidth limit (using dladm -p maxbw=xxx). Now moving on to flows - Flows operate on a per link basis (primary or VNIC). Flows configured on one link should not interact with flows configured on other links (unless we don''t have enough CPUs and threads processing links and flows start interacting with each other since they are running on same CPU). In such cases, priority is the way to differentiate. To be really honest, priorities are still work in progress. We see them working in some cases (like Real time etc) but in some cases they become meaning less because app is running a lower priority. I think going forward, we need to do some more work in making sure they are as simple to use as other Crossbow features (like VNICs, B/W partitioning, cpu binding, flows etc). So to answer your specific questions, if fairness is all you want, then don''t configure priorities or B/W partition but instead just make sure that individual links (for PVM guests) have their own CPUs (use the dladm -C option if necessary). On top, if you want policy based B/W sharing (the guy has only paid for 100Mb of usage), then configure the B/W limits on links using dladm -p maxbw=xxx. The use of priority in these scenarios might make things complicated for you. Let me know if you still have questions. Cheers, Sunay -- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow
Sunay Tripathi wrote:> Vasiliy Baranov wrote: >> Vasiliy Baranov wrote: >>> 1. Suppose I have multiple VNICs with multiple flows per VNIC, >>> operating over a single physical NIC. If I specify some priority for >>> a flow, is that priority relative to the priorities of all other >>> flows across all VNICs or only those flows that belong to the same >>> VNIC? For example, if I have two VNICs - vnic1 and vnic2 - and two >>> flows - flow1-1 and flow1-2 - defined on vnic1 and three flows - >>> flow2-1, flow2-2, flow2-3 - defined on vnic2, on a fully loaded >>> system, with the priority set to the same value on all VNICs and >>> flows, will the bandwidth be distributed as: >>> >>> OBJECT BANDWIDTH FRACTION >>> vnic1 50% >>> flow1-1 25% >>> flow1-2 25% >>> vnic2 50% >>> flow2-1 16.7% >>> flow2-2 16.7% >>> flow2-3 16.7% >>> >>> or as: >>> >>> OBJECT BANDWIDTH FRACTION >>> vnic1 40% >>> flow1-1 20% >>> flow1-2 20% >>> vnic2 60% >>> flow2-1 20% >>> flow2-2 20% >>> flow2-3 20% >>> >>> I expect the former. >>> http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says: >>> >>> "When you create a flow, you can allocate bandwidth to it and >>> then give the flow a relative priority among all flows on the >>> interface." >>> >>> but I''d like to confirm whether that applies to VNICs operating over >>> the same physical interface. >> >> I studied some more documentation and got even more confused. This >> thread >> http://markmail.org/message/m4mgdpgfixjqfzyz#query:+page:1+mid:xtip474mo76vqzwu+state:results >> seems to imply that NIC and flow priorities actually map to >> priorities of threads processing the corresponding stacks. Now it is >> unclear to me how deterministic the effect of priority on bandwidth >> distribution across competing stacks is. Say if I have two VNICs with >> the same priority and on both VNICs try to consume more than the >> total available bandwidth, will the bandwidth be distributed 50%/50% >> between the VNICs? >> >> Adding flows and flow priorities back here, when I have multiple >> VNICs with multiple flows per VNIC, is the distribution of bandwidth >> between VNICs determined solely by the priorities of those VNICs? Do >> flow priorities affect distribution of bandwidth only within >> respective VNICs or also between VNICs? >> >> Am I asking stupid questions? > > No, you are asking pretty reasonable questions so let me try and answer > them. The VNICs are currently the key. They along with the primary > (what you used to see has e1000g, nxge, ixgbe, etc) are all links in > crossbow world. The primary link has the default or primary mac address > but other than that its no different (by default it gets more Rx > rings as well). > > Now links are what maintain fairness across all links. So in > principal, if the links have a Rx/Tx ring assigned to them and they > get a separate CPUs (either there were enough or administrator > specified CPUs via dladm -C), the priorities have no role to play. > The fairness should be there by default and if you want to share > (or partition) bandwidth based on policy, then specify the > bandwidth limit (using dladm -p maxbw=xxx). > > Now moving on to flows - Flows operate on a per link basis > (primary or VNIC). Flows configured on one link should not > interact with flows configured on other links (unless we > don''t have enough CPUs and threads processing links and > flows start interacting with each other since they are running > on same CPU). In such cases, priority is the way to differentiate. > > To be really honest, priorities are still work in progress. We > see them working in some cases (like Real time etc) but in > some cases they become meaning less because app is running a > lower priority. I think going forward, we need to do some more > work in making sure they are as simple to use as other > Crossbow features (like VNICs, B/W partitioning, cpu binding, > flows etc). > > So to answer your specific questions, if fairness is all you > want, then don''t configure priorities or B/W partition but > instead just make sure that individual links (for PVM guests) > have their own CPUs (use the dladm -C option if necessary). On > top, if you want policy based B/W sharing (the guy has only > paid for 100Mb of usage), then configure the B/W limits on > links using dladm -p maxbw=xxx. The use of priority in these > scenarios might make things complicated for you. > > Let me know if you still have questions.Thank you very much for your reply. Indeed, I need fairness between VNICs, and I want to use flows for accounting of different traffic types (as a side note, I am evaluating other accounting mechanisms too). I am only interested in flow statistic and, in all other senses, want things to work as if there were no flows. As I understand, if I assign different CPUs to different links and don''t set priorities on flows, that should be close to what I want. Thank you very much, Vasiliy
Vasiliy Baranov wrote:> Sunay Tripathi wrote: >> Vasiliy Baranov wrote: >>> Vasiliy Baranov wrote: >>>> 1. Suppose I have multiple VNICs with multiple flows per VNIC, >>>> operating over a single physical NIC. If I specify some priority >>>> for a flow, is that priority relative to the priorities of all >>>> other flows across all VNICs or only those flows that belong to the >>>> same VNIC? For example, if I have two VNICs - vnic1 and vnic2 - and >>>> two flows - flow1-1 and flow1-2 - defined on vnic1 and three flows >>>> - flow2-1, flow2-2, flow2-3 - defined on vnic2, on a fully loaded >>>> system, with the priority set to the same value on all VNICs and >>>> flows, will the bandwidth be distributed as: >>>> >>>> OBJECT BANDWIDTH FRACTION >>>> vnic1 50% >>>> flow1-1 25% >>>> flow1-2 25% >>>> vnic2 50% >>>> flow2-1 16.7% >>>> flow2-2 16.7% >>>> flow2-3 16.7% >>>> >>>> or as: >>>> >>>> OBJECT BANDWIDTH FRACTION >>>> vnic1 40% >>>> flow1-1 20% >>>> flow1-2 20% >>>> vnic2 60% >>>> flow2-1 20% >>>> flow2-2 20% >>>> flow2-3 20% >>>> >>>> I expect the former. >>>> http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says: >>>> >>>> "When you create a flow, you can allocate bandwidth to it and >>>> then give the flow a relative priority among all flows on the >>>> interface." >>>> >>>> but I''d like to confirm whether that applies to VNICs operating >>>> over the same physical interface. >>> >>> I studied some more documentation and got even more confused. This >>> thread >>> http://markmail.org/message/m4mgdpgfixjqfzyz#query:+page:1+mid:xtip474mo76vqzwu+state:results >>> seems to imply that NIC and flow priorities actually map to >>> priorities of threads processing the corresponding stacks. Now it is >>> unclear to me how deterministic the effect of priority on bandwidth >>> distribution across competing stacks is. Say if I have two VNICs >>> with the same priority and on both VNICs try to consume more than >>> the total available bandwidth, will the bandwidth be distributed >>> 50%/50% between the VNICs? >>> >>> Adding flows and flow priorities back here, when I have multiple >>> VNICs with multiple flows per VNIC, is the distribution of bandwidth >>> between VNICs determined solely by the priorities of those VNICs? >>> Do flow priorities affect distribution of bandwidth only within >>> respective VNICs or also between VNICs? >>> >>> Am I asking stupid questions? >> >> No, you are asking pretty reasonable questions so let me try and answer >> them. The VNICs are currently the key. They along with the primary >> (what you used to see has e1000g, nxge, ixgbe, etc) are all links in >> crossbow world. The primary link has the default or primary mac address >> but other than that its no different (by default it gets more Rx >> rings as well). >> >> Now links are what maintain fairness across all links. So in >> principal, if the links have a Rx/Tx ring assigned to them and they >> get a separate CPUs (either there were enough or administrator >> specified CPUs via dladm -C), the priorities have no role to play. >> The fairness should be there by default and if you want to share >> (or partition) bandwidth based on policy, then specify the >> bandwidth limit (using dladm -p maxbw=xxx). >> >> Now moving on to flows - Flows operate on a per link basis >> (primary or VNIC). Flows configured on one link should not >> interact with flows configured on other links (unless we >> don''t have enough CPUs and threads processing links and >> flows start interacting with each other since they are running >> on same CPU). In such cases, priority is the way to differentiate. >> >> To be really honest, priorities are still work in progress. We >> see them working in some cases (like Real time etc) but in >> some cases they become meaning less because app is running a >> lower priority. I think going forward, we need to do some more >> work in making sure they are as simple to use as other >> Crossbow features (like VNICs, B/W partitioning, cpu binding, >> flows etc). >> >> So to answer your specific questions, if fairness is all you >> want, then don''t configure priorities or B/W partition but >> instead just make sure that individual links (for PVM guests) >> have their own CPUs (use the dladm -C option if necessary). On >> top, if you want policy based B/W sharing (the guy has only >> paid for 100Mb of usage), then configure the B/W limits on >> links using dladm -p maxbw=xxx. The use of priority in these >> scenarios might make things complicated for you. >> >> Let me know if you still have questions.One more question. Where is the -C option of dladm documented? According to http://dlc.sun.com/osol/netvirt/downloads/20081031/dladm.1m.txt, if I am parsing it correctly, processor binding is specified via the "cpus" property: dladm create-vnic -l bge0 -p cpus=0 vnic0 Am I missing something? Thank you, Vasiliy
Vasiliy Baranov wrote:> Vasiliy Baranov wrote: >> Sunay Tripathi wrote: >>> Vasiliy Baranov wrote: >>>> Vasiliy Baranov wrote: >>>>> 1. Suppose I have multiple VNICs with multiple flows per VNIC, >>>>> operating over a single physical NIC. If I specify some priority >>>>> for a flow, is that priority relative to the priorities of all >>>>> other flows across all VNICs or only those flows that belong to the >>>>> same VNIC? For example, if I have two VNICs - vnic1 and vnic2 - and >>>>> two flows - flow1-1 and flow1-2 - defined on vnic1 and three flows >>>>> - flow2-1, flow2-2, flow2-3 - defined on vnic2, on a fully loaded >>>>> system, with the priority set to the same value on all VNICs and >>>>> flows, will the bandwidth be distributed as: >>>>> >>>>> OBJECT BANDWIDTH FRACTION >>>>> vnic1 50% >>>>> flow1-1 25% >>>>> flow1-2 25% >>>>> vnic2 50% >>>>> flow2-1 16.7% >>>>> flow2-2 16.7% >>>>> flow2-3 16.7% >>>>> >>>>> or as: >>>>> >>>>> OBJECT BANDWIDTH FRACTION >>>>> vnic1 40% >>>>> flow1-1 20% >>>>> flow1-2 20% >>>>> vnic2 60% >>>>> flow2-1 20% >>>>> flow2-2 20% >>>>> flow2-3 20% >>>>> >>>>> I expect the former. >>>>> http://opensolaris.org/os/project/crossbow/Docs/vnicresource.pdf says: >>>>> >>>>> "When you create a flow, you can allocate bandwidth to it and >>>>> then give the flow a relative priority among all flows on the >>>>> interface." >>>>> >>>>> but I''d like to confirm whether that applies to VNICs operating >>>>> over the same physical interface. >>>> >>>> I studied some more documentation and got even more confused. This >>>> thread >>>> http://markmail.org/message/m4mgdpgfixjqfzyz#query:+page:1+mid:xtip474mo76vqzwu+state:results >>>> seems to imply that NIC and flow priorities actually map to >>>> priorities of threads processing the corresponding stacks. Now it is >>>> unclear to me how deterministic the effect of priority on bandwidth >>>> distribution across competing stacks is. Say if I have two VNICs >>>> with the same priority and on both VNICs try to consume more than >>>> the total available bandwidth, will the bandwidth be distributed >>>> 50%/50% between the VNICs? >>>> >>>> Adding flows and flow priorities back here, when I have multiple >>>> VNICs with multiple flows per VNIC, is the distribution of bandwidth >>>> between VNICs determined solely by the priorities of those VNICs? >>>> Do flow priorities affect distribution of bandwidth only within >>>> respective VNICs or also between VNICs? >>>> >>>> Am I asking stupid questions? >>> >>> No, you are asking pretty reasonable questions so let me try and answer >>> them. The VNICs are currently the key. They along with the primary >>> (what you used to see has e1000g, nxge, ixgbe, etc) are all links in >>> crossbow world. The primary link has the default or primary mac address >>> but other than that its no different (by default it gets more Rx >>> rings as well). >>> >>> Now links are what maintain fairness across all links. So in >>> principal, if the links have a Rx/Tx ring assigned to them and they >>> get a separate CPUs (either there were enough or administrator >>> specified CPUs via dladm -C), the priorities have no role to play. >>> The fairness should be there by default and if you want to share >>> (or partition) bandwidth based on policy, then specify the >>> bandwidth limit (using dladm -p maxbw=xxx). >>> >>> Now moving on to flows - Flows operate on a per link basis >>> (primary or VNIC). Flows configured on one link should not >>> interact with flows configured on other links (unless we >>> don''t have enough CPUs and threads processing links and >>> flows start interacting with each other since they are running >>> on same CPU). In such cases, priority is the way to differentiate. >>> >>> To be really honest, priorities are still work in progress. We >>> see them working in some cases (like Real time etc) but in >>> some cases they become meaning less because app is running a >>> lower priority. I think going forward, we need to do some more >>> work in making sure they are as simple to use as other >>> Crossbow features (like VNICs, B/W partitioning, cpu binding, >>> flows etc). >>> >>> So to answer your specific questions, if fairness is all you >>> want, then don''t configure priorities or B/W partition but >>> instead just make sure that individual links (for PVM guests) >>> have their own CPUs (use the dladm -C option if necessary). On >>> top, if you want policy based B/W sharing (the guy has only >>> paid for 100Mb of usage), then configure the B/W limits on >>> links using dladm -p maxbw=xxx. The use of priority in these >>> scenarios might make things complicated for you. >>> >>> Let me know if you still have questions. > > One more question. Where is the -C option of dladm documented? According > to http://dlc.sun.com/osol/netvirt/downloads/20081031/dladm.1m.txt, if I > am parsing it correctly, processor binding is specified via the "cpus" > property: > > dladm create-vnic -l bge0 -p cpus=0 vnic0 > > Am I missing something?Sorry. cpus is correct. I forgot that we had changed the syntax few months back making them link properties. Thanks, Sunay -- Sunay Tripathi Distinguished Engineer Solaris Core Operating System Sun MicroSystems Inc. Solaris Networking: http://www.opensolaris.org/os/community/networking Project Crossbow: http://www.opensolaris.org/os/project/crossbow