Following is a short description on assigning priorities to a link (via
dladm). Flows will have a similar mechanism (via flowadm).
thanks,
-venu
----------------------------------------------------------------------------
The components associated with a link or flow, such as Soft Ring Sets,
Soft Rings and squeues, have kernel threads processing inbound/outbound
packets. The scheduler processes threads based on a thread''s priority
level.
The priority level of a thread determine how it is scheduled w.r.t other
kernel threads. Thus by assigning a priority to a link/flow, we can control
how the threads associated with that link/flow is processed w.r.t other
link/flows. When the system is busy, threads with higher priority will have
a greater slice of CPU time, than the ones with lower priority.
The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI
(100) for threads in the system class.
We define three levels of priority - low, medium and high within the range
MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when creating
a virtual link (say VNIC) or can be modified for an existing link using
dladm(1m). The priority level is a property of the link.
Setting a priority essentially means setting priorities for the following
elements associated with a MAC client[1], specifically:
Soft Ring Set(s), SRS, (Rx and Tx):
(srs_pri)
o Worker thread
o Poll Thread (Rx)
o Associated Soft Ring(s)
-> Worker Thread
-> Poll Thread (Rx)
-> squeue
(sq_priority)
worker thread
poll thread (Rx)
Setting a Default Priority
--------------------------
There are two ways a MAC client can be opened, either implicitly when plumbing
a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual
link, e.g VNIC. In the case of the latter, it is quite possible that the
properties of the link, including priority, is present when the MAC client
is created.
When a MAC client is created, we always set a default priority. The default
priority is set to be medium. Subsequently, the priority can be modified,
if specified (e.g. in the case of a VNIC), as part of creating the virtual
link itself.
We add the following members to mac_client_impl_t
pri_t mci_pri;
pri_t mci_max_pri;
pri_t mci_min_pri;
where mci_pri is the priority assigned with the MAC client. mci_min_pri and
mci_max_pri is the available range corresponding to the priority for
the MAC client. The range will be used for any flows that are created on
top of the MAC client.
Given a range min and max, we assign low, medium and high value as follows:
priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) +
1)))
where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high).
Additionally, we obtain the range for the given priority as:
min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri)))
max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS))
for e.g.: when we create a MAC client given the range of MINCLSYSPRI and
MAXCLSYSPRI, we set a default priority of medium, which translates to the
following for the MAC client:
mci_pri = 86
mci_min_pri = 73
mci_max_pri = 86
All the threads associated with the MAC client elements (listed above) will
have a thread priority of 86. Any flow created on the MAC client will now
derive its priority value using the specified priority (low, medium and high)
for that flow within the range 73 and 86.
If we create a virtual link, say a VNIC, with a specific priority, say high,
mac_client_open() will initialize the values for the VNIC''s MAC client
with
the default priority. Subsequently, mac_resource_ctl_set() will result in
modifying the priority values for the MAC client to the value corresponding
to high.
Modifying Priority
------------------
A set-linkprop after the MAC client has been opened can be used to modify
the priority of a link. The result is a new priority value and range for
the MAC client based on the new priority. Additionally, any flows on top
of the MAC client will also need to be reprioritized as the range for the
MAC client will change.
Setting priority before opening a MAC client
---------------------------------------------
A set-linkprop before a MAC client has been created will require the priority
values (in fact, any property) to be cached in the mac_impl_t. When the
primary MAC client is opened, the associated properties can be set on the
newly created MAC client post mac_client_open().
_______________________________________________________________________________
References:
[1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf
venugopal iyer wrote:> Following is a short description on assigning priorities to a link (via > dladm). Flows will have a similar mechanism (via flowadm). > > thanks, > > -venu > > ---------------------------------------------------------------------------- > > The components associated with a link or flow, such as Soft Ring Sets, > Soft Rings and squeues, have kernel threads processing inbound/outbound > packets. The scheduler processes threads based on a thread''s priority > level. > > The priority level of a thread determine how it is scheduled w.r.t other > kernel threads. Thus by assigning a priority to a link/flow, we can control > how the threads associated with that link/flow is processed w.r.t other > link/flows. When the system is busy, threads with higher priority will have > a greater slice of CPU time, than the ones with lower priority. > > The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI > (100) for threads in the system class. > > We define three levels of priority - low, medium and high within the range > MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when creating > a virtual link (say VNIC) or can be modified for an existing link using > dladm(1m). The priority level is a property of the link. > > Setting a priority essentially means setting priorities for the following > elements associated with a MAC client[1], specifically: > > Soft Ring Set(s), SRS, (Rx and Tx): > (srs_pri) > o Worker thread > o Poll Thread (Rx) > o Associated Soft Ring(s) > -> Worker Thread > -> Poll Thread (Rx) > -> squeue > (sq_priority) > worker thread > poll thread (Rx) > > > Setting a Default Priority > -------------------------- > There are two ways a MAC client can be opened, either implicitly when plumbing > a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual > link, e.g VNIC. In the case of the latter, it is quite possible that the > properties of the link, including priority, is present when the MAC client > is created. > > When a MAC client is created, we always set a default priority. The default > priority is set to be medium. Subsequently, the priority can be modified, > if specified (e.g. in the case of a VNIC), as part of creating the virtual > link itself. > > We add the following members to mac_client_impl_t > > pri_t mci_pri; > pri_t mci_max_pri; > pri_t mci_min_pri; > > where mci_pri is the priority assigned with the MAC client. mci_min_pri and > mci_max_pri is the available range corresponding to the priority for > the MAC client. The range will be used for any flows that are created on > top of the MAC client. > > Given a range min and max, we assign low, medium and high value as follows: > > priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) + 1))) > > where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high). > Additionally, we obtain the range for the given priority as: > > min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri))) > > max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS)) > > for e.g.: when we create a MAC client given the range of MINCLSYSPRI and > MAXCLSYSPRI, we set a default priority of medium, which translates to the > following for the MAC client: > > mci_pri = 86 > mci_min_pri = 73 > mci_max_pri = 86 > > All the threads associated with the MAC client elements (listed above) will > have a thread priority of 86. Any flow created on the MAC client will now > derive its priority value using the specified priority (low, medium and high) > for that flow within the range 73 and 86. >Since there are only three priority levels, can you list the thread priorities (min/max) for the low and high cases also (instead of me calculating them ;-)). Thanks, -krgopi> If we create a virtual link, say a VNIC, with a specific priority, say high, > mac_client_open() will initialize the values for the VNIC''s MAC client with > the default priority. Subsequently, mac_resource_ctl_set() will result in > modifying the priority values for the MAC client to the value corresponding > to high. > > Modifying Priority > ------------------ > > A set-linkprop after the MAC client has been opened can be used to modify > the priority of a link. The result is a new priority value and range for > the MAC client based on the new priority. Additionally, any flows on top > of the MAC client will also need to be reprioritized as the range for the > MAC client will change. > > Setting priority before opening a MAC client > --------------------------------------------- > > A set-linkprop before a MAC client has been created will require the priority > values (in fact, any property) to be cached in the mac_impl_t. When the > primary MAC client is opened, the associated properties can be set on the > newly created MAC client post mac_client_open(). > > _______________________________________________________________________________ > References: > > [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
On Mon, 26 Nov 2007, Rajagopal Kunhappan wrote:> > > venugopal iyer wrote: >> Following is a short description on assigning priorities to a link (via >> dladm). Flows will have a similar mechanism (via flowadm). >> >> thanks, >> >> -venu >> >> >> ---------------------------------------------------------------------------- >> >> The components associated with a link or flow, such as Soft Ring Sets, >> Soft Rings and squeues, have kernel threads processing inbound/outbound >> packets. The scheduler processes threads based on a thread''s priority >> level. >> >> The priority level of a thread determine how it is scheduled w.r.t other >> kernel threads. Thus by assigning a priority to a link/flow, we can control >> how the threads associated with that link/flow is processed w.r.t other >> link/flows. When the system is busy, threads with higher priority will have >> a greater slice of CPU time, than the ones with lower priority. >> >> The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI >> (100) for threads in the system class. >> >> We define three levels of priority - low, medium and high within the range >> MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when >> creating >> a virtual link (say VNIC) or can be modified for an existing link using >> dladm(1m). The priority level is a property of the link. >> >> Setting a priority essentially means setting priorities for the following >> elements associated with a MAC client[1], specifically: >> >> Soft Ring Set(s), SRS, (Rx and Tx): >> (srs_pri) >> o Worker thread >> o Poll Thread (Rx) >> o Associated Soft Ring(s) >> -> Worker Thread >> -> Poll Thread (Rx) >> -> squeue >> (sq_priority) >> worker thread >> poll thread (Rx) >> >> >> Setting a Default Priority >> -------------------------- >> There are two ways a MAC client can be opened, either implicitly when >> plumbing >> a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual >> link, e.g VNIC. In the case of the latter, it is quite possible that the >> properties of the link, including priority, is present when the MAC client >> is created. >> >> When a MAC client is created, we always set a default priority. The default >> priority is set to be medium. Subsequently, the priority can be modified, >> if specified (e.g. in the case of a VNIC), as part of creating the virtual >> link itself. >> >> We add the following members to mac_client_impl_t >> >> pri_t mci_pri; >> pri_t mci_max_pri; >> pri_t mci_min_pri; >> >> where mci_pri is the priority assigned with the MAC client. mci_min_pri and >> mci_max_pri is the available range corresponding to the priority for >> the MAC client. The range will be used for any flows that are created on >> top of the MAC client. >> >> Given a range min and max, we assign low, medium and high value as follows: >> >> priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) >> + 1))) >> >> where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high). >> Additionally, we obtain the range for the given priority as: >> >> min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri))) >> >> max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS)) >> >> for e.g.: when we create a MAC client given the range of MINCLSYSPRI and >> MAXCLSYSPRI, we set a default priority of medium, which translates to the >> following for the MAC client: >> >> mci_pri = 86 >> mci_min_pri = 73 >> mci_max_pri = 86 >> >> All the threads associated with the MAC client elements (listed above) will >> have a thread priority of 86. Any flow created on the MAC client will now >> derive its priority value using the specified priority (low, medium and >> high) >> for that flow within the range 73 and 86. >> > Since there are only three priority levels, can you list the thread > priorities (min/max) for the low and high cases also (instead of me > calculating them ;-)).Within the range MINCLSYSPRI and MAXCLSYSPRI: low would be: mci_pri = 73 mci_min_pri = 60 mci_max_pri = 73 high would be: mci_pri = 99 mci_min_pri = 86 mci_max_pri = 99 -venu> > Thanks, > -krgopi >> If we create a virtual link, say a VNIC, with a specific priority, say >> high, >> mac_client_open() will initialize the values for the VNIC''s MAC client with >> the default priority. Subsequently, mac_resource_ctl_set() will result in >> modifying the priority values for the MAC client to the value corresponding >> to high. >> >> Modifying Priority >> ------------------ >> >> A set-linkprop after the MAC client has been opened can be used to modify >> the priority of a link. The result is a new priority value and range for >> the MAC client based on the new priority. Additionally, any flows on top >> of the MAC client will also need to be reprioritized as the range for the >> MAC client will change. >> >> Setting priority before opening a MAC client >> --------------------------------------------- >> >> A set-linkprop before a MAC client has been created will require the >> priority >> values (in fact, any property) to be cached in the mac_impl_t. When the >> primary MAC client is opened, the associated properties can be set on the >> newly created MAC client post mac_client_open(). >> >> _______________________________________________________________________________ >> References: >> >> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>
Hi Venu, venugopal iyer wrote:> Following is a short description on assigning priorities to a link (via > dladm). Flows will have a similar mechanism (via flowadm). > > thanks, > > -venu > > ---------------------------------------------------------------------------- > > The components associated with a link or flow, such as Soft Ring Sets, > Soft Rings and squeues, have kernel threads processing inbound/outbound > packets. The scheduler processes threads based on a thread''s priority > level. >will link/flow priority be static or dynamic (can re-adjust as per flows/links get added)?> The priority level of a thread determine how it is scheduled w.r.t other > kernel threads. Thus by assigning a priority to a link/flow, we can control > how the threads associated with that link/flow is processed w.r.t other > link/flows. When the system is busy, threads with higher priority will have > a greater slice of CPU time, than the ones with lower priority. > > The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI > (100) for threads in the system class. > > We define three levels of priority - low, medium and high within the range > MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when creating > a virtual link (say VNIC) or can be modified for an existing link using > dladm(1m). The priority level is a property of the link. > > Setting a priority essentially means setting priorities for the following > elements associated with a MAC client[1], specifically: > > Soft Ring Set(s), SRS, (Rx and Tx): > (srs_pri) > o Worker thread > o Poll Thread (Rx) > o Associated Soft Ring(s) > -> Worker Thread > -> Poll Thread (Rx) > -> squeue > (sq_priority) > worker thread > poll thread (Rx) > > > Setting a Default Priority > -------------------------- > There are two ways a MAC client can be opened, either implicitly when plumbing > a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual > link, e.g VNIC. In the case of the latter, it is quite possible that the > properties of the link, including priority, is present when the MAC client > is created. >in case of link priorities, would it make sense to take into account the bandwidth associated with link? (cause, the thread servicing link with higher bandwidth cap, would make sense to have higher priority).> When a MAC client is created, we always set a default priority. The default > priority is set to be medium. Subsequently, the priority can be modified, > if specified (e.g. in the case of a VNIC), as part of creating the virtual > link itself. > > We add the following members to mac_client_impl_t > > pri_t mci_pri; > pri_t mci_max_pri; > pri_t mci_min_pri; > >why min_pri and max_pri range is different per mac_client_imp based? are there any min/max bandwidth ranges as well and there is any association with above priority ranges?> where mci_pri is the priority assigned with the MAC client. mci_min_pri and > mci_max_pri is the available range corresponding to the priority for > the MAC client. The range will be used for any flows that are created on > top of the MAC client. > >are the min, max ranges dynamic/static? what could be the deciding factor to calculate min/max ranges for given mac_impl? can you have this min,max ranges static based on type whether NIC, VNIC or FLOW?> Given a range min and max, we assign low, medium and high value as follows: > > priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) + 1))) > > where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high). > Additionally, we obtain the range for the given priority as: > > min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri))) > > max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS)) > > for e.g.: when we create a MAC client given the range of MINCLSYSPRI and > MAXCLSYSPRI, we set a default priority of medium, which translates to the > following for the MAC client: > > mci_pri = 86 > mci_min_pri = 73 > mci_max_pri = 86 > > All the threads associated with the MAC client elements (listed above) will > have a thread priority of 86.you mean vnics and physical NICs?> Any flow created on the MAC client will now > derive its priority value using the specified priority (low, medium and high) > for that flow within the range 73 and 86. >so flows created on NICs or VNICs will have priority upto NICs or vnics existing priority. so threads processing packets on flows will have priorities lower than thread processing packets deposited on vnics?> If we create a virtual link, say a VNIC, with a specific priority, say high, > mac_client_open() will initialize the values for the VNIC''s MAC client with > the default priority.I think we would dis-honor user''s request of a high priority in that case? I may be wrong. if it is not in the range, do we error out? what should be the default priority in this case?> Subsequently, mac_resource_ctl_set()at what events, it would get called?> will result in > modifying the priority values for the MAC client to the value corresponding > to high. > > Modifying Priority > ------------------ > > A set-linkprop after the MAC client has been opened can be used to modify > the priority of a link. The result is a new priority value and range for > the MAC client based on the new priority. Additionally, any flows on top > of the MAC client will also need to be reprioritized as the range for the > MAC client will change. >what events could trigger re-adjusting the thread priorities? I mean, can user change it through flowadm or dladm? can addition/removal of more vnics or flows on given NICs could cause this change? or change of bandwidth on given link or flow cause this? if, there are multiple threads with equal priorities on the system, does solaris scheduler is designed to resolve this contention, does it fall back on say round-robin etc? (I understand, scope of your changes for crossbow is limited to assigning and maintaining right priorities, but just curious. :-))> Setting priority before opening a MAC client > --------------------------------------------- > > A set-linkprop before a MAC client has been created will require the priority > values (in fact, any property) to be cached in the mac_impl_t. When the > primary MAC client is opened, the associated properties can be set on the > newly created MAC client post mac_client_open(). >would above initialize the priority to be used, or max priority limit? -Deepti> _______________________________________________________________________________ > References: > > [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
Thanks for looking at this, Deepti. On Mon, 26 Nov 2007, deepti dhokte wrote:> Hi > Venu, > > venugopal iyer wrote: >> Following is a short description on assigning priorities to a link (via >> dladm). Flows will have a similar mechanism (via flowadm). >> >> thanks, >> >> -venu >> >> ---------------------------------------------------------------------------- >> >> The components associated with a link or flow, such as Soft Ring Sets, >> Soft Rings and squeues, have kernel threads processing inbound/outbound >> packets. The scheduler processes threads based on a thread''s priority >> level. >> > will link/flow priority be static or dynamic (can re-adjust as per > flows/links get added)?static.>> The priority level of a thread determine how it is scheduled w.r.t other >> kernel threads. Thus by assigning a priority to a link/flow, we can control >> how the threads associated with that link/flow is processed w.r.t other >> link/flows. When the system is busy, threads with higher priority will have >> a greater slice of CPU time, than the ones with lower priority. >> >> The kernel provides priority range between MINCLSYSPRI (60) and MAXCLSYSPRI >> (100) for threads in the system class. >> >> We define three levels of priority - low, medium and high within the range >> MINCLSYSPRI and MAXCLSYSPRI. The priority level can be assigned when creating >> a virtual link (say VNIC) or can be modified for an existing link using >> dladm(1m). The priority level is a property of the link. >> >> Setting a priority essentially means setting priorities for the following >> elements associated with a MAC client[1], specifically: >> >> Soft Ring Set(s), SRS, (Rx and Tx): >> (srs_pri) >> o Worker thread >> o Poll Thread (Rx) >> o Associated Soft Ring(s) >> -> Worker Thread >> -> Poll Thread (Rx) >> -> squeue >> (sq_priority) >> worker thread >> poll thread (Rx) >> >> >> Setting a Default Priority >> -------------------------- >> There are two ways a MAC client can be opened, either implicitly when plumbing >> a physical link, e.g. bge0 (by DLS) or explicitly when creating a virtual >> link, e.g VNIC. In the case of the latter, it is quite possible that the >> properties of the link, including priority, is present when the MAC client >> is created. >> > in case of link priorities, would it make sense to take into account the > bandwidth associated with link? > (cause, the thread servicing link with higher bandwidth cap, would make > sense to have higher priority).Bandwidth and link properties are configured by the user. There may be implications on how they interwork, but implicitly we don''t try to make any association between them. When we support guarantee, we might want to look into this. So, this is something that we might want to look at in the future.>> When a MAC client is created, we always set a default priority. The default >> priority is set to be medium. Subsequently, the priority can be modified, >> if specified (e.g. in the case of a VNIC), as part of creating the virtual >> link itself. >> >> We add the following members to mac_client_impl_t >> >> pri_t mci_pri; >> pri_t mci_max_pri; >> pri_t mci_min_pri; >> >> > why min_pri and max_pri range is different per mac_client_imp based? > are there any min/max bandwidth ranges as well and there is any > association with above priority ranges?No, but the priority is relative, in the sense high, low and medium, w.r.t to some base, which is the range.> >> where mci_pri is the priority assigned with the MAC client. mci_min_pri and >> mci_max_pri is the available range corresponding to the priority for >> the MAC client. The range will be used for any flows that are created on >> top of the MAC client. >> >> > are the min, max ranges dynamic/static?static.> what could be the deciding factor to calculate min/max ranges for given > mac_impl?You mean for a MAC client? The priority for the MAC client determines the range, note that the range is not for that MAC client itself, but any flows etc. create on top of the MAC client.> can you have this min,max ranges static based on type whether NIC, VNIC > or FLOW?I don''t follow. The range is for a MAC client to be used for any flows created on top of it. i.e we could create a VNIC with a medium priority, which will determine the range for the MAC client corresponding to the VNIC. Now, when we create a flow with a high priority on top of the VNIC, the high is with the range of that MAC client.> >> Given a range min and max, we assign low, medium and high value as follows: >> >> priority = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * ((pri) + 1))) >> >> where MAC_PRI_LEVELS is 3 and pri is 0 (low), 1 (medium) or 2 (high). >> Additionally, we obtain the range for the given priority as: >> >> min_pri = ((min) + ((((max) - (min)) / MAC_PRI_LEVELS) * (pri))) >> >> max_pri = ((min_pri) + (((max) - (min)) / MAC_PRI_LEVELS)) >> >> for e.g.: when we create a MAC client given the range of MINCLSYSPRI and >> MAXCLSYSPRI, we set a default priority of medium, which translates to the >> following for the MAC client: >> >> mci_pri = 86 >> mci_min_pri = 73 >> mci_max_pri = 86 >> >> All the threads associated with the MAC client elements (listed above) will >> have a thread priority of 86. > you mean vnics and physical NICs? >> Any flow created on the MAC client will now >> derive its priority value using the specified priority (low, medium and high) >> for that flow within the range 73 and 86. >> > so flows created on NICs or VNICs will have priority upto NICs or vnics > existing priority.Yes, based on the NIC/VNIC.> so threads processing packets on flows will have priorities lower than > thread processing packets > deposited on vnics?A flow with a high priority will have the same priority value as the MAC client, while one with low/medium will have lower.>> If we create a virtual link, say a VNIC, with a specific priority, say high, >> mac_client_open() will initialize the values for the VNIC''s MAC client with >> the default priority. > I think we would dis-honor user''s request of a high priority in that case? > I may be wrong.mac_client_open() will always start with a default priority. For a VNIC with a high priority, mac_resource_ctl_set() (part of the same dladm create-vnic call) will set it appropriately, so we don''t dishonor the request.> if it is not in the range, do we error out? > what should be the default priority in this case?Let me know if these are still unanswered.>> Subsequently, mac_resource_ctl_set() > at what events, it would get called?It is part of the creation of the virtual NIC itself.>> will result in >> modifying the priority values for the MAC client to the value corresponding >> to high. >> >> Modifying Priority >> ------------------ >> >> A set-linkprop after the MAC client has been opened can be used to modify >> the priority of a link. The result is a new priority value and range for >> the MAC client based on the new priority. Additionally, any flows on top >> of the MAC client will also need to be reprioritized as the range for the >> MAC client will change. >> > what events could trigger re-adjusting the thread priorities?dladm modify-vnic> > I mean, > can user change it through flowadm or dladm?yes.> can addition/removal of more vnics or flows on given NICs could cause > this change?no.> or change of bandwidth on given link or flow cause this?not now, at least. This may change in the future.> > if, there are multiple threads with equal priorities on the system, does > solaris scheduler > is designed to resolve this contention, does it fall back on say > round-robin etc?I would assume so, but I am not absolutely sure.> (I understand, scope of your changes for crossbow is limited to assigning > and maintaining right priorities, but just curious. :-))8)> > > >> Setting priority before opening a MAC client >> --------------------------------------------- >> >> A set-linkprop before a MAC client has been created will require the priority >> values (in fact, any property) to be cached in the mac_impl_t. When the >> primary MAC client is opened, the associated properties can be set on the >> newly created MAC client post mac_client_open(). >> > would above initialize the priority to be used, or max priority limit?It would start with the default, but as part of the creation, it would update the priority to that cached, if at all. thanks, Deepti. -venu> -Deepti >> _______________________________________________________________________________ >> References: >> >> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >> > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
venugopal iyer wrote:> >> what could be the deciding factor to calculate min/max ranges for given >> mac_impl? >> > > You mean for a MAC client? The priority for the MAC client determines the > range, note that the range is not for that MAC client itself, but any > flows etc. create on top of the MAC client. >I see, now, I follow that part better .> >> can you have this min,max ranges static based on type whether NIC, VNIC >> or FLOW? >> > > I don''t follow. The range is for a MAC client to be used for any flows > created on top of it. i.e we could create a VNIC with a medium priority, > which will determine the range for the MAC client corresponding to the > VNIC. Now, when we create a flow with a high priority on top of the VNIC, > the high is with the range of that MAC client. >ok, say I have mac_client, would it have priority say x,y and min,max and z as assigned/used.? now I add flow on top of it. what would be the min,max and assigned priorities for it? since th "high for the flow being added to vnic should be with "in?" the range of the mac client? also, wonder if flow a mac_client as well? (here the priority on flow could it be the one and only priority and that''s what is its max priority?)>> so threads processing packets on flows will have priorities lower than >> thread processing packets >> deposited on vnics? >> > > A flow with a high priority will have the same priority value as the MAC > client,do you mean a flow created on vnic,aggr etc ? so will there be any min, max and medium priority ranges for flows and will be derived from mac_client and will be exact same as mac_client? or flows on top of mac_clients get only one static priority thruout its lifetime?> while one with low/medium will have lower. > > >>> If we create a virtual link, say a VNIC, with a specific priority, say high, >>> mac_client_open() will initialize the values for the VNIC''s MAC client with >>> the default priority. >>> >> I think we would dis-honor user''s request of a high priority in that case? >> I may be wrong. >> > > mac_client_open() will always start with a default priority. For a VNIC > with a high priority, mac_resource_ctl_set() (part of the same > dladm create-vnic call) will set it appropriately, so we don''t dishonor > the request. > >when you set default priority you set default values for mci_pri, mci_max_pri, mci_min_pri or you set default priority for just mci_pri? I thought all 3 could be set. is mci_pri a priority being currently used priority of the thread servicing flow (SRS based) or vnic (mac_impl based). ?>> if it is not in the range, do we error out? >> what should be the default priority in this case? >> > > Let me know if these are still unanswered. > >I don''t understand it. sorry. say you create vnic, then add flow ? - how min, max and mci_pri be set for vnic and does the same equivalence is also there for flow (i.e. mac_min_pri, mci_pri, mci_max_pri etc.)? or flow just get one priority. or when you add flow to a NIC what are priority relations on these? (I think in another context, I am asking the same question again :-() -Deepti>> -Deepti >> >>> _______________________________________________________________________________ >>> References: >>> >>> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>> >>> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >> >> > > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
Hi, Deepti: On Mon, 26 Nov 2007, deepti dhokte wrote:> venugopal iyer wrote: >> >>> what could be the deciding factor to calculate min/max ranges for given >>> mac_impl? >>> >> >> You mean for a MAC client? The priority for the MAC client determines the >> range, note that the range is not for that MAC client itself, but any >> flows etc. create on top of the MAC client. >> > I see, now, I follow that part better . >> >>> can you have this min,max ranges static based on type whether NIC, VNIC >>> or FLOW? >>> >> >> I don''t follow. The range is for a MAC client to be used for any flows >> created on top of it. i.e we could create a VNIC with a medium priority, >> which will determine the range for the MAC client corresponding to the >> VNIC. Now, when we create a flow with a high priority on top of the VNIC, >> the high is with the range of that MAC client. >> > ok, > say I have mac_client, would it have priority say x,y and min,max and z > as assigned/used.?Yes, it would have a priority and a range (min and max) when created. It could be the default if not specified at creation time, or the specified one.> > now I add flow on top of it. > what would be the min,max and assigned priorities for it?When you add a flow, you don''t create a MAC client, so it won''t have a range, only a priority based on its configured priority level (low, high, medium) and the range of its underlying MAC client.> since th "high for the flow being added to vnic should be with "in?" the > range of the mac client?the high is w.r.t to the range of the VNIC/MAC client, so it will be within the range.> also, wonder if flow a mac_client as well?No.> > (here the priority on flow could it be the one and only priority and > that''s what is its max > priority?) >>> so threads processing packets on flows will have priorities lower than >>> thread processing packets >>> deposited on vnics? >>> >> >> A flow with a high priority will have the same priority value as the MAC >> client, > do you mean a flow created on vnic,aggr etc ?Yes, it could be a VNIC or a NIC, i.e any MAC client.> > so will there be any min, max and medium priority ranges for flows and > will be derived > from mac_client and will be exact same as mac_client?No, there won''t be a range, just the priority based on the range of the underlying MAC client (and its configured level - high, low, medium).> > or flows on top of mac_clients get only one static priority thruout its > lifetime?It gets a static priority, but it can be modified using flowadm modify-flow.>> while one with low/medium will have lower. >> >> >>>> If we create a virtual link, say a VNIC, with a specific priority, say high, >>>> mac_client_open() will initialize the values for the VNIC''s MAC client with >>>> the default priority. >>>> >>> I think we would dis-honor user''s request of a high priority in that case? >>> I may be wrong. >>> >> >> mac_client_open() will always start with a default priority. For a VNIC >> with a high priority, mac_resource_ctl_set() (part of the same >> dladm create-vnic call) will set it appropriately, so we don''t dishonor >> the request. >> >> > when you set default priority you set default values for > mci_pri, mci_max_pri, mci_min_pri or you set default priority for just > mci_pri? > I thought all 3 could be set.All 3 will be set with a default. All 3 will be changed if a priority is specified.> > is mci_pri a priority being currently used priority of the thread > servicing flow (SRS based)mci_pri is for a MAC client.> or vnic (mac_impl based). ?Yes.> >>> if it is not in the range, do we error out? >>> what should be the default priority in this case? >>> >> >> Let me know if these are still unanswered. >> >> > I don''t understand it. sorry. > > say you create vnic, then add flow ? > - how min, max and mci_pri be set for vnicIf it is not specified, it defaults to medium.> and does the same equivalence is also there for flow (i.e. mac_min_pri, > mci_pri, mci_max_pri etc.)? > or flow just get one priority.Just one priority.> > or when you add flow to a NIC what are priority relations on these?The NIC has a MAC client associated, so it is not different from a flow on a VNIC. -venu> (I think in another context, I am asking the same question again :-()Let me know if you still have questions, -venu> -Deepti >>> -Deepti >>> >>>> _______________________________________________________________________________ >>>> References: >>>> >>>> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >>>> >>>> _______________________________________________ >>>> crossbow-discuss mailing list >>>> crossbow-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>> >>>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>> >>> >> >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >> > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >
venugopal iyer wrote:> > Hi, Deepti: > > On Mon, 26 Nov 2007, deepti dhokte wrote: > >> venugopal iyer wrote: >>> >>>> what could be the deciding factor to calculate min/max ranges for >>>> given >>>> mac_impl? >>>> >>> >>> You mean for a MAC client? The priority for the MAC client >>> determines the >>> range, note that the range is not for that MAC client itself, but any >>> flows etc. create on top of the MAC client. >>> >> I see, now, I follow that part better . >>> >>>> can you have this min,max ranges static based on type whether NIC, >>>> VNIC >>>> or FLOW? >>>> >>> >>> I don''t follow. The range is for a MAC client to be used for any flows >>> created on top of it. i.e we could create a VNIC with a medium >>> priority, >>> which will determine the range for the MAC client corresponding to the >>> VNIC. Now, when we create a flow with a high priority on top of the >>> VNIC, >>> the high is with the range of that MAC client. >>> >> ok, >> say I have mac_client, would it have priority say x,y and min,max and z >> as assigned/used.? > > Yes, it would have a priority and a range (min and max) when created. It > could be the default if not specified at creation time, or the > specified one. > >> >> now I add flow on top of it. >> what would be the min,max and assigned priorities for it? > > When you add a flow, you don''t create a MAC client, so it won''t have a > range, only a priority based on its configured priority level (low, high, > medium) and the range of its underlying MAC client. >ok. so say I have a vnic that has> pri_t mci_pri =86 > pri_t mci_max_pri=73 > pri_t mci_min_pri=86 >when I add a flow on top of, how you will decide whether flows priority should be low(0), high(2) or medium (1)? why notions are different in case of flow vs. vnic?> >> >> so will there be any min, max and medium priority ranges for flows and >> will be derived >> from mac_client and will be exact same as mac_client? > > No, there won''t be a range, just the priority based on the range of > the underlying MAC client (and its configured level - high, low, medium). >, I have same question as above.>> >> or flows on top of mac_clients get only one static priority thruout its >> lifetime? > > It gets a static priority, but it can be modified using flowadm > modify-flow.if that priority isn''t within the range of mac_client''s mci_mri_max, we error out?> >> >> is mci_pri a priority being currently used priority of the thread >> servicing flow (SRS based) > > mci_pri is for a MAC client.what is the corresponding member-field/s for flows. do you save it in flow_entry_t ?>>>> if it is not in the range, do we error out? >>>> what should be the default priority in this case? >>>> >>> >>> Let me know if these are still unanswered. >>> >>> >> I don''t understand it. sorry. >> >> say you create vnic, then add flow ? >> - how min, max and mci_pri be set for vnic > > > If it is not specified, it defaults to medium.what is that value? how do you decide it is a medium priority?> >> and does the same equivalence is also there for flow (i.e. mac_min_pri, >> mci_pri, mci_max_pri etc.)? >> or flow just get one priority. > > Just one priority.and do you set it in flow_entry_t? (If this doc. meant to be only for mac_impl_t based. mac_impl/link''s priorities) if so, you can defer explaining it, I am just curious.) -Deepti> > > Let me know if you still have questions, > > -venu > >> -Deepti >>>> -Deepti >>>> >>>>> _______________________________________________________________________________ >>>>> >>>>> References: >>>>> >>>>> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >>>>> >>>>> _______________________________________________ >>>>> crossbow-discuss mailing list >>>>> crossbow-discuss at opensolaris.org >>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>> >>>>> >>>> _______________________________________________ >>>> crossbow-discuss mailing list >>>> crossbow-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>> >>>> >>> >>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>> >> >> _______________________________________________ >> crossbow-discuss mailing list >> crossbow-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>
Deepti and I resolved this offline, but just for the record, comments inline. thanks, -venu On Mon, 26 Nov 2007, deepti dhokte wrote:> venugopal iyer wrote: >> >> Hi, Deepti: >> >> On Mon, 26 Nov 2007, deepti dhokte wrote: >> >>> venugopal iyer wrote: >>>> >>>>> what could be the deciding factor to calculate min/max ranges for >>>>> given >>>>> mac_impl? >>>>> >>>> >>>> You mean for a MAC client? The priority for the MAC client >>>> determines the >>>> range, note that the range is not for that MAC client itself, but any >>>> flows etc. create on top of the MAC client. >>>> >>> I see, now, I follow that part better . >>>> >>>>> can you have this min,max ranges static based on type whether NIC, >>>>> VNIC >>>>> or FLOW? >>>>> >>>> >>>> I don''t follow. The range is for a MAC client to be used for any flows >>>> created on top of it. i.e we could create a VNIC with a medium >>>> priority, >>>> which will determine the range for the MAC client corresponding to the >>>> VNIC. Now, when we create a flow with a high priority on top of the >>>> VNIC, >>>> the high is with the range of that MAC client. >>>> >>> ok, >>> say I have mac_client, would it have priority say x,y and min,max and z >>> as assigned/used.? >> >> Yes, it would have a priority and a range (min and max) when created. It >> could be the default if not specified at creation time, or the >> specified one. >> >>> >>> now I add flow on top of it. >>> what would be the min,max and assigned priorities for it? >> >> When you add a flow, you don''t create a MAC client, so it won''t have a >> range, only a priority based on its configured priority level (low, high, >> medium) and the range of its underlying MAC client. >> > ok. so say I have a vnic that has > >> pri_t mci_pri =86 >> pri_t mci_max_pri=73 >> pri_t mci_min_pri=86 >> > > when I add a flow on top of, > how you will decide whether flows priority should be > low(0), high(2) or medium (1)? >The default will be medium, so if you don''t specify a priority using flowadm, then the priority will be set to medium, i.e 81. If you specify anything using flowadm (i.e. low, high, medium), it will be either 73 (low), 81 (medium) or 86 (high).> why notions are different in case of flow vs. vnic?A flow is created on top of a MAC client (NIC, VNIC etc.).> >> >>> >>> so will there be any min, max and medium priority ranges for flows and >>> will be derived >>> from mac_client and will be exact same as mac_client? >> >> No, there won''t be a range, just the priority based on the range of >> the underlying MAC client (and its configured level - high, low, medium). >> > , I have same question as above. >>> >>> or flows on top of mac_clients get only one static priority thruout its >>> lifetime? >> >> It gets a static priority, but it can be modified using flowadm >> modify-flow. > if that priority isn''t within the range of mac_client''s mci_mri_max, we > error out?it can''t be out of the range, as the priority levels are w.r.t to the underlying range.>> >>> >>> is mci_pri a priority being currently used priority of the thread >>> servicing flow (SRS based) >> >> mci_pri is for a MAC client. > what is the corresponding member-field/s for flows. do you save it in > flow_entry_t ? >>>>> if it is not in the range, do we error out? >>>>> what should be the default priority in this case? >>>>> >>>> >>>> Let me know if these are still unanswered. >>>> >>>> >>> I don''t understand it. sorry. >>> >>> say you create vnic, then add flow ? >>> - how min, max and mci_pri be set for vnic >> >> >> If it is not specified, it defaults to medium. > what is that value? how do you decide it is a medium priority?we have medium as the default.>> >>> and does the same equivalence is also there for flow (i.e. mac_min_pri, >>> mci_pri, mci_max_pri etc.)? >>> or flow just get one priority. >> >> Just one priority. > and do you set it in flow_entry_t? > (If this doc. meant to be only for mac_impl_t based. mac_impl/link''s > priorities) > if so, you can defer explaining it, I am just curious.)Yes, this is text is specifically for Link. The flow''s priority would probably be set in some member in flow_entry_t (TBD).> -Deepti >> >> >> Let me know if you still have questions, >> >> -venu >> >>> -Deepti >>>>> -Deepti >>>>> >>>>>> _______________________________________________________________________________ >>>>>> >>>>>> References: >>>>>> >>>>>> [1]http://opensolaris.org/os/project/crossbow/Docs/crossbow-virt.pdf >>>>>> >>>>>> _______________________________________________ >>>>>> crossbow-discuss mailing list >>>>>> crossbow-discuss at opensolaris.org >>>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> crossbow-discuss mailing list >>>>> crossbow-discuss at opensolaris.org >>>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> crossbow-discuss mailing list >>>> crossbow-discuss at opensolaris.org >>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>>> >>> >>> _______________________________________________ >>> crossbow-discuss mailing list >>> crossbow-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >>> > > _______________________________________________ > crossbow-discuss mailing list > crossbow-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss >