crossbow discuss - Jan 2008 - Crossbow and HW-assisted virtualization

In what way does Crossbow make use of functionalities of PCI-SIG IOV-capable
NICs that export multiple virtual NICs to the OS? Is this already supported in
the current Crossbow implementation and if yes, how much performance impact does
it have to use IOV-capable NICs with Crossbow?

Thanks,
Anna

Hi Anna,
Current Crossbow architecture doesn''t make use of PCI-SIG IOV 
functionalities. In fact, Crossbow can virtualize any NIC even the 
hardware doesn''t support any PCI-SIG IOV technologies.

Crossbow framework manage all hardware resources exposed by driver and 
hardware so that IOV-capable NICs can be handled well if the driver can 
configure the device properly.

We are considering to have explicit interfaces for IOV functionalities 
in the future.

Thanks,

Roamer

Fischer, Anna wrote:
> In what way does Crossbow make use of functionalities of PCI-SIG
IOV-capable NICs that export multiple virtual NICs to the OS? Is this already
supported in the current Crossbow implementation and if yes, how much
performance impact does it have to use IOV-capable NICs with Crossbow?
> 
> Thanks,
> Anna
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
-- 

# telnet (650)-786-6759 (x86759)
Connected to Solaris.Sun.COM.
login: Lu, Yunsong
Last login: January 2, 2007 from beyond.sfbay
Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
[Roamer at Solaris Networking]# cd ..

A current trend in virtualization seems to be to move more functionality into
hardware and also to remove the hypervisor (like e.g. Xen) from the main I/O
data path. This means that VMs can directly transmit on the physical hardware
(that might be shared with between multiple VMs, e.g. through PCI-SIG IOV
capabilities) which has a big positive performance impact. Do you support this
kind of direct I/O with Crossbow? I can imagine that it''s difficult to
do rate (bandwidth) controlling with such a design, and this seems to be one of
the most important features of Crossbow.

Also, can Crossbow make use of new Intel VT-d features? If not, is this
something you look at for future work?

Thanks,
Anna
> -----Original Message-----
> From: Roamer.Lu at sun.com [mailto:Roamer.Lu at sun.com] On Behalf
> Of Yunsong (Roamer) Lu
> Sent: 09 January 2008 15:22
> To: Fischer, Anna
> Cc: crossbow-discuss at opensolaris.org
> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
> virtualization
>
> Hi Anna,
> Current Crossbow architecture doesn''t make use of PCI-SIG IOV
> functionalities. In fact, Crossbow can virtualize any NIC
> even the hardware doesn''t support any PCI-SIG IOV technologies.
>
> Crossbow framework manage all hardware resources exposed by
> driver and hardware so that IOV-capable NICs can be handled
> well if the driver can configure the device properly.
>
> We are considering to have explicit interfaces for IOV
> functionalities in the future.
>
> Thanks,
>
> Roamer
>
> Fischer, Anna wrote:
> > In what way does Crossbow make use of functionalities of
> PCI-SIG IOV-capable NICs that export multiple virtual NICs to
> the OS? Is this already supported in the current Crossbow
> implementation and if yes, how much performance impact does
> it have to use IOV-capable NICs with Crossbow?
> >
> > Thanks,
> > Anna
> > _______________________________________________
> > crossbow-discuss mailing list
> > crossbow-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>
> --
>
> # telnet (650)-786-6759 (x86759)
> Connected to Solaris.Sun.COM.
> login: Lu, Yunsong
> Last login: January 2, 2007 from beyond.sfbay
> Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
> [Roamer at Solaris Networking]# cd ..
>

Fischer, Anna wrote:
> A current trend in virtualization seems to be to move more functionality
into hardware and also to remove the hypervisor (like e.g. Xen) from the main
I/O data path. This means that VMs can directly transmit on the physical
hardware (that might be shared with between multiple VMs, e.g. through PCI-SIG
IOV capabilities) which has a big positive performance impact. Do you support
this kind of direct I/O with Crossbow? I can imagine that it''s
difficult to do rate (bandwidth) controlling with such a design, and this seems
to be one of the most important features of Crossbow.
>   
There are two kinds of rate limiting.

The first is inbound (RX) traffic.   The second is outbound (TX) traffic.

For inbound, you have no control over the bits on the wire.  So you can 
only ensure that one domain (or ring) cannot fill up and thereby cause 
another one to fill up.  PCI-IOV is complementary to this.  (A VM may 
have its rings fill up because it can''t process them quickly enough... 
perhaps due to CPU resource considerations ... but it can''t create a
DoS
for other VMs with different rings.)

For outbound, we have a different problem.  It sort of depends on the 
hardware.  One can imagine that you can assign rings to VMs, but the 
question is whether they have a priority associated with them or not.  
If the rings are all fair-share, then rate limiting is not necessary.  
(One assumes that then you want to allow a VM to use all resource 
available, but not more than its fair share if other VMs are 
competing.)   But if you assign a high-priority (high QoS) ring to a VM, 
then that VM could starve other VMs.  So I guess it depends on the hardware.

As far as limiting the bandwidth used by a particular ethernet port 
overall, if you want that level of control, then you probably need to 
use Crossbow soft rings instead of hardware rings.  I don''t see any 
other solution.  But most people only want to use resource control to 
divvy up the bandwidth between different VMs or processes, and would 
like to use the maximum bandwidth to the box if possible.  So PCI IOV is 
useful there.

Btw, there are other solutions besides PCI IOV.  Any NIC which has 
multiple rings and separate rx/tx interrupts can have the same feature 
with Crossbow.  The nxge hardware is an example of this.  I believe the 
Intel Zoar NIC is also multiple-ring capable, though I believe that the 
driver that just got integrated does not have support it yet.
> Also, can Crossbow make use of new Intel VT-d features? If not, is this
something you look at for future work?
>   
I believe we (Sun) are planning on enhancements to support VT-d.  Much 
of what we want to do with VT-d is not just virtualization.  VT-d can 
also be used to provide fault isolation and reduce or eliminate the need 
for DMA bounce buffers. 

    -- Garrett
> Thanks,
> Anna
>
>   
>> -----Original Message-----
>> From: Roamer.Lu at sun.com [mailto:Roamer.Lu at sun.com] On Behalf
>> Of Yunsong (Roamer) Lu
>> Sent: 09 January 2008 15:22
>> To: Fischer, Anna
>> Cc: crossbow-discuss at opensolaris.org
>> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
>> virtualization
>>
>> Hi Anna,
>> Current Crossbow architecture doesn''t make use of PCI-SIG IOV
>> functionalities. In fact, Crossbow can virtualize any NIC
>> even the hardware doesn''t support any PCI-SIG IOV
technologies.
>>
>> Crossbow framework manage all hardware resources exposed by
>> driver and hardware so that IOV-capable NICs can be handled
>> well if the driver can configure the device properly.
>>
>> We are considering to have explicit interfaces for IOV
>> functionalities in the future.
>>
>> Thanks,
>>
>> Roamer
>>
>> Fischer, Anna wrote:
>>     
>>> In what way does Crossbow make use of functionalities of
>>>       
>> PCI-SIG IOV-capable NICs that export multiple virtual NICs to
>> the OS? Is this already supported in the current Crossbow
>> implementation and if yes, how much performance impact does
>> it have to use IOV-capable NICs with Crossbow?
>>     
>>> Thanks,
>>> Anna
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>>>       
>> --
>>
>> # telnet (650)-786-6759 (x86759)
>> Connected to Solaris.Sun.COM.
>> login: Lu, Yunsong
>> Last login: January 2, 2007 from beyond.sfbay
>> Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
>> [Roamer at Solaris Networking]# cd ..
>>
>>     
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>

Fischer, Anna wrote:
> A current trend in virtualization seems to be to move more functionality
into hardware and also to remove the hypervisor (like e.g. Xen) from the main
I/O data path. This means that VMs can directly transmit on the physical
hardware (that might be shared with between multiple VMs, e.g. through PCI-SIG
IOV capabilities) which has a big positive performance impact. Do you support
this kind of direct I/O with Crossbow?
Not yet, though Crossbow moves one step towards that direction.
In Dom0, we offer the ability to assign hw NIC Rx and Tx rings to VNICs,
which are in turn associated with a back-end driver instance that talks 
with its
vis-a-vis front end driver running in the guest VM.
The step to finish is to define a programming interface that gives a 
safe and controlled access to the front end driver
to directly interact with the rx and tx rings (submit packets to ''n
pick
up packets from Rx and Tx rings, control
interrupts at the ring level, etc.)
>  I can imagine that it''s difficult to do rate (bandwidth)
controlling with such a design, and this seems to be one of the most important
features of Crossbow.
>   
difficult but feasable with some assist from the device driver. We could 
adjust the allocated rings capacity (the number of descriptors available 
for rx and tx operations)
based on the share of bandwidth allowed for the client that "owns" the
rings.
> Also, can Crossbow make use of new Intel VT-d features? If not, is this
something you look at for future work?
>   
We''re looking closely at the features involved in Intel''s
VT-d. Brutus
(Cc''ed) could say more.

    Kais.
> Thanks,
> Anna
>
>   
>> -----Original Message-----
>> From: Roamer.Lu at sun.com [mailto:Roamer.Lu at sun.com] On Behalf
>> Of Yunsong (Roamer) Lu
>> Sent: 09 January 2008 15:22
>> To: Fischer, Anna
>> Cc: crossbow-discuss at opensolaris.org
>> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
>> virtualization
>>
>> Hi Anna,
>> Current Crossbow architecture doesn''t make use of PCI-SIG IOV
>> functionalities. In fact, Crossbow can virtualize any NIC
>> even the hardware doesn''t support any PCI-SIG IOV
technologies.
>>
>> Crossbow framework manage all hardware resources exposed by
>> driver and hardware so that IOV-capable NICs can be handled
>> well if the driver can configure the device properly.
>>
>> We are considering to have explicit interfaces for IOV
>> functionalities in the future.
>>
>> Thanks,
>>
>> Roamer
>>
>> Fischer, Anna wrote:
>>     
>>> In what way does Crossbow make use of functionalities of
>>>       
>> PCI-SIG IOV-capable NICs that export multiple virtual NICs to
>> the OS? Is this already supported in the current Crossbow
>> implementation and if yes, how much performance impact does
>> it have to use IOV-capable NICs with Crossbow?
>>     
>>> Thanks,
>>> Anna

> Not yet, though Crossbow moves one step towards that direction.
> In Dom0, we offer the ability to assign hw NIC Rx and Tx
> rings to VNICs, which are in turn associated with a back-end
> driver instance that talks with its vis-a-vis front end
> driver running in the guest VM.
> The step to finish is to define a programming interface that
> gives a safe and controlled access to the front end driver to
> directly interact with the rx and tx rings (submit packets to
> ''n pick up packets from Rx and Tx rings, control interrupts
> at the ring level, etc.)
So in your design you always have a VNIC in between? Are you working on
enhancing front end drivers for direct HW/soft ring access for the Xen
hypervisor? I have seen that Xen is stated very often in your support documents.
How much of the rate (bandwidth) controlling capabilities work for Xen? I guess
all of it as long as you always have a Crossbow VNIC in between?

Thanks,
Anna

> As far as limiting the bandwidth used by a particular
> ethernet port overall, if you want that level of control,
> then you probably need to use Crossbow soft rings instead of
> hardware rings.  I don''t see any other solution.  But most
> people only want to use resource control to divvy up the
> bandwidth between different VMs or processes, and would like
> to use the maximum bandwidth to the box if possible.  So PCI
> IOV is useful there.
Can you give a rough estimate about how much performance impact it has to use
Crossbow soft rings instead of hardware rings (when using bandwidth limiting)?

> Btw, there are other solutions besides PCI IOV.  Any NIC
> which has multiple rings and separate rx/tx interrupts can
> have the same feature with Crossbow.  The nxge hardware is an
> example of this.  I believe the Intel Zoar NIC is also
> multiple-ring capable, though I believe that the driver that
> just got integrated does not have support it yet.
Have you had a look at the Intel VMDq feature, too? Is this something that
Crossbow can benefit from?

Thanks,
Anna

Anna,

Yes, you are correct that getting Xen dom0 out of the picture for
domU (guest) to outside communication is good. Crossbow is providing
the framework to do just that. Their are various terms used like
Hybrid I/O (for SPARC implementation) and it can be done with
existing NICs like Sun''s Neptune and Neterion''s 10Gb NICs but
primarily on SPARC. With IOV, we will get the needed IOMMU
protection to achieve this with any IOV capable NIC. And yes, if
you directly map the DMA channel and associated Rx ring into a domU,
the bandwidth limiting/guarantees, priority and CPU association
becomes moot because you are dependent on "dynamic polling" from
MAC layer for some of these features. The moment the MAC layer is
part of domU (or guest OS) and controlled by a hostile entity,
theoretically they can circumvent the MAC layer running in domU
(which they control).

So we build more smarts into the system where the highest priority
VNICs which have unrestricted access are the ones that get to
use these feature while others who have a B/W limit/guarantee
specified get to go through the dom0 where we don''t have to
worry about it.

As part of the MAC layer design, we implement a
pseudo H/W layer which has its S/W classification (more powerful
than the H/W) and S/W rings (called soft rings) etc. They can
have H/W resources associated with them (like Rx/Tx rings or Intel''s
VMDq and H/W classification) in which case the pseudo H/W layer is
pass through but it allows us to move H/W resources in and out of
any VNICs (or flows) as and when we want it. Also, this allows us
to provide the uniform capabilities from the stack without
depending on the type of NIC and its capabilities. You can create
1000s of VNICs and Flows with their associated properties (like
B/W limits, priorities, degree of fanout, CPU bindings, etc) and
the most important ones get to use whatever H/W support is available.

Cheers,
Sunay

Fischer, Anna wrote:
> A current trend in virtualization seems to be to move more functionality
into
> hardware and also to remove the hypervisor (like e.g. Xen) from the main
I/O
 > data path. This means that VMs can directly transmit on the physical
 > hardware (that might be shared with between multiple VMs, e.g. 
through PCI-SIG
 > IOV capabilities) which has a big positive performance impact. Do
you
> support  this kind of direct I/O with Crossbow? I can imagine that
it''s
 > difficult to do rate (bandwidth) controlling with such a design,
and
> this seems to be one of the most important features of Crossbow.
> 
> Also, can Crossbow make use of new Intel VT-d features? If not, is 
> this something you look at for future work?
> 
> Thanks,
> Anna
> 
>> -----Original Message-----
>> From: Roamer.Lu at sun.com [mailto:Roamer.Lu at sun.com] On Behalf
>> Of Yunsong (Roamer) Lu
>> Sent: 09 January 2008 15:22
>> To: Fischer, Anna
>> Cc: crossbow-discuss at opensolaris.org
>> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
>> virtualization
>>
>> Hi Anna,
>> Current Crossbow architecture doesn''t make use of PCI-SIG IOV
>> functionalities. In fact, Crossbow can virtualize any NIC
>> even the hardware doesn''t support any PCI-SIG IOV
technologies.
>>
>> Crossbow framework manage all hardware resources exposed by
>> driver and hardware so that IOV-capable NICs can be handled
>> well if the driver can configure the device properly.
>>
>> We are considering to have explicit interfaces for IOV
>> functionalities in the future.
>>
>> Thanks,
>>
>> Roamer
>>
>> Fischer, Anna wrote:
>>> In what way does Crossbow make use of functionalities of
>> PCI-SIG IOV-capable NICs that export multiple virtual NICs to
>> the OS? Is this already supported in the current Crossbow
>> implementation and if yes, how much performance impact does
>> it have to use IOV-capable NICs with Crossbow?
>>> Thanks,
>>> Anna
>>> _______________________________________________
>>> crossbow-discuss mailing list
>>> crossbow-discuss at opensolaris.org
>>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>> --
>>
>> # telnet (650)-786-6759 (x86759)
>> Connected to Solaris.Sun.COM.
>> login: Lu, Yunsong
>> Last login: January 2, 2007 from beyond.sfbay
>> Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
>> [Roamer at Solaris Networking]# cd ..
>>
> _______________________________________________
> crossbow-discuss mailing list
> crossbow-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Fischer, Anna wrote:
>> Not yet, though Crossbow moves one step towards that direction.
>> In Dom0, we offer the ability to assign hw NIC Rx and Tx
>> rings to VNICs, which are in turn associated with a back-end
>> driver instance that talks with its vis-a-vis front end
>> driver running in the guest VM.
>> The step to finish is to define a programming interface that
>> gives a safe and controlled access to the front end driver to
>> directly interact with the rx and tx rings (submit packets to
>> ''n pick up packets from Rx and Tx rings, control interrupts
>> at the ring level, etc.)
> 
> So in your design you always have a VNIC in between? Are you 
 > working on enhancing front end drivers for direct HW/soft
 > ring access for the Xen hypervisor? I have seen that Xen
 > is stated very often in your support documents. How much
 > of the rate (bandwidth) controlling capabilities work for
 > Xen? I guess all of it as long as you always have a
 > Crossbow VNIC in between?

Yes, that is correct. The VNICs don''t have any overhead by
themselves. We can run them with Containers etc with zero
performance penalty. With Xen, the penalty comes from going
through dom0 as I said earlier we can pick and choose when
we allow the dom0 to be bypassed on a per domU basis. So
if there is no B/W restriction and we have H/W resources
available we can choose to go directly to domU and when
dom0 specifies a B/W limit etc, we can just make the data
flow go via dom0 again without any interruption.

This is designed in although not implemented yet. The
current bit always go through dom0 and have a VNIC in
between which allows is to provide all the resource
control capabilities.

cheers,
Sunay

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Fischer, Anna wrote:
>> As far as limiting the bandwidth used by a particular
>> ethernet port overall, if you want that level of control,
>> then you probably need to use Crossbow soft rings instead of
>> hardware rings.  I don''t see any other solution.  But most
>> people only want to use resource control to divvy up the
>> bandwidth between different VMs or processes, and would like
>> to use the maximum bandwidth to the box if possible.  So PCI
>> IOV is useful there.
> 
> Can you give a rough estimate about how much performance impact it has to
use Crossbow soft rings instead of hardware rings (when using bandwidth
limiting)?
> 
It depends on lot of things like size of packets etc but
our current measurements with iperf etc show that S/W
classification and soft rings cause 4-5% performance
regression compared to using Rx rings and H/W classification.
> 
>> Btw, there are other solutions besides PCI IOV.  Any NIC
>> which has multiple rings and separate rx/tx interrupts can
>> have the same feature with Crossbow.  The nxge hardware is an
>> example of this.  I believe the Intel Zoar NIC is also
>> multiple-ring capable, though I believe that the driver that
>> just got integrated does not have support it yet.
> 
> Have you had a look at the Intel VMDq feature, too? Is this something that
Crossbow can benefit from?
VMDq is just like Rx rings from what I understand and I think
it fits into Crossbow framework without much change needed.

Cheers,
Sunay

-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:     http://www.opensolaris.org/os/community/networking
Project Crossbow:       http://www.opensolaris.org/os/project/crossbow

Sunay Tripathi wrote:
> Fischer, Anna wrote:
>>> As far as limiting the bandwidth used by a particular
>>> ethernet port overall, if you want that level of control,
>>> then you probably need to use Crossbow soft rings instead of
>>> hardware rings.  I don''t see any other solution.  But most
>>> people only want to use resource control to divvy up the
>>> bandwidth between different VMs or processes, and would like
>>> to use the maximum bandwidth to the box if possible.  So PCI
>>> IOV is useful there.
>> Can you give a rough estimate about how much performance impact it has
to use Crossbow soft rings instead of hardware rings (when using bandwidth
limiting)?
>>
> 
> It depends on lot of things like size of packets etc but
> our current measurements with iperf etc show that S/W
> classification and soft rings cause 4-5% performance
> regression compared to using Rx rings and H/W classification.
> 
>>> Btw, there are other solutions besides PCI IOV.  Any NIC
>>> which has multiple rings and separate rx/tx interrupts can
>>> have the same feature with Crossbow.  The nxge hardware is an
>>> example of this.  I believe the Intel Zoar NIC is also
>>> multiple-ring capable, though I believe that the driver that
>>> just got integrated does not have support it yet.
>> Have you had a look at the Intel VMDq feature, too? Is this something
that Crossbow can benefit from?
> 
> VMDq is just like Rx rings from what I understand and I think
> it fits into Crossbow framework without much change needed.
Right, current Crossbow framework can handle Intel''s VMDq fairly well. 
You may want to refer to the crossbow design document for Hardware 
Resources Management. Ring grouping feature is the interface for VMDq.

Thanks,

Roamer


-- 

# telnet (650)-786-6759 (x86759)
Connected to Solaris.Sun.COM.
login: Lu, Yunsong
Last login: January 2, 2007 from beyond.sfbay
Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
[Roamer at Solaris Networking]# cd ..

How much of the rate controlling work and your interface design has been going
into the public Xen tree? I know that Xen has always been lacking proper network
resource control features (for non-Solaris systems at least). Is Crossbow fully
integrated with the Xen API and management tools?

Thanks,
Anna
> -----Original Message-----
> From: Sunay Tripathi [mailto:Sunay.Tripathi at Sun.Com]
> Sent: 10 January 2008 07:58
> To: Fischer, Anna
> Cc: crossbow-discuss at opensolaris.org
> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
> virtualization
>
> Anna,
>
> Yes, you are correct that getting Xen dom0 out of the picture
> for domU (guest) to outside communication is good. Crossbow
> is providing the framework to do just that. Their are various
> terms used like Hybrid I/O (for SPARC implementation) and it
> can be done with existing NICs like Sun''s Neptune and
> Neterion''s 10Gb NICs but primarily on SPARC. With IOV, we
> will get the needed IOMMU protection to achieve this with any
> IOV capable NIC. And yes, if you directly map the DMA channel
> and associated Rx ring into a domU, the bandwidth
> limiting/guarantees, priority and CPU association becomes
> moot because you are dependent on "dynamic polling" from MAC
> layer for some of these features. The moment the MAC layer is
> part of domU (or guest OS) and controlled by a hostile
> entity, theoretically they can circumvent the MAC layer
> running in domU (which they control).
>
> So we build more smarts into the system where the highest
> priority VNICs which have unrestricted access are the ones
> that get to use these feature while others who have a B/W
> limit/guarantee specified get to go through the dom0 where we
> don''t have to worry about it.
>
> As part of the MAC layer design, we implement a pseudo H/W
> layer which has its S/W classification (more powerful than
> the H/W) and S/W rings (called soft rings) etc. They can have
> H/W resources associated with them (like Rx/Tx rings or
> Intel''s VMDq and H/W classification) in which case the pseudo
> H/W layer is pass through but it allows us to move H/W
> resources in and out of any VNICs (or flows) as and when we
> want it. Also, this allows us to provide the uniform
> capabilities from the stack without depending on the type of
> NIC and its capabilities. You can create 1000s of VNICs and
> Flows with their associated properties (like B/W limits,
> priorities, degree of fanout, CPU bindings, etc) and the most
> important ones get to use whatever H/W support is available.
>
> Cheers,
> Sunay
>
> Fischer, Anna wrote:
> > A current trend in virtualization seems to be to move more
> > functionality into hardware and also to remove the hypervisor (like
> > e.g. Xen) from the main I/O
>  > data path. This means that VMs can directly transmit on
> the physical  > hardware (that might be shared with between
> multiple VMs, e.g.
> through PCI-SIG
>  > IOV capabilities) which has a big positive performance
> impact. Do you
> > support  this kind of direct I/O with Crossbow? I can imagine that
> > it''s
>  > difficult to do rate (bandwidth) controlling with such a
> design, and
> > this seems to be one of the most important features of Crossbow.
> >
> > Also, can Crossbow make use of new Intel VT-d features? If not, is
> > this something you look at for future work?
> >
> > Thanks,
> > Anna
> >
> >> -----Original Message-----
> >> From: Roamer.Lu at sun.com [mailto:Roamer.Lu at sun.com] On Behalf
Of
> >> Yunsong (Roamer) Lu
> >> Sent: 09 January 2008 15:22
> >> To: Fischer, Anna
> >> Cc: crossbow-discuss at opensolaris.org
> >> Subject: Re: [crossbow-discuss] Crossbow and HW-assisted
> >> virtualization
> >>
> >> Hi Anna,
> >> Current Crossbow architecture doesn''t make use of PCI-SIG
IOV
> >> functionalities. In fact, Crossbow can virtualize any NIC even the
> >> hardware doesn''t support any PCI-SIG IOV technologies.
> >>
> >> Crossbow framework manage all hardware resources exposed by driver
> >> and hardware so that IOV-capable NICs can be handled well if the
> >> driver can configure the device properly.
> >>
> >> We are considering to have explicit interfaces for IOV
> >> functionalities in the future.
> >>
> >> Thanks,
> >>
> >> Roamer
> >>
> >> Fischer, Anna wrote:
> >>> In what way does Crossbow make use of functionalities of
> >> PCI-SIG IOV-capable NICs that export multiple virtual NICs
> to the OS?
> >> Is this already supported in the current Crossbow
> implementation and
> >> if yes, how much performance impact does it have to use
> IOV-capable
> >> NICs with Crossbow?
> >>> Thanks,
> >>> Anna
> >>> _______________________________________________
> >>> crossbow-discuss mailing list
> >>> crossbow-discuss at opensolaris.org
> >>> http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
> >> --
> >>
> >> # telnet (650)-786-6759 (x86759)
> >> Connected to Solaris.Sun.COM.
> >> login: Lu, Yunsong
> >> Last login: January 2, 2007 from beyond.sfbay
> >> Yunsong.Lu at Sun.COM    v1.03    Since Mon Dec. 22, 2003
> >> [Roamer at Solaris Networking]# cd ..
> >>
> > _______________________________________________
> > crossbow-discuss mailing list
> > crossbow-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/crossbow-discuss
>
>
> --
> Sunay Tripathi
> Distinguished Engineer
> Solaris Core Operating System
> Sun MicroSystems Inc.
>
> Solaris Networking:
> http://www.opensolaris.org/os/community/networking
> Project Crossbow:       http://www.opensolaris.org/os/project/crossbow
>
>
>

* anna.fischer at hp.com [2008-01-10 10:57:50]
> How much of the rate controlling work and your interface design has
> been going into the public Xen tree?
The work is almost all inside Solaris (the Xen bits are largely
confined to control tool tweaks), so the interesting part will not
make it back into the Xen tree.

dme.

dme.
-- 
David Edmondson, Sun Microsystems, http://www.dme.org

> Yes, you are correct that getting Xen dom0 out of the picture
> for domU (guest) to outside communication is good. Crossbow
> is providing the framework to do just that. Their are various
> terms used like Hybrid I/O (for SPARC implementation) and it
> can be done with existing NICs like Sun''s Neptune and
> Neterion''s 10Gb NICs but primarily on SPARC. With IOV, we
> will get the needed IOMMU protection to achieve this with any
> IOV capable NIC. And yes, if you directly map the DMA channel
> and associated Rx ring into a domU, the bandwidth
> limiting/guarantees, priority and CPU association becomes
> moot because you are dependent on "dynamic polling" from MAC
> layer for some of these features. The moment the MAC layer is
> part of domU (or guest OS) and controlled by a hostile
> entity, theoretically they can circumvent the MAC layer
> running in domU (which they control).
So with IOV and direct I/O access from within a DomU rate controlling can only
be done in the actual H/W NIC. This means that you might loose a richer set of
classification features, but you could gain in network performance. Do you
collaborate with H/W NIC developers in this space, or provide some input in what
way NICs can be improved to better support rate controlling for virtualized
systems?

Cheers,
Anna

Fischer, Anna wrote:
>> Yes, you are correct that getting Xen dom0 out of the picture
>> for domU (guest) to outside communication is good. Crossbow
>> is providing the framework to do just that. Their are various
>> terms used like Hybrid I/O (for SPARC implementation) and it
>> can be done with existing NICs like Sun''s Neptune and
>> Neterion''s 10Gb NICs but primarily on SPARC. With IOV, we
>> will get the needed IOMMU protection to achieve this with any
>> IOV capable NIC. And yes, if you directly map the DMA channel
>> and associated Rx ring into a domU, the bandwidth
>> limiting/guarantees, priority and CPU association becomes
>> moot because you are dependent on "dynamic polling" from MAC
>> layer for some of these features. The moment the MAC layer is
>> part of domU (or guest OS) and controlled by a hostile
>> entity, theoretically they can circumvent the MAC layer
>> running in domU (which they control).
> 
> So with IOV and direct I/O access from within a 
 > DomU rate controlling can only be done in the actual
 > H/W NIC. This means that you might loose a richer
 > set of classification features, but you could gain
 > in network performance. Do you collaborate with H/W
 > NIC developers in this space, or provide some input
 > in what way NICs can be improved to better support
 > rate controlling for virtualized systems?
> 
Its more complicated than that. You really don''t want the
H/W to start queuing on Rx and Tx and dropping packets
is not such a great idea (TCP really hurts on dropped packet).
So for instance on Tx, when you start exceeding the allowed
rate, the Crossbow MAC layer propogates the flow control all the
way upto the higher level queues (like TCP and UDP Squeues)
which can block the application generating the traffic by
putting it to sleep or failing the write. Similarly on the
Rx side, you drop packets only as a last resort and you want
to keep some elasticity. On 10Gig NIC, this elasticity or
ability to queue some before you drop can start becoming
expensive. The key thing is that H/W should not be keeping
any state. Its very slippery slope.

BTW, would you mind telling us what you are trying to do and
what is your interest?

Cheers,
Sunay



-- 
Sunay Tripathi
Distinguished Engineer
Solaris Core Operating System
Sun MicroSystems Inc.

Solaris Networking:	http://www.opensolaris.org/os/community/networking
Project Crossbow:	http://www.opensolaris.org/os/project/crossbow

> >> Yes, you are correct that getting Xen dom0 out of the picture for
> >> domU (guest) to outside communication is good. Crossbow is
> providing
> >> the framework to do just that. Their are various terms used like
> >> Hybrid I/O (for SPARC implementation) and it can be done with
> >> existing NICs like Sun''s Neptune and Neterion''s
10Gb NICs but
> >> primarily on SPARC. With IOV, we will get the needed IOMMU
> protection
> >> to achieve this with any IOV capable NIC. And yes, if you directly
> >> map the DMA channel and associated Rx ring into a domU,
> the bandwidth
> >> limiting/guarantees, priority and CPU association becomes moot
> >> because you are dependent on "dynamic polling" from MAC
layer for
> >> some of these features. The moment the MAC layer is part
> of domU (or
> >> guest OS) and controlled by a hostile entity,
> theoretically they can
> >> circumvent the MAC layer running in domU (which they control).
> >
> > So with IOV and direct I/O access from within a
>  > DomU rate controlling can only be done in the actual  >
> H/W NIC. This means that you might loose a richer  > set of
> classification features, but you could gain  > in network
> performance. Do you collaborate with H/W  > NIC developers in
> this space, or provide some input  > in what way NICs can be
> improved to better support  > rate controlling for
> virtualized systems?
> >
>
> Its more complicated than that. You really don''t want the H/W
> to start queuing on Rx and Tx and dropping packets is not
> such a great idea (TCP really hurts on dropped packet).
> So for instance on Tx, when you start exceeding the allowed
> rate, the Crossbow MAC layer propogates the flow control all
> the way upto the higher level queues (like TCP and UDP
> Squeues) which can block the application generating the
> traffic by putting it to sleep or failing the write.
> Similarly on the Rx side, you drop packets only as a last
> resort and you want to keep some elasticity. On 10Gig NIC,
> this elasticity or ability to queue some before you drop can
> start becoming expensive. The key thing is that H/W should
> not be keeping any state. Its very slippery slope.
So that means you are doing L3/L4 classification even when you run on
virtualized systems? When running VMs, how do you know L3 information for VMs as
this is not in your control, but only in the hands of the VM user. It can change
at any time. Is L3/L4 classification for virtualized systems efficient, as you
would normally not inspect L3/L4 information within Dom0?

> BTW, would you mind telling us what you are trying to do and
> what is your interest?
Thanks very much for giving great feedback on all my points. I work in the area
of network virtualization, and I''m investigating how to do virtual
network resource control in a very efficient and flexible manner.

Cheers,
Anna