Xen devel - May 2007 - [PATCH 0/4] [Net] Support Xen accelerated network plugin modules

This is a repost of some earlier patches to the xen-devel mailing list,
with a number of changes thanks to some useful suggestions from others.
Apologies for the short delay in getting this next version ready.

I''ve also CC''d netdev@vger.kernel.org as some of the files
being patched
may be merged into upstream linux soon, and so folks there may have
opinions too.

This set of patches provides the hooks and support necessary for
accelerated network plugin modules to attach to Xen''s netback and
netfront.  These modules provide a fast path for network traffic where
there is hardware support available for the netfront driver to send and
receive packets directly to a NIC (such as those available from
Solarflare).

As there are currently no available plugins, I''ve attached a couple of
dummy ones to illustrate how the hooks could be used.  These are
incomplete (and clearly wouldn''t even compile) in that they only
include
code to show the interface between the accelerated module and
netfront/netback.  A lot of the comments hint at what code should go
where.  They don''t show any interface between the accelerated frontend
and accelerated backend, or hardware access, for example, as those would
both be specific to the implementation.  I hope they help illustrate
this, but if you have any questions I''m happy to provide more
information.

A brief overview of the operation of the plugins:  When the accelerated
modules are loaded, a VI is created by the accelerated backend to allow
the accelerated frontend to safely access portions of the NIC.  For RX,
when packets are received by the accelerated backend, it will examine
them and if appropriate insert filters into the NIC to deliver future
packets on that address directly to the accelerated frontend''s VI.  For
TX, netfront gives each accelerated frontend the option of sending each
packet, which it can accept (if it wants to send it directly to the
hardware) or decline (if it thinks this is more appropriate to send via
the normal network path).

We have tried to ensure that the hooks are hardware-agnostic, i.e. would
be relevant to hardware other than our own, without providing all
possible ways of doing each task (but if others need to extend it, that
would be welcomed).

We have found that using this approach to accelerating network traffic,
domU to domU connections (across the network) can achieve close to the
performance of dom0 to dom0 connections on a 10Gbps ethernet.  This is
roughly double the bandwidth seen with unmodified Xen. 

Kieran



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, May 18, 2007 at 02:15:48PM +0100, Kieran Mansley
wrote:
> 
> I''ve also CC''d netdev@vger.kernel.org as some of the
files being patched
> may be merged into upstream linux soon, and so folks there may have
> opinions too.
Jeff and Stephen, could you both take a look at these proposed patches
against the Xen netfront driver and see whether the hooks that they add
look OK from a Linux network driver point of view?
> This set of patches provides the hooks and support necessary for
> accelerated network plugin modules to attach to Xen''s netback and
> netfront.  These modules provide a fast path for network traffic where
> there is hardware support available for the netfront driver to send and
> receive packets directly to a NIC (such as those available from
> Solarflare).
The RX side looks reasonble.  For TX, what sort of conditions are you
likely to use to decide whether a packet gets fast path or not?

Also for TX, you''ll need a better way of managing queue
stopping/starting
since presumably you only want to wake the queue when both fast path and
slow path have free slots.
> We have found that using this approach to accelerating network traffic,
> domU to domU connections (across the network) can achieve close to the
> performance of dom0 to dom0 connections on a 10Gbps ethernet.  This is
> roughly double the bandwidth seen with unmodified Xen. 
That sounds good.  I wonder how much of this difference goes away if we
had LRO though.

Here''s an interesting experiment, what if you only used these hooks on
the receive side for your 10Gbps transfer? Of course, you need to make
sure that TSO is working properly on the sending side.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/18/07, Kieran Mansley <kmansley@solarflare.com>
wrote:
> This is a repost of some earlier patches to the xen-devel mailing list,
> with a number of changes thanks to some useful suggestions from others.
> Apologies for the short delay in getting this next version ready.
>
> I''ve also CC''d netdev@vger.kernel.org as some of the
files being patched
> may be merged into upstream linux soon, and so folks there may have
> opinions too.
>
> This set of patches provides the hooks and support necessary for
> accelerated network plugin modules to attach to Xen''s netback and
> netfront.  These modules provide a fast path for network traffic where
> there is hardware support available for the netfront driver to send and
> receive packets directly to a NIC (such as those available from
> Solarflare).
>
Hi Kieran,

This is interesting. Just a question: is this possible for more than 1
domain to access directly to NIC, or at a time only 1 domain can do
that?

Thanks,
Jun

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2007-05-21 at 18:14 +1000, Herbert Xu wrote:
> On Fri, May 18, 2007 at 02:15:48PM +0100, Kieran Mansley wrote:
> > This set of patches provides the hooks and support necessary for
> > accelerated network plugin modules to attach to Xen''s netback
and
> > netfront.  These modules provide a fast path for network traffic where
> > there is hardware support available for the netfront driver to send
and
> > receive packets directly to a NIC (such as those available from
> > Solarflare).
> 
> The RX side looks reasonble.  For TX, what sort of conditions are you
> likely to use to decide whether a packet gets fast path or not?
For us, at the moment, we TX-fast-path packets to MAC addresses that we
would also RX-fast-path from.  i.e. Those for which we have a filter in
the NIC for.  This boils down to a hash table lookup on the MAC
addresses.  We do a couple of other checks too (make sure it''s IPV4,
TCP
or UDP, unicast) to keep things simple.
> Also for TX, you''ll need a better way of managing queue
stopping/starting
> since presumably you only want to wake the queue when both fast path and
> slow path have free slots.
Hmm, good point.  At the moment the queue is stopped if either fast or
slow path is busy, and restarted if either fast or slow path becomes
not-busy.  I suppose this would work OK if we could guarantee that the
slow path wouldn''t restart the queue that had been stopped by the fast
path, and vice versa, but at the moment that''s not the case, so
you''re
right it needs looking at.

Kieran


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2007-05-21 at 19:02 +0900, Jun Koi wrote:
> On 5/18/07, Kieran Mansley <kmansley@solarflare.com> wrote:
> > This set of patches provides the hooks and support necessary for
> > accelerated network plugin modules to attach to Xen''s netback
and
> > netfront.  These modules provide a fast path for network traffic where
> > there is hardware support available for the netfront driver to send
and
> > receive packets directly to a NIC (such as those available from
> > Solarflare).
> >
> 
> Hi Kieran,
> 
> This is interesting. Just a question: is this possible for more than 1
> domain to access directly to NIC, or at a time only 1 domain can do
> that?
Many domains can directly access the NIC at the same time.  The number
of virtual interfaces that can be created would depend on what NIC
hardware you''re using.  

Kieran


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel