Xen users - Jul 2009 - Quagga on Xen - Latency / Bandwidth?

I was wondering if anyone is running Quagga on Xen? What is
throughput/latency like?

Was looking for about a gig of mixed packet size BGP throughput.
Hardware is Opteron 4 Way Quad cores, was thinking paravirt with a
couple of dedicated cores and maybe the addition of Solarflare 10GB
Direct IO NICs might give the required performance level but I am
unsure.  The other option is multi-queue Intel or Mellanox NICs and an
upgrade for Centos Kernel and Xen.

The SAQ Group

Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ
SAQ is the trading name of SEMTEC Limited. Registered in England & Wales
Company Number: 06481952

http://www.saqnet.co.uk AS29219

SAQ Group Delivers high quality, honestly priced communication and I.T. services
to UK Business.

Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit :
Backups : Managed Networks : Remote Support.

ISPA Member


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Jul 29, 2009, at 4:04 AM, Robert Dunkley wrote:
> I was wondering if anyone is running Quagga on Xen? What is
> throughput/latency like?
>
> Was looking for about a gig of mixed packet size BGP throughput.
Presumably you mean internet traffic.

On bare metal (no Xen), that will work fine for mixed Internet  
traffic, but you need to think about corner cases. What do you expect  
to have happen when you get DDoSed with minimum-size packets? I don''t  
think even the multiqueue GbE cards will let you handle that, but I  
have not tested that, and I''d love to be proven wrong.

I''m also under the impression that Linux 2.6.30 kernels have some  
significant patches to make (better?) use of multiqueue cards but I  
don''t remember the details. Maybe 2.6.30 can distribute ksoftirqd load
over multiple cores? If so that''s a big win, as that was the killer  
with large PPS when we tested on somewhat less powerful hardware with  
an older kernel, and you can probably handle a full gbps of min-sized  
packets if you give it a number of cores (and you have enough queues).

Another question to think about (sorry, no answers here, just  
questions): If you do run under Xen, is there interrupt load in both  
the dom0 and the domU? In that case you''ll get pounded. Maybe  
dedicating an entire Ethernet port to the domU is a way to work around  
that. I know Xen3 has provisions for that sort of thing but I haven''t  
used it.

/a

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Jul 29, 2009 at 04:52:47AM -0400, Alexis Rosen
wrote:
> On Jul 29, 2009, at 4:04 AM, Robert Dunkley wrote:
> >I was wondering if anyone is running Quagga on Xen? What is
> >throughput/latency like?
> >
> >Was looking for about a gig of mixed packet size BGP throughput.
> 
> Presumably you mean internet traffic.
> 
> On bare metal (no Xen), that will work fine for mixed Internet  
> traffic, but you need to think about corner cases. What do you expect  
> to have happen when you get DDoSed with minimum-size packets? I
don''t
> think even the multiqueue GbE cards will let you handle that, but I  
> have not tested that, and I''d love to be proven wrong.
> 
> I''m also under the impression that Linux 2.6.30 kernels have some
> significant patches to make (better?) use of multiqueue cards but I  
> don''t remember the details. Maybe 2.6.30 can distribute ksoftirqd
load
> over multiple cores? If so that''s a big win, as that was the
killer
> with large PPS when we tested on somewhat less powerful hardware with  
> an older kernel, and you can probably handle a full gbps of min-sized  
> packets if you give it a number of cores (and you have enough queues).
> 
Here''s some recent thread on lkml about linux 10 gbit routing
performance:
http://groups.google.com/group/linux.kernel/browse_thread/thread/70e62d8a85cd3241

"We also achieved nearly 80 Gbps in bidirectional TCP tests (40 Gbps
simultaneously in each direction):"

But that was on baremetal.
> Another question to think about (sorry, no answers here, just  
> questions): If you do run under Xen, is there interrupt load in both  
> the dom0 and the domU? In that case you''ll get pounded. Maybe  
> dedicating an entire Ethernet port to the domU is a way to work around  
> that. I know Xen3 has provisions for that sort of thing but I
haven''t
> used it.
> 
Other link about Xen network performance with Solarflare 10 Gbit
"Accelerated" NIC:

http://blog.xen.org/index.php/2009/04/07/white-paper-getting-10-gbs-from-xen-safe-and-fast-device-access-from-unprivileged-domains/
http://blog.xen.org/wp-content/uploads/2009/04/getting_10gbps_from_xen_.pdf

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Jul 29, 2009, at 5:17 AM, sthaug@nethelp.no wrote:
>> I was wondering if anyone is running Quagga on Xen? What is
>> throughput/latency like?
>
> This is a function of kernel forwarding performance. Quagga
doesn''t
> do forwarding.
But the question is still of significant importance to a lot of Quagga  
users. Thus, I suggest that it is an appropriate topic for the list.

Here''s my earlier reply, which I (again, damnit) failed to send to the
list from an authorized address:
> Presumably you mean internet traffic.
>
> On bare metal (no Xen), that will work fine for mixed Internet  
> traffic, but you need to think about corner cases. What do you  
> expect to have happen when you get DDoSed with minimum-size packets?  
> I don''t think even the multiqueue GbE cards will let you handle  
> that, but I have not tested that, and I''d love to be proven wrong.
>
> I''m also under the impression that Linux 2.6.30 kernels have some
> significant patches to make (better?) use of multiqueue cards but I  
> don''t remember the details. Maybe 2.6.30 can distribute ksoftirqd
> load over multiple cores? If so that''s a big win, as that was the
> killer with large PPS when we tested on somewhat less powerful  
> hardware with an older kernel, and you can probably handle a full  
> gbps of min-sized packets if you give it a number of cores (and you  
> have enough queues).
>
> Another question to think about (sorry, no answers here, just  
> questions): If you do run under Xen, is there interrupt load in both  
> the dom0 and the domU? In that case you''ll get pounded. Maybe  
> dedicating an entire Ethernet port to the domU is a way to work  
> around that. I know Xen3 has provisions for that sort of thing but I  
> haven''t used it.
/a

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Jul 29, 2009 at 02:57:04PM -0400, Alexis Rosen
wrote:
> But the question is still of significant importance to a lot of Quagga  
> users. Thus, I suggest that it is an appropriate topic for the list.
> 
> Here''s my earlier reply, which I (again, damnit) failed to send to
the
> list from an authorized address:
Well i haven''t tried xen myself, but from what I hve read I/O
performance is not as fast as native would be.  kvm seems to be the only
virtualization system with just about native I/O speed.  paravirt is
close too, but requires of course a paravirt compatible guest.

-- 
Len Sorensen

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/29/09 11:57 AM, Alexis Rosen wrote:
> On Jul 29, 2009, at 5:17 AM, sthaug@nethelp.no wrote:
>>> I was wondering if anyone is running Quagga on Xen? What is
>>> throughput/latency like?
>>
>> This is a function of kernel forwarding performance. Quagga
doesn''t
>> do forwarding.
At my company, we have done extensive testing of the forwarding
performance of Linux vms on Xen.  We use Quagga as our routing suite,
but as previously mentioned it has nothing to do with forwarding
performance.  I removed the Quagga list from this thread to stop the
cross post.

For testing we follow rfc2544.  To give you some representative numbers,
we see anywhere between 100-150mbps zero loss throughput for
bi-directional 64byte packet streams on a 3.0ghz Intel quad core
processor.  This follows the typical bandwidth curve up to roughly
1.6gig for large packet sizes.  We are currently running a Linux 2.6.30
pv_ops enabled kernel in the domU.  We have noticed that if we share a
physical processor core with more than one vm we will take a roughly 2%
hit to overall performance.  Interestingly, a third or fourth vm on the
same core still only incurs the same 2% penalty.  Throughput is highly
dependent on the system; i.e. processor model, motherboard chipsets, bus
type and location of the card on the bus, etc...  Throughput also has a
fairly high jitter factor.  The system can be tuned to mitigate the
jitter, but at a loss of overall throughput and an average increase in
latency.

If the system is configured for PCI pass through, expect a much higher
throughput.  It''s more on the order of 650mbps zero-loss for
bi-directional streams of small packet sizes.  HVM domUs aren''t even
worth using for networking.

Hope that helps.

Cheers,
Robert.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Jul 29, 2009 at 03:26:37PM -0700, Robert Bays
wrote:
> 
> On 7/29/09 11:57 AM, Alexis Rosen wrote:
> > On Jul 29, 2009, at 5:17 AM, sthaug@nethelp.no wrote:
> >>> I was wondering if anyone is running Quagga on Xen? What is
> >>> throughput/latency like?
> >>
> >> This is a function of kernel forwarding performance. Quagga
doesn''t
> >> do forwarding.
> 
> At my company, we have done extensive testing of the forwarding
> performance of Linux vms on Xen.  We use Quagga as our routing suite,
> but as previously mentioned it has nothing to do with forwarding
> performance.  I removed the Quagga list from this thread to stop the
> cross post.
> 
> For testing we follow rfc2544.  To give you some representative numbers,
> we see anywhere between 100-150mbps zero loss throughput for
> bi-directional 64byte packet streams on a 3.0ghz Intel quad core
> processor.  This follows the typical bandwidth curve up to roughly
> 1.6gig for large packet sizes.  We are currently running a Linux 2.6.30
> pv_ops enabled kernel in the domU.  We have noticed that if we share a
> physical processor core with more than one vm we will take a roughly 2%
> hit to overall performance.  Interestingly, a third or fourth vm on the
> same core still only incurs the same 2% penalty.  Throughput is highly
> dependent on the system; i.e. processor model, motherboard chipsets, bus
> type and location of the card on the bus, etc...  Throughput also has a
> fairly high jitter factor.  The system can be tuned to mitigate the
> jitter, but at a loss of overall throughput and an average increase in
> latency.
> 
Interesting. Did you profile what limits the performance, or uses the cpu? 
Bridging in dom0? Xen? 

Are you familiar with the netchannel2 development stuff? 
> If the system is configured for PCI pass through, expect a much higher
> throughput.  It''s more on the order of 650mbps zero-loss for
> bi-directional streams of small packet sizes.  HVM domUs aren''t
even
> worth using for networking.
> 
Yeah, PV guests are much easier, faster and stable for this purpose.

(and yeah I know you can use PV-on-HVM drivers on HVM domain).

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Jul 29, 2009, at 9:09 AM, Pasi Kärkkäinen wrote:
> On Wed, Jul 29, 2009 at 04:52:47AM -0400, Alexis Rosen wrote:
>> On Jul 29, 2009, at 4:04 AM, Robert Dunkley wrote:
>>> I was wondering if anyone is running Quagga on Xen? What is
>>> throughput/latency like?
>>>
>>> Was looking for about a gig of mixed packet size BGP throughput.
>>
>> Presumably you mean internet traffic. [...]
>
> Here''s some recent thread on lkml about linux 10 gbit routing  
> performance:
>
http://groups.google.com/group/linux.kernel/browse_thread/thread/70e62d8a85cd3241
>
> "We also achieved nearly 80 Gbps in bidirectional TCP tests (40 Gbps
> simultaneously in each direction):"
>
> But that was on baremetal.
For a real-world internet connection, that kind of throughput is  
irrelevantly large. The more interesting question is how many PPS you  
can handle.

The good news for Robert is that 80gbps at 1500B/packet is over 7mpps,  
which is way more than you need for 1gbps at 64B/packet (about 2mpps).  
So apparently recent kernels have made gigantic advances here.

/a
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Wed, Jul 29, 2009 at 03:19:44PM -0400, Lennart Sorensen
wrote:
> On Wed, Jul 29, 2009 at 02:57:04PM -0400, Alexis Rosen wrote:
> > But the question is still of significant importance to a lot of Quagga
> > users. Thus, I suggest that it is an appropriate topic for the list.
> > 
> > Here''s my earlier reply, which I (again, damnit) failed to
send to the
> > list from an authorized address:
> 
> Well i haven''t tried xen myself, but from what I hve read I/O
> performance is not as fast as native would be.  kvm seems to be the only
> virtualization system with just about native I/O speed.  paravirt is
> close too, but requires of course a paravirt compatible guest.
> 
Getting a bit offtopic, but I believe Xen paravirt guests are still faster 
on I/O speed than KVM guests (using virtio drivers), based on recent IBM
benchmarks.

-- Pasi

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, Jul 30, 2009 at 05:06:14PM +0300, Pasi Kärkkäinen
wrote:
> On Wed, Jul 29, 2009 at 03:19:44PM -0400, Lennart Sorensen wrote:
> > On Wed, Jul 29, 2009 at 02:57:04PM -0400, Alexis Rosen wrote:
> > > But the question is still of significant importance to a lot of
Quagga
> > > users. Thus, I suggest that it is an appropriate topic for the
list.
> > > 
> > > Here''s my earlier reply, which I (again, damnit) failed
to send to the
> > > list from an authorized address:
> > 
> > Well i haven''t tried xen myself, but from what I hve read I/O
> > performance is not as fast as native would be.  kvm seems to be the
only
> > virtualization system with just about native I/O speed.  paravirt is
> > close too, but requires of course a paravirt compatible guest.
> > 
> 
> Getting a bit offtopic, but I believe Xen paravirt guests are still faster 
> on I/O speed than KVM guests (using virtio drivers), based on recent IBM
benchmarks.
Well I haven''t tried xen.  I am certainly not complaining about the
speed of kvm though.  The e1000 emulation is very fast and works well
enough for my needs.

And I did mean paravirt was close to native hardware speed, not that it
was close to kvm speed.

-- 
Len Sorensen

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users