fabi.a169@gmail.com
2008-Dec-22 16:36 UTC
[Xen-users] 10Gbit ethernet NIC gives slow throughput
Hello, I would like to use a server having myricom 10gbit ethernet interfaces (driver myri10ge) for Xen. But between dom0s (no domU running), I get only a throughput of about 2.3Gbit/s (I use Xen 3.2 with the 2.6.18 kernel and debian in dom0), whether xend is running or not. However, on linux kernel with the same version but without xen patch, I get more than 6Gbit/s. Clearly, with Xen, the CPU is not the bottleneck, beeing used at much less than 100%. Does anybody have experiences with Xen and 10Gbit ethernet ? Fabi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2008-Dec-22 17:14 UTC
Re: [Xen-users] 10Gbit ethernet NIC gives slow throughput
On Mon, Dec 22, 2008 at 05:36:40PM +0100, fabi.a169@gmail.com wrote:> Hello, > > I would like to use a server having myricom 10gbit ethernet interfaces > (driver myri10ge) for Xen. But between dom0s (no domU running), I get only a > throughput of about 2.3Gbit/s (I use Xen 3.2 with the 2.6.18 kernel and > debian in dom0), whether xend is running or not. However, on linux kernel > with the same version but without xen patch, I get more than 6Gbit/s. > Clearly, with Xen, the CPU is not the bottleneck, beeing used at much less > than 100%. > > Does anybody have experiences with Xen and 10Gbit ethernet ? >With Xen, are you using bridged networking? The default network configuration in Xen creates virtual ethernet interface called ''eth0'', which is bridged to the real NIC (which is renamed to peth0). So, when making comparisons, make sure you''re using the actual physical interface. Dunno if there''s something to optimize in the bridge-setup.. have you done all the normal optimizations? txqueuelen for all interfaces (including vif interfaces), tcp window sizes etc. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
fabi.a169@gmail.com
2008-Dec-22 17:48 UTC
Re: [Xen-users] 10Gbit ethernet NIC gives slow throughput
Pasi Kärkkäinen wrote:> On Mon, Dec 22, 2008 at 05:36:40PM +0100, fabi.a169@gmail.com wrote: > >> Hello, >> >> I would like to use a server having myricom 10gbit ethernet interfaces >> (driver myri10ge) for Xen. But between dom0s (no domU running), I get only a >> throughput of about 2.3Gbit/s (I use Xen 3.2 with the 2.6.18 kernel and >> debian in dom0), whether xend is running or not. However, on linux kernel >> with the same version but without xen patch, I get more than 6Gbit/s. >> Clearly, with Xen, the CPU is not the bottleneck, beeing used at much less >> than 100%. >> >> Does anybody have experiences with Xen and 10Gbit ethernet ? >> >> > > With Xen, are you using bridged networking? > > The default network configuration in Xen creates virtual ethernet interface called > ''eth0'', which is bridged to the real NIC (which is renamed to peth0). > > So, when making comparisons, make sure you''re using the actual physical interface. > > Dunno if there''s something to optimize in the bridge-setup.. have you done > all the normal optimizations? txqueuelen for all interfaces (including vif > interfaces), tcp window sizes etc. > > -- Pasi > >I don''t even use the brigdes, for example without starting xend. Also, when I measure throughput on the loopback interface, I get the same throughput bounded to 2.29 Gbps. There should be something bounding everything to this value, maybe related to the vcpus. If I send for example 2 flows in parallel, I get the double of this throughput. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2008-Dec-22 18:42 UTC
Re: [Xen-users] 10Gbit ethernet NIC gives slow throughput
On Mon, Dec 22, 2008 at 06:48:21PM +0100, fabi.a169@gmail.com wrote:> Pasi Kärkkäinen wrote: > >On Mon, Dec 22, 2008 at 05:36:40PM +0100, fabi.a169@gmail.com wrote: > > > >>Hello, > >> > >>I would like to use a server having myricom 10gbit ethernet interfaces > >>(driver myri10ge) for Xen. But between dom0s (no domU running), I get > >>only a > >>throughput of about 2.3Gbit/s (I use Xen 3.2 with the 2.6.18 kernel and > >>debian in dom0), whether xend is running or not. However, on linux kernel > >>with the same version but without xen patch, I get more than 6Gbit/s. > >>Clearly, with Xen, the CPU is not the bottleneck, beeing used at much less > >>than 100%. > >> > >>Does anybody have experiences with Xen and 10Gbit ethernet ? > >> > >> > > > >With Xen, are you using bridged networking? > > > >The default network configuration in Xen creates virtual ethernet > >interface called > >''eth0'', which is bridged to the real NIC (which is renamed to peth0). > > > >So, when making comparisons, make sure you''re using the actual physical > >interface. > >Dunno if there''s something to optimize in the bridge-setup.. have you done > >all the normal optimizations? txqueuelen for all interfaces (including vif > >interfaces), tcp window sizes etc. > > > >-- Pasi > > > > > I don''t even use the brigdes, for example without starting xend. Also, > when I measure throughput on the loopback interface, I get the same > throughput bounded to 2.29 Gbps. There should be something bounding > everything to this value, maybe related to the vcpus. If I send for > example 2 flows in parallel, I get the double of this throughput.OK. I''m wondering if that could be caused by Xen scheduling timeslices or something like that.. You might want to ask about this issue from xen-devel list. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users