Xen users - Nov 2007 - Speed of Xen Network Bridge Interface (10/100/1000?)

Hi,

I would like to learn the speed of the network bridge interfaces created by
XEN.

More specifically, on xenbr0, given that the traffic only occurs between my
Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or 1000 Mbps?
Does it depend on something such as ethernet card capability (even though
the packets don''t go out of the card and stay inside Dom0)?

I plan to use iScsi or Ata-over-Ethernet, that''s why I''m
asking this
question,

Thanks a lot,

Emre


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 11/27/07, Emre Erenoglu <erenoglu@gmail.com>
wrote:
> Hi,
>
> I would like to learn the speed of the network bridge interfaces created by
> XEN.
>
> More specifically, on xenbr0, given that the traffic only occurs between my
> Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or 1000
Mbps?
> Does it depend on something such as ethernet card capability (even though
> the packets don''t go out of the card and stay inside Dom0)?
>
> I plan to use iScsi or Ata-over-Ethernet, that''s why I''m
asking this
> question,
AFAIK, there''s no hard speed limit.  on HVM guests, the eth card
appears as a 10Mbit model, but usually performs much faster than that.

OTOH, any block device protocol with the initiators on DomU is
certainly doable, but you get huge overheads.  be sure to dedicate at
least one physical CPU core to Dom0, so that you don''t get two
context-switchs (with associated blocking) for each packet.

IMHO, you always want to use highest-level, biggest-block abstractions
on DomU''s, with any low-level, small-block implementations terminated
at Dom0.  in your case, the iSCSI initiator or AoE module would be on
Dom0, and DomU''s would see only block devices, without any special
drivers.

why would you want to run AoE/iSCSI over the emulated LAN? is it to
make it easier to live-resize the block devices?  there should be a
better way...

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Emre Erenoglu wrote:
| Hi,
|
| I would like to learn the speed of the network bridge interfaces created
| by XEN.
|
| More specifically, on xenbr0, given that the traffic only occurs between
| my Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or
| 1000 Mbps? Does it depend on something such as ethernet card capability
| (even though the packets don''t go out of the card and stay inside
Dom0)?
|
| I plan to use iScsi or Ata-over-Ethernet, that''s why I''m
asking this
| question,
|
| Thanks a lot,
|
| Emre

	In Debian, you can use mii-tool command to limit speed and other
configurations of an network interface. This command is present on
net-tools package. I don''t known if this solution is the best way for
your problem.

- --
http://www.goes.eti.br
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHTFOwDo8Z2PVlFN8RArYcAJ9or5/RomnNt5XoMaq/9x/mJp5poACfeCbA
xaG9I6Lhn6EO8i9alJggbAU=JQIq
-----END PGP SIGNATURE-----

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> I would like to learn the speed of the network bridge interfaces created by
> XEN.
>
> More specifically, on xenbr0, given that the traffic only occurs between my
> Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or 1000
> Mbps? Does it depend on something such as ethernet card capability (even
> though the packets don''t go out of the card and stay inside Dom0)?
It''s not limited by your physical ethernet card, nor is it restricted
to any
particular maximum.  It''s basically limited by how fast the Xen virtual
network drivers and the Linux bridging code can move the data around.  This 
used to actually be slower a domU accessing the physical ethernet due to the 
extra memory operations that were required (and used a fair bit of CPU).  I 
think there have been some changes to reduce the bottleneck and improve 
intrahost performance since then, so it would be faster than I remember it.  
I''m not sure if it''s currently faster than GigE; possibly.

It ought to be significantly faster than 100Mbps on a modern machine. 
It''ll
act like a really fast ethernet card, with no hard limit on the transmission 
speed (instead, transmission speed will be limited by how powerful your 
machine is and how efficient the virtual ethernet code is).
> I plan to use iScsi or Ata-over-Ethernet, that''s why I''m
asking this
> question,
Is that from dom0 to domU?  Do you have a particular reason for doing that?  
Using blkback / blkfront would be simpler and more efficient.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> > More specifically, on xenbr0, given that the traffic only occurs
between
> > my Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or
1000
> > Mbps? Does it depend on something such as ethernet card capability
(even
> > though the packets don''t go out of the card and stay inside
Dom0)?
> >
> > I plan to use iScsi or Ata-over-Ethernet, that''s why
I''m asking this
> > question,
>
> AFAIK, there''s no hard speed limit.  on HVM guests, the eth card
> appears as a 10Mbit model, but usually performs much faster than that.
HVM incurs more overhead, so for HVM virtual ethernet it''ll be slower
than if
you use the PV virtual ethernet.

The overhead for HVM is due to the inefficiencies of emulating real hardware 
under Xen, and should not (as Javier points out) be limited by the fact that 
the device pretends to be 10Mbps.

Cheers,
Mark
> OTOH, any block device protocol with the initiators on DomU is
> certainly doable, but you get huge overheads.  be sure to dedicate at
> least one physical CPU core to Dom0, so that you don''t get two
> context-switchs (with associated blocking) for each packet.
>
> IMHO, you always want to use highest-level, biggest-block abstractions
> on DomU''s, with any low-level, small-block implementations
terminated
> at Dom0.  in your case, the iSCSI initiator or AoE module would be on
> Dom0, and DomU''s would see only block devices, without any special
> drivers.
>
> why would you want to run AoE/iSCSI over the emulated LAN? is it to
> make it easier to live-resize the block devices?  there should be a
> better way...


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi!

Just to add some salt to the original posters question, I''m going to
migrate
clustermembers into domU''s on one big machine.
Well, it''s not a full featered HA cluster, it consists of one huge
nfs/nis
server and a lot of diskless servers with as little as necessary failover
management.
First tests showed, that booting one of the machines as domU results in
random disk throughput of about 10 MB/s against about 80-95MB/s when running
on bare-matel.
I don''t necessarilly need to keep the current infrastructure, but
I''ll definitely
need one mountpoint available on many (expandable) machines.
Is there some best-practice description on how to get one mountpoint available
to
a lot of domU''s ?

Thanks for any suggestion!

Stephan



Mark Williamson schrieb:
>> I would like to learn the speed of the network bridge interfaces
created by
>> XEN.
>>
>> More specifically, on xenbr0, given that the traffic only occurs
between my
>> Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps or 1000
>> Mbps? Does it depend on something such as ethernet card capability
(even
>> though the packets don''t go out of the card and stay inside
Dom0)?
> 
> It''s not limited by your physical ethernet card, nor is it
restricted to any
> particular maximum.  It''s basically limited by how fast the Xen
virtual
> network drivers and the Linux bridging code can move the data around.  This
> used to actually be slower a domU accessing the physical ethernet due to
the
> extra memory operations that were required (and used a fair bit of CPU).  I
> think there have been some changes to reduce the bottleneck and improve 
> intrahost performance since then, so it would be faster than I remember it.
> I''m not sure if it''s currently faster than GigE;
possibly.
> 
> It ought to be significantly faster than 100Mbps on a modern machine. 
It''ll
> act like a really fast ethernet card, with no hard limit on the
transmission
> speed (instead, transmission speed will be limited by how powerful your 
> machine is and how efficient the virtual ethernet code is).
> 
>> I plan to use iScsi or Ata-over-Ethernet, that''s why
I''m asking this
>> question,
> 
> Is that from dom0 to domU?  Do you have a particular reason for doing that?
> Using blkback / blkfront would be simpler and more efficient.
> 
> Cheers,
> Mark
> 

-- 
Stephan Seitz
Senior System Administrator

*netz-haut* e.K.
multimediale kommunikation

zweierweg 22
97074 würzburg

fon: +49 931 2876247
fax: +49 931 2876248

web: www.netz-haut.de <http://www.netz-haut.de/>

registriergericht: amtsgericht würzburg, hra 5054



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi, thanks to everybody who commented on my question, now it''s clear
for me.


Let me explain a little deeper why I asked this question (AoE issue). While
surfing on the net and looking for solutions how to run PV drivers inside a
HVM DomU linux guest (like the PV drivers for Windows), I came accross this
website:
http://et.redhat.com/~jmh/pv_drivers/PV_HowTo.txt<http://et.redhat.com/%7Ejmh/pv_drivers/PV_HowTo.txt>

This is explaining in some detail how to build and use PV drivers for
storage and networking. Everything''s fine.

However, reading through, even though you can use this PV VBD driver for
other filesystems, you can''t use it for the  "root" device . 
Show begins
here :)

As far as I understood from the text, the PV networking driver can be
enabled at boot time, so I had this crazy idea of using AtaOverEthernet to
boot up the system through the PV network card. I don''t know if
it''s a good
idea anyways, but I wanted to try.

Of course the issue here would be, if the PV driver based network card can
sustain a disk throughput similar to the Dom0 throughput, or would I be
somewhat limited to the speed of the card or driver itself... (appearantly
not).

One last thing, I asked to Jan (owner of text) about it, he commented that
it''s not possible to use the PV VBD on "boot" device, but not
on other file
systems, so if my / "root" filesystem resides on a different VBD
device, it
may work. I will also try this. I will just put ioemu device for the /boot
partition, the rest will be PV VBD.

What do you think?

Thanks,

Emre



On Nov 27, 2007 11:14 PM, Mark Williamson <mark.williamson@cl.cam.ac.uk>
wrote:
> > > More specifically, on xenbr0, given that the traffic only occurs
> between
> > > my Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps
or
> 1000
> > > Mbps? Does it depend on something such as ethernet card
capability
> (even
> > > though the packets don''t go out of the card and stay
inside Dom0)?
> > >
> > > I plan to use iScsi or Ata-over-Ethernet, that''s why
I''m asking this
> > > question,
> >
> > AFAIK, there''s no hard speed limit.  on HVM guests, the eth
card
> > appears as a 10Mbit model, but usually performs much faster than that.
>
> HVM incurs more overhead, so for HVM virtual ethernet it''ll be
slower than
> if
> you use the PV virtual ethernet.
>
> The overhead for HVM is due to the inefficiencies of emulating real
> hardware
> under Xen, and should not (as Javier points out) be limited by the fact
> that
> the device pretends to be 10Mbps.
>
> Cheers,
> Mark
>
> > OTOH, any block device protocol with the initiators on DomU is
> > certainly doable, but you get huge overheads.  be sure to dedicate at
> > least one physical CPU core to Dom0, so that you don''t get
two
> > context-switchs (with associated blocking) for each packet.
> >
> > IMHO, you always want to use highest-level, biggest-block abstractions
> > on DomU''s, with any low-level, small-block implementations
terminated
> > at Dom0.  in your case, the iSCSI initiator or AoE module would be on
> > Dom0, and DomU''s would see only block devices, without any
special
> > drivers.
> >
> > why would you want to run AoE/iSCSI over the emulated LAN? is it to
> > make it easier to live-resize the block devices?  there should be a
> > better way...
>
>
>
> --
> Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
>


-- 
Emre Erenoglu
erenoglu@gmail.com


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Just to add some salt to the original posters question, I''m going
to
> migrate clustermembers into domU''s on one big machine.
> Well, it''s not a full featered HA cluster, it consists of one huge
nfs/nis
> server and a lot of diskless servers with as little as necessary failover
> management.
> First tests showed, that booting one of the machines as domU results in
> random disk throughput of about 10 MB/s against about 80-95MB/s when
> running on bare-matel.
I''m not entirely clear where your domU is accessing storage from vs the
bare
metal case.  The domU is accessing disk via NFS?  How about the bare metal 
machine in your example?
> I don''t necessarilly need to keep the current infrastructure, but
I''ll
> definitely need one mountpoint available on many (expandable) machines.
> Is there some best-practice description on how to get one mountpoint
> available to a lot of domU''s ?
Well, if you can arrange for it to be readonly then the obvious thing to do is 
to export readonly VBDs to all the domUs.  That way they should all get 
access to it at a speed similar to local disk access.

If you needed them to have private writeable access you could look at some 
kind of layered copy-on-write access (e.g. run unionfs in the domUs?).

If they actually needed shared writeable access then at the moment I guess the 
best option is either NFS or to set them all up with a cluster filesystem 
such as GFS or OCFS2.  You might get better performance with a cluster 
filesystem, I guess - I''m not aware of any benchmarks of cluster FSes
on Xen
though.

Various projects, such as my XenFS filesystem are aiming to provide high 
performance NFS-like functionality on Xen, but I don''t know of any
ready for
production use.

Cheers,
Mark
> Thanks for any suggestion!
>
> Stephan
>
> Mark Williamson schrieb:
> >> I would like to learn the speed of the network bridge interfaces
created
> >> by XEN.
> >>
> >> More specifically, on xenbr0, given that the traffic only occurs
between
> >> my Dom0 host and PV DomU guest, am I limited to 10 Mbps, 100 Mbps
or
> >> 1000 Mbps? Does it depend on something such as ethernet card
capability
> >> (even though the packets don''t go out of the card and
stay inside Dom0)?
> >
> > It''s not limited by your physical ethernet card, nor is it
restricted to
> > any particular maximum.  It''s basically limited by how fast
the Xen
> > virtual network drivers and the Linux bridging code can move the data
> > around.  This used to actually be slower a domU accessing the physical
> > ethernet due to the extra memory operations that were required (and
used
> > a fair bit of CPU).  I think there have been some changes to reduce
the
> > bottleneck and improve intrahost performance since then, so it would
be
> > faster than I remember it. I''m not sure if it''s
currently faster than
> > GigE; possibly.
> >
> > It ought to be significantly faster than 100Mbps on a modern machine. 
> > It''ll act like a really fast ethernet card, with no hard
limit on the
> > transmission speed (instead, transmission speed will be limited by how
> > powerful your machine is and how efficient the virtual ethernet code
is).
> >
> >> I plan to use iScsi or Ata-over-Ethernet, that''s why
I''m asking this
> >> question,
> >
> > Is that from dom0 to domU?  Do you have a particular reason for doing
> > that? Using blkback / blkfront would be simpler and more efficient.
> >
> > Cheers,
> > Mark


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users