Xen devel - Apr 2005 - what''s maximum speed setting for xen network?

For a physical network card, it has a maximum transmission speed such as
100Mb. How about network in a virtual machine? It is a virtual network
card. Is there any speed limit? Or it can transmit as fast as possible? If
so, I guess network transmission happened within a physical host should be
very fast (great than 100Mb). But this is not true in my experiment.
What''s the bottleneck then?

Thanks in advance for your advice.

Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Machine local networking is something that is currently not handled
well by xen. One ends up context switching for every packet sent. I
believe this is on the TODO list.

                                    -Kip 

On 4/20/05, Xin Zhao <zhaoxin@eecs.umich.edu>
wrote:
> For a physical network card, it has a maximum transmission speed such as
> 100Mb. How about network in a virtual machine? It is a virtual network
> card. Is there any speed limit? Or it can transmit as fast as possible? If
> so, I guess network transmission happened within a physical host should be
> very fast (great than 100Mb). But this is not true in my experiment.
> What''s the bottleneck then?
> 
> Thanks in advance for your advice.
> 
> Xin
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> For a physical network card, it has a maximum transmission speed such as
> 100Mb. How about network in a virtual machine? It is a virtual network
> card. Is there any speed limit? Or it can transmit as fast as possible? 
> If so, I guess network transmission happened within a physical host 
> should be very fast (great than 100Mb). But this is not true in my 
> experiment.
Out of curiousity, what kind of testing have you run, and what kind 
of speeds are you getting?

I''ve got a configuration with DomU talking to Dom0 over a vif, and seem
to
be seeing UDP losses, even at low data rates. (I haven''t dug into this 
deeply to very the problem yet, though.) So I''d be interested if
you''re
seeing something similar.

Thanks,
mukesh

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I just setup network and tried to copy files between dom0 and domU. The
speed is around 100Mb/s. So I once thought it is simulating 100Mb network,
but after looking into the network driver, I failed to find anything about
the flow control.

I didn''t try UDP, so have no idea about UDP losses.

Xin


On Fri, 22 Apr 2005, mukesh agrawal wrote:
>
> > For a physical network card, it has a maximum transmission speed such
as
> > 100Mb. How about network in a virtual machine? It is a virtual network
> > card. Is there any speed limit? Or it can transmit as fast as
possible?
> > If so, I guess network transmission happened within a physical host
> > should be very fast (great than 100Mb). But this is not true in my
> > experiment.
>
> Out of curiousity, what kind of testing have you run, and what kind
> of speeds are you getting?
>
> I''ve got a configuration with DomU talking to Dom0 over a vif, and
seem to
> be seeing UDP losses, even at low data rates. (I haven''t dug into
this
> deeply to very the problem yet, though.) So I''d be interested if
you''re
> seeing something similar.
>
> Thanks,
> mukesh
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

It is possible to rate limit vifs but it won''t be active by default.

What are you using to copy the files across?  If you''re using scp, bear
in
bind that both virtual machines will have to do crypto, so you could actually 
be limited by CPU performance rather than virtual network.

Also, uniprocessor systems are a worst case for this sort of test because 
you''ll incur a lot of context switches.  Hyperthreading should be able
to
mitigate this somewhat.

There are plans to optimise host-internal networking somewhat.

Cheers,
Mark

On Friday 22 April 2005 21:14, Xin Zhao wrote:
> I just setup network and tried to copy files between dom0 and domU. The
> speed is around 100Mb/s. So I once thought it is simulating 100Mb network,
> but after looking into the network driver, I failed to find anything about
> the flow control.
>
> I didn''t try UDP, so have no idea about UDP losses.
>
> Xin
>
> On Fri, 22 Apr 2005, mukesh agrawal wrote:
> > > For a physical network card, it has a maximum transmission speed
such
> > > as 100Mb. How about network in a virtual machine? It is a virtual
> > > network card. Is there any speed limit? Or it can transmit as
fast as
> > > possible? If so, I guess network transmission happened within a
> > > physical host should be very fast (great than 100Mb). But this is
not
> > > true in my experiment.
> >
> > Out of curiousity, what kind of testing have you run, and what kind
> > of speeds are you getting?
> >
> > I''ve got a configuration with DomU talking to Dom0 over a
vif, and seem
> > to be seeing UDP losses, even at low data rates. (I haven''t
dug into this
> > deeply to very the problem yet, though.) So I''d be interested
if you''re
> > seeing something similar.
> >
> > Thanks,
> > mukesh
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yes. I tried scp. I know CPU could be bottleneck. But I also tried ftp and
NFS, they do not have much better performance.  :) So it might be virtual
network issue.

xin


On Fri, 22 Apr 2005, Mark Williamson wrote:
> It is possible to rate limit vifs but it won''t be active by
default.
>
> What are you using to copy the files across?  If you''re using scp,
bear in
> bind that both virtual machines will have to do crypto, so you could
actually
> be limited by CPU performance rather than virtual network.
>
> Also, uniprocessor systems are a worst case for this sort of test because
> you''ll incur a lot of context switches.  Hyperthreading should be
able to
> mitigate this somewhat.
>
> There are plans to optimise host-internal networking somewhat.
>
> Cheers,
> Mark
>
> On Friday 22 April 2005 21:14, Xin Zhao wrote:
> > I just setup network and tried to copy files between dom0 and domU.
The
> > speed is around 100Mb/s. So I once thought it is simulating 100Mb
network,
> > but after looking into the network driver, I failed to find anything
about
> > the flow control.
> >
> > I didn''t try UDP, so have no idea about UDP losses.
> >
> > Xin
> >
> > On Fri, 22 Apr 2005, mukesh agrawal wrote:
> > > > For a physical network card, it has a maximum transmission
speed such
> > > > as 100Mb. How about network in a virtual machine? It is a
virtual
> > > > network card. Is there any speed limit? Or it can transmit
as fast as
> > > > possible? If so, I guess network transmission happened
within a
> > > > physical host should be very fast (great than 100Mb). But
this is not
> > > > true in my experiment.
> > >
> > > Out of curiousity, what kind of testing have you run, and what
kind
> > > of speeds are you getting?
> > >
> > > I''ve got a configuration with DomU talking to Dom0 over
a vif, and seem
> > > to be seeing UDP losses, even at low data rates. (I
haven''t dug into this
> > > deeply to very the problem yet, though.) So I''d be
interested if you''re
> > > seeing something similar.
> > >
> > > Thanks,
> > > mukesh
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22 Apr 2005, at 20:53, mukesh agrawal wrote:
> I''ve got a configuration with DomU talking to Dom0 over a vif, and
> seem to be seeing UDP losses, even at low data rates. (I haven''t
dug
> into this deeply to very the problem yet, though.) So I''d be 
> interested if you''re seeing something similar.
We have no back-pressure / flow control on inter-domain networking. 
This leads to embarrassing performance for UDP when the sender and 
receiver share a CPU. The sender fills the receiver''s buffers but 
continues to send even though he drops further packets (no buffer space 
at receiver) -- then the receiver gets a time slice, receives a ring''s 
worth of packets -- then the sender gets to run for another time slice, 
most of which again is wasted -- and so on.

We need a sensible mechanism for ensuring batched packet delivery, but 
also to ensure that comms rings do not so easily get overflowed. This 
would vastly improve the inter-domain performance, far more than 
striving for zero copy or other such tweaks.

It would be great if someone else is interested in investigating this. 
:-)

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Apr 23, 2005 7:31 AM,  Keir Fraser wrote:
> We have no back-pressure / flow control on inter-domain networking.
> This leads to embarrassing performance for UDP when the sender and
> receiver share a CPU. The sender fills the receiver''s buffers but
> continues to send even though he drops further packets (no buffer space
> at receiver) -- then the receiver gets a time slice, receives a
ring''s
> worth of packets -- then the sender gets to run for another time slice,
> most of which again is wasted -- and so on.
FWIW, I don''t think this is the problem I''m running into. The
reason being
that I don''t think the workload I''m using is heavy enough to
trigger
overflow. I''m sending 102 packets over the course of 2 seconds. Of 
these, 5 are dropped.

Assuming the ring size is given by NETIF_RX_RING_SIZE (which is defined to 
256 in io/netif.h), I''m thinking that the drops can''t be
attributed to
overflow.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23 Apr 2005, at 17:14, mukesh agrawal wrote:
> FWIW, I don''t think this is the problem I''m running into.
The reason
> being that I don''t think the workload I''m using is heavy
enough to
> trigger overflow. I''m sending 102 packets over the course of 2 
> seconds. Of these, 5 are dropped.
>
> Assuming the ring size is given by NETIF_RX_RING_SIZE (which is 
> defined to 256 in io/netif.h), I''m thinking that the drops
can''t be
> attributed to overflow.
We also dynamically size the receive ring. It starts off small (8 
buffers) and is doubled in size every time packets may have been lost. 
It then slowly (linearly) shrinks when loss is not detected. The 
parameters for this are RX_MIN_TARGET and RX_MAX_TARGET in netfront.c 
-- currently they are not configurable at run-time, nor easily 
configurable at compile-time.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 23 Apr 2005, Keir Fraser wrote:
> We also dynamically size the receive ring. It starts off small (8 buffers) 
> and is doubled in size every time packets may have been lost. It then
slowly
> (linearly) shrinks when loss is not detected. The parameters for this are 
> RX_MIN_TARGET and RX_MAX_TARGET in netfront.c -- currently they are not 
> configurable at run-time, nor easily configurable at compile-time.
If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, would 
that be enough to eliminate the adaptation (i.e. the ring size would 
always be 256)?

Thanks,
mukesh


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> On 22 Apr 2005, at 20:53, mukesh agrawal wrote:
> 
>> I''ve got a configuration with DomU talking to Dom0 over a vif,
and
>> seem to be seeing UDP losses, even at low data rates. (I
haven''t dug
>> into this deeply to very the problem yet, though.) So I''d be 
>> interested if you''re seeing something similar.
> 
> 
> We have no back-pressure / flow control on inter-domain networking. This 
> leads to embarrassing performance for UDP when the sender and receiver 
> share a CPU. The sender fills the receiver''s buffers but continues
to
> send even though he drops further packets (no buffer space at receiver) 
> -- then the receiver gets a time slice, receives a ring''s worth of
> packets -- then the sender gets to run for another time slice, most of 
> which again is wasted -- and so on.
> 
> We need a sensible mechanism for ensuring batched packet delivery, but 
> also to ensure that comms rings do not so easily get overflowed. This 
> would vastly improve the inter-domain performance, far more than 
> striving for zero copy or other such tweaks.
> 
> It would be great if someone else is interested in investigating this. :-)
Hi Keir,

Yep, I''m one of the people at least looking into this, don''t
know
if others are too :). Thrown some instrumentation in and doing some
testing as we speak. Really needed some profiling so was trying to
get the oprofile stuff to work in unstable. From what I saw from
his last post, Jose is working on it.

A few people have kindly agreed to do some testing for me too.

We''ll throw some stuff out on the list after the weekend.  I am
tracking all the reports of poor networking performance so if
people do see some issues please continue to post on the list..

thanks,
Nivedita


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23 Apr 2005, at 17:47, mukesh agrawal wrote:
> If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, 
> would that be enough to eliminate the adaptation (i.e. the ring size 
> would always be 256)?
Yes, that would do it.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Nivedita,

  Yes, I am working to get oprofile in xen-unstable.
  I was only able to spend a few  hours on this
  My own port to xen-unstable shows the same behavior as 
  the one done by Andrew Theurer.
  I will start debugging xenoprof this week to nail down 
  the problem. 
  Renato
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Nivedita Singhvi
> Sent: Saturday, April 23, 2005 10:37 AM
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com; zhaoxin@eecs.umich.edu; 
> mukesh agrawal
> Subject: Re: [Xen-devel] what''s maximum speed setting for xen
network?
> 
> 
> Keir Fraser wrote:
> > 
> > On 22 Apr 2005, at 20:53, mukesh agrawal wrote:
> > 
> >> I''ve got a configuration with DomU talking to Dom0 over a
vif, and
> >> seem to be seeing UDP losses, even at low data rates. (I 
> haven''t dug 
> >> into this deeply to very the problem yet, though.) So I''d
be
> >> interested if you''re seeing something similar.
> > 
> > 
> > We have no back-pressure / flow control on inter-domain networking. 
> > This
> > leads to embarrassing performance for UDP when the sender 
> and receiver 
> > share a CPU. The sender fills the receiver''s buffers but 
> continues to 
> > send even though he drops further packets (no buffer space 
> at receiver) 
> > -- then the receiver gets a time slice, receives a ring''s
worth of
> > packets -- then the sender gets to run for another time 
> slice, most of 
> > which again is wasted -- and so on.
> > 
> > We need a sensible mechanism for ensuring batched packet 
> delivery, but
> > also to ensure that comms rings do not so easily get 
> overflowed. This 
> > would vastly improve the inter-domain performance, far more than 
> > striving for zero copy or other such tweaks.
> > 
> > It would be great if someone else is interested in 
> investigating this. 
> > :-)
> 
> Hi Keir,
> 
> Yep, I''m one of the people at least looking into this,
don''t
> know if others are too :). Thrown some instrumentation in and 
> doing some testing as we speak. Really needed some profiling 
> so was trying to get the oprofile stuff to work in unstable. 
> From what I saw from his last post, Jose is working on it.
> 
> A few people have kindly agreed to do some testing for me too.
> 
> We''ll throw some stuff out on the list after the weekend.  I 
> am tracking all the reports of poor networking performance so 
> if people do see some issues please continue to post on the list..
> 
> thanks,
> Nivedita
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> On 23 Apr 2005, at 17:47, mukesh agrawal wrote:
>
>> If I just set RX_MIN_TARGET to NETIF_RX_RING_SIZE, and recompile, would
>> that be enough to eliminate the adaptation (i.e. the ring size would
always
>> be 256)?
FWIW, between this hack, and giving the domains more memory, the drops 
have gone away. (I haven''t tried the changes separately, so I
can''t say
for certain that either one alone is responsible.)

Thanks,
mukesh


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel