Xen devel - Nov 2005 - xen-unstable: TX/RX ring buffer exhaustion and NR_GRANT_FRAMES

Hi,

it seems as if it is not possible to have more than 3 nics per domU
right now.  If I specify more, I get the usual

  [...]
  xen_net: Initialising virtual ethernet driver.
  #### netfront can''t alloc rx grant refs
  vif: probe of vif-3 failed with error -12
  [...]

message - see bug #183.  It was pointed out to me that it might be
possible to adjust this manually in xen/include/xen/grant_table.h,
however, setting ORDER_GRANT_FRAMES to 3 to get more pages of grant
tables doesn''t change anything.  Sooo, question #1: how can I alter the
code to get more grant tables such that more nics per domU will become
available?  Question #2: is it planned to make this configurable, e.g.
by boot option for dom0?


Regards,
-- 
Birger Tödtmann                   email:btoedtmann@iem.uni-due.de
Technik der Rechnernetze, Institut für Experimentelle Mathematik
Universität Duisburg-Essen, Campus Essen, Germany.
skype:birger.toedtmann  pgp:0x6FB166C9  phone: +49-(0)201-1837662

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 11 Nov 2005, at 08:59, Birger Toedtmann wrote:
> message - see bug #183.  It was pointed out to me that it might be
> possible to adjust this manually in xen/include/xen/grant_table.h,
> however, setting ORDER_GRANT_FRAMES to 3 to get more pages of grant
> tables doesn''t change anything.  Sooo, question #1: how can I
alter the
> code to get more grant tables such that more nics per domU will become
> available?  Question #2: is it planned to make this configurable, e.g.
> by boot option for dom0?
1: you need to edit NR_GRANT_FRAMES in linux/include/asm-xen/gnttab.h

2: yes, it will eventually be manually configurable with a higher 
default ''maximum''

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The xenidc code has the concept of a buffer resource provider which
implements an anti-deadlock resource reservation.  Each client has uses
a pool with it''s own buffer resource provider and the anti-deadlock
resource reservations decouple the clients from one another.  The
underlying implementation of the buffer resource provider is currently
trivial (individual resource pools) but the intention is that the buffer
resource providers will share resources from a common underlying pool.

This resource management strategy allows you to make better use of
constrained resources like grant tables whilst avoiding deadlock.

On Fri, 2005-11-11 at 10:48 +0000, Keir Fraser wrote:
> On 11 Nov 2005, at 08:59, Birger Toedtmann wrote:
> 
> > message - see bug #183.  It was pointed out to me that it might be
> > possible to adjust this manually in xen/include/xen/grant_table.h,
> > however, setting ORDER_GRANT_FRAMES to 3 to get more pages of grant
> > tables doesn''t change anything.  Sooo, question #1: how can I
alter the
> > code to get more grant tables such that more nics per domU will become
> > available?  Question #2: is it planned to make this configurable, e.g.
> > by boot option for dom0?
> 
> 1: you need to edit NR_GRANT_FRAMES in linux/include/asm-xen/gnttab.h
> 
> 2: yes, it will eventually be manually configurable with a higher 
> default ''maximum''
> 
>   -- Keir
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

domUs are yet not allowed to run more than 3 NICs as far as I can see
when running tests with 3.0, and the ''old'' way to provide more
interfaces by enhancing the grant table structures do not work anymore
(see below).  I''d really appreciate any suggestions where I could
modify
the source to get more NICs.


Am Freitag, den 11.11.2005, 12:11 +0000 schrieb harry:
> The xenidc code has the concept of a buffer resource provider which
> implements an anti-deadlock resource reservation.  Each client has uses
> a pool with it''s own buffer resource provider and the
anti-deadlock
> resource reservations decouple the clients from one another.  The
> underlying implementation of the buffer resource provider is currently
> trivial (individual resource pools) but the intention is that the buffer
> resource providers will share resources from a common underlying pool.
> 
> This resource management strategy allows you to make better use of
> constrained resources like grant tables whilst avoiding deadlock.
Interesting thing, but I could not find documentation on xenidc - where
do I get it?  Where is the code, is it already inside Xen?

> 
> On Fri, 2005-11-11 at 10:48 +0000, Keir Fraser wrote:
> > On 11 Nov 2005, at 08:59, Birger Toedtmann wrote:
> > 
> > > message - see bug #183.  It was pointed out to me that it might
be
> > > possible to adjust this manually in
xen/include/xen/grant_table.h,
> > > however, setting ORDER_GRANT_FRAMES to 3 to get more pages of
grant
> > > tables doesn''t change anything.  Sooo, question #1: how
can I alter the
> > > code to get more grant tables such that more nics per domU will
become
> > > available?  Question #2: is it planned to make this configurable,
e.g.
> > > by boot option for dom0?
> > 
> > 1: you need to edit NR_GRANT_FRAMES in linux/include/asm-xen/gnttab.h
> > 
> > 2: yes, it will eventually be manually configurable with a higher 
> > default ''maximum''
Unfortunately, this method does not work anymore since November/
December.  After modifying NR_GRANT_FRAMES etc., the dom0 would say
"g.e. still in use!" on boot and halt after this message.  Any hints
how
I can adjust the grant tables to get more NICs?



Regards,
-- 
Birger Tödtmann                   email:btoedtmann@iem.uni-due.de
Technik der Rechnernetze, Institut für Experimentelle Mathematik
Universität Duisburg-Essen, Campus Essen, Germany.
skype:birger.toedtmann  pgp:0x6FB166C9  phone: +49-(0)201-1837662


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2006-01-10 at 16:22 +0100, Birger Tödtmann wrote:
> Am Freitag, den 11.11.2005, 12:11 +0000 schrieb harry:
> > The xenidc code has the concept of a buffer resource provider which
> > implements an anti-deadlock resource reservation.  Each client has
uses
> > a pool with it''s own buffer resource provider and the
anti-deadlock
> > resource reservations decouple the clients from one another.  The
> > underlying implementation of the buffer resource provider is currently
> > trivial (individual resource pools) but the intention is that the
buffer
> > resource providers will share resources from a common underlying pool.
> > 
> > This resource management strategy allows you to make better use of
> > constrained resources like grant tables whilst avoiding deadlock.
> 
> Interesting thing, but I could not find documentation on xenidc - where
> do I get it?  Where is the code, is it already inside Xen?
No, xenidc is not already inside Xen. The most recent version of xenidc
and the USB patch was posted here:

http://lists.xensource.com/archives/html/xen-devel/2005-12/msg00601.html


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi

Here''s the quick and dirty explanation of why you can''t just
boost that
#define:

In the data structure that tracks a domain''s list of foreign frames
currently mapped by using grant references (''maptrack''), 11
bits are used to
record the foreign grant reference index. This is to keep the data structure
packed into 32 bits because you end up needing a lot of them if you''re
mapping a lot of foreign frames.

So the foreign domain can''t hand you a grant reference index >=
2^11, so
that''s 2048 maximum grant entries (ie. frames you can allow others to
map)
per domain.
The sizeof the shared data structure stored in the shared grant table is 64
bits == 8 bytes. So 2048 of 8 bytes gives you 4 frames, which is why
NR_GRANT_FRAMES is maxed out at 4 right now.

If you do just boost it above 4 then your index ends up colliding with the
flags packed into the other bits and I''m not surprised it gets nasty.

Mea culpa: I hadn''t anticipated each NIC needing as many entries as
they
actually do.
Fixing this is surgery.

For right now, if you can live with giving up some performance to get your
extra NICs, you could try halving the ring sizes NET_TX_RING_SIZE and
NET_RX_RING_SIZE defined in sparse/drivers/xen/netback/common.h

Christopher

On 1/10/06, Birger Tödtmann <btoedtmann@iem.uni-due.de>
wrote:
>
>
>
> Hi,
>
> domUs are yet not allowed to run more than 3 NICs as far as I can see
> when running tests with 3.0, and the ''old'' way to provide
more
> interfaces by enhancing the grant table structures do not work anymore
> (see below).  I''d really appreciate any suggestions where I could
modify
> the source to get more NICs.
>
>
> Am Freitag, den 11.11.2005, 12:11 +0000 schrieb harry:
> > The xenidc code has the concept of a buffer resource provider which
> > implements an anti-deadlock resource reservation.  Each client has
uses
> > a pool with it''s own buffer resource provider and the
anti-deadlock
> > resource reservations decouple the clients from one another.  The
> > underlying implementation of the buffer resource provider is currently
> > trivial (individual resource pools) but the intention is that the
buffer
> > resource providers will share resources from a common underlying pool.
> >
> > This resource management strategy allows you to make better use of
> > constrained resources like grant tables whilst avoiding deadlock.
>
> Interesting thing, but I could not find documentation on xenidc - where
> do I get it?  Where is the code, is it already inside Xen?
>
>
> >
> > On Fri, 2005-11-11 at 10:48 +0000, Keir Fraser wrote:
> > > On 11 Nov 2005, at 08:59, Birger Toedtmann wrote:
> > >
> > > > message - see bug #183.  It was pointed out to me that it
might be
> > > > possible to adjust this manually in
xen/include/xen/grant_table.h,
> > > > however, setting ORDER_GRANT_FRAMES to 3 to get more pages
of grant
> > > > tables doesn''t change anything.  Sooo, question #1:
how can I alter
> the
> > > > code to get more grant tables such that more nics per domU
will
> become
> > > > available?  Question #2: is it planned to make this
configurable,
> e.g.
> > > > by boot option for dom0?
> > >
> > > 1: you need to edit NR_GRANT_FRAMES in
linux/include/asm-xen/gnttab.h
> > >
> > > 2: yes, it will eventually be manually configurable with a higher
> > > default ''maximum''
>
> Unfortunately, this method does not work anymore since November/
> December.  After modifying NR_GRANT_FRAMES etc., the dom0 would say
> "g.e. still in use!" on boot and halt after this message.  Any
hints how
> I can adjust the grant tables to get more NICs?
>
>
>
> Regards,
> --
> Birger Tödtmann                   email:btoedtmann@iem.uni-due.de
> Technik der Rechnernetze, Institut für Experimentelle Mathematik
> Universität Duisburg-Essen, Campus Essen, Germany.
> skype:birger.toedtmann  pgp:0x6FB166C9  phone: +49-(0)201-1837662
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel