Xen devel - Feb 2010 - increasing the number of guests (NR_DYNIRQS)

So I''m setting up a server to host many very small domains, and I ran
into a problem where after I create about 60 guests, any addtional guests
start, but hang at 0 cpu usage-  they never do anything.  

grep Dynamic-irq /proc/interrupts | wc -l
256


I did some digging and found some old discussion of the topic[1]
and but the patches discussed there didn''t work, I assume because the
linux/xen codebase has evolved since then.  I dug around and made this 
change:

xen-3.4-testing.hg/linux-2.6.18-xen.hg/include/asm-x86_64/mach-xen/irq_vectors.h

I changed:

#define NR_DYNIRQS           1024

(it defaults to 256)

two questions:  first, is this stupid?  domains appear to be working, and
I''m currently running north of 200 guests on the server.  

second, if this is indeed the right way to fix the problem, if I 
submitted a CONFIG_NR_DYNIRQS patch as Kier suggested in the referenced
post, would that be considered for acceptance upstream?



[1]http://lists.xensource.com/archives/html/xen-devel/2006-12/msg00353.html


-- 
Luke S. Crawford
http://prgmr.com/xen/         -   Hosting for the technically adept
http://nostarch.com/xen.htm   -   We don''t assume you are stupid.  

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 2010-02-20 at 05:53 -0500, Luke S Crawford
wrote:
> So I''m setting up a server to host many very small domains, and I
ran
> into a problem where after I create about 60 guests, any addtional guests
> start, but hang at 0 cpu usage-  they never do anything.  
> 
> grep Dynamic-irq /proc/interrupts | wc -l
> 256
> 
> 
> I did some digging and found some old discussion of the topic[1]
> and but the patches discussed there didn''t work, I assume because
the
> linux/xen codebase has evolved since then.  I dug around and made this 
> change:
> 
>
xen-3.4-testing.hg/linux-2.6.18-xen.hg/include/asm-x86_64/mach-xen/irq_vectors.h
> 
> I changed:
> 
> #define NR_DYNIRQS           1024
> 
> (it defaults to 256)
I''ve done the same thing several times without breaking anything. What
I
could not see is how much more expensive 1024 is than 256. Like you, I
had hosts with guests way north of 200 that did very little, so ...

Cheers,
--Tim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 20/02/2010 10:53, "Luke S Crawford" <lsc@prgmr.com> wrote:
> two questions:  first, is this stupid?  domains appear to be working, and
> I''m currently running north of 200 guests on the server.
> 
> second, if this is indeed the right way to fix the problem, if I
> submitted a CONFIG_NR_DYNIRQS patch as Kier suggested in the referenced
> post, would that be considered for acceptance upstream?
Yes, I''d take that for the 2.6.18 tree.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us> wrote:
>> #define NR_DYNIRQS           1024
>> 
>> (it defaults to 256)
> 
> I''ve done the same thing several times without breaking anything.
What I
> could not see is how much more expensive 1024 is than 256. Like you, I
> had hosts with guests way north of 200 that did very little, so ...
Yeah, could even just change to 1024 statically. I don''t think
it''s very
expensive.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sat, 2010-02-20 at 13:41 +0000, Keir Fraser wrote:
> On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us> wrote:
> 
> >> #define NR_DYNIRQS           1024
> >> 
> >> (it defaults to 256)
> > 
> > I''ve done the same thing several times without breaking
anything. What I
> > could not see is how much more expensive 1024 is than 256. Like you, I
> > had hosts with guests way north of 200 that did very little, so ...
> 
> Yeah, could even just change to 1024 statically. I don''t think
it''s very
> expensive.
> 
>  -- Keir
I think xenstat is capped way below that, so changing this is probably
going to require some love there too, else xentop might not show all
guests.  


AFAIK, it was 256, ICBW, I''ll check when I get back to my desk. I
mention it so this doesn''t go in without checking.

Cheers,
--Tim



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

IIRC from a preliminary investigation, there are some
memory allocations for which the size is determined by
multiplying by number of dynirqs.  I don''t recall whether
this already results in a ("post-boottime") order>0 allocation,
but quadrupling the number might.

Not necessarily a big problem (yet), I''m just being
watchful to see that the fragmentation problem due to
dynamic order>0 allocations doesn''t continue to get worse.
> -----Original Message-----
> From: Tim Post [mailto:echo@echoreply.us]
> Sent: Saturday, February 20, 2010 8:44 AM
> To: Keir Fraser
> Cc: Luke S Crawford; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] increasing the number of guests (NR_DYNIRQS)
> 
> On Sat, 2010-02-20 at 13:41 +0000, Keir Fraser wrote:
> > On 20/02/2010 11:54, "Tim Post" <echo@echoreply.us>
wrote:
> >
> > >> #define NR_DYNIRQS           1024
> > >>
> > >> (it defaults to 256)
> > >
> > > I''ve done the same thing several times without breaking
anything.
> What I
> > > could not see is how much more expensive 1024 is than 256. Like
> you, I
> > > had hosts with guests way north of 200 that did very little, so
...
> >
> > Yeah, could even just change to 1024 statically. I don''t
think it''s
> very
> > expensive.
> >
> >  -- Keir
> 
> I think xenstat is capped way below that, so changing this is probably
> going to require some love there too, else xentop might not show all
> guests.
> 
> 
> AFAIK, it was 256, ICBW, I''ll check when I get back to my desk. I
> mention it so this doesn''t go in without checking.
> 
> Cheers,
> --Tim
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel