I''ve finally hit the NR_IRQ limit in Xen 3.2-1. I''d like to increase this, which I suspect is as simple as increasing NR_VECTORS in include/ asm-x86/mach-default/irq_vectors.h; has anyone else had to do this, is this all that I''ll need to do, or will I run into other issues? I will need to increase the limit from its default of 256 to 1536. My other question is, will I need to modify or otherwise recompile my Linux kernel (2.8.18-xen.hg, rev=416)? As a suggestion, I''d really like to see this limit user-configurable at compile time, without having to edit the source... my server is only at approximately 25% of its expected capacity, so I''d imagine that even those with smaller machines could be hitting this limit. Thank you, Eric Windisch GrokThis.net PH: 1-800-630-0476 IM: support@grokthis.net (jabber/xmpp) IM: grokthissupport (AIM) IM: grokthisnet (Yahoo) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Eric, I think you are trying to boot large numbers of VMs. In that case, you will need to change the NR_DYNIRQS value in ./linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/irq_vectors.h and recompile your Dom0. You might also find it useful to increase Xen''s heap (using the boot time option xenheap_megabytes). Raj>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of >Eric Windisch >Sent: Thursday, February 28, 2008 1:37 PM >To: xen-users@lists.xensource.com >Subject: [Xen-devel] NR_IRQ limit > >(Originally posted to -users but I received no response there. I >figure this question might be better suited to those on -devel >anyway...) > >I''ve finally hit the NR_IRQ limit in Xen 3.2-1. I''d like to >increase this, which I suspect is as simple as increasing >NR_VECTORS in include/ >asm-x86/mach-default/irq_vectors.h. Has anyone else had to do this, >is this all that I''ll need to do, or will I run into other issues? I >will need to increase the limit from its default of 256 to 1536. > >My other question is, will I need to modify or otherwise >recompile my Linux kernel (2.8.18-xen.hg, rev=416)? > >As a suggestion, I''d really like to see this limit >user-configurable at compile time, without having to edit the >source... my server is only at approximately 25% of its >expected capacity, so I''d imagine that even those with smaller >machines could be hitting this limit. >Particularly since I built such a bulky (and expensive) >server, I''d really like to have Xen scale... > >Thank you, > >Eric Windisch >GrokThis.net > >PH: 1-800-630-0476 >IM: support@grokthis.net (jabber/xmpp) >IM: grokthissupport (AIM) >IM: grokthisnet (Yahoo) > > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Feb 29, 2008, at 5:13 PM, Subrahmanian, Raj wrote:> Eric, > > In that case, you will need to change the NR_DYNIRQS value in ./ > linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/irq_vectors.h and > recompile your Dom0.Thank you. I assume I''ll still have to change NR_VECTORS in Xen itself, right?> I think you are trying to boot large numbers of VMs.I''m currently booting 80 guests, each with two vdb, one vif. I am prevented from creating more at this time due to the limited number of IRQs (80*3=240 + various others for console/xenbus/etc). Ultimately, I''d like to try bringing this up to 160, or even 320 guests if the hardware allows. My kernels are compiling now, I''ll test them out on my test machine, and hopefully I''ll be able to get these changes into production tomorrow. Regards, Eric Windisch GrokThis.net PH: 1-800-630-0476 IM: support@grokthis.net (jabber/xmpp) IM: grokthissupport (AIM) IM: grokthisnet (Yahoo) _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Eric, I don''t think that you need to change the NR_VECTORS value in Xen. My understanding is that these IRQs are being doled out of Dom0, and not Xen itself. Incidentally, we did run 320 VMs here at Unisys. Any reason you are shooting for 320 in particular? Also, as you increase the number of VMs, you will run out of Xen''s heap (which is set at 10 or 12 MB), soon and then you will need to change that with the boot option. We set it at 64 MB, just to be safe. Raj>-----Original Message----- >From: Eric Windisch [mailto:eric@grokthis.net] >Sent: Friday, February 29, 2008 5:52 PM >To: Subrahmanian, Raj >Cc: xen-users@lists.xensource.com >Subject: Re: [Xen-devel] NR_IRQ limit > > >On Feb 29, 2008, at 5:13 PM, Subrahmanian, Raj wrote: > >> Eric, >> >> In that case, you will need to change the NR_DYNIRQS value in ./ >> linux-2.6-xen-sparse/include/asm-x86_64/mach-xen/irq_vectors.h and >> recompile your Dom0. > >Thank you. I assume I''ll still have to change NR_VECTORS in >Xen itself, right? > >> I think you are trying to boot large numbers of VMs. > > >I''m currently booting 80 guests, each with two vdb, one vif. I am >prevented from creating more at this time due to the limited >number of IRQs (80*3=240 + various others for >console/xenbus/etc). Ultimately, I''d like to try bringing >this up to 160, or even 320 guests if the hardware allows. > >My kernels are compiling now, I''ll test them out on my test >machine, and hopefully I''ll be able to get these changes into >production tomorrow. > >Regards, > >Eric Windisch >GrokThis.net > >PH: 1-800-630-0476 >IM: support@grokthis.net (jabber/xmpp) >IM: grokthissupport (AIM) >IM: grokthisnet (Yahoo) > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Feb 29, 2008, at 5:58 PM, Subrahmanian, Raj wrote:> Eric, > I don''t think that you need to change the NR_VECTORS value in Xen. > My understanding is that these IRQs are being doled out of Dom0, and > not Xen itself.Hmm, okay. I guess the less to change, the better.> > Incidentally, we did run 320 VMs here at Unisys. Any reason you are > shooting for 320 in particular?Each guest has 64MB-256MB of ram. We plan to put 32GB of ram into this server. Considering we''re slightly above 25% capacity with 80 guests, I''m guessing 4*80 = 320 is a good estimate of where we will be at 100% capacity (in terms of RAM anyway).> Also, as you increase the number of VMs, you will run out of Xen''s > heap (which is set at 10 or 12 MB), soon and then you will need to > change that with the boot option. We set it at 64 MB, just to be safe.It is 16MB on x86_64, according to the docs. I''ll increase to 64MB as well, from what I''ve read I think ~320 guests will need at least 32MB. Thanks again, Eric Windisch GrokThis.net PH: 1-800-630-0476 IM: support@grokthis.net (jabber/xmpp) IM: grokthissupport (AIM) IM: grokthisnet (Yahoo) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel