Hello, Can somebody explain why the number of CPUs Xen can handle is limited to 32? I have a machine with 48 cores, and would love to virtualise it, but if I do it, I loose 16 CPUs, which isnt very good... :-( Where is this limit coming from? Or maybe it is possible to change it somewhere in the source files? Cheers, Martin This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200. This communication and any attachments contain information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it. E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept liability for any such matters or their consequences. Anyone who communicates with us by e-mail is taken to accept the risks in doing so. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Mar 11, 2010 at 04:08:38PM +0000, Martin Lukasik wrote:> Hello, > > Can somebody explain why the number of CPUs Xen can handle is limited to 32? > I have a machine with 48 cores, and would love to virtualise it, but if > I do it, I loose 16 CPUs, which isnt very good... :-( > Where is this limit coming from? Or maybe it is possible to change it > somewhere in the source files? >I think the number of virtual cpus (vcpus) per guest is (was) 32, and now in Xen 4.0.0 you can have up to 64 vcpus per guest. The max number of physical cpus is 128 or so.. -- Pasi> Cheers, > Martin > > > This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200. > > This communication and any attachments contain information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it. > > E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept liability for any such matters or their consequences. Anyone who communicates with us by e-mail is taken to accept the risks in doing so. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 11.03.10 17:08 >>> >Can somebody explain why the number of CPUs Xen can handle is limited to 32? >I have a machine with 48 cores, and would love to virtualise it, but if >I do it, I loose 16 CPUs, which isnt very good... :-( >Where is this limit coming from? Or maybe it is possible to change it >somewhere in the source files?This is done on the make command line, by adding "max_phys_cpus=...". Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich wrote:>>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 11.03.10 17:08 >>> >>>> >> Can somebody explain why the number of CPUs Xen can handle is limited to 32? >> I have a machine with 48 cores, and would love to virtualise it, but if >> I do it, I loose 16 CPUs, which isnt very good... :-( >> Where is this limit coming from? Or maybe it is possible to change it >> somewhere in the source files? >> > > This is done on the make command line, by adding "max_phys_cpus=...". > > Jan >Yeah... I thought that installing it from RPM wasn''t a good idea... ;-) Thanks for the info. Will give it a try some time this week. BTW.: Wouldn''t it make sense to include support for max number of CPUs when building RPMs? Martin -- ----------------------------------------------------------------------- Martin Lukasik tel: +44 20 7269 3115 Bioinformatics Officer Bioinformatics and Biostatistics Cancer Research UK, London Research Institute 44 Lincoln''s Inn Fields, London, WC2A 3PX, UK ----------------------------------------------------------------------- This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200. This communication and any attachments contain information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it. E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept liability for any such matters or their consequences. Anyone who communicates with us by e-mail is taken to accept the risks in doing so. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 29.03.10 18:03 >>> >BTW.: Wouldn''t it make sense to include support for max number of CPUs >when building RPMs?Sure - anyone can easily do so in their .spec file. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Latest Xen supports up to 128 CPUs by default. Earlier versions you can try rebuilding (for example): "make xen max_phys_cpus=64". -- Keir On 11/03/2010 17:08, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:> Hello, > > Can somebody explain why the number of CPUs Xen can handle is limited to 32? > I have a machine with 48 cores, and would love to virtualise it, but if > I do it, I loose 16 CPUs, which isnt very good... :-( > Where is this limit coming from? Or maybe it is possible to change it > somewhere in the source files? > > Cheers, > Martin > > > This communication is from Cancer Research UK. Our website is at > www.cancerresearchuk.org. We are a charity registered under number 1089464 and > a company limited by guarantee registered in England & Wales under number > 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. > Our central telephone number is 020 7242 0200. > > This communication and any attachments contain information which is > confidential and may also be privileged. It is for the exclusive use of the > intended recipient(s). If you are not the intended recipient(s) please note > that any form of disclosure, distribution, copying or use of this > communication or the information in it or in any attachments is strictly > prohibited and may be unlawful. If you have received this communication in > error, please notify the sender and delete the email and destroy any copies of > it. > > E-mail communications cannot be guaranteed to be secure or error free, as > information could be intercepted, corrupted, amended, lost, destroyed, arrive > late or incomplete, or contain viruses. We do not accept liability for any > such matters or their consequences. Anyone who communicates with us by e-mail > is taken to accept the risks in doing so. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>> >I think the number of virtual cpus (vcpus) per guest is (was) 32, >and now in Xen 4.0.0 you can have up to 64 vcpus per guest.I don''t think the tools are up to anything beyond 32 yet. The (64-bit) hypervisor allows up to 8192 vCPU-s iirc (but that''s a truly theoretical limit, as Dom0 or a guest likely won''t be able to bring up that many due to there only being 4096 event channels; current Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote:> >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>> > >I think the number of virtual cpus (vcpus) per guest is (was) 32, > >and now in Xen 4.0.0 you can have up to 64 vcpus per guest. > > I don''t think the tools are up to anything beyond 32 yet.Hmm.. what''s actually missing from the tools to handle >32 vcpus?> The (64-bit) hypervisor allows up to 8192 vCPU-s iirc (but that''s a truly > theoretical limit, as Dom0 or a guest likely won''t be able to bring up > that many due to there only being 4096 event channels; current > Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ). >Wow.. I wasn''t aware the (theoretical) limit is that big :) -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Pasi Kärkkäinen<pasik@iki.fi> 30.03.10 09:53 >>> >On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote: >> >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>> >> >I think the number of virtual cpus (vcpus) per guest is (was) 32, >> >and now in Xen 4.0.0 you can have up to 64 vcpus per guest. >> >> I don''t think the tools are up to anything beyond 32 yet. > >Hmm.. what''s actually missing from the tools to handle >32 vcpus?Any place referencing XEN_LEGACY_MAX_VCPUS needs revisiting (seems like the one obvious place that needs fixing is in xc_domain_restore()). Beyond that I really don''t know - I just know too little about the tools. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen wrote:> I think the number of virtual cpus (vcpus) per guest is (was) 32, > and now in Xen 4.0.0 you can have up to 64 vcpus per guest. > > The max number of physical cpus is 128 or so.. >When I said "CPUs" I meant the number of physical CPUs (or strictly speaking: cores) visible by Dom0. I thought I got my RPM from RedHat''s repo, but there is a possibility that I''m wrong here... and I thought that Xen won''t support more than 32 cores. I''m not really after setting up 48 cores for one virtual machine, just wanted to be able to have all 48 visible in Dom0 (and then have up to 16 per VM). Thanks for the answers; interesting read. Kind regards, Martin -- ----------------------------------------------------------------------- Martin Lukasik tel: +44 20 7269 3115 Bioinformatics Officer Bioinformatics and Biostatistics Cancer Research UK, London Research Institute 44 Lincoln''s Inn Fields, London, WC2A 3PX, UK ----------------------------------------------------------------------- This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200. This communication and any attachments contain information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it. E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept liability for any such matters or their consequences. Anyone who communicates with us by e-mail is taken to accept the risks in doing so. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30/03/2010 10:16, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:> I''m not really after setting up 48 cores for one virtual machine, just > wanted to be able to have all 48 visible in Dom0 (and then have up to 16 > per VM).Dom0 is itself a VM. Though of course it doesn''t need tools support to set it up since it gets auto-constructed by the hypervisor itself. But it''s subject to hypervisor limits just like any other VM. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 30, 2010 at 10:16:36AM +0100, Martin Lukasik wrote:> Pasi Kärkkäinen wrote: >> I think the number of virtual cpus (vcpus) per guest is (was) 32, >> and now in Xen 4.0.0 you can have up to 64 vcpus per guest. >> >> The max number of physical cpus is 128 or so.. >> > > When I said "CPUs" I meant the number of physical CPUs (or strictly > speaking: cores) visible by Dom0. > I thought I got my RPM from RedHat''s repo, but there is a possibility > that I''m wrong here... and I thought that Xen won''t support more than 32 > cores. > I''m not really after setting up 48 cores for one virtual machine, just > wanted to be able to have all 48 visible in Dom0 (and then have up to 16 > per VM). > > Thanks for the answers; interesting read. >Why do you need to have all the cores visible to dom0? It''s usually enough to have a couple of cores for dom0, and the rest free/available in Xen hypervisor for other guests. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen wrote:> > Why do you need to have all the cores visible to dom0? > > It''s usually enough to have a couple of cores for dom0, > and the rest free/available in Xen hypervisor for other guests. > >I read three books about Xen, but surely they weren''t a good read. Many things haven''t been even mentioned (btw: can anyone recommend a *good* book on Xen?) I thought that the way it works is: you set up Dom0 with max number of CPUs and then allocate them to VMs. But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor. So I need a hypervisor to support 48 cores, then set up Dom0 with let''s say 2 vCPUs and 1GB of RAM, and create other VMs according to my requirements. Am I right? Thank you, Martin -- ----------------------------------------------------------------------- Martin Lukasik tel: +44 20 7269 3115 Bioinformatics Officer Bioinformatics and Biostatistics Cancer Research UK, London Research Institute 44 Lincoln''s Inn Fields, London, WC2A 3PX, UK ----------------------------------------------------------------------- This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln''s Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200. This communication and any attachments contain information which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful. If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it. E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses. We do not accept liability for any such matters or their consequences. Anyone who communicates with us by e-mail is taken to accept the risks in doing so. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Mar 30, 2010 at 10:32:29AM +0100, Martin Lukasik wrote:> Pasi Kärkkäinen wrote: >> >> Why do you need to have all the cores visible to dom0? >> >> It''s usually enough to have a couple of cores for dom0, and the rest >> free/available in Xen hypervisor for other guests. >> >> > > I read three books about Xen, but surely they weren''t a good read. Many > things haven''t been even mentioned (btw: can anyone recommend a *good* > book on Xen?) > I thought that the way it works is: you set up Dom0 with max number of > CPUs and then allocate them to VMs. > But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor. > > So I need a hypervisor to support 48 cores, then set up Dom0 with let''s > say 2 vCPUs and 1GB of RAM, and create other VMs according to my > requirements. > Am I right? >Yes, that''s correct. See here for some tips: http://wiki.xensource.com/xenwiki/XenBestPractices -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 30/03/2010 10:32, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:> I read three books about Xen, but surely they weren''t a good read. Many > things haven''t been even mentioned (btw: can anyone recommend a *good* > book on Xen?) > I thought that the way it works is: you set up Dom0 with max number of > CPUs and then allocate them to VMs. > But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor. > > So I need a hypervisor to support 48 cores, then set up Dom0 with let''s > say 2 vCPUs and 1GB of RAM, and create other VMs according to my > requirements. > Am I right?Yeah, that would be sensible. All dom0 is doing is VM management (i.e., running control tools like xend/xm) and also handling I/O, so two cores is probably plenty. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: Pasi Kärkkäinen [mailto:pasik@iki.fi] > Sent: Tuesday, March 30, 2010 1:53 AM > To: Jan Beulich > Cc: xen-devel@lists.xensource.com; Martin Lukasik > Subject: Re: [Xen-devel] Xen CPU limit? > > On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote: > > >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>> > > >I think the number of virtual cpus (vcpus) per guest is (was) 32, > > >and now in Xen 4.0.0 you can have up to 64 vcpus per guest. > > > > I don''t think the tools are up to anything beyond 32 yet. > > Hmm.. what''s actually missing from the tools to handle >32 vcpus? > > > The (64-bit) hypervisor allows up to 8192 vCPU-s iirc (but that''s a > truly > > theoretical limit, as Dom0 or a guest likely won''t be able to bring > up > > that many due to there only being 4096 event channels; current > > Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ). > > > > Wow.. I wasn''t aware the (theoretical) limit is that big :)But, if I am reading this correctly, 4096 event channels is the next scalability barrier. If, on a "big" machine, one tries to run 64 (nearly always idle) guests each configured with 16 vcpus (because they run a busy database load when they are not idle), it won''t work due to the event channel limit? This scenario seems quite possible in a cloud/hosting environment. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>>> Dan Magenheimer <dan.magenheimer@oracle.com> 30.03.10 18:14 >>> >But, if I am reading this correctly, 4096 event channels >is the next scalability barrier. If, on a "big" machine, one >tries to run 64 (nearly always idle) guests each configured >with 16 vcpus (because they run a busy database load when >they are not idle), it won''t work due to the event channel limit?No, the event channel limit is a per-domain one. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel