Hi all, I''ve read a few posts about this and have pinned my domUs. I''ve a dual quad core Opteron. I''ve also set dom0-cpus 1 in my xend-config file. xm vcpu-list shows Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 59582.6 any cpu Domain-0 0 1 - --p 13227.9 any cpu Domain-0 0 2 - --p 13527.5 any cpu Domain-0 0 3 - --p 12448.3 any cpu Domain-0 0 4 - --p 13520.6 any cpu Domain-0 0 5 - --p 17223.0 any cpu Domain-0 0 6 - --p 12614.1 any cpu Domain-0 0 7 - --p 13570.9 any cpu bugs 24 0 2 -b- 598.5 2 logantools 26 0 3 -b- 147.8 3 nello 27 0 5 -b- 246.7 5 zmail 25 0 6 -b- 8064.4 6 zmail 25 1 7 r-- 7681.2 7 Every few seconds, the first occurrence of Domain-0 changes to 4, then back to 0. Why? If I change dom0-cpus 2 so that it uses 2 cpus, xend won''t restart and errors; [2009-10-21 10:36:54 26309] INFO (XendDomainInfo:157) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0 [2009-10-21 10:36:54 26309] DEBUG (XendDomain:452) Adding Domain: 0 [2009-10-21 10:36:54 26309] DEBUG (XendDomain:386) number of vcpus to use is 2 [2009-10-21 10:36:54 26309] ERROR (SrvDaemon:349) Exception starting xend (Cannot set vcpus greater than max vcpus on running domain) Traceback (most recent call last): So it appears I don''t understand cpu pinning when it comes to dom0. - Brian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Brian, You need to set this in your kernel config as well as xend-config.sxp, eg. add the following to your kernel line in grub.conf: dom0_max_vcpus=2 Set dom0-cpus to 2 in xend-config.sxp as well and restart Dom0, should behave as you expect then. Rob ________________________________ From: xen-users-bounces@lists.xensource.com on behalf of Brian Krusic Sent: Wed 21/10/2009 18:39 To: Xen Users Subject: [Xen-users] xen cpu affinity, ie; pinning cpus Hi all, I''ve read a few posts about this and have pinned my domUs. I''ve a dual quad core Opteron. I''ve also set dom0-cpus 1 in my xend-config file. xm vcpu-list shows Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 59582.6 any cpu Domain-0 0 1 - --p 13227.9 any cpu Domain-0 0 2 - --p 13527.5 any cpu Domain-0 0 3 - --p 12448.3 any cpu Domain-0 0 4 - --p 13520.6 any cpu Domain-0 0 5 - --p 17223.0 any cpu Domain-0 0 6 - --p 12614.1 any cpu Domain-0 0 7 - --p 13570.9 any cpu bugs 24 0 2 -b- 598.5 2 logantools 26 0 3 -b- 147.8 3 nello 27 0 5 -b- 246.7 5 zmail 25 0 6 -b- 8064.4 6 zmail 25 1 7 r-- 7681.2 7 Every few seconds, the first occurrence of Domain-0 changes to 4, then back to 0. Why? If I change dom0-cpus 2 so that it uses 2 cpus, xend won''t restart and errors; [2009-10-21 10:36:54 26309] INFO (XendDomainInfo:157) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0 [2009-10-21 10:36:54 26309] DEBUG (XendDomain:452) Adding Domain: 0 [2009-10-21 10:36:54 26309] DEBUG (XendDomain:386) number of vcpus to use is 2 [2009-10-21 10:36:54 26309] ERROR (SrvDaemon:349) Exception starting xend (Cannot set vcpus greater than max vcpus on running domain) Traceback (most recent call last): So it appears I don''t understand cpu pinning when it comes to dom0. - Brian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users The SAQ Group Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ SAQ is the trading name of SEMTEC Limited. Registered in England & Wales Company Number: 06481952 http://www.saqnet.co.uk AS29219 SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business. Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support. ISPA Member _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Robert, I did the grub mods. Here what I have now; Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 216.6 any cpu fm 0 0.0 1 intranet 0 0.0 2 michel 3 0 3 -b- 10.1 3 michel 3 1 4 -b- 6.6 4 vault 1 0 5 -b- 23.2 5 zello 2 0 6 -b- 21.0 6 zello 2 1 7 -b- 20.1 7 Why does Domain-0 say any cpu under CPU Affinity? I still notice it changes from 0 to 4 to 0 to 3 to ... Functions much better overall. Just curious how to properly interpret the behavior. - Brian On Oct 21, 2009, at 12:03 PM, Robert Dunkley wrote:> Hi Brian, > > You need to set this in your kernel config as well as xend- > config.sxp, eg. add the following to your kernel line in grub.conf: > dom0_max_vcpus=2 > > Set d cpu affinity, ie; pinning cpus > > Hi all, > > I''ve read a few posts about this and have pinned my domUs. > > I''ve a dual quad core Opteron. > > I''ve also set dom0-cpus 1 in my xend-config file. > > xm vcpu-list shows > > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 r-- 59582.6 any > cpu > Domain-0 0 1 - --p 13227.9 any > cpu > Domain-0 0 2 - --p 13527.5 any > cpu > Domain-0 0 3 - --p 12448.3 any > cpu > Domain-0 0 4 - --p 13520.6 any > cpu > Domain-0 0 5 - --p 17223.0 any > cpu > Domain-0 0 6 - --p 12614.1 any > cpu > Domain-0 0 7 - --p 13570.9 any > cpu > bugs 24 0 2 -b- 598.5 2 > logantools 26 0 3 -b- 147.8 3 > nello 27 0 5 -b- 246.7 5 > zmail 25 0 6 -b- 8064.4 6 > zmail 25 1 7 r-- 7681.2 7 > > Every few seconds, the first occurrence of Domain-0 changes to 4, then > back to 0. > > Why? > > If I change dom0-cpus 2 so that it uses 2 cpus, xend won''t restart and > errors; > > [2009-10-21 10:36:54 26309] INFO (XendDomainInfo:157) Recreating > domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/ > domain/0 > [2009-10-21 10:36:54 26309] DEBUG (XendDomain:452) Adding Domain: 0 > [2009-10-21 10:36:54 26309] DEBUG (XendDomain:386) number of vcpus to > use is 2 > [2009-10-21 10:36:54 26309] ERROR (SrvDaemon:349) Exception starting > xend (Cannot set vcpus greater than max vcpus on running domain) > Traceback (most recent call last): > > So it appears I don''t understand cpu pinning when it comes to dom0. > > - Brian > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > The SAQ Group > Registered Office: 18 Chapel Street, Petersfield, Hampshire GU32 3DZ > SAQ is the trading name of SEMTEC Limited. Registered in England & > Wales > Company Number: 06481952 > > http://www.saqnet.co.uk AS29219 > SAQ Group Delivers high quality, honestly priced communication and > I.T. services to UK Business. > Broadband : Domains : Email : Hosting : CoLo : Servers : Racks : > Transit : Backups : Managed Networks : Remote Support. > > > > ISPA Member > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pekka.Panula@sofor.fi
2009-Oct-22 06:17 UTC
RE: [Xen-users] xen cpu affinity, ie; pinning cpus
> You need to set this in your kernel config as well as xend-config.sxp,eg.> add the following to your kernel line in grub.conf: > dom0_max_vcpus=2For best practices use, is it better to limit dom0 cpu count to 1? Eg. if i have 2*quad core, is there harm to let dom-0 to see all cpus or is it best to limit dom-0 count to for example to 1 cpu? I am currently limiting dom0 memory to 512MB using dom0_mem parameter, i also set sched credit schedurer weight to 6000 for dom-0 and for DomU''s i keep weight atm at 256. I would like to know how you are doing things, eg. pinning, dom-0 cpu count, dom-0 memory limits and weight distribution? Terveisin/Regards, Pekka Panula, Sofor Oy - Jatkuvat palvelut _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi, Yes, i read several article about having dom0 pinned to 1 cpu. Funny thing is that I do this via xend-config-sxp and grub-conf but xm vcpu-list still shows dom0 as "any" under affinity column. When I run xm vcpu-pin Domain-0 0 0, xm vcpu-list then shows "0" under the affinity column until I reboot. Any ideas? - Brian On Oct 21, 2009, at 11:17 PM, Pekka.Panula@sofor.fi wrote:> > You need to set this in your kernel config as well as xend- > config.sxp, eg. > > add the following to your kernel line in grub.conf: > > dom0_max_vcpus=2 > > For best practices use, is it better to limit dom0 cpu count to 1? > Eg. if i have 2*quad core, is there harm to let dom-0 to see all > cpus or is it best to limit dom-0 count to for example to 1 cpu? > I am currently limiting dom0 memory to 512MB using dom0_mem > parameter, i also set sched credit schedurer weight to 6000 for > dom-0 and for DomU''s i keep weight atm at 256. > I would like to know how you are doing things, eg. pinning, dom-0 > cpu count, dom-0 memory limits and weight distribution? > > Terveisin/Regards, > Pekka Panula, Sofor Oy - Jatkuvat palvelut > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, 2009-10-22 at 10:25 -0700, Brian Krusic wrote:> Hi, > > > Yes, i read several article about having dom0 pinned to 1 cpu. > > > Funny thing is that I do this via xend-config-sxp and grub-conf but xm > vcpu-list still shows dom0 as "any" under affinity column. > > > When I run xm vcpu-pin Domain-0 0 0, xm vcpu-list then shows "0" under > the affinity column until I reboot. > > > Any ideas? >After setting (dom0-cpus 1) in /etc/xen/xend-config.sxp did you restart the xend service? If I remember correctly, it should be safe to do when domU''s are running. This should pin Domain-0 to cpu0. eg. [root@xxxxxx ]# xm vcpu-list Name ID VCPUs CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 1648038.3 0 Domain-0 0 1 - --p 3.1 1 Domain-0 0 2 - --p 3.9 2 Domain-0 0 3 - --p 3.8 3 Domain-0 0 4 - --p 3.7 4 Domain-0 0 5 - --p 3.1 5 Domain-0 0 6 - --p 3.2 6 Domain-0 0 7 - --p 3.6 7 xendomainhere 1 0 4 -b- 300000.1 any cpu That shows that my vm host has 8 cores, but is only using one for Domain-0.. the rest are in a paused state, so now what I can do is pin them from the config files to use a core from cpus 1-7 or pin the domU to a specific one as well. Tait _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Oct 22, 2009, at 12:13 PM, Tait Clarridge wrote:> On Thu, 2009-10-22 at 10:25 -0700, Brian Krusic wrote: >> Hi, >> >> >> Yes, i read several article about having dom0 pinned to 1 cpu. >> >> >> Funny thing is that I do this via xend-config-sxp and grub-conf but >> xm >> vcpu-list still shows dom0 as "any" under affinity column. >> >> >> When I run xm vcpu-pin Domain-0 0 0, xm vcpu-list then shows "0" >> under >> the affinity column until I reboot. >> >> >> Any ideas? >> > > After setting (dom0-cpus 1) in /etc/xen/xend-config.sxp did you > restart > the xend service? > > If I remember correctly, it should be safe to do when domU''s are > running. > > This should pin Domain-0 to cpu0. > > eg. > > [root@xxxxxx ]# xm vcpu-list > Name ID VCPUs CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 r-- 1648038.3 0 > Domain-0 0 1 - --p 3.1 1 > Domain-0 0 2 - --p 3.9 2 > Domain-0 0 3 - --p 3.8 3 > Domain-0 0 4 - --p 3.7 4 > Domain-0 0 5 - --p 3.1 5 > Domain-0 0 6 - --p 3.2 6 > Domain-0 0 7 - --p 3.6 7 > xendomainhere 1 0 4 -b- 300000.1 any cpu > > That shows that my vm host has 8 cores, but is only using one for > Domain-0.. the rest are in a paused state, so now what I can do is pin > them from the config files to use a core from cpus 1-7 or pin the domU > to a specific one as well. > > TaitHi, Heres my grub.conf; kernel /boot/xen.gz-3.4.1 dom0_max_vcpus=1 dom0_mem=1024M xend-config.sxp; (dom0-cpus 1) Heres my xm vcpu-list after a reboot of the system; Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 216.6 any cpu Heres my xm vcpu-list after xm vcpu-pin; Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 r-- 12585.6 0 I tired to format this nice for you so please excuse if it come out like crap. Notice the the CPU Affinity column. Any ideas? - Brian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, 2009-10-22 at 12:24 -0700, Brian Krusic wrote:> On Oct 22, 2009, at 12:13 PM, Tait Clarridge wrote: > > > On Thu, 2009-10-22 at 10:25 -0700, Brian Krusic wrote: > >> Hi, > >> > >> > >> Yes, i read several article about having dom0 pinned to 1 cpu. > >> > >> > >> Funny thing is that I do this via xend-config-sxp and grub-conf but > >> xm > >> vcpu-list still shows dom0 as "any" under affinity column. > >> > >> > >> When I run xm vcpu-pin Domain-0 0 0, xm vcpu-list then shows "0" > >> under > >> the affinity column until I reboot. > >> > >> > >> Any ideas? > >> > > > > After setting (dom0-cpus 1) in /etc/xen/xend-config.sxp did you > > restart > > the xend service? > > > > If I remember correctly, it should be safe to do when domU''s are > > running. > > > > This should pin Domain-0 to cpu0. > > > > eg. > > > > [root@xxxxxx ]# xm vcpu-list > > Name ID VCPUs CPU State Time(s) CPU > > Affinity > > Domain-0 0 0 0 r-- 1648038.3 0 > > Domain-0 0 1 - --p 3.1 1 > > Domain-0 0 2 - --p 3.9 2 > > Domain-0 0 3 - --p 3.8 3 > > Domain-0 0 4 - --p 3.7 4 > > Domain-0 0 5 - --p 3.1 5 > > Domain-0 0 6 - --p 3.2 6 > > Domain-0 0 7 - --p 3.6 7 > > xendomainhere 1 0 4 -b- 300000.1 any cpu > > > > That shows that my vm host has 8 cores, but is only using one for > > Domain-0.. the rest are in a paused state, so now what I can do is pin > > them from the config files to use a core from cpus 1-7 or pin the domU > > to a specific one as well. > > > > Tait > > Hi, > > Heres my grub.conf; > > kernel /boot/xen.gz-3.4.1 dom0_max_vcpus=1 dom0_mem=1024M > > xend-config.sxp; > > (dom0-cpus 1) > > Heres my xm vcpu-list after a reboot of the system; > > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 r-- 216.6 any cpu > > > Heres my xm vcpu-list after xm vcpu-pin; > > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 r-- 12585.6 0 > > I tired to format this nice for you so please excuse if it come out > like crap. > > Notice the the CPU Affinity column. > > Any ideas? >Well it looks like it is still grabbing cpu0 for Domain-0 to use. Does it ever use any other CPU for dom0 before pinning it manually? I am using the default xen version from CentOS 5.3 so maybe this is how 3.4 addresses pinning? Can anyone running xen 3.4 confirm that pinning leads the CPU Affinity section to display "any cpu"? _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Oct 22, 2009, at 1:13 PM, Tait Clarridge wrote:> On Thu, 2009-10-22 at 12:24 -0700, Brian Krusic wrote: >> On Oct 22, 2009, at 12:13 PM, Tait Clarridge wrote: >> >>> On Thu, 2009-10-22 at 10:25 -0700, Brian Krusic wrote: >>>> Hi, >>>> >>>> >>>> Yes, i read several article about having dom0 pinned to 1 cpu. >>>> >>>> >>>> Funny thing is that I do this via xend-config-sxp and grub-conf but >>>> xm >>>> vcpu-list still shows dom0 as "any" under affinity column. >>>> >>>> >>>> When I run xm vcpu-pin Domain-0 0 0, xm vcpu-list then shows "0" >>>> under >>>> the affinity column until I reboot. >>>> >>>> >>>> Any ideas? >>>> >>> >>> After setting (dom0-cpus 1) in /etc/xen/xend-config.sxp did you >>> restart >>> the xend service? >>> >>> If I remember correctly, it should be safe to do when domU''s are >>> running. >>> >>> This should pin Domain-0 to cpu0. >>> >>> eg. >>> >>> [root@xxxxxx ]# xm vcpu-list >>> Name ID VCPUs CPU State Time(s) CPU >>> Affinity >>> Domain-0 0 0 0 r-- 1648038.3 0 >>> Domain-0 0 1 - --p 3.1 1 >>> Domain-0 0 2 - --p 3.9 2 >>> Domain-0 0 3 - --p 3.8 3 >>> Domain-0 0 4 - --p 3.7 4 >>> Domain-0 0 5 - --p 3.1 5 >>> Domain-0 0 6 - --p 3.2 6 >>> Domain-0 0 7 - --p 3.6 7 >>> xendomainhere 1 0 4 -b- 300000.1 any cpu >>> >>> That shows that my vm host has 8 cores, but is only using one for >>> Domain-0.. the rest are in a paused state, so now what I can do is >>> pin >>> them from the config files to use a core from cpus 1-7 or pin the >>> domU >>> to a specific one as well. >>> >>> Tait >> >> Hi, >> >> Heres my grub.conf; >> >> kernel /boot/xen.gz-3.4.1 dom0_max_vcpus=1 dom0_mem=1024M >> >> xend-config.sxp; >> >> (dom0-cpus 1) >> >> Heres my xm vcpu-list after a reboot of the system; >> >> Name ID VCPU CPU State Time(s) CPU Affinity >> Domain-0 0 0 0 r-- 216.6 any cpu >> >> >> Heres my xm vcpu-list after xm vcpu-pin; >> >> Name ID VCPU CPU State Time(s) CPU Affinity >> Domain-0 0 0 0 r-- 12585.6 0 >> >> I tired to format this nice for you so please excuse if it come out >> like crap. >> >> Notice the the CPU Affinity column. >> >> Any ideas? >> > > Well it looks like it is still grabbing cpu0 for Domain-0 to use. Does > it ever use any other CPU for dom0 before pinning it manually?Yes, it moves around from proc to proc. Mostly 0, sometimes, 3,4,6, etc... Kinda weird. Just wondering how to interpret this. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Brian Krusic <brian@krusic.com> writes:> Heres my grub.conf; > > kernel /boot/xen.gz-3.4.1 dom0_max_vcpus=1 dom0_mem=1024M > > xend-config.sxp; > > (dom0-cpus 1) > > Heres my xm vcpu-list after a reboot of the system; > > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 r-- 216.6 any cpu > > > Heres my xm vcpu-list after xm vcpu-pin; > > Name ID VCPU CPU State Time(s) CPU Affinity > Domain-0 0 0 0 r-- 12585.6 0 > > I tired to format this nice for you so please excuse if it come out > like crap. > > Notice the the CPU Affinity column.here is my kernel: line kernel /boot/xen.gz dom0_mem=1024M dom0_max_vcpus=1 dom0_vcpus_pin=true console=com1 com1=115200,8n1 note the dom0_vcpus_pin=true - that''s the key bit; it means pin vcpu0 to cpu0, vcpu1 to cpu1, etc... once you have that, the dom0_max_vcpus setting isn''t so important, you can just set dom0_cpus or what have you in xend-config.sxp. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Oct 23, 2009, at 8:05 PM, Luke S Crawford wrote:> Brian Krusic <brian@krusic.com> writes: > >> Heres my grub.conf; >> >> kernel /boot/xen.gz-3.4.1 dom0_max_vcpus=1 dom0_mem=1024M >> >> xend-config.sxp; >> >> (dom0-cpus 1) >> >> Heres my xm vcpu-list after a reboot of the system; >> >> Name ID VCPU CPU State Time(s) CPU Affinity >> Domain-0 0 0 0 r-- 216.6 any cpu >> >> >> Heres my xm vcpu-list after xm vcpu-pin; >> >> Name ID VCPU CPU State Time(s) CPU Affinity >> Domain-0 0 0 0 r-- 12585.6 0 >> >> I tired to format this nice for you so please excuse if it come out >> like crap. >> >> Notice the the CPU Affinity column. > > here is my kernel: line > > kernel /boot/xen.gz dom0_mem=1024M dom0_max_vcpus=1 > dom0_vcpus_pin=true console=com1 com1=115200,8n1 > > > note the dom0_vcpus_pin=true - that''s the key bit; it means pin > vcpu0 to cpu0, vcpu1 to cpu1, etc... once you have that, the > dom0_max_vcpus setting isn''t so important, you can just set > dom0_cpus or > what have you in xend-config.sxp.Yep, that did it. Many many thanks Luke. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users