hi
let''s suppose i have something like that:
---
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 24842.7 any cpu
Domain-0 0 1 1 r-- 4306.2 any cpu
Domain-0 0 2 2 -b- 5171.3 any cpu
Domain-0 0 3 3 -b- 2013.0 any cpu
[snip]
trip 18 0 2 -b- 1189.8 any cpu
uni147 19 0 0 r-- 112415.0 0
uni147 19 1 1 -b- 35259.4 1
[snip]
---
trip have 1 vcpu, cpus blank ("").
uni147 have 2 vcpus, cpus = "0-3" (i have 4 opteron 844).
which is the difference between dom0 and uni147, or better why one have
"any
cpu" and the other have "0" and "1" in cpu affinity?
how xen can decide to allocate to one cpu or another?
is supported something like "if cpu 0 will get full, move vcpu 0 of uni147
to
cpu 3"?
and for trip?
am i wrong if i consider cpu0 the one that will get most of IO of the HW?
the cpu time of dom0 is affected by the usage of the domU for that real cpu?
thanks
Daniele
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Petersson, Mats
2007-Feb-08 10:08 UTC
RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Daniele Palumbo > Sent: 08 February 2007 09:47 > To: xen-users@lists.xensource.com > Subject: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > hi > > let''s suppose i have something like that: > --- > Name ID VCPU CPU State > Time(s) CPU Affinity > Domain-0 0 0 0 -b- > 24842.7 any cpu > Domain-0 0 1 1 r-- > 4306.2 any cpu > Domain-0 0 2 2 -b- > 5171.3 any cpu > Domain-0 0 3 3 -b- > 2013.0 any cpu > [snip] > trip 18 0 2 -b- > 1189.8 any cpu > uni147 19 0 0 r-- 112415.0 0 > uni147 19 1 1 -b- 35259.4 1 > [snip] > --- > > trip have 1 vcpu, cpus blank (""). > uni147 have 2 vcpus, cpus = "0-3" (i have 4 opteron 844). > > which is the difference between dom0 and uni147, or better > why one have "any > cpu" and the other have "0" and "1" in cpu affinity? > > how xen can decide to allocate to one cpu or another? > is supported something like "if cpu 0 will get full, move > vcpu 0 of uni147 to > cpu 3"? > and for trip?If you "hard-assign" the cpus in your config file, then that''s the CPU''s you get. If you don''t, Xen just assings "at random". Xen''s Credit scheduler, which is what''s in Xen by default since 3.0.2 (I think), can, and will, move VCPU''s between physical CPUs "for best utilization of the CPU resources". There''s no "automatic" move things around between physical CPU''s when you have hard-coded which CPU''s you want (at least not outside the set!). As to what works best for any particular setup, that''s a much harder to answer question - and one that you''ll probably be better placed to answer than I am.> > am i wrong if i consider cpu0 the one that will get most of > IO of the HW?That depends on the hardware on your motherboard. The AMD setup for IO is that each CPU has three HT-links, which can either be used for I/O or interprocessor communication, so each CPU can have it''s own set of hardware that it can access. However, in practice, most motherboards will attach most of the IO-devices to CPU0, since if you want to run the system without a full set of CPU''s, the first CPU is the one that you have to use... But there are systems with two PCI-X controllers, one for CPU0 and another for CPU1 or some such (in which case, you need to have two CPU''s to access all PCI-X slots). That doesn''t in itself mean that CPU0 is doing more work tho'', as CPU1, 2, 3 can all access the hardware that is attached to CPU0, just that it takes a fraction longer (but we''re talking about PCI access with a 33/66/133MHz bus, compared to the 800MHz-1GHz of the HT-link, so the latency is most likely not at all noticable - the overhead is roughly 40ns per hop, where CPU{1,3} will have 1 hop, CPU2 has two hops). Note that a PCI access lasts at least 6-7 cycles in general (200ns+), and not unusually longer than that.> the cpu time of dom0 is affected by the usage of the domU for > that real cpu?Dom0''s "usage" will be entirely based on what Dom0 is doing, but of course, if DomU is sharing time with DomU, then the time used by DomU is obviously not available to Dom0. Not sure what you''re asking here...> > > thanks > Daniele > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Daniele Palumbo
2007-Feb-08 10:38 UTC
Re: [Xen-users] vcpus and cpus, plus cpu affinity = ??
On Thursday 08 February 2007 11:08, Petersson, Mats wrote:> > the cpu time of dom0 is affected by the usage of the domU for > > that real cpu? > > Dom0''s "usage" will be entirely based on what Dom0 is doing, but of > course, if DomU is sharing time with DomU, then the time used by DomU is > obviously not available to Dom0. Not sure what you''re asking here...you have cpu time in list. is dom0 time in xm vcpu-list affected by domU times? bye d. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-Feb-08 11:06 UTC
RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Daniele Palumbo > Sent: 08 February 2007 10:38 > To: Petersson, Mats > Cc: xen-users@lists.xensource.com > Subject: Re: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > On Thursday 08 February 2007 11:08, Petersson, Mats wrote: > > > the cpu time of dom0 is affected by the usage of the domU for > > > that real cpu? > > > > Dom0''s "usage" will be entirely based on what Dom0 is doing, but of > > course, if DomU is sharing time with DomU, then the time > used by DomU is > > obviously not available to Dom0. Not sure what you''re asking here... > > you have cpu time in list. > > is dom0 time in xm vcpu-list affected by domU times?What do you mean by "affected". CPU0 seems to have most of the time for both Dom0 and uni147, so it''s hard to say if this is because for both domains, CPU0 is the one doing most of the work, but it''s rarely running 100%, or it wants to run 100% for any given domain, but can''t because the other domain is also using the CPU... Only by using "xm top" can you find out which CPU is used how much - if none are at 100% then there''s no problem of contention. Of course, another factor to consider is the overhead of a "world-switch". By giving Dom0 one physical CPU that isn''t shared with other domains, Dom0 is "always ready" on that CPU. This may of course waste some available CPU-time, so it''s a balance to be struck between reducing the overhead on Dom0 and getting work done on DomU''s. Since Dom0 has all four CPU''s available at this point, it''s clearly not getting completely starved of CPU-time, particularly as CPU3 is ONLY Dom0, and it''s got the lowest CPU-time of all CPU''s in Dom0 - which means to me that Dom0 isn''t using all the CPU-time it may use - a lot of time Dom0 is sitting idle. -- Mats> > bye > d. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Mats, You mentioned a very interesting feature of AMD CPU: The AMD setup for IO is that each CPU has three HT-links, which can either be used for I/O or interprocessor communication, so each CPU can have it''s own set of hardware that it can access. Three questions (please bear with me if I misunderstood you): 1. How to bind a HT-link in a physcial CPU to I/O or IPC? Did you do this in BIOS? 2. Do you know if there is any CPU from Intel supports such kind of feature? 3. Each CPU has three HT-links. Is the HT-link you mentioned the same as the Hyper Threads in Intel CPU? I heard most of CPUs only have two HT. Thanks, Liang ----- Original Message ----- From: "Petersson, Mats" <Mats.Petersson@amd.com> To: "Daniele Palumbo" <daniele@retaggio.net>; <xen-users@lists.xensource.com> Sent: Thursday, February 08, 2007 3:08 AM Subject: RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Daniele Palumbo > Sent: 08 February 2007 09:47 > To: xen-users@lists.xensource.com > Subject: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > hi > > let''s suppose i have something like that: > --- > Name ID VCPU CPU State > Time(s) CPU Affinity > Domain-0 0 0 0 -b- > 24842.7 any cpu > Domain-0 0 1 1 r-- > 4306.2 any cpu > Domain-0 0 2 2 -b- > 5171.3 any cpu > Domain-0 0 3 3 -b- > 2013.0 any cpu > [snip] > trip 18 0 2 -b- > 1189.8 any cpu > uni147 19 0 0 r-- 112415.0 0 > uni147 19 1 1 -b- 35259.4 1 > [snip] > --- > > trip have 1 vcpu, cpus blank (""). > uni147 have 2 vcpus, cpus = "0-3" (i have 4 opteron 844). > > which is the difference between dom0 and uni147, or better > why one have "any > cpu" and the other have "0" and "1" in cpu affinity? > > how xen can decide to allocate to one cpu or another? > is supported something like "if cpu 0 will get full, move > vcpu 0 of uni147 to > cpu 3"? > and for trip?If you "hard-assign" the cpus in your config file, then that''s the CPU''s you get. If you don''t, Xen just assings "at random". Xen''s Credit scheduler, which is what''s in Xen by default since 3.0.2 (I think), can, and will, move VCPU''s between physical CPUs "for best utilization of the CPU resources". There''s no "automatic" move things around between physical CPU''s when you have hard-coded which CPU''s you want (at least not outside the set!). As to what works best for any particular setup, that''s a much harder to answer question - and one that you''ll probably be better placed to answer than I am.> > am i wrong if i consider cpu0 the one that will get most of > IO of the HW?That depends on the hardware on your motherboard. The AMD setup for IO is that each CPU has three HT-links, which can either be used for I/O or interprocessor communication, so each CPU can have it''s own set of hardware that it can access. However, in practice, most motherboards will attach most of the IO-devices to CPU0, since if you want to run the system without a full set of CPU''s, the first CPU is the one that you have to use... But there are systems with two PCI-X controllers, one for CPU0 and another for CPU1 or some such (in which case, you need to have two CPU''s to access all PCI-X slots). That doesn''t in itself mean that CPU0 is doing more work tho'', as CPU1, 2, 3 can all access the hardware that is attached to CPU0, just that it takes a fraction longer (but we''re talking about PCI access with a 33/66/133MHz bus, compared to the 800MHz-1GHz of the HT-link, so the latency is most likely not at all noticable - the overhead is roughly 40ns per hop, where CPU{1,3} will have 1 hop, CPU2 has two hops). Note that a PCI access lasts at least 6-7 cycles in general (200ns+), and not unusually longer than that.> the cpu time of dom0 is affected by the usage of the domU for > that real cpu?Dom0''s "usage" will be entirely based on what Dom0 is doing, but of course, if DomU is sharing time with DomU, then the time used by DomU is obviously not available to Dom0. Not sure what you''re asking here...> > > thanks > Daniele > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 8 Feb 2007 at 10:34, Liang Yang wrote:> Hi Mats, > > You mentioned a very interesting feature of AMD CPU: The AMD setup for IO > is that each CPU has three HT-links, which can either be used for I/O or > interprocessor communication, so each CPU can have it''s own set of > hardware that it can access. > > Three questions (please bear with me if I misunderstood you): > 1. How to bind a HT-link in a physcial CPU to I/O or IPC? Did you do this in > BIOS? > > 2. Do you know if there is any CPU from Intel supports such kind of feature? > > 3. Each CPU has three HT-links. Is the HT-link you mentioned the same as the > Hyper Threads in Intel CPU? I heard most of CPUs only have two HT.Intel: HT == HyperThreading AMD: HT == Hypertransport AFAIK, Wikipedia.org should get you jump-started ;-) Also as HT(AMD) is hardware, it''s unlikely that you can re-route it through BIOS settings. It would be becoming slower if redirectable I guess. Ulrich _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Petersson, Mats
2007-Feb-09 10:01 UTC
RE: [Xen-users] vcpus and cpus, plus cpu affinity = ??
> -----Original Message----- > From: Liang Yang [mailto:multisyncfe991@hotmail.com] > Sent: 08 February 2007 17:35 > To: Petersson, Mats > Cc: Daniele Palumbo; xen-users@lists.xensource.com > Subject: Re: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > Hi Mats, > > You mentioned a very interesting feature of AMD CPU: The AMD > setup for IO > is that each CPU has three HT-links, which can either be used > for I/O or > interprocessor communication, so each CPU can have it''s own set of > hardware that it can access. > > Three questions (please bear with me if I misunderstood you): > 1. How to bind a HT-link in a physcial CPU to I/O or IPC? Did > you do this in > BIOS?The BIOS (or some similar startup software) will "enumerate" the links, similar to how PCI devices are "detected", and also set up routing tables to indicate which device connects to what other device (you can have a PCI-X bridge that connects to a "southbridge" on consecutive links. The actualy physical linking of devices is hard-linked by the motherboard designer. Also, all processors can access any other processors I/O hardware by using the routing tables, as well as using the links for cache-coherency. Finally, as each of the processors have it''s own memory controller, the HT links are also used to do "remote memory reads", i.e. processor 2 reading procesor 0''s memory via the HT link.> > 2. Do you know if there is any CPU from Intel supports such > kind of feature?Not as of current - I have no idea what Intel plans for the future are.> > 3. Each CPU has three HT-links. Is the HT-link you mentioned > the same as the > Hyper Threads in Intel CPU? I heard most of CPUs only have two HT.HT in this case stands for HyperTransport, which is a chip interconnect protocol. See: http://www.hypertransport.org/ HyperThreads is a mechanism to make a processor behave like it''s a dual core processor without implementing (much of) the second core - it''s advantageous on a processor with very long pipelines, as it allows a second stream of instruction to be fed to the core of the processor, and thus reduce the latency caused by a pipe-line stall. -- Mats> > Thanks, > > Liang > > ----- Original Message ----- > From: "Petersson, Mats" <Mats.Petersson@amd.com> > To: "Daniele Palumbo" <daniele@retaggio.net>; > <xen-users@lists.xensource.com> > Sent: Thursday, February 08, 2007 3:08 AM > Subject: RE: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > > > > > -----Original Message----- > > From: xen-users-bounces@lists.xensource.com > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > > Daniele Palumbo > > Sent: 08 February 2007 09:47 > > To: xen-users@lists.xensource.com > > Subject: [Xen-users] vcpus and cpus, plus cpu affinity = ?? > > > > hi > > > > let''s suppose i have something like that: > > --- > > Name ID VCPU CPU State > > Time(s) CPU Affinity > > Domain-0 0 0 0 -b- > > 24842.7 any cpu > > Domain-0 0 1 1 r-- > > 4306.2 any cpu > > Domain-0 0 2 2 -b- > > 5171.3 any cpu > > Domain-0 0 3 3 -b- > > 2013.0 any cpu > > [snip] > > trip 18 0 2 -b- > > 1189.8 any cpu > > uni147 19 0 0 r-- 112415.0 0 > > uni147 19 1 1 -b- 35259.4 1 > > [snip] > > --- > > > > trip have 1 vcpu, cpus blank (""). > > uni147 have 2 vcpus, cpus = "0-3" (i have 4 opteron 844). > > > > which is the difference between dom0 and uni147, or better > > why one have "any > > cpu" and the other have "0" and "1" in cpu affinity? > > > > how xen can decide to allocate to one cpu or another? > > is supported something like "if cpu 0 will get full, move > > vcpu 0 of uni147 to > > cpu 3"? > > and for trip? > > If you "hard-assign" the cpus in your config file, then > that''s the CPU''s > you get. > If you don''t, Xen just assings "at random". > > Xen''s Credit scheduler, which is what''s in Xen by default > since 3.0.2 (I > think), can, and will, move VCPU''s between physical CPUs "for best > utilization of the CPU resources". There''s no "automatic" move things > around between physical CPU''s when you have hard-coded which CPU''s you > want (at least not outside the set!). > > As to what works best for any particular setup, that''s a much > harder to > answer question - and one that you''ll probably be better placed to > answer than I am. > > > > am i wrong if i consider cpu0 the one that will get most of > > IO of the HW? > > That depends on the hardware on your motherboard. The AMD setup for IO > is that each CPU has three HT-links, which can either be used > for I/O or > interprocessor communication, so each CPU can have it''s own set of > hardware that it can access. However, in practice, most motherboards > will attach most of the IO-devices to CPU0, since if you want > to run the > system without a full set of CPU''s, the first CPU is the one that you > have to use... But there are systems with two PCI-X > controllers, one for > CPU0 and another for CPU1 or some such (in which case, you > need to have > two CPU''s to access all PCI-X slots). > > That doesn''t in itself mean that CPU0 is doing more work > tho'', as CPU1, > 2, 3 can all access the hardware that is attached to CPU0, > just that it > takes a fraction longer (but we''re talking about PCI access with a > 33/66/133MHz bus, compared to the 800MHz-1GHz of the HT-link, so the > latency is most likely not at all noticable - the overhead is roughly > 40ns per hop, where CPU{1,3} will have 1 hop, CPU2 has two hops). Note > that a PCI access lasts at least 6-7 cycles in general > (200ns+), and not > unusually longer than that. > > > the cpu time of dom0 is affected by the usage of the domU for > > that real cpu? > > Dom0''s "usage" will be entirely based on what Dom0 is doing, but of > course, if DomU is sharing time with DomU, then the time used > by DomU is > obviously not available to Dom0. Not sure what you''re asking here... > > > > > > > thanks > > Daniele > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > > > > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users