The set of patches implements virtual NUMA-enlightenment to support NUMA-aware PV guests. In more detail, the patch implements the following : * For the NUMA systems, the following memory allocation strategies are implemented : - CONFINE : Confine the VM memory allocation to a single node. As opposed to the current method of doing this in python, the patch implements this in libxc(along with other strategies) and with assurance that the memory actually comes from the selected node. - STRIPE : If the VM memory doesn''t fit in a single node and if the VM is not compiled with guest-numa-support, the memory is allocated striped across a selected max-set of nodes. - SPLIT : If the VM memory doesn''t fit in a single node and if the VM is compiled with guest-numa-support, the memory is allocated split (equally for now) from the min-set of nodes. The VM is then made aware of this NUMA allocation (virtual NUMA enlightenment). -DEFAULT : This is the existing allocation scheme. * If the numa-guest support is compiled into the PV guest, we add numa-guest-support to xen features elfnote. The xen tools use this to determine if SPLIT strategy can be applied. * The PV guest uses the virtual NUMA enlightenment to setup its NUMA layout (at the time of initmem_init) Please comment. -dulloor Signed-off-by: Dulloor Rao <dulloor@gatech.edu> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I would like Acks from the people working on HVM NUMA for this patch series. At the very least it would be nice to have a single user interface for setting this up, regardless of whether for a PV or HVM guest. Hopefully code in the toolstack also can be shared. So I''m cc''ing Dexuan and Andre, as I know they are involved in the HVM NUMA work. Thanks, Keir On 04/04/2010 20:30, "Dulloor" <dulloor@gmail.com> wrote:> The set of patches implements virtual NUMA-enlightenment to support > NUMA-aware PV guests. In more detail, the patch implements the > following : > > * For the NUMA systems, the following memory allocation strategies are > implemented : > - CONFINE : Confine the VM memory allocation to a single node. As > opposed to the current method of doing this in python, the patch > implements this in libxc(along with other strategies) and with > assurance that the memory actually comes from the selected node. > - STRIPE : If the VM memory doesn''t fit in a single node and if the VM > is not compiled with guest-numa-support, the memory is allocated > striped across a selected max-set of nodes. > - SPLIT : If the VM memory doesn''t fit in a single node and if the VM > is compiled with guest-numa-support, the memory is allocated split > (equally for now) from the min-set of nodes. The VM is then made > aware of this NUMA allocation (virtual NUMA enlightenment). > -DEFAULT : This is the existing allocation scheme. > > * If the numa-guest support is compiled into the PV guest, we add > numa-guest-support to xen features elfnote. The xen tools use this to > determine if SPLIT strategy can be applied. > > * The PV guest uses the virtual NUMA enlightenment to setup its NUMA > layout (at the time of initmem_init) > > Please comment. > > -dulloor > > Signed-off-by: Dulloor Rao <dulloor@gatech.edu>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Could you comment on if/how these work when memory is more dynamically allocated (e.g. via an active balloon driver in a guest)? Specifically, I''m wondering if you are running multiple domains, all are actively ballooning, and there is a mix of guest NUMA policies, how do you ensure that non-CONFINE''d domains don''t starve a CONFINE''d domain? Thanks, Dan> -----Original Message----- > From: Dulloor [mailto:dulloor@gmail.com] > Sent: Sunday, April 04, 2010 1:30 PM > To: xen-devel@lists.xensource.com > Cc: Keir Fraser > Subject: [Xen-devel] [PATCH 00/11] PV NUMA Guests > > The set of patches implements virtual NUMA-enlightenment to support > NUMA-aware PV guests. In more detail, the patch implements the > following : > > * For the NUMA systems, the following memory allocation strategies are > implemented : > - CONFINE : Confine the VM memory allocation to a single node. As > opposed to the current method of doing this in python, the patch > implements this in libxc(along with other strategies) and with > assurance that the memory actually comes from the selected node. > - STRIPE : If the VM memory doesn''t fit in a single node and if the VM > is not compiled with guest-numa-support, the memory is allocated > striped across a selected max-set of nodes. > - SPLIT : If the VM memory doesn''t fit in a single node and if the VM > is compiled with guest-numa-support, the memory is allocated split > (equally for now) from the min-set of nodes. The VM is then made > aware of this NUMA allocation (virtual NUMA enlightenment). > -DEFAULT : This is the existing allocation scheme. > > * If the numa-guest support is compiled into the PV guest, we add > numa-guest-support to xen features elfnote. The xen tools use this to > determine if SPLIT strategy can be applied. > > * The PV guest uses the virtual NUMA enlightenment to setup its NUMA > layout (at the time of initmem_init) > > Please comment. > > -dulloor > > Signed-off-by: Dulloor Rao <dulloor@gatech.edu> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan, Sorry I missed one of your previous mails on the topic too, so I have copied answers to those too.> Could you comment on if/how these work when memory is more > dynamically allocated (e.g. via an active balloon driver > in a guest)?The balloon driver is also made numa-aware and uses (the same) enlightenment to derive the guest-node to physical-node mapping. Please refer to my previously submitted patch for this (http://old.nabble.com/Xen-devel--XEN-PATCH---Linux-PVOPS--ballooning-on-numa-domains-td26262334.html). I intend to send out a refreshed patch once the basic guest numa is checked in.> Specifically, I''m wondering if you are running > multiple domains, all are actively ballooning, and there > is a mix of guest NUMA policies, how do you ensure that > non-CONFINE''d domains don''t starve a CONFINE''d domain?We first try to CONFINE a domain and only then proceed to STRIPE or SPLIT(if capable) the domain. So, in this (automatic) global domain memory allocation scheme, there is no possibility of starvation from memory pov. Hope I got your question right.> I''d be interested in your thoughts on numa-aware tmem > as well as the other dynamic memory mechanisms in Xen 4.0. > Tmem is special in that it uses primarily full-page copies > from/to tmem-space to/from guest-space so, assuming the > interconnect can pipeline/stream a memcpy, overhead of > off-node memory vs on-node memory should be less > noticeable. However tmem uses large data structures > (rbtrees and radix-trees) and the lookup process might > benefit from being NUMA-aware.For the tmem, I was thinking of the ability to specify a set of nodes from which the tmem-space memory is preferred which could be derived from the domain''s numa enlightenment, but as you mentioned the full-page copy overhead is less noticeable (at least on my smaller NUMA machine). But, the rate would determine if we should do this to reduce inter-node traffic. What do you suggest ? I was looking at the data structures too.> Also, I will be looking into adding some page-sharing > techniques into tmem in the near future. This (and the > existing page sharing feature just added to 4.0) may > create some other interesting challenges for NUMA-awareness.I have just started reading up on the memsharing feature of Xen. I would be glad to get your input on NUMA challenges over there. thanks dulloor On Mon, Apr 5, 2010 at 10:52 AM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> Could you comment on if/how these work when memory is more > dynamically allocated (e.g. via an active balloon driver > in a guest)? Specifically, I''m wondering if you are running > multiple domains, all are actively ballooning, and there > is a mix of guest NUMA policies, how do you ensure that > non-CONFINE''d domains don''t starve a CONFINE''d domain? > > Thanks, > Dan > >> -----Original Message----- >> From: Dulloor [mailto:dulloor@gmail.com] >> Sent: Sunday, April 04, 2010 1:30 PM >> To: xen-devel@lists.xensource.com >> Cc: Keir Fraser >> Subject: [Xen-devel] [PATCH 00/11] PV NUMA Guests >> >> The set of patches implements virtual NUMA-enlightenment to support >> NUMA-aware PV guests. In more detail, the patch implements the >> following : >> >> * For the NUMA systems, the following memory allocation strategies are >> implemented : >> - CONFINE : Confine the VM memory allocation to a single node. As >> opposed to the current method of doing this in python, the patch >> implements this in libxc(along with other strategies) and with >> assurance that the memory actually comes from the selected node. >> - STRIPE : If the VM memory doesn''t fit in a single node and if the VM >> is not compiled with guest-numa-support, the memory is allocated >> striped across a selected max-set of nodes. >> - SPLIT : If the VM memory doesn''t fit in a single node and if the VM >> is compiled with guest-numa-support, the memory is allocated split >> (equally for now) from the min-set of nodes. The VM is then made >> aware of this NUMA allocation (virtual NUMA enlightenment). >> -DEFAULT : This is the existing allocation scheme. >> >> * If the numa-guest support is compiled into the PV guest, we add >> numa-guest-support to xen features elfnote. The xen tools use this to >> determine if SPLIT strategy can be applied. >> >> * The PV guest uses the virtual NUMA enlightenment to setup its NUMA >> layout (at the time of initmem_init) >> >> Please comment. >> >> -dulloor >> >> Signed-off-by: Dulloor Rao <dulloor@gatech.edu> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
In general, I am of the opinion that in a virtualized world, one gets best flexibility or best performance, but not both. There may be a couple of reasonable points on this "slider selector", but I''m not sure in general if it will be worth a huge time investment as real users will not understand the subtleties of their workloads well enough to choose from a large number of (perhaps more than two) points on the performance/flexibility spectrum. So customers that want highest performance should be prepared to pin their guests and not use ballooning. And those that want the flexibility of migration and ballooning etc should expect to see a performance hit (including NUMA consequences). But since I don''t get to make that decision, let''s look at the combination of NUMA + dynamic memory utilization...> Please refer to my previously submitted patch for this > (http://old.nabble.com/Xen-devel--XEN-PATCH---Linux-PVOPS--ballooning- > on-numa-domains-td26262334.html). > I intend to send out a refreshed patch once the basic guest numa is > checked in.OK, will wait and take a look at that later.> We first try to CONFINE a domain and only then proceed to STRIPE or > SPLIT(if capable) the domain. So, in this (automatic) global domain > memory allocation scheme, there is no possibility of starvation from > memory pov. Hope I got your question right.The example I''m concerned with is: 1) Domain A is CONFINE''d to node A and domain B/C/D/etc are not CONFINE''d 2) Domain A uses less than the total memory on node A and/or balloons down so it uses even less than when launched. 3) Domains B/C/D have an increasing memory need, and semi-randomly absorb memory from all nodes, including node A. After (3), free memory is somewhat randomly distributed across all nodes. Then: 4) Domain A suddenly has an increasing memory need... but there''s not enough free memory remaining on node A (in fact possibly there is none at all) to serve its need. But by definition of CONFINE, domain A is not allowed to use memory other than on node A. What happens now? It appears to me that other domains have (perhaps even maliciously) starved domain A. I think this is a dynamic bin-packing problem which is unsolvable in general form. So the choice of heuristics is going to be important.> For the tmem, I was thinking of the ability to specify a set of nodes > from which the tmem-space memory is preferred which could be derived > from the domain''s numa enlightenment, but as you mentioned the > full-page copy overhead is less noticeable (at least on my smaller > NUMA machine). But, the rate would determine if we should do this to > reduce inter-node traffic. What do you suggest ? I was looking at the > data structures too.Since tmem allocates individual xmalloc-tlsf memory pools per domain, it should be possible to inform tmem of node preferences, but I don''t know that it will be feasible to truly CONFINE a domain''s tmem. On the other hand, because of the page copying, affinity by itself may be sufficient.> > Also, I will be looking into adding some page-sharing > > techniques into tmem in the near future. This (and the > > existing page sharing feature just added to 4.0) may > > create some other interesting challenges for NUMA-awareness. > I have just started reading up on the memsharing feature of Xen. I > would be glad to get your input on NUMA challenges over there.Note that the tmem patch that does sharing (tmem calls it "page deduplication") was just accepted into xen-unstable. Basically some memory may belong to more than one domain, so NUMA affects and performance/memory tradeoffs may get very complicated. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> I would like Acks from the people working on HVM NUMA for this patch > series. At the very least it would be nice to have a single user > interface for setting this up, regardless of whether for a PV or HVM > guest. Hopefully code in the toolstack also can be shared. So I''mYes, I strongly agree we should share one interterface, e.g., The XENMEM_numa_op hypercalls implemented by Dulloor could be re-used in the hvm numa case and some parts of the toolstack could be shared, I think. I also replied in another thead and supplied some similarity I found in Andre/Dulloor''s patches.> cc''ing Dexuan and Andre, as I know they are involved in the HVM NUMA > work. > > Thanks, > Keir > > On 04/04/2010 20:30, "Dulloor" <dulloor@gmail.com> wrote: > >> The set of patches implements virtual NUMA-enlightenment to support >> NUMA-aware PV guests. In more detail, the patch implements the >> following : >> >> * For the NUMA systems, the following memory allocation strategies >> are implemented : >> - CONFINE : Confine the VM memory allocation to a >> single node. As opposed to the current method of doing this in >> python, the patch implements this in libxc(along with other >> strategies) and with assurance that the memory actually comes from >> the selected node. > > - STRIPE : If the VM memory doesn''t fit in a >> single node and if the VM is not compiled with guest-numa-support, >> the memory is allocated striped across a selected max-set of nodes. >> - SPLIT : If the VM memory doesn''t fit in a single node and if the VM >> is compiled with guest-numa-support, the memory is allocated split >> (equally for now) from the min-set of nodes. The VM is then made >> aware of this NUMA allocation (virtual NUMA enlightenment). >> -DEFAULT : This is the existing allocation scheme. >> >> * If the numa-guest support is compiled into the PV guest, we add >> numa-guest-support to xen features elfnote. The xen tools use this to >> determine if SPLIT strategy can be applied. >>I think this looks too complex to allow a real user to easily determine which one to use... About the CONFINE stragegy -- looks this is not a useful usage model to me -- do we really think it''s a typical usage model to ensure a VM''s memory can only be allocated on a specified node? The definitions of STRIPE and SPLIT also doesn''t sound like typical usage models to me. Why must tools know if the PV kernel is built with guest numa support or not? If a user configures guest numa to "on" for a pv guest, the tools can supply the numa info to PV kernel even if the pv kernel is not built with guest numa support -- the pv kernel will neglect the info safely; If a user configures guest numa to "off" for a pv guest and the tools don''t supply the numa info to PV kernel, and if the pv kernel is built with guest numa support, the pv kernel can easily detect this by your new hypercall and will not enable numa. When a user finds the computing capability of a single node can''t satisfy the actual need and hence wants to use guest numa, since the user has specified the amount of guest memory and the number of vcpus in guest config file, I think the user only needs to specify how many guest nodes (the "guestnodes" option in Andre''s patch) the guest will see, and the tools and the hypervisor should co-work to distribute guest memory and vcpus uniformly among the guest nodes(I think we may not want to support non-uniform nodes as that doesn''t look like a typical usage model) -- of course, maybe a specified node doesn''t have the expected amount of memory -- in this case, the guest can continue to run with a slower speed (we can print a warning message to the user); or, if the user does care about predictable guest performance, the guest creation should fail. How do you like this? My thought is we can make things simple in the first step. :-) Thanks, -- Dexuan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Apr 6, 2010 at 1:18 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> In general, I am of the opinion that in a virtualized world, > one gets best flexibility or best performance, but not both. > There may be a couple of reasonable points on this "slider > selector", but I''m not sure in general if it will be worth > a huge time investment as real users will not understand the > subtleties of their workloads well enough to choose from > a large number of (perhaps more than two) points on the > performance/flexibility spectrum. > > So customers that want highest performance should be prepared > to pin their guests and not use ballooning. And those that > want the flexibility of migration and ballooning etc should > expect to see a performance hit (including NUMA consequences).In principle, I agree with you. For the same reason, in this first version, I have tried to keep the configurable parameters to minimum. Wrt ballooning, we could work out simple solutions that work. Migration would be problematic though.> > But since I don''t get to make that decision, let''s look > at the combination of NUMA + dynamic memory utilization... > >> Please refer to my previously submitted patch for this >> (http://old.nabble.com/Xen-devel--XEN-PATCH---Linux-PVOPS--ballooning- >> on-numa-domains-td26262334.html). >> I intend to send out a refreshed patch once the basic guest numa is >> checked in. > > OK, will wait and take a look at that later. > >> We first try to CONFINE a domain and only then proceed to STRIPE or >> SPLIT(if capable) the domain. So, in this (automatic) global domain >> memory allocation scheme, there is no possibility of starvation from >> memory pov. Hope I got your question right. > > The example I''m concerned with is: > 1) Domain A is CONFINE''d to node A and domain B/C/D/etc are not > CONFINE''d > 2) Domain A uses less than the total memory on node A and/or > balloons down so it uses even less than when launched. > 3) Domains B/C/D have an increasing memory need, and semi-randomly > absorb memory from all nodes, including node A. > > After (3), free memory is somewhat randomly distributed across > all nodes. Then: > > 4) Domain A suddenly has an increasing memory need... but there''s > not enough free memory remaining on node A (in fact possibly > there is none at all) to serve its need. But by definition > of CONFINE, domain A is not allowed to use memory other than > on node A. > > What happens now? It appears to me that other domains have > (perhaps even maliciously) starved domain A. > > I think this is a dynamic bin-packing problem which is unsolvable > in general form. So the choice of heuristics is going to be > important. >In the proposed solution, the domain could be either CONFINED, SPLIT (NUMA-aware), or STRIPED. But, in each case, the domain is aware of how much memory is allocated from each of the nodes at the time of start-up. The enlightened ballooning attempts to keep the state similar to that during startup. But, we might have to allocate from any node under extreme memory pressure. For that (hopefully less likely) case, we can implement dynamic mechanisms to converge to the original state, by sweeping through the memory and exchanging memory reservations whenever possible. I already have the means of doing this, as part of the ballooning changes. For CONFINED/SPLIT domains, I am using the Best-Fit-Decreasing heuristic, whereas for STRIPED, I am using First-Fit-Increasing strategy (as a means to reduce the fragmentation of free node memory).>> For the tmem, I was thinking of the ability to specify a set of nodes >> from which the tmem-space memory is preferred which could be derived >> from the domain''s numa enlightenment, but as you mentioned the >> full-page copy overhead is less noticeable (at least on my smaller >> NUMA machine). But, the rate would determine if we should do this to >> reduce inter-node traffic. What do you suggest ? I was looking at the >> data structures too. > > Since tmem allocates individual xmalloc-tlsf memory pools per domain, > it should be possible to inform tmem of node preferences, but I don''t > know that it will be feasible to truly CONFINE a domain''s tmem. > On the other hand, because of the page copying, affinity by itself > may be sufficient. >Yeah, I guess affinity should suffice for the CONFINED domains, but I was thinking of node-preferences for the NUMA (SPLIT)Guests.>> > Also, I will be looking into adding some page-sharing >> > techniques into tmem in the near future. This (and the >> > existing page sharing feature just added to 4.0) may >> > create some other interesting challenges for NUMA-awareness. >> I have just started reading up on the memsharing feature of Xen. I >> would be glad to get your input on NUMA challenges over there. > > Note that the tmem patch that does sharing (tmem calls it "page > deduplication") was just accepted into xen-unstable. Basically > some memory may belong to more than one domain, so NUMA affects > and performance/memory tradeoffs may get very complicated. >Thanks for sharing. I will read this very soon.> Dan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Apr 7, 2010 at 3:57 AM, Cui, Dexuan <dexuan.cui@intel.com> wrote:> Keir Fraser wrote: >> I would like Acks from the people working on HVM NUMA for this patch >> series. At the very least it would be nice to have a single user >> interface for setting this up, regardless of whether for a PV or HVM >> guest. Hopefully code in the toolstack also can be shared. So I''m > Yes, I strongly agree we should share one interterface, e.g., The XENMEM_numa_op hypercalls implemented by Dulloor could be >re-used in the hvm numa case and some parts of the toolstack could be shared, I think. I also replied in another thead and >supplied some similarity I found in Andre/Dulloor''s patches. >IMO PV NUMA guests and HVM NUMA guests could share most of the code from toolstack - for instance, getting the current state of machine, deciding on a strategy for domain memory allocation, selection of nodes, etc. They diverge only at the actual point of domain construction. PV NUMA uses enlightenments, whereas HVM would need working with hvmloader to export SLIT/SRAT ACPI tables. So, I agree that we need to converge.>> cc''ing Dexuan and Andre, as I know they are involved in the HVM NUMA >> work. >> >> Thanks, >> Keir >> >> On 04/04/2010 20:30, "Dulloor" <dulloor@gmail.com> wrote: >> >>> The set of patches implements virtual NUMA-enlightenment to support >>> NUMA-aware PV guests. In more detail, the patch implements the >>> following : >>> >>> * For the NUMA systems, the following memory allocation strategies >>> are implemented : >>> - CONFINE : Confine the VM memory allocation to a >>> single node. As opposed to the current method of doing this in >>> python, the patch implements this in libxc(along with other >>> strategies) and with assurance that the memory actually comes from >>> the selected node. >> > - STRIPE : If the VM memory doesn''t fit in a >>> single node and if the VM is not compiled with guest-numa-support, >>> the memory is allocated striped across a selected max-set of nodes. >>> - SPLIT : If the VM memory doesn''t fit in a single node and if the VM >>> is compiled with guest-numa-support, the memory is allocated split >>> (equally for now) from the min-set of nodes. The VM is then made >>> aware of this NUMA allocation (virtual NUMA enlightenment). >>> -DEFAULT : This is the existing allocation scheme. >>> >>> * If the numa-guest support is compiled into the PV guest, we add >>> numa-guest-support to xen features elfnote. The xen tools use this to >>> determine if SPLIT strategy can be applied. >>> > I think this looks too complex to allow a real user to easily determine which one to use...I think you misunderstood this. For the first version, I have implemented an automatic global domain memory allocation scheme, which (when enabled) applies to all domains on a NUMA machine. I am of opinion that users are seldom in a state to determine which strategy to use. They would want the best possible performance for their VM at any point of time, and we can only guarantee the best possible performance, given the current state of the system (how the free memory is scattered across nodes, distance between those nodes, etc). In that regard, this solution is the simplest.> About the CONFINE stragegy -- looks this is not a useful usage model to me -- do we really think it''s a typical usage model to > ensure a VM''s memory can only be allocated on a specified node?Not all VMs are large enough not to fit into a single node (note that user doesn''t specify a node). And, if a VM can be fit into a single node, that is obviously the best possible option for a VM.> The definitions of STRIPE and SPLIT also doesn''t sound like typical usage models to me.There are only two possibilities. Either the VM fits in a single node or it doesn''t. The mentioned strategies (SPLIT, STRIPE) try to optimize the solution when the VM doesn''t fit in a single node. The aim is to reduce the number of inter-node accesses(SPLIT) and/or provide a more predictable performance(STRIPE).> Why must tools know if the PV kernel is built with guest numa support or not?What is the point of arranging the memory amenable for construction of nodes in guest if the guest itself is not compiled to do so.> If a user configures guest numa to "on" for a pv guest, the tools can supply the numa info to PV kernel even if the pv kernel is not > built with guest numa support -- the pv kernel will neglect the info safely; > If a user configures guest numa to "off" for a pv guest and the tools don''t supply the numa info to PV kernel, and if the pv kernel > is built with guest numa support, the pv kernel can easily detect this by your new hypercall and will not enable numa.These error checks are done even now. But, by checking if the PV kernel is built with guest numa support, we don''t require the user to configure yet another parameter. Wasn''t that your concern too in the very first point ?> > When a user finds the computing capability of a single node can''t satisfy the actual need and hence wants to use guest numa, > since the user has specified the amount of guest memory and the number of vcpus in guest config file, I think the user only needs >to specify how many guest nodes (the "guestnodes" option in Andre''s patch) the guest will see, and the tools and the hypervisor >should co-work to distribute guest memory and vcpus uniformly among the guest nodes(I think we may not want to support non- >uniform nodes as that doesn''t look like a typical usage model) -- of course, maybe a specified node doesn''t have the expected >amount of memory -- in this case, the guest can continue to run with a slower speed (we can print a warning message to the >user); or, if the user does care about predictable guest performance, the guest creation should fail.Please observe that the patch does all these things plus some more. For one, "guestnodes" option doesn''t make sense, since as you observe, it needs the user to carefully read the state of the system when starting the domain and also the user needs to make sure that the guest itself is compiled with numa support. The aim should be to automate this part and provide the best performance, given the current state. The patch attempts to do that. Secondly, when the guests are not compiled with numa support, they would still want a more predictable (albeit average) performance. And, by striping the memory across the nodes and by pinning the domain vcpus to the union of those nodes'' processors, applications (of substantial sizes) could be expected to see more predictable performance.> > How do you like this? My thought is we can make things simple in the first step. :-)Please let me know if my comments are not clear. I agree that we should shoot for simplicity and also for a common interface. Hope we will get there :)> > Thanks, > -- Dexuan >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> The set of patches implements virtual NUMA-enlightenment to support > NUMA-aware PV guests. In more detail, the patch implements the > following : > > * For the NUMA systems, the following memory allocation strategies are > implemented : > - CONFINE : Confine the VM memory allocation to a single node. As > opposed to the current method of doing this in python, the patch > implements this in libxc(along with other strategies) and with > assurance that the memory actually comes from the selected node.Do you use the VCPU affinity masks to determine the node in question, or is there another parameter to specify this? Thanks, Ian> - STRIPE : If the VM memory doesn''t fit in a single node and if the VM > is not compiled with guest-numa-support, the memory is allocated > striped across a selected max-set of nodes. > - SPLIT : If the VM memory doesn''t fit in a single node and if the VM > is compiled with guest-numa-support, the memory is allocated split > (equally for now) from the min-set of nodes. The VM is then made > aware of this NUMA allocation (virtual NUMA enlightenment). > -DEFAULT : This is the existing allocation scheme. > > * If the numa-guest support is compiled into the PV guest, we add > numa-guest-support to xen features elfnote. The xen tools use this to > determine if SPLIT strategy can be applied. > > * The PV guest uses the virtual NUMA enlightenment to setup its NUMA > layout (at the time of initmem_init) > > Please comment. > > -dulloor > > Signed-off-by: Dulloor Rao <dulloor@gatech.edu> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, Apr 9, 2010 at 7:34 AM, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:>> The set of patches implements virtual NUMA-enlightenment to support >> NUMA-aware PV guests. In more detail, the patch implements the >> following : >> >> * For the NUMA systems, the following memory allocation strategies are >> implemented : >> - CONFINE : Confine the VM memory allocation to a single node. As >> opposed to the current method of doing this in python, the patch >> implements this in libxc(along with other strategies) and with >> assurance that the memory actually comes from the selected node. > > Do you use the VCPU affinity masks to determine the node in question, or is there another parameter to specify this?As of now, the node is selected solely based on the distribution of free memory (across nodes).> > Thanks, > Ian > > >> - STRIPE : If the VM memory doesn''t fit in a single node and if the VM >> is not compiled with guest-numa-support, the memory is allocated >> striped across a selected max-set of nodes. >> - SPLIT : If the VM memory doesn''t fit in a single node and if the VM >> is compiled with guest-numa-support, the memory is allocated split >> (equally for now) from the min-set of nodes. The VM is then made >> aware of this NUMA allocation (virtual NUMA enlightenment). >> -DEFAULT : This is the existing allocation scheme. >> >> * If the numa-guest support is compiled into the PV guest, we add >> numa-guest-support to xen features elfnote. The xen tools use this to >> determine if SPLIT strategy can be applied. >> >> * The PV guest uses the virtual NUMA enlightenment to setup its NUMA >> layout (at the time of initmem_init) >> >> Please comment. >> >> -dulloor >> >> Signed-off-by: Dulloor Rao <dulloor@gatech.edu> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dulloor wrote:> On Wed, Apr 7, 2010 at 3:57 AM, Cui, Dexuan <dexuan.cui@intel.com> > wrote: >> Keir Fraser wrote: >>> I would like Acks from the people working on HVM NUMA for this patch >>> series. At the very least it would be nice to have a single user >>> interface for setting this up, regardless of whether for a PV or HVM >>> guest. Hopefully code in the toolstack also can be shared. So I''m >> Yes, I strongly agree we should share one interterface, e.g., The >> XENMEM_numa_op hypercalls implemented by Dulloor could be >re-used >> in the hvm numa case and some parts of the toolstack could be >> shared, I think. I also replied in another thead and >supplied some >> similarity I found in Andre/Dulloor''s patches. >> > IMO PV NUMA guests and HVM NUMA guests could share most of the code > from toolstack - for instance, getting the current state of machine, > deciding on a strategy for domain memory allocation, selection of > nodes, etc. They diverge only at the actual point of domain > construction. PV NUMA uses enlightenments, whereas HVM would need > working with hvmloader to export SLIT/SRAT ACPI tables. So, I agree > that we need to converge.Hi Dulloor, In your patches, the toolstack tries to figure out the "best fit nodes" for a PV guest and invokes a hypercall set_domain_numa_layout to tell the hypervisor to remember the info, and later the PV guest invokes a hypercall get_domain_numa_layout to retrieve the info from the hypervisor. Can this be changed to: the toolstack writes the guest numa info directly into a new field in the start_info(or the share_info) (maybe in the starndard format of the SRAT/SLIT) and later PV guest reads the info and uses acpi_numa_init() to parse the info? I think in this way the new hypercalls can be avoided and the pv numa enlightenment code in guest kernel can be minimized. I''m asking this because this is the way how HVM numa patches of Andure do(the toolstack passes the info to hvmloader and the latter builds SRAT/SLIT for guest) xc_select_best_fit_nodes() decides the "min-set" of host nodes that will be used for the guest. It only considers the current memory usage of the system. Maybe we should also condider the cpu load? And the number of the nodes must be 2^^n? And how to handle the case #vcpu is < #vnode? And looks your patches only consider the guest''s memory requirement -- guest''s vcpu requirement is neglected? e.g., a guest may not need a very large amount of memory while it needs many vcpus. xc_select_best_fit_nodes() should consider this when determining the number of vnode.>>> On 04/04/2010 20:30, "Dulloor" <dulloor@gmail.com> wrote: >>> >>>> The set of patches implements virtual NUMA-enlightenment to support >>>> NUMA-aware PV guests. In more detail, the patch implements the >>>> following : >>>> >>>> * For the NUMA systems, the following memory allocation strategies >>>> are implemented : - CONFINE : Confine the VM memory allocation to a >>>> single node. As opposed to the current method of doing this in >>>> python, the patch implements this in libxc(along with other >>>> strategies) and with assurance that the memory actually comes from >>>> the selected node. - STRIPE : If the VM memory doesn''t fit in a >>>> single node and if the VM is not compiled with guest-numa-support, >>>> the memory is allocated striped across a selected max-set of nodes. >>>> - SPLIT : If the VM memory doesn''t fit in a single node and if the >>>> VM is compiled with guest-numa-support, the memory is allocated >>>> split (equally for now) from the min-set of nodes. The VM is then >>>> made aware of this NUMA allocation (virtual NUMA enlightenment). >>>> -DEFAULT : This is the existing allocation scheme. >>>> >>>> * If the numa-guest support is compiled into the PV guest, we add >>>> numa-guest-support to xen features elfnote. The xen tools use this >>>> to determine if SPLIT strategy can be applied. >>>> >> I think this looks too complex to allow a real user to easily >> determine which one to use... > I think you misunderstood this. For the first version, I have > implemented an automatic global domain memory allocation scheme, which > (when enabled) applies to all domains on a NUMA machine. I am of > opinion that users are seldom in a state to determine which strategy > to use. They would want the best possible performance for their VM at > any point of time, and we can only guarantee the best possible > performance, given the current state of the system (how the free > memory is scattered across nodes, distance between those nodes, etc). > In that regard, this solution is the simplest.Ok, I see. BTW: I think actually currently Xen can handle the case CONFINE pretty well, e.g, when no vcpu affinity is explicitly specified, the toolstack tries to choose a "best" host node for the guest and pins all vcpus of the guest to the host node.>> About the CONFINE stragegy -- looks this is not a useful usage model >> to me -- do we really think it''s a typical usage model to >> ensure a VM''s memory can only be allocated on a specified node? > Not all VMs are large enough not to fit into a single node (note that > user doesn''t specify a node). And, if a VM can be fit into a single > node, that is obviously the best possible option for a VM. > >> The definitions of STRIPE and SPLIT also doesn''t sound like typical >> usage models to me. > There are only two possibilities. Either the VM fits in a single node > or it doesn''t. The mentioned strategies (SPLIT, STRIPE) try to > optimize the solution when the VM doesn''t fit in a single node. The > aim is to reduce the number of inter-node accesses(SPLIT) and/or > provide a more predictable performance(STRIPE). > >> Why must tools know if the PV kernel is built with guest numa >> support or not? > What is the point of arranging the memory amenable for construction of > nodes in guest if the guest itself is not compiled to do so.I meant: to simplify the implementation, the toolstack can always supply the numa config info to the guest *if necessary*, no matter if the guest kernel is numa-enabled or not (even if the guest kernel isn''t numa-enabled, the guest performance may be better if the toolstack decides to supply a numa config to the guest) About the "*if necessary*": Andure and I think the user should supply an option "guestnode" in the guest config file, and you think the toolstack should be able to automatically determine a "best" value. I raised some questions about xc_select_best_fit_nodes() in the above paragraph. Hi Andre, would you like to comment on this?> >> If a user configures guest numa to "on" for a pv guest, the tools >> can supply the numa info to PV kernel even if the pv kernel is not > >> built with guest numa support -- the pv kernel will neglect the info >> safely; >> If a user configures guest numa to "off" for a pv guest and the >> tools don''t supply the numa info to PV kernel, and if the pv kernel >> > is built with guest numa support, the pv kernel can easily detect >> this by your new hypercall and will not enable numa. > These error checks are done even now. But, by checking if the PV > kernel is built with guest numa support, we don''t require the user to > configure yet another parameter. Wasn''t that your concern too in the > very first point ? > >> >> When a user finds the computing capability of a single node can''t >> satisfy the actual need and hence wants to use guest numa, >> since the user has specified the amount of guest memory and the >> number of vcpus in guest config file, I think the user only needs >> to specify how many guest nodes (the "guestnodes" option in Andre''s >> patch) the guest will see, and the tools and the hypervisor >> should co-work to distribute guest memory and vcpus uniformly among >> the guest nodes(I think we may not want to support non- >> uniform nodes as that doesn''t look like a typical usage model) -- of >> course, maybe a specified node doesn''t have the expected >> amount of memory -- in this case, the guest can continue to run with >> a slower speed (we can print a warning message to the >> user); or, if the user does care about predictable guest >> performance, the guest creation should fail. > > Please observe that the patch does all these things plus some more. > For one, "guestnodes" option doesn''t make sense, since as you observe, > it needs the user to carefully read the state of the system when > starting the domain and also the user needs to make sure that the > guest itself is compiled with numa support. The aim should be toI think it''s not difficult for a user to specify "guestnodes" and to check if a PV/HVM guest kernel is numa-enabled or not(anyway, a user needs to ensure that to achieve the optimal peformance). "xm info/list/vcpu-list" should already supply enough info. I think it''s reasonable to assume a numa user has more knowledge than a preliminary user. :-) I suppose Andure would argue more for the "guestnodes" option. PV guest can use the ELFnote as a hit to the toolstack. This may be used as a kind of optimization. HVM guest can''t use this.> automate this part and provide the best performance, given the current > state. The patch attempts to do that. Secondly, when the guests are > not compiled with numa support, they would still want a more > predictable (albeit average) performance. And, by striping the memory > across the nodes and by pinning the domain vcpus to the union of those > nodes'' processors, applications (of substantial sizes) could be > expected to see more predictable performance. >> >> How do you like this? My thought is we can make things simple in the >> first step. :-) > Please let me know if my comments are not clear. I agree that we > should shoot for simplicity and also for a common interface. Hope we > will get there :)Thanks a lot for all the explanation and discussion. Yes, we need to agree on a common interface to avoid confusion. And I still think the "guestnodes/uniform_nodes" idea is more straightforward and the implementatin is simpler. :-) Thanks, -- Dexuan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Apr 14, 2010 at 1:18 AM, Cui, Dexuan <dexuan.cui@intel.com> wrote:> Dulloor wrote: >> On Wed, Apr 7, 2010 at 3:57 AM, Cui, Dexuan <dexuan.cui@intel.com> >> wrote: >>> Keir Fraser wrote: >>>> I would like Acks from the people working on HVM NUMA for this patch >>>> series. At the very least it would be nice to have a single user >>>> interface for setting this up, regardless of whether for a PV or HVM >>>> guest. Hopefully code in the toolstack also can be shared. So I''m >>> Yes, I strongly agree we should share one interterface, e.g., The >>> XENMEM_numa_op hypercalls implemented by Dulloor could be >re-used >>> in the hvm numa case and some parts of the toolstack could be >>> shared, I think. I also replied in another thead and >supplied some >>> similarity I found in Andre/Dulloor''s patches. >>> >> IMO PV NUMA guests and HVM NUMA guests could share most of the code >> from toolstack - for instance, getting the current state of machine, >> deciding on a strategy for domain memory allocation, selection of >> nodes, etc. They diverge only at the actual point of domain >> construction. PV NUMA uses enlightenments, whereas HVM would need >> working with hvmloader to export SLIT/SRAT ACPI tables. So, I agree >> that we need to converge. > Hi Dulloor, > In your patches, the toolstack tries to figure out the "best fit nodes" for a PV guest and >invokes a hypercall set_domain_numa_layout to tell the hypervisor to remember the >info, and later the PV guest invokes a hypercall get_domain_numa_layout to retrieve the >info from the hypervisor. > Can this be changed to: the toolstack writes the guest numa info directly into a new >field in the start_info(or the share_info) (maybe in the starndard format of the SRAT/SLIT) >and later PV guest reads the info and uses acpi_numa_init() to parse the info? I think in >this way the new hypercalls can be avoided and the pv numa enlightenment code in >guest kernel can be minimized. > I''m asking this because this is the way how HVM numa patches of Andure do(the >toolstack passes the info to hvmloader and the latter builds SRAT/SLIT for guest)Hi Cui, In my first version of patches (for making dom0 a numa guest), I had put this information into start_info (http://lists.xensource.com/archives/html/xen-devel/2010-02/msg00630.html). But, after that I thought this new approach is better (for pv numa and maybe even hvm numa) for following reasons : - For PV NUMA guests, there are more places where the enlightenment might be useful. For instance, in the attached (refreshed)patch, I have used the enlightenment to support ballooning (without changing node mappings) for PV NUMA guests. Similarly, there are other places within the hypervisor as well as in the VM where I plan to use the domain_numa_layout. That''s the main reason for choosing this approach. Although I am not sure, I think this could be useful for HVM too (maybe with PV on HVM). - Using the hypercall interface is equally simple. And, also with start-info, I wasn''t sure if it looks clean to add feature-specific variables (useful only with PV NUMA guests) to start-info (or even shared info), changing the xen-vm interface, adding (unnecessary) changes for compat, etc. Please let me know your thoughts.> > xc_select_best_fit_nodes() decides the "min-set" of host nodes that will be used for the >guest. It only considers the current memory usage of the system. Maybe we should also >condider the cpu load? And the number of the nodes must be 2^^n? And how to handle >the case #vcpu is < #vnode? > And looks your patches only consider the guest''s memory requirement -- guest''s vcpu >requirement is neglected? e.g., a guest may not need a very large amount of memory >while it needs many vcpus. xc_select_best_fit_nodes() should consider this when >determining the number of vnode.I agree with you. I was planning to consider vcpu load as the next step. Also, I am looking for a good heuristic. I looked at the nodeload heuristic (currently in xen), but found it too naive. But, if you/Andre think it is a good heuristic, I will add the support. Actually, I think in future we should do away with strict vcpu-affinities and rely more on a scheduler with necessary NUMA support to complement our placement strategies. As of now, we don''t SPLIT, if #vcpu < #vnode. We use STRIPING in that case.> >>>> On 04/04/2010 20:30, "Dulloor" <dulloor@gmail.com> wrote: >>>> >>>>> The set of patches implements virtual NUMA-enlightenment to support >>>>> NUMA-aware PV guests. In more detail, the patch implements the >>>>> following : >>>>> >>>>> * For the NUMA systems, the following memory allocation strategies >>>>> are implemented : - CONFINE : Confine the VM memory allocation to a >>>>> single node. As opposed to the current method of doing this in >>>>> python, the patch implements this in libxc(along with other >>>>> strategies) and with assurance that the memory actually comes from >>>>> the selected node. - STRIPE : If the VM memory doesn''t fit in a >>>>> single node and if the VM is not compiled with guest-numa-support, >>>>> the memory is allocated striped across a selected max-set of nodes. >>>>> - SPLIT : If the VM memory doesn''t fit in a single node and if the >>>>> VM is compiled with guest-numa-support, the memory is allocated >>>>> split (equally for now) from the min-set of nodes. The VM is then >>>>> made aware of this NUMA allocation (virtual NUMA enlightenment). >>>>> -DEFAULT : This is the existing allocation scheme. >>>>> >>>>> * If the numa-guest support is compiled into the PV guest, we add >>>>> numa-guest-support to xen features elfnote. The xen tools use this >>>>> to determine if SPLIT strategy can be applied. >>>>> >>> I think this looks too complex to allow a real user to easily >>> determine which one to use... >> I think you misunderstood this. For the first version, I have >> implemented an automatic global domain memory allocation scheme, which >> (when enabled) applies to all domains on a NUMA machine. I am of >> opinion that users are seldom in a state to determine which strategy >> to use. They would want the best possible performance for their VM at >> any point of time, and we can only guarantee the best possible >> performance, given the current state of the system (how the free >> memory is scattered across nodes, distance between those nodes, etc). >> In that regard, this solution is the simplest. > Ok, I see. > BTW: I think actually currently Xen can handle the case CONFINE pretty well, e.g, when > no vcpu affinity is explicitly specified, the toolstack tries to choose a "best" host node > for the guest and pins all vcpus of the guest to the host node.But, currently it is done in python code and also it doesn''t use exact_node interface. I added this to the libxc toolstack for the sake of completeness (CONFINE is just a special case of SPLIT). Also, with libxl catching up, we might anyway want to do these things in libxc, where it is accessible to both xm and xl.> >>> About the CONFINE stragegy -- looks this is not a useful usage model >>> to me -- do we really think it''s a typical usage model to >>> ensure a VM''s memory can only be allocated on a specified node? >> Not all VMs are large enough not to fit into a single node (note that >> user doesn''t specify a node). And, if a VM can be fit into a single >> node, that is obviously the best possible option for a VM. >> >>> The definitions of STRIPE and SPLIT also doesn''t sound like typical >>> usage models to me. >> There are only two possibilities. Either the VM fits in a single node >> or it doesn''t. The mentioned strategies (SPLIT, STRIPE) try to >> optimize the solution when the VM doesn''t fit in a single node. The >> aim is to reduce the number of inter-node accesses(SPLIT) and/or >> provide a more predictable performance(STRIPE). >> >>> Why must tools know if the PV kernel is built with guest numa >>> support or not? >> What is the point of arranging the memory amenable for construction of >> nodes in guest if the guest itself is not compiled to do so. > I meant: to simplify the implementation, the toolstack can always supply the numa > config info to the guest *if necessary*, no matter if the guest kernel is numa-enabled or > not (even if the guest kernel isn''t numa-enabled, the guest performance may be better > if the toolstack decides to supply a numa config to the guest) > About the "*if necessary*": Andure and I think the user should supply an option > "guestnode" in the guest config file, and you think the toolstack should be able to > automatically determine a "best" value. I raised some questions about > xc_select_best_fit_nodes() in the above paragraph. > Hi Andre, would you like to comment on this?How about an "automatic" global option along with a VM-level "guestnode" option. These options could be work independently or with each other ("guestnode" would take preference over global "automatic" option). We can work out finer details.> >> >>> If a user configures guest numa to "on" for a pv guest, the tools >>> can supply the numa info to PV kernel even if the pv kernel is not > >>> built with guest numa support -- the pv kernel will neglect the info >>> safely; >>> If a user configures guest numa to "off" for a pv guest and the >>> tools don''t supply the numa info to PV kernel, and if the pv kernel >>> > is built with guest numa support, the pv kernel can easily detect >>> this by your new hypercall and will not enable numa. >> These error checks are done even now. But, by checking if the PV >> kernel is built with guest numa support, we don''t require the user to >> configure yet another parameter. Wasn''t that your concern too in the >> very first point ? >> >>> >>> When a user finds the computing capability of a single node can''t >>> satisfy the actual need and hence wants to use guest numa, >>> since the user has specified the amount of guest memory and the >>> number of vcpus in guest config file, I think the user only needs >>> to specify how many guest nodes (the "guestnodes" option in Andre''s >>> patch) the guest will see, and the tools and the hypervisor >>> should co-work to distribute guest memory and vcpus uniformly among >>> the guest nodes(I think we may not want to support non- >>> uniform nodes as that doesn''t look like a typical usage model) -- of >>> course, maybe a specified node doesn''t have the expected >>> amount of memory -- in this case, the guest can continue to run with >>> a slower speed (we can print a warning message to the >>> user); or, if the user does care about predictable guest >>> performance, the guest creation should fail. >> >> Please observe that the patch does all these things plus some more. >> For one, "guestnodes" option doesn''t make sense, since as you observe, >> it needs the user to carefully read the state of the system when >> starting the domain and also the user needs to make sure that the >> guest itself is compiled with numa support. The aim should be to > I think it''s not difficult for a user to specify "guestnodes" and to check if a PV/HVM guest > kernel is numa-enabled or not(anyway, a user needs to ensure that to achieve the > optimal peformance). "xm info/list/vcpu-list" should already supply enough info. I think > it''s reasonable to assume a numa user has more knowledge than a preliminary user. :-) > > I suppose Andure would argue more for the "guestnodes" option. > > PV guest can use the ELFnote as a hit to the toolstack. This may be used as a kind of optimization. > HVM guest can''t use this.As mentioned above, I think we have a good case for both global and VM-level options. What do you think ?> >> automate this part and provide the best performance, given the current >> state. The patch attempts to do that. Secondly, when the guests are >> not compiled with numa support, they would still want a more >> predictable (albeit average) performance. And, by striping the memory >> across the nodes and by pinning the domain vcpus to the union of those >> nodes'' processors, applications (of substantial sizes) could be >> expected to see more predictable performance. >>> >>> How do you like this? My thought is we can make things simple in the >>> first step. :-) >> Please let me know if my comments are not clear. I agree that we >> should shoot for simplicity and also for a common interface. Hope we >> will get there :) > Thanks a lot for all the explanation and discussion. > Yes, we need to agree on a common interface to avoid confusion. > And I still think the "guestnodes/uniform_nodes" idea is more straightforward and the > implementatin is simpler. :-) > > Thanks, > -- Dexuanthanks dulloor _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel