I''ve been looking at the alloc_heap_pages() and alloc_domheap_pages() routines with an eye on making things NUMA (allocate local to a node) aware. I''ve got some counters and I''m seeing something around 35 to 40 calls to alloc_heap_pages() to get xen and dom0 up. Then when I launch domU, the number of allocs goes up by 65k. This is incurred for every domU launch. When launching the domain, a dom_mem_op ([1]increase_reservation) is being called which turns into a alloc_dom_mem() which calls alloc_domheap_pages(). I''ve yet to find what is calling all of those increase_reservation memops. Can anyone help explain this or point me to the code I should be looking at? Thanks in advance. 1. Instrumented alloc_dom_mem() output: (XEN) mem_op: DOM1 start(0) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(880) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(1751) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(26b2) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(3589) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(4514) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(547b) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(6364) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(71df) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(803a) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(8f52) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(9e70) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(add4) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(bd2e) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(cc95) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(db94) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(ea77) nr_extents(65536) order(0) (XEN) mem_op: DOM1 start(f89f) nr_extents(65536) order(0) -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
* Ryan Harper <ryanh@us.ibm.com> [2005-07-22 15:56]:> I''ve been looking at the alloc_heap_pages() and alloc_domheap_pages() > routines with an eye on making things NUMA (allocate local to a node) > aware. I''ve got some counters and I''m seeing something around 35 to > 40 calls to alloc_heap_pages() to get xen and dom0 up. Then when I > launch domU, the number of allocs goes up by 65k. This is incurred for > every domU launch. When launching the domain, a dom_mem_op > ([1]increase_reservation) is being called which turns into a > alloc_dom_mem() which calls alloc_domheap_pages(). I''ve yet to find > what is calling all of those increase_reservation memops. Can anyone > help explain this or point me to the code I should be looking at? > > Thanks in advance. > > 1. Instrumented alloc_dom_mem() output: > (XEN) mem_op: DOM1 start(0) nr_extents(65536) order(0)After some more digging, this is normal. I just didn''t know what was going on. The domU was allocated 256MB, that is 262144 KB, or 65536 4K pages. libxc issues a increase_reservation MEMOP and converts the mem value requested into number of pages. I didn''t make the connection since the argument was memkb / 4 , nor that nr_extents is related to number of pages. I also naively expected to see fewer allocations for domU memory, but after thinking about it, domU needs 256M of memory, and it _doesn''t_ have to be contiguous, and by asking for a page at a time, we allow the allocator to help decrease fragmentation. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel