I''m having a problem with NUMA on an AMD G34 system. Xen seems to think
that out of the 16 total cores, there are 12 on node0 and 4 on node3.
The box is:
SuperMicro H8DGU (dual socket G34)
Pair of AMD 6128 CPUs (8 core each)
32 GB of RAM.
BIOS has been upgraded three times - no help.
xen-unstable from Feb 12 (same problem on 3.4.x).
I''m including the output from the "u" command below to show
Xen''s idea
of the CPU->node mapping. Complete boot log is here:
http://www.panix.com/~marcotte/g34/boot.txt
Any help would be appreciated.
Thanks.
- Brian
(XEN) Xen version 4.1.0-rc5-pre (root@nyc.access.net) (gcc version 4.3.2 (Debian
4.3.2-1.1) ) Sat Feb 12 18:53:49 EST 2011
(XEN) Command line: dom0_mem=600M com1=57600,8n1 console=com1 no-mce
loglvl=debug
(XEN) ''u'' pressed -> dumping numa info (now-0x2:5A61B881)
(XEN) idx0 -> NODE0 start->0 size->4325376
(XEN) phys_to_nid(0000000000001000) -> 0 should be 0
(XEN) idx3 -> NODE3 start->4325376 size->4194304
(XEN) phys_to_nid(0000000420001000) -> 3 should be 3
(XEN) CPU0 -> NODE0
(XEN) CPU1 -> NODE0
(XEN) CPU2 -> NODE0
(XEN) CPU3 -> NODE0
(XEN) CPU4 -> NODE0
(XEN) CPU5 -> NODE0
(XEN) CPU6 -> NODE0
(XEN) CPU7 -> NODE0
(XEN) CPU8 -> NODE3
(XEN) CPU9 -> NODE3
(XEN) CPU10 -> NODE3
(XEN) CPU11 -> NODE3
(XEN) CPU12 -> NODE0
(XEN) CPU13 -> NODE0
(XEN) CPU14 -> NODE0
(XEN) CPU15 -> NODE0
(XEN) Memory location of each domain:
(XEN) Domain 0 (total: 153600):
(XEN) Node 0: 132232
(XEN) Node 3: 21368
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Brian Marcotte wrote:> I''m having a problem with NUMA on an AMD G34 system. Xen seems to think > that out of the 16 total cores, there are 12 on node0 and 4 on node3. > > The box is: > > SuperMicro H8DGU (dual socket G34) > Pair of AMD 6128 CPUs (8 core each) > 32 GB of RAM. > BIOS has been upgraded three times - no help. > > xen-unstable from Feb 12 (same problem on 3.4.x). > > I''m including the output from the "u" command below to show Xen''s idea > of the CPU->node mapping. Complete boot log is here: > > http://www.panix.com/~marcotte/g34/boot.txt > > Any help would be appreciated.Have you checked that you populated the DIMM slots properly? The SRAT CPU table seems sane, but the memory table seems to say that there is no memory on node 1 and 3, thus Xen will simply put those CPUs to node 0 as it cannot cope with memory-less nodes. If I understand the manual right, you first have to populate all the blue slots, then the black ones. Hope that helps, Andre.> > Thanks. > > - Brian > > (XEN) Xen version 4.1.0-rc5-pre (root@nyc.access.net) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) Sat Feb 12 18:53:49 EST 2011 > (XEN) Command line: dom0_mem=600M com1=57600,8n1 console=com1 no-mce loglvl=debug > > (XEN) ''u'' pressed -> dumping numa info (now-0x2:5A61B881) > (XEN) idx0 -> NODE0 start->0 size->4325376 > (XEN) phys_to_nid(0000000000001000) -> 0 should be 0 > (XEN) idx3 -> NODE3 start->4325376 size->4194304 > (XEN) phys_to_nid(0000000420001000) -> 3 should be 3 > (XEN) CPU0 -> NODE0 > (XEN) CPU1 -> NODE0 > (XEN) CPU2 -> NODE0 > (XEN) CPU3 -> NODE0 > (XEN) CPU4 -> NODE0 > (XEN) CPU5 -> NODE0 > (XEN) CPU6 -> NODE0 > (XEN) CPU7 -> NODE0 > (XEN) CPU8 -> NODE3 > (XEN) CPU9 -> NODE3 > (XEN) CPU10 -> NODE3 > (XEN) CPU11 -> NODE3 > (XEN) CPU12 -> NODE0 > (XEN) CPU13 -> NODE0 > (XEN) CPU14 -> NODE0 > (XEN) CPU15 -> NODE0 > (XEN) Memory location of each domain: > (XEN) Domain 0 (total: 153600): > (XEN) Node 0: 132232 > (XEN) Node 3: 21368 > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> The SRAT CPU table seems sane, but the memory table seems to say > that there is no memory on node 1 and 3, thus Xen will simply put > those CPUs to node 0 as it cannot cope with memory-less nodes.Ah. I think that''s what was causing my confusion. Since the cpu-to-node mapping was unbalanced and Xen wasn''t complaining, I thought for sure this was a problem with Xen and newer CPUs and I''d need to help with debugging the problem. In this case, I suppose I''ll remove a CPU and rearrange the memory until it''s time to add more memory. Thanks. -- - Brian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Brian Marcotte wrote:>> The SRAT CPU table seems sane, but the memory table seems to say >> that there is no memory on node 1 and 3, thus Xen will simply put >> those CPUs to node 0 as it cannot cope with memory-less nodes. > > Ah. I think that''s what was causing my confusion. Since the cpu-to-node > mapping was unbalanced and Xen wasn''t complaining, I thought for sure > this was a problem with Xen and newer CPUs and I''d need to help with > debugging the problem.I prepared Xen to handle this situation (nodes without memory) before the M-C launch, before that it simply crashed. I agree that it is not obvious that you need to populate at least 4 DIMMs per CPU.> In this case, I suppose I''ll remove a CPU and rearrange the memory > until it''s time to add more memory.Do you have only 4 DIMMs? If yes, you could do so if you can cope with half of the CPU power. If you don''t want to sacrifice the second CPU, you could go ahead and use the system with this kind of strange NUMA setup. Depending on your workload you will benefit more from this setup, as the impact of remote memory on most application is not that huge (compared to less cores, at least). But if you have 8 DIMMs, you should keep the second CPU and put all DIMMs in the blue slots. Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I prepared Xen to handle this situation (nodes without memory) > before the M-C launch, before that it simply crashed. I agree that > it is not obvious that you need to populate at least 4 DIMMs per > CPU.I suppose that Xen could arrange the cpu to node mapping such that the memoryless CPUs were assigned to the "nearest" node making it so you could benefit from the shared L3 cache. I don''t know that it''s worth anyone''s time though. Perhaps a warning would suffice?> Do you have only 4 DIMMs?Yes, but I don''t expect that to last too long. We find that over the lifetime of a machine, we max out the RAM slots long before we run out of CPU. The second CPU is mostly just so we can (eventually) fill all the RAM slots. It''s not a problem to run with one CPU for while. I may also just pin the domains to particular CPU cores rather than pull the CPU. Thanks for your help. -- - Brian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Brian Marcotte wrote:>> I prepared Xen to handle this situation (nodes without memory) >> before the M-C launch, before that it simply crashed. I agree that >> it is not obvious that you need to populate at least 4 DIMMs per >> CPU. > > I suppose that Xen could arrange the cpu to node mapping such that > the memoryless CPUs were assigned to the "nearest" node making it so > you could benefit from the shared L3 cache. > > I don''t know that it''s worth anyone''s time though. Perhaps a warning > would suffice?I also considered more possible action back then, but found that it is probably not worth the effort, as this kind of configuration was considered kind of bogus. But a warning message sounds like a good idea.> >> Do you have only 4 DIMMs? > > Yes, but I don''t expect that to last too long. We find that over the > lifetime of a machine, we max out the RAM slots long before we run out > of CPU. The second CPU is mostly just so we can (eventually) fill all > the RAM slots. It''s not a problem to run with one CPU for while. I may > also just pin the domains to particular CPU cores rather than pull the > CPU.Yes, just put all the DIMMs in the blue slots of the first CPU and boot Xen with maxcpus=8. This should give you the best results without much fiddling with the hardware. Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel