Srinivasan, Arunkumar
2018-Aug-17 15:11 UTC
[Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
Dear R-devel list, R's detectCores() function internally calls "ncpus" function to get the total number of logical processors. However, this doesnot seem to take NUMA into account on Windows machines. On a machine having 48 processors (24 cores) in total and windows server 2012 installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If NUMA is disabled, detectCores() returns 48. Similarly, on a machine with 88 cores (176 processors) and windows server 2012, detectCores() with NUMA disabled only returns the maximum value of 64. If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will only return 44. This is particularly limiting since we cannot get to use all processors by enabling/disabling NUMA in this case. We think this is because R's ncpus.c file uses "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). Specifically, quoting from the first link: "On systems with more than 64 logical processors, the?GetLogicalProcessorInformation?function retrieves logical processor information about processors in the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx?to which the calling thread is currently assigned. Use the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx?function to retrieve information about processors in all processor groups on the system." Therefore, it might be possible to get the right count of total processors even with NUMA enabled by using "GetLogicalProcessorInformationEX". It'd be nice to know what you think. Thank you very much, Arun. -- Arun Srinivasan Analyst, Millennium Management LLC 50 Berkeley Street | London, W1J 8HD ###################################################################### The information contained in this communication is confi...{{dropped:30}}
Tomas Kalibera
2018-Aug-21 10:50 UTC
[Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
Dear Arun, thank you for the report. I agree with the analysis, detectCores() will only report logical processors in the NUMA group in which R is running. I don't have a system to test on, could you please check these workarounds for me on your systems? # number of logical processors - what detectCores() should return out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of cores - what detectCores(FALSE) should return out <- system("wmic cpu get numberofcores", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of physical processors - as a sanity check system("wmic computersystem get numberofprocessors") Thanks, Tomas On 08/17/2018 05:11 PM, Srinivasan, Arunkumar wrote:> Dear R-devel list, > > R's detectCores() function internally calls "ncpus" function to get the total number of logical processors. However, this doesnot seem to take NUMA into account on Windows machines. > > On a machine having 48 processors (24 cores) in total and windows server 2012 installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If NUMA is disabled, detectCores() returns 48. > > Similarly, on a machine with 88 cores (176 processors) and windows server 2012, detectCores() with NUMA disabled only returns the maximum value of 64. If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will only return 44. This is particularly limiting since we cannot get to use all processors by enabling/disabling NUMA in this case. > > We think this is because R's ncpus.c file uses "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). Specifically, quoting from the first link: > > "On systems with more than 64 logical processors, the?GetLogicalProcessorInformation?function retrieves logical processor information about processors in the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx?to which the calling thread is currently assigned. Use the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx?function to retrieve information about processors in all processor groups on the system." > > Therefore, it might be possible to get the right count of total processors even with NUMA enabled by using "GetLogicalProcessorInformationEX". It'd be nice to know what you think. > > Thank you very much, > Arun. > > -- > Arun Srinivasan > Analyst, Millennium Management LLC > 50 Berkeley Street | London, W1J 8HD > > > ###################################################################### > > The information contained in this communication is con...{{dropped:11}}
Srinivasan, Arunkumar
2018-Aug-21 12:53 UTC
[Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
Dear Tomas, thank you for looking into this. Here's the output: # number of logical processors - what detectCores() should return out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE) [1] "NumberOfLogicalProcessors \r" "22 \r" "22 \r" [4] "20 \r" "22 \r" "\r" sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # [1] 86 [I've asked the IT team to understand why one of the values is 20 instead of 22]. # number of cores - what detectCores(FALSE) should return out <- system("wmic cpu get numberofcores", intern=TRUE) [1] "NumberOfCores \r" "22 \r" "22 \r" "20 \r" "22 \r" [6] "\r" sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # [1] 86 [Currently hyperthreading is disabled. So this output being identical to the previous output makes sense]. system("wmic computersystem get numberofprocessors") NumberOfProcessors 4 In addition, I'd also bring to your attention this documentation: https://docs.microsoft.com/en-us/windows/desktop/ProcThread/processor-groups on processor groups which explain how one should go about running a process ro run on multiple groups (which seems to be different to NUMA). All this seems overly complicated to allow a process to use all cores by default TBH. Here's a project on Github 'fio' where the issue of running a process on more than 1 processor group has come up - https://github.com/axboe/fio/issues/527 and is addressed - https://github.com/axboe/fio/blob/c479640d6208236744f0562b1e79535eec290e2b/os/os-windows-7.h . I am not sure though if this is entirely relevant since we would be forking new processes in R instead of allowing a single process to use all cores. Apologies if this is utterly irrelevant. Thank you, Arun. From: Tomas Kalibera <tomas.kalibera at gmail.com> Sent: 21 August 2018 11:50 To: Srinivasan, Arunkumar <Arunkumar.Srinivasan at uk.mlp.com>; r-devel at r-project.org Subject: Re: [Rd] Get Logical processor count correctly whether NUMA is enabled or disabled Dear Arun, thank you for the report. I agree with the analysis, detectCores() will only report logical processors in the NUMA group in which R is running. I don't have a system to test on, could you please check these workarounds for me on your systems? # number of logical processors - what detectCores() should return out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of cores - what detectCores(FALSE) should return out <- system("wmic cpu get numberofcores", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))) # number of physical processors - as a sanity check system("wmic computersystem get numberofprocessors") Thanks, Tomas On 08/17/2018 05:11 PM, Srinivasan, Arunkumar wrote: Dear R-devel list, R's detectCores() function internally calls "ncpus" function to get the total number of logical processors. However, this doesnot seem to take NUMA into account on Windows machines. On a machine having 48 processors (24 cores) in total and windows server 2012 installed, if NUMA is enabled and has 2 nodes (node 0 and node 1 each having 24 CPUs), then R's detectCores() only detects 24 instead of the total 48. If NUMA is disabled, detectCores() returns 48. Similarly, on a machine with 88 cores (176 processors) and windows server 2012, detectCores() with NUMA disabled only returns the maximum value of 64. If NUMA is enabled with 4 nodes (44 processors each), then detectCores() will only return 44. This is particularly limiting since we cannot get to use all processors by enabling/disabling NUMA in this case. We think this is because R's ncpus.c file uses "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION" (https://msdn.microsoft.com/en-us/library/windows/desktop/ms683194(v=vs.85).aspx) instead of "PSYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX" (https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx). Specifically, quoting from the first link: "On systems with more than 64 logical processors, the?GetLogicalProcessorInformation?function retrieves logical processor information about processors in the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405503(v=vs.85).aspx?to which the calling thread is currently assigned. Use the?https://msdn.microsoft.com/en-us/library/windows/desktop/dd405488(v=vs.85).aspx?function to retrieve information about processors in all processor groups on the system." Therefore, it might be possible to get the right count of total processors even with NUMA enabled by using "GetLogicalProcessorInformationEX". It'd be nice to know what you think. Thank you very much, Arun. -- Arun Srinivasan Analyst, Millennium Management LLC 50 Berkeley Street | London, W1J 8HD ###################################################################### The information contained in this communication is confi...{{dropped:30}} ______________________________________________ mailto:R-devel at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel ###################################################################### The information contained in this communication is confidential and intended only for the individual(s) named above. If you are not a named addressee, please notify the sender immediately and delete this email from your system and do not disclose the email or any part of it to any person. The views expressed in this email are the views of the author and do not necessarily represent the views of Millennium Capital Partners LLP (MCP LLP) or any of its affiliates. Outgoing and incoming electronic communications of MCP LLP and its affiliates, including telephone communications, may be electronically archived and subject to review and/or disclosure to someone other than the recipient. MCP LLP is authorized and regulated by the Financial Conduct Authority. Millennium Capital Partners LLP is a limited liability partnership registered in England & Wales with number OC312897 and with its registered office at 50 Berkeley Street, London, W1J 8HD. ######################################################################
Maybe Matching Threads
- Get Logical processor count correctly whether NUMA is enabled or disabled
- Get Logical processor count correctly whether NUMA is enabled or disabled
- Get Logical processor count correctly whether NUMA is enabled or disabled
- Get Logical processor count correctly whether NUMA is enabled or disabled
- Get Logical processor count correctly whether NUMA is enabled or disabled