Hi Denis,> could you please take a look at which process is taking so much RAM? > Is there a pile of smbd process around? If you are using Bind-dlz, how > much memory named is eating? Is is the increase in RAM consumption is > linear over time? > > If it is a "samba" process that is eating all the RAM, please check > the PID and the samba-tool processes command to report which part of > samba is weaking havoc the system. > > I'd say you should be fine with 4GiB with such a setup... >Seems lots of samba processes related to rpc_server are making the server crawl (please see attachment) after just 6 days of uptime. I can provide a strace of the most CPU/IO hungry process upon request. Thanks Laurent -------------- next part -------------- Service: PID ----------------------------- dnsupdate 619 cldap_server 612 rpc_server 607 rpc_server 607 rpc_server 607 rpc_server 2895 rpc_server 4252 rpc_server 5977 rpc_server 6365 rpc_server 6433 rpc_server 7695 rpc_server 12060 rpc_server 19006 rpc_server 20669 rpc_server 21124 rpc_server 22408 rpc_server 24379 rpc_server 24537 rpc_server 24674 rpc_server 25816 rpc_server 29316 rpc_server 31377 rpc_server 2812 rpc_server 3875 rpc_server 4511 rpc_server 5109 rpc_server 5433 rpc_server 5503 rpc_server 5534 rpc_server 5684 rpc_server 5738 rpc_server 5795 rpc_server 6100 rpc_server 6117 rpc_server 6219 rpc_server 6420 rpc_server 6487 rpc_server 6492 rpc_server 6545 rpc_server 6814 rpc_server 6893 rpc_server 6919 rpc_server 7076 rpc_server 7499 rpc_server 7545 rpc_server 7556 rpc_server 7564 rpc_server 7619 rpc_server 7633 rpc_server 7793 rpc_server 7936 rpc_server 7950 rpc_server 8068 rpc_server 8217 rpc_server 8270 rpc_server 8288 rpc_server 8355 rpc_server 8361 rpc_server 8537 rpc_server 8735 rpc_server 8839 rpc_server 8888 rpc_server 8926 rpc_server 8948 rpc_server 9012 rpc_server 9026 rpc_server 9094 rpc_server 9362 rpc_server 9368 rpc_server 9396 rpc_server 9438 rpc_server 9498 rpc_server 9527 rpc_server 9537 rpc_server 9554 rpc_server 9573 rpc_server 9590 rpc_server 9623 rpc_server 9640 rpc_server 9654 rpc_server 9701 rpc_server 9855 rpc_server 9872 rpc_server 9892 rpc_server 9906 rpc_server 9930 rpc_server 10038 rpc_server 10123 rpc_server 10133 rpc_server 10141 rpc_server 10178 rpc_server 10246 rpc_server 10330 rpc_server 10347 rpc_server 10360 rpc_server 10469 rpc_server 10474 rpc_server 10532 rpc_server 10546 rpc_server 10587 rpc_server 10601 rpc_server 10608 rpc_server 10624 rpc_server 10639 rpc_server 10661 rpc_server 10688 rpc_server 10710 rpc_server 10736 rpc_server 10851 rpc_server 10973 rpc_server 10979 rpc_server 11001 rpc_server 11103 rpc_server 11140 rpc_server 11186 rpc_server 11200 rpc_server 11553 rpc_server 11573 rpc_server 11589 rpc_server 11598 rpc_server 11628 rpc_server 11690 rpc_server 11718 rpc_server 11741 rpc_server 11754 rpc_server 11845 rpc_server 12057 rpc_server 12374 rpc_server 12402 rpc_server 12613 rpc_server 12919 rpc_server 13585 rpc_server 13620 rpc_server 14301 rpc_server 17864 rpc_server 19112 rpc_server 19751 rpc_server 21992 rpc_server 22147 rpc_server 22310 rpc_server 23937 rpc_server 24338 rpc_server 24529 rpc_server 25531 rpc_server 27219 rpc_server 27449 rpc_server 28529 rpc_server 29515 rpc_server 30658 rpc_server 31823 rpc_server 2049 rpc_server 3080 rpc_server 607 rpc_server 10061 rpc_server 10394 rpc_server 10687 rpc_server 12948 rpc_server 15003 rpc_server 16508 rpc_server 17763 rpc_server 19118 rpc_server 19732 rpc_server 20169 rpc_server 20714 rpc_server 21374 rpc_server 21577 rpc_server 23288 rpc_server 26608 rpc_server 27091 rpc_server 27685 rpc_server 28045 rpc_server 29584 rpc_server 30020 rpc_server 30692 rpc_server 30830 rpc_server 30867 rpc_server 31159 rpc_server 31333 rpc_server 31364 rpc_server 31394 rpc_server 31592 rpc_server 31604 rpc_server 31632 rpc_server 31831 rpc_server 31992 rpc_server 32045 rpc_server 32057 rpc_server 32556 rpc_server 32704 rpc_server 696 rpc_server 762 rpc_server 860 rpc_server 886 rpc_server 918 rpc_server 1075 rpc_server 1228 rpc_server 1330 rpc_server 1525 rpc_server 1547 rpc_server 1727 rpc_server 1739 rpc_server 1869 rpc_server 2116 rpc_server 2127 rpc_server 2175 rpc_server 2281 rpc_server 2309 rpc_server 2444 rpc_server 2626 rpc_server 2713 rpc_server 2729 rpc_server 2948 rpc_server 2964 rpc_server 3104 rpc_server 3170 rpc_server 3244 rpc_server 3266 rpc_server 3314 rpc_server 3350 rpc_server 3481 rpc_server 3593 rpc_server 3642 rpc_server 3699 rpc_server 3729 rpc_server 3864 rpc_server 3872 rpc_server 4034 rpc_server 4068 rpc_server 4102 rpc_server 4163 rpc_server 4189 rpc_server 4274 rpc_server 4312 rpc_server 4329 rpc_server 4825 rpc_server 4929 rpc_server 4990 rpc_server 5042 rpc_server 5148 rpc_server 5393 rpc_server 5637 rpc_server 5775 rpc_server 5881 rpc_server 6391 rpc_server 7105 rpc_server 9093 rpc_server 9312 rpc_server 11742 rpc_server 12450 rpc_server 14203 rpc_server 14317 rpc_server 19157 rpc_server 20383 rpc_server 20975 rpc_server 22658 rpc_server 22730 rpc_server 23008 rpc_server 23759 rpc_server 24014 rpc_server 24847 rpc_server 26653 rpc_server 26740 rpc_server 31528 rpc_server 32186 rpc_server 409 rpc_server 1429 rpc_server 1624 rpc_server 607 rpc_server 9188 rpc_server 9619 rpc_server 10573 rpc_server 11326 rpc_server 14150 rpc_server 14583 rpc_server 16908 rpc_server 17826 rpc_server 18050 rpc_server 18353 rpc_server 18530 rpc_server 19083 rpc_server 19456 rpc_server 20560 rpc_server 20655 rpc_server 22920 rpc_server 27576 rpc_server 27642 rpc_server 28839 rpc_server 29861 rpc_server 30270 rpc_server 30636 rpc_server 30682 rpc_server 31124 rpc_server 31466 rpc_server 31510 rpc_server 31598 rpc_server 31616 rpc_server 31635 rpc_server 31645 rpc_server 31891 rpc_server 32086 rpc_server 32092 rpc_server 32128 rpc_server 32143 rpc_server 32412 rpc_server 32448 rpc_server 32472 rpc_server 32506 rpc_server 32587 rpc_server 448 rpc_server 534 rpc_server 899 rpc_server 922 rpc_server 1025 rpc_server 1134 rpc_server 1211 rpc_server 1275 rpc_server 1294 rpc_server 1433 rpc_server 1464 rpc_server 1523 rpc_server 1582 rpc_server 1648 rpc_server 1660 rpc_server 1857 rpc_server 1891 rpc_server 1903 rpc_server 1918 rpc_server 1975 rpc_server 2067 rpc_server 2156 rpc_server 2234 rpc_server 2290 rpc_server 2337 rpc_server 2375 rpc_server 2424 rpc_server 2447 rpc_server 2580 rpc_server 2606 rpc_server 2635 rpc_server 2689 rpc_server 2712 rpc_server 2835 rpc_server 2862 rpc_server 2951 rpc_server 3218 rpc_server 3283 rpc_server 3347 rpc_server 3659 rpc_server 3923 rpc_server 4045 rpc_server 5171 rpc_server 5249 rpc_server 5283 rpc_server 5351 rpc_server 5405 rpc_server 5710 rpc_server 6179 rpc_server 6537 rpc_server 7435 rpc_server 7605 rpc_server 9539 rpc_server 10230 rpc_server 11767 rpc_server 12403 rpc_server 17572 rpc_server 19788 rpc_server 20088 rpc_server 21149 rpc_server 25503 rpc_server 25767 rpc_server 26716 rpc_server 28188 rpc_server 29235 rpc_server 32278 rpc_server 32442 rpc_server 3726 rpc_server 6156 rpc_server 6714 rpc_server 10847 rpc_server 11076 rpc_server 12244 rpc_server 13302 rpc_server 16601 rpc_server 17910 rpc_server 19073 rpc_server 20179 rpc_server 21139 rpc_server 23468 rpc_server 24809 rpc_server 25978 rpc_server 25988 rpc_server 26050 rpc_server 26103 rpc_server 26121 rpc_server 26717 rpc_server 26727 rpc_server 26937 rpc_server 27002 rpc_server 27197 rpc_server 27314 rpc_server 27485 rpc_server 27553 rpc_server 27585 rpc_server 27703 rpc_server 28180 rpc_server 28448 rpc_server 28515 rpc_server 28542 rpc_server 28553 rpc_server 28599 rpc_server 28676 rpc_server 28803 rpc_server 28819 rpc_server 28961 rpc_server 29010 rpc_server 29100 rpc_server 29146 rpc_server 29233 rpc_server 29370 rpc_server 29530 rpc_server 29608 rpc_server 29690 rpc_server 29841 rpc_server 29894 rpc_server 29973 rpc_server 30193 rpc_server 30236 rpc_server 30264 rpc_server 30318 rpc_server 30359 rpc_server 30420 rpc_server 30485 rpc_server 30536 rpc_server 30555 rpc_server 30646 rpc_server 30685 rpc_server 30881 rpc_server 31012 rpc_server 31169 rpc_server 31268 rpc_server 31332 rpc_server 31370 rpc_server 31507 rpc_server 31538 rpc_server 31670 rpc_server 31889 rpc_server 31947 rpc_server 32081 rpc_server 32109 rpc_server 32156 rpc_server 32264 rpc_server 32301 rpc_server 3627 rpc_server 3654 rpc_server 3799 rpc_server 4843 rpc_server 8184 rpc_server 9095 rpc_server 9601 rpc_server 9707 rpc_server 10859 rpc_server 14672 rpc_server 14795 rpc_server 15344 rpc_server 16541 rpc_server 20064 rpc_server 21008 rpc_server 21662 rpc_server 22689 rpc_server 23300 rpc_server 24508 rpc_server 24630 rpc_server 26198 rpc_server 26741 rpc_server 26862 rpc_server 27303 rpc_server 28422 rpc_server 28541 rpc_server 29502 rpc_server 30634 rpc_server 31063 rpc_server 31901 rpc_server 32447 rpc_server 32500 rpc_server 779 rpc_server 1160 rpc_server 1338 rpc_server 1365 rpc_server 1382 rpc_server 1567 rpc_server 1806 rpc_server 1911 rpc_server 2075 rpc_server 2269 rpc_server 2521 rpc_server 2586 rpc_server 2733 rpc_server 2827 rpc_server 3109 rpc_server 3524 rpc_server 3682 rpc_server 3777 rpc_server 3917 rpc_server 3971 rpc_server 3981 rpc_server 4030 rpc_server 4179 rpc_server 4351 rpc_server 4362 rpc_server 4415 rpc_server 4617 rpc_server 4798 rpc_server 5080 rpc_server 5309 rpc_server 5368 rpc_server 5557 rpc_server 5735 rpc_server 5790 rpc_server 5880 rpc_server 6102 rpc_server 6222 rpc_server 6291 rpc_server 6345 rpc_server 6384 rpc_server 6544 rpc_server 6627 rpc_server 7006 rpc_server 7012 rpc_server 7034 rpc_server 7163 rpc_server 7213 rpc_server 7226 rpc_server 7241 rpc_server 7269 rpc_server 7429 rpc_server 7475 rpc_server 7532 rpc_server 7595 rpc_server 7764 rpc_server 7925 rpc_server 8100 rpc_server 8162 rpc_server 8169 rpc_server 8542 rpc_server 8877 rpc_server 8983 rpc_server 9019 rpc_server 9054 rpc_server 9645 rpc_server 9968 rpc_server 12771 rpc_server 13556 rpc_server 13700 rpc_server 16144 rpc_server 16417 rpc_server 16781 rpc_server 17262 rpc_server 17279 rpc_server 17825 rpc_server 18061 rpc_server 18207 rpc_server 18970 rpc_server 19839 rpc_server 20354 rpc_server 20763 rpc_server 21491 rpc_server 21935 rpc_server 22130 rpc_server 22239 rpc_server 22427 rpc_server 22922 rpc_server 23408 rpc_server 23547 rpc_server 24037 rpc_server 24113 rpc_server 24305 rpc_server 24411 rpc_server 25051 rpc_server 25201 rpc_server 25226 rpc_server 25401 rpc_server 25894 rpc_server 26240 rpc_server 26573 rpc_server 26655 rpc_server 27036 rpc_server 27145 rpc_server 27426 rpc_server 27949 rpc_server 28007 rpc_server 28016 rpc_server 28104 rpc_server 28167 rpc_server 28650 rpc_server 28846 rpc_server 29055 rpc_server 29171 rpc_server 29781 rpc_server 29835 rpc_server 29964 rpc_server 30211 rpc_server 30251 rpc_server 30407 rpc_server 30500 rpc_server 30525 rpc_server 32233 rpc_server 32649 rpc_server 713 rpc_server 1021 rpc_server 1684 rpc_server 2767 rpc_server 3719 rpc_server 5782 rpc_server 6175 rpc_server 607 rpc_server 9133 rpc_server 11656 rpc_server 14177 rpc_server 15021 rpc_server 15217 rpc_server 15633 rpc_server 16901 rpc_server 16975 rpc_server 17474 rpc_server 18180 rpc_server 19161 rpc_server 22647 rpc_server 23333 rpc_server 23663 rpc_server 24443 rpc_server 26322 rpc_server 27192 rpc_server 27791 rpc_server 28336 rpc_server 28455 rpc_server 29284 rpc_server 29898 rpc_server 30266 rpc_server 30923 rpc_server 31212 rpc_server 31720 rpc_server 32260 rpc_server 478 rpc_server 1131 rpc_server 4462 rpc_server 4982 rpc_server 5269 rpc_server 5413 rpc_server 6675 rpc_server 7577 rpc_server 7694 rpc_server 9567 rpc_server 9811 rpc_server 10762 rpc_server 11480 rpc_server 11731 rpc_server 13396 rpc_server 13509 rpc_server 14196 rpc_server 14202 rpc_server 14208 rpc_server 14244 rpc_server 15337 rpc_server 15679 rpc_server 15683 rpc_server 607 rpc_server 607 rpc_server 607 rpc_server 607 rpc_server 607 rpc_server 607 nbt_server 608 winbind_server 622 wrepl_server 610 kdc_server 614 notify-daemon 637 wins_server 608 ldap_server 611 kccsrv 618 samba 0 dreplsrv 615 dnssrv 620
Hi, It looks like what is happening is the Samba RPC server is forking a separate process for each NETLOGON connection it accepts. Normally, when the connection is closed, the child process will be cleaned up. On your setup, the connections appear to be left open (presumably due to the constant mail SSO queries), and so the rpc_server processes just accumulate. You had 669 rpc_server processes, which sounds like roughly one per user. On Samba v4.10, there will be better handling of these NETLOGON connections using the prefork process model. Instead of forking a separate process for each NETLOGON connection, a set number of worker processes will be pre-forked when samba first starts up, and the RPC connections will be shared amongst these worker processes. To use prefork, specify '--model=prefork' as arguments when starting Samba. https://www.samba.org/samba/docs/current/man-html/samba.8.html Unfortunately, the scheduled release date for v4.10 isn't until 5 March 2019. Note that the prefork option does exist in v4.9, however, there is no prefork support in 4.9 for the RPC server process. I'm not sure what you can do in the meantime, other than restart the DCs regularly. Best regards, Tim On 29/01/19 5:19 AM, Laurent CARON via samba wrote:> Hi Denis, > > >> could you please take a look at which process is taking so much RAM? >> Is there a pile of smbd process around? If you are using Bind-dlz, >> how much memory named is eating? Is is the increase in RAM >> consumption is linear over time? >> >> If it is a "samba" process that is eating all the RAM, please check >> the PID and the samba-tool processes command to report which part of >> samba is weaking havoc the system. >> >> I'd say you should be fine with 4GiB with such a setup... >> > > Seems lots of samba processes related to rpc_server are making the > server crawl (please see attachment) after just 6 days of uptime. > > > I can provide a strace of the most CPU/IO hungry process upon request. > > > Thanks > > > Laurent > >
Hi, Actually, you could try starting samba with --model=prefork on 4.9. Samba 4.9 doesn't properly support distributing the NETLOGON connections amongst separate prefork worker processes. But using the prefork process model on 4.9 will mean that there is a single rpc_server process to handle all NETLOGON connections (instead of forking a separate process for each connection, like it's doing currently). Having a single rpc_server process should be sufficient to handle ~700 connections. Best regards, Tim On 29/01/19 9:53 AM, Tim Beale via samba wrote:> Hi, > > It looks like what is happening is the Samba RPC server is forking a > separate process for each NETLOGON connection it accepts. Normally, when > the connection is closed, the child process will be cleaned up. On your > setup, the connections appear to be left open (presumably due to the > constant mail SSO queries), and so the rpc_server processes just > accumulate. You had 669 rpc_server processes, which sounds like roughly > one per user. > > On Samba v4.10, there will be better handling of these NETLOGON > connections using the prefork process model. Instead of forking a > separate process for each NETLOGON connection, a set number of worker > processes will be pre-forked when samba first starts up, and the RPC > connections will be shared amongst these worker processes. To use > prefork, specify '--model=prefork' as arguments when starting Samba. > https://www.samba.org/samba/docs/current/man-html/samba.8.html > > Unfortunately, the scheduled release date for v4.10 isn't until 5 March > 2019. Note that the prefork option does exist in v4.9, however, there is > no prefork support in 4.9 for the RPC server process. I'm not sure what > you can do in the meantime, other than restart the DCs regularly. > > Best regards, > Tim > > On 29/01/19 5:19 AM, Laurent CARON via samba wrote: >> Hi Denis, >> >> >>> could you please take a look at which process is taking so much RAM? >>> Is there a pile of smbd process around? If you are using Bind-dlz, >>> how much memory named is eating? Is is the increase in RAM >>> consumption is linear over time? >>> >>> If it is a "samba" process that is eating all the RAM, please check >>> the PID and the samba-tool processes command to report which part of >>> samba is weaking havoc the system. >>> >>> I'd say you should be fine with 4GiB with such a setup... >>> >> Seems lots of samba processes related to rpc_server are making the >> server crawl (please see attachment) after just 6 days of uptime. >> >> >> I can provide a strace of the most CPU/IO hungry process upon request. >> >> >> Thanks >> >> >> Laurent >> >>