I?m looking for ways to make a ?smbd? daemon use less memory. Typically on our system each process uses 160-200MB virtual memory with ~120-140MB as RSS. Now with ~1400 such processes (and clients connected) this uses a lot of memory? Even though some may be shared. It doesn?t look like a memory leak though since it?s pretty stable at those levels. But it still feels pretty bloated... This particular server has 256GB RAM right now but will have more added. I?m not sure exactly where all this memory is used but I think the fact that our AD has 130,000 users (not all active) and a gazillion groups (but just a few (~2000) with ?gidNumber? set) contributes a bit to it. Any suggestions on ways to trim this would be welcome.> # ps auxdwwww > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > ... > root 30577 0.3 0.0 159492 117936 - Ss 07:00 2:55.80 |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 98861 7.8 0.1 172628 135424 - S 13:54 0:01.56 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > rodro16 30602 0.6 0.1 192128 143028 - S 07:00 4:28.45 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > gorra80 37534 0.6 0.1 201744 147004 - S 12:13 0:00.33 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > kerjo10 35974 0.5 0.1 203952 142024 - S 07:03 0:01.42 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > marma13 47632 0.4 0.1 205100 170616 - S 07:10 0:16.69 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > poasa83 93542 0.1 0.1 199500 156160 - S 13:49 0:00.51 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > vinya32 98774 0.1 0.1 201460 135048 - S 13:53 0:00.09 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 174 0.0 0.1 186616 137720 - I 07:43 0:00.12 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 226 0.0 0.1 204168 140004 - S 07:43 0:01.21 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 227 0.0 0.1 184724 138456 - S 07:43 0:01.11 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 228 0.0 0.1 201652 150152 - S 07:43 0:03.73 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 261 0.0 0.1 196656 143840 - I 11:50 0:00.08 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 334 0.0 0.1 201380 140928 - I 11:51 0:00.80 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > root 361 0.0 0.1 199468 153156 - I 11:51 0:01.39 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.confLooking at the memory allocation output (procstat -v on a FreeBSD machine, a test server with not much activity) on a master ?smbd? with VSZ 160MB and RSS 117 it looks like 102MB of it is allocated memory (the rest is shared libraries) spread out as: Size Allocations 4096 8868 4K * 88868 = 36MB 8192 1 16384 1 32768 1 45056 1 49152 2 73728 1 135168 1 143360 1 180224 1 270336 1 544768 1 2093056 16 2*16 = 32 MB 2097152 12 2*12 = 24 MB 6291456 1 6 MB Since this is on the master smbd then it?ll be inherited by the ones it forks off too. On a busy server (1317 smbd processes currently) the master smbd uses 138MB allocated: 4096 8915 36MB 8192 1 12288 1 106496 2 671744 1 2093056 24 50MB 2097152 22 46MB 4194304 1 4MB And one one of the user smbd processes - also 138MB allocated: 4096 8924 36MB 8192 1 12288 1 106496 2 671744 1 2093056 24 50MB 2097152 22 46MB 4194304 1 If one could reduce the memory usage by say 10MB per process then that would translate to 14GB ?saved? that probably could be of better use somewhere else :-) For reference our smb.conf: (Our AD contains uidNumbers from 10001 to 2000010001 so I guess I could trim the ?AD? range below a little bit, a test didn?t show much improvement though)> ; /liu/etc/samba/smb.conf > > [global] > private directory = /liu/etc/samba/private > lock directory = /liu/var/samba/locks > cache directory = /liu/var/samba/cache > state directory = /liu/var/samba/state > ncalrpc dir = /liu/var/samba/ncalrpc > > ;; Network interfaces > bind interfaces only = true > interfaces = lagg0 > > ;; Server names > server string = Filur06 File Server > netbios name = FILUR06 > > ;; Security type > security = ADS > realm = AD.LIU.SE > workgroup = AD > > ;; ID Mappings > idmap config * : backend = tdb > idmap config * : range = 3000000001-3001000000 > idmap config AD : backend = ad > idmap config AD : range = 1000-3000000000 > idmap config AD : schema_mode = rfc2307 > idmap config AD : unix_primary_group = yes > > > ;; Enable shared robust mutexes > dbwrap_tdb_mutexes:* = yes > > ;; VFS objects to enable > vfs objects = shadow_copy2 zfsacl full_audit > > ;; Auditinig > full_audit:prefix = %u|%I > full_audit:success = none > full_audit:failure = none > full_audit:facility = LOCAL7 > full_audit:priority = notice> > shadow:snapdir = .zfs/snapshot > shadow:format = auto-%Y-%m-%d.%H:%M:%S > shadow:sort = desc > shadow:localtime = yes > shadow:snapdirseverywhere = yes > > ;; Handle ZFS ACLs > zfsacl:acesort = dontcare > > nfs4:mode = special > nfs4:acedup = merge > ; nfs4:chown = yes > > ;; Options from Puppet/Hiera (data/common.yaml and nodes/$HOSTNAME.yaml) > kerberos method = system keytab > kernel change notify = false > socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536 > client ldap sasl wrapping = seal > socket listen backlog = 1024 > invalid users = root > min protocol = SMB2 > smb encrypt = auto > time server = true > hostname lookups = false > follow symlinks = true > wide links = false > unix extensions = false > winbind nested groups = false > winbind enum users = false > winbind enum groups = false > winbind use default domain = yes > winbind normalize names = yes > winbind max clients = 2000 > winbind max domain connections = 5 > winbind nss info = template > template homedir = /home/%U > template shell = /bin/sh > nt acl support = yes > inherit acls = yes > inherit permissions = yes > store dos attributes = yes > ea support = false > map hidden = false > map system = false> map archive = false > map read only = permissions > case sensitive = auto > logging = syslog:1 file:2 > log level = 1 auth:2 winbind:2 acls:0 > load printers = false > printing = bsd > printcap name = /dev/null > disable spoolss = true > veto files = /.zfs/ > hide files = /__--None--__/ > delete veto files = yes > > ;; ============================ Share Definitions =============================> > [homes] > browseable = false > printable = false > public = false > writeable = true> [admin$] > copy = homes > full_audit:success = connect disconnect opendir mkdir rmdir closedir open close read pread write pwrite sendfile rename unlink chmod fchmod chown fchown chdir ftruncate lock symlink readlink link mknod realpath > full_audit:failure = connect > comment = Admin Access > path = /export > valid users = @superusers > admin users = @superusers > preexec close = yes > preexec = /liu/libexec/mfa_auth %u > > [students] > copy = homes > comment = Student Home Directories > path = /export/students > inherit owner = yes > > [staff] > copy = homes > comment = Staff Home Directories > path = /export/staff > inherit owner = yesTDB file sizes (on the busy server):> # du -hs */*.tdb > 877K cache/netsamlogon_cache.tdb > 17K locks/brlock.tdb > 4.5K locks/g_lock.tdb > 17K locks/gencache_notrans.tdb > 23M locks/gencache.tdb > 109K locks/leases.tdb > 5.8M locks/locking.tdb > 512B locks/mutex.tdb > 4.5K locks/names.tdb > 4.5K locks/printer_list.tdb > 25K locks/serverid.tdb > 13K locks/smbd_cleanupd.tdb > 512B locks/smbXsrv_client_global.tdb > 197K locks/smbXsrv_open_global.tdb > 2.8M locks/smbXsrv_session_global.tdb > 109K locks/smbXsrv_tcon_global.tdb > 4.5K locks/smbXsrv_version_global.tdb > 25K state/account_policy.tdb > 512B state/group_mapping.tdb > 33K state/registry.tdb > 25K state/share_info.tdb > 6.6M state/winbindd_cache.tdb > 13K state/winbindd_idmap.tdbAny suggestions before I start digging deeper ? - Peter
On Tue, Mar 31, 2020 at 03:58:49PM +0200, Peter Eriksson via samba wrote:> I?m looking for ways to make a ?smbd? daemon use less memory. Typically on our system each process uses 160-200MB virtual memory with ~120-140MB as RSS. Now with ~1400 such processes (and clients connected) this uses a lot of memory? Even though some may be shared. It doesn?t look like a memory leak though since it?s pretty stable at those levels. But it still feels pretty bloated... > > This particular server has 256GB RAM right now but will have more added. > > I?m not sure exactly where all this memory is used but I think the fact that our AD has 130,000 users (not all active) and a gazillion groups (but just a few (~2000) with ?gidNumber? set) contributes a bit to it. > > > Any suggestions on ways to trim this would be welcome. > > > # ps auxdwwww > > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND > > ... > > root 30577 0.3 0.0 159492 117936 - Ss 07:00 2:55.80 |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 98861 7.8 0.1 172628 135424 - S 13:54 0:01.56 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > rodro16 30602 0.6 0.1 192128 143028 - S 07:00 4:28.45 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > gorra80 37534 0.6 0.1 201744 147004 - S 12:13 0:00.33 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > kerjo10 35974 0.5 0.1 203952 142024 - S 07:03 0:01.42 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > marma13 47632 0.4 0.1 205100 170616 - S 07:10 0:16.69 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > poasa83 93542 0.1 0.1 199500 156160 - S 13:49 0:00.51 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > vinya32 98774 0.1 0.1 201460 135048 - S 13:53 0:00.09 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 174 0.0 0.1 186616 137720 - I 07:43 0:00.12 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 226 0.0 0.1 204168 140004 - S 07:43 0:01.21 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 227 0.0 0.1 184724 138456 - S 07:43 0:01.11 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 228 0.0 0.1 201652 150152 - S 07:43 0:03.73 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 261 0.0 0.1 196656 143840 - I 11:50 0:00.08 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 334 0.0 0.1 201380 140928 - I 11:51 0:00.80 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > root 361 0.0 0.1 199468 153156 - I 11:51 0:01.39 | |-- /liu/sbin/smbd --daemon --configfile=/liu/etc/samba/smb.conf > > > Looking at the memory allocation output (procstat -v on a FreeBSD machine, a test server with not much activity) on a master ?smbd? with VSZ 160MB and RSS 117 it looks like 102MB of it is allocated memory (the rest is shared libraries) spread out as: > > Size Allocations > 4096 8868 4K * 88868 = 36MB > 8192 1 > 16384 1 > 32768 1 > 45056 1 > 49152 2 > 73728 1 > 135168 1 > 143360 1 > 180224 1 > 270336 1 > 544768 1 > 2093056 16 2*16 = 32 MB > 2097152 12 2*12 = 24 MB > 6291456 1 6 MB > > Since this is on the master smbd then it?ll be inherited by the ones it forks off too.Does this included the mmap'ed tdb databases ? If so it's probably not as bad as it looks. Is there a way you can split this out to see filenames of mmap'ed regions ? The other thing to do is use smbcontrol to query the named or unnamed talloc allocations (we keep track of them all).