Hi Samba gurus, ? Samba 4.19.9 on FreeBSD 14.3 with a ZFS pool for storage. ? This is storage for roughly 30 million image files that our site displays via a number of Windows 2019 servers running IIS. There are a few other uses, however that is the primary usage. ? A few weeks back a software update was released for the web site running on those IIS servers and we noticed a marked increase in load averages after this. I do believe its related to these open file messages.? ? We are seeing errors in our log about too many open files: Sep 30 11:13:11 XXXXX smbd[1111]: [2025/09/30 11:13:11.346275, 0] ../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:11 XXXXX smbd[1111]: reopen_one_log: Unable to open new log file '/var/log/samba4/log.smbd': Too many open files Sep 30 11:13:14 XXXXX smbd[1111]: [2025/09/30 11:13:14.603146, 0] ../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:14 XXXXX smbd[1111]: reopen_one_log: Unable to open new log file '/var/log/samba4/log.smbd': Too many open files ? The log file is being written to, however I don't know if process 1111 is writing to it. This is notifyd [XXXXXXX /var/log]# ps aux |grep 1111 XXXXXX 1111 0.0 0.5 285832 79212 - S Thu22 2:17.17 smbd: notifyd (smbd) XXXXXX 38195 0.0 0.0 13836 1920 0 S+ 11:16 0:00.00 grep 1111 ? If I run an fstat command to sum up the open files, I see it is hitting the limit set for the process which is 36000. ? [XXXXXXX /var/log]# fstat|cut -w -f1,2,3|sort |uniq -c|sort -rn | grep smbd? 33603 XXXXXX smbd 1111 ? Yesterday I increased this from 25600 to 33600 and immediately this process hit that limit. The smbd processes handing the file shares usually stay in the range of 2000-3000 open files per process. I am trying to figure out what is causing this. The FreeBSD server's load averages are higher than they historically have been. I know the code is C# that the IIS servers run and I do not have access to it. I understand that a new filestream operation was introduced that allowed the browser to pick a 'medium' sized file and these are properly disposed of in the code as I am being told. I am looking for any suggestions or troubleshooting steps to help identify what is causing these file locks, or what I can do about it besides increasing the limit of open files? eventually that will just overload the system as we discovered when we did try to increase it. The system became unusable at roughly 520,000 open files which happened minutes after smbd was started. Below is the sanitized smb4.conf * [XXXXXXX /var/log]# cat /XXX/smb4.conf realm = DOMAIN.DOMAIN workgroup = WORKGROUP security = ADS guest account = User winbind separator = + winbind enum groups = Yes winbind enum users = Yes winbind nss info = rfc2307 winbind offline logon = yes socket options = TCP_NODELAY IPTOS_LOWDELAY idmap config * : range = 1000-200000 idmap config * : backend = tdb template shell = /xxx/xxx/xxx/xxx template homedir = /home/%D/%U ntlm auth = yes passdb backend samba_dsdb server role = member server username map script = /bin/echo log level = 6 follow symlinks = yes unix extensions = no nt acl support = yes inherit acls = no map acl inherit = yes deadtime = 10 [share] path = /path/on/zfs guest ok = yes writeable = yes browsable = yes inherit permissions = yes inherit owner = yes locking = yes oplocks = yes kernel oplocks = no level2 oplocks = true strict locking = auto Thank you in advance!
Devon Johnson via samba said on Wed, 01 Oct 2025 15:24:22 -0700>Hi Samba gurus, >? >Samba 4.19.9 on FreeBSD 14.3 with a ZFS pool for storage. >? >This is storage for roughly 30 million image files that our site >displays via a number of Windows 2019 servers running IIS. There are a >few other uses, however that is the primary usage.That's a lot of files. Do you have them in a directory hierarchy, and if so, how deep and how wide?>A few weeks back a >software update was released for the web site running on those IIS >servers and we noticed a marked increase in load averages after this. >I do believe its related to these open file messages. We are seeing >errors in our log about too many open files: Sep 30 11:13:11 XXXXX >smbd[1111]: [2025/09/30 11:13:11.346275, 0] >../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:11 XXXXX >smbd[1111]: reopen_one_log: Unable to open new log file >'/var/log/samba4/log.smbd': Too many open files Sep 30 11:13:14 XXXXX >smbd[1111]: [2025/09/30 11:13:14.603146, 0] >../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:14 XXXXX >smbd[1111]: reopen_one_log: Unable to open new log file >'/var/log/samba4/log.smbd': Too many open files The log file is being >written to, however I don't know if process 1111 is writing to it. >This is notifyd [XXXXXXX /var/log]# ps aux |grep 1111 XXXXXX 1111 0.0 >0.5 285832 79212 - S Thu22 2:17.17 smbd: notifyd (smbd) XXXXXX 38195 >0.0 0.0 13836 1920 0 S+ 11:16 0:00.00 grep 1111 If I run an fstat >command to sum up the open files, I see it is hitting the limit set >for the process which is 36000. [XXXXXXX /var/log]# fstat|cut -w >-f1,2,3|sort |uniq -c|sort -rn | grep smbd 33603 XXXXXX smbd 1111 >Yesterday I increased this from 25600 to 33600 and immediately this >process hit that limit.Just for fun, make a shellscript that detects zombie processes and see how many there are. On my computer, a poorly written Python program (written by me unfortunately) leaves zombies that hold files open. If you have a lot of zombies, that problem needs to be fixed whether it's the root cause or not. I don't have fstat on my computer so I can't comment on how similar or different it is than lsof, but lsof redirected to a file can give you a snapshot of open files and who has them open. This file, which I assume will be at least 33600 lines long, might give you clues if you look for patterns within it. These are a couple easy general diagnostic tests. I don't have the knowledge to suggest anything further. SteveT Steve Litt http://444domains.com
On Wed, 01 Oct 2025 15:24:22 -0700 Devon Johnson via samba <samba at lists.samba.org> wrote:> Hi Samba gurus, > ? > Samba 4.19.9 on FreeBSD 14.3 with a ZFS pool for storage. > ? > This is storage for roughly 30 million image files that our site > displays via a number of Windows 2019 servers running IIS. There are > a few other uses, however that is the primary usage. A few weeks back > a software update was released for the web site running on those IIS > servers and we noticed a marked increase in load averages after this. > I do believe its related to these open file messages. We are seeing > errors in our log about too many open files: Sep 30 11:13:11 XXXXX > smbd[1111]: [2025/09/30 11:13:11.346275, 0] > ../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:11 XXXXX > smbd[1111]: reopen_one_log: Unable to open new log file > '/var/log/samba4/log.smbd': Too many open files Sep 30 11:13:14 XXXXX > smbd[1111]: [2025/09/30 11:13:14.603146, 0] > ../../lib/util/debug.c:1274(reopen_one_log) Sep 30 11:13:14 XXXXX > smbd[1111]: reopen_one_log: Unable to open new log file > '/var/log/samba4/log.smbd': Too many open files The log file is being > written to, however I don't know if process 1111 is writing to it. > This is notifyd [XXXXXXX /var/log]# ps aux |grep 1111 XXXXXX 1111 0.0 > 0.5 285832 79212 - S Thu22 2:17.17 smbd: notifyd (smbd) XXXXXX 38195 > 0.0 0.0 13836 1920 0 S+ 11:16 0:00.00 grep 1111 If I run an fstat > command to sum up the open files, I see it is hitting the limit set > for the process which is 36000. [XXXXXXX /var/log]# fstat|cut -w > -f1,2,3|sort |uniq -c|sort -rn | grep smbd 33603 XXXXXX smbd 1111 > Yesterday I increased this from 25600 to 33600 and immediately this > process hit that limit. > > The smbd processes handing the file shares usually stay in the range > of 2000-3000 open files per process. > > I am trying to figure out what is causing this. The FreeBSD server's > load averages are higher than they historically have been. I know the > code is C# that the IIS servers run and I do not have access to it. I > understand that a new filestream operation was introduced that > allowed the browser to pick a 'medium' sized file and these are > properly disposed of in the code as I am being told. > > I am looking for any suggestions or troubleshooting steps to help > identify what is causing these file locks, or what I can do about it > besides increasing the limit of open files? eventually that will just > overload the system as we discovered when we did try to increase it. > The system became unusable at roughly 520,000 open files which > happened minutes after smbd was started. > > Below is the sanitized smb4.conf * > [XXXXXXX /var/log]# cat /XXX/smb4.conf > realm = DOMAIN.DOMAIN > workgroup = WORKGROUP > security = ADS > guest account = User > winbind separator = + > winbind enum groups = Yes > winbind enum users = Yes > winbind nss info = rfc2307 > winbind offline logon = yes > socket options = TCP_NODELAY IPTOS_LOWDELAY > idmap config * : range = 1000-200000 > idmap config * : backend = tdb > template shell = /xxx/xxx/xxx/xxx > template homedir = /home/%D/%U > ntlm auth = yes > passdb backend samba_dsdb > server role = member server > username map script = /bin/echo > log level = 6 > follow symlinks = yes > unix extensions = no > nt acl support = yes > inherit acls = no > map acl inherit = yes > deadtime = 10 > > [share] > path = /path/on/zfs > guest ok = yes > writeable = yes > browsable = yes > inherit permissions = yes > inherit owner = yes > locking = yes > oplocks = yes > kernel oplocks = no > level2 oplocks = true > strict locking = auto > > Thank you in advance!I am sorry, but there isn't really a polite way of saying this, but your smb.conf file is rubbish. Lets start with the pasdb backend, the one in your smb.conf is only meant to be used on a Samba AD DC, your smb.conf isn't for a Samba AD DC. However, it might be being ignored because there isn't an equals sign, '=' , between 'passdb backend' and 'samba_dsdb'. You haven't set the required idmap backend for 'WORKGROUP' (I take it that there are AD DCs somewhere), this means that everything is going into the default '*' domain. Do you have a local user called 'User' ? The real question is, why did you set this, you do not appear to be using guest access. You think you are, you have 'guest ok = yes' set in '[share]', but you do not have 'map to guest = bad user' set in '[global]' and without it, there is no guest access. By setting the 'enum' lines you are just slowing things down. Why sanitise the template shell, it should be something like /bin/bash (or the bsd equivalent) ? Why are you using '/bin/echo' for the 'username map script' ? Rowland