Tinco Andringa
2022-Jun-29 10:24 UTC
[Samba] Leaking file descriptors in Version 4.13.17-Ubuntu
Hi,
For about 2 weeks, I think since we updated smbd to version Version
4.13.17-Ubuntu we have encountered a "too many files open" error. I
set out
to investigate and besides the limit being relatively low (~16500) I
noticed a very strange set of directory handles allocated by smbd:
```
root at big-data-1:/proc/11929# lsof |grep smbd |grep DIR| awk '{ print $9
}'|sort| uniq -c
15 /
16527 /big-data/wachtrij
2 /big-data/wachtrij/processing
4 /big-data/wachtrij/processing/P-CUSTOMER/Leeuwarden
10 /big-data/wachtrij/processing/P-CUSTOMER/WF
14 /big-data/wachtrij/processing/WF
```
There's over 16500 handles open to the root of one of the shares.
Restarting smbd dropped it to 0, and then over a period of 10 minutes it
has already increased to a thousand. There's only about 30 machines in this
network, of which only 10 are actively using this share for rendering
projects. Most of the clients are running windows, but there are 4 linux
clients as well. The rendering clients probably make very intensive use of
this samba share.
Is there a known (hopefully fixed) bug in 4.13.17, or is there something
wrong with our configuration? I am not much of an expert, and I did
recently blindly followed some performance optimization guides I found by
Googling around a bit, maybe I did something dumb:
[global]
workgroup = WORKGROUP
server string = %h server (Samba, Ubuntu)
log level = 1 auth_audit:3 auth_json_audit:3
log file = /var/log/samba/log.%m
max log size = 1000
logging = file
panic action = /usr/share/samba/panic-action %d
server role = standalone server
obey pam restrictions = yes
unix password sync = yes
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:*
%n\n *password\supdated\ssuccessfully* .
pam password change = yes
map to guest = bad user
usershare allow guests = yes
allow insecure wide links = yes
[wachtrij]
path = /big-data/wachtrij
valid users = processing,preprocessing
comment
browseable = yes
writeable = yes
read only = no
force create mode = 0660
force directory mode = 2770
force user = processing
force group = processing
follow symlinks = yes
wide links = yes
store dos attributes = no
ea support = no
map archive = no
map hidden = no
map system = no
map readonly = no
If there's no known bugs, and I haven't made any obvious configuration
mistakes, what's the best way to figure out what's going on, for example
if
the handles are somehow legitimate? I suppose upgrading to the latest
upstream samba release would be the first step.
Kind regards,
Tinco
Rowland Penny
2022-Jun-29 10:52 UTC
[Samba] Leaking file descriptors in Version 4.13.17-Ubuntu
On Wed, 2022-06-29 at 12:24 +0200, Tinco Andringa via samba wrote:> Hi, > > For about 2 weeks, I think since we updated smbd to version Version > 4.13.17-Ubuntu we have encountered a "too many files open" error. I > set out > to investigate and besides the limit being relatively low (~16500) I > noticed a very strange set of directory handles allocated by smbd: > > ``` > > root at big-data-1:/proc/11929# lsof |grep smbd |grep DIR| awk '{ print > $9 > }'|sort| uniq -c > > 15 / > > 16527 /big-data/wachtrij > > 2 /big-data/wachtrij/processing > > 4 /big-data/wachtrij/processing/P-CUSTOMER/Leeuwarden > > 10 /big-data/wachtrij/processing/P-CUSTOMER/WF > > 14 /big-data/wachtrij/processing/WF > ``` > > > There's over 16500 handles open to the root of one of the shares. > Restarting smbd dropped it to 0, and then over a period of 10 minutes > it > has already increased to a thousand. There's only about 30 machines > in this > network, of which only 10 are actively using this share for rendering > projects. Most of the clients are running windows, but there are 4 > linux > clients as well. The rendering clients probably make very intensive > use of > this samba share. > > Is there a known (hopefully fixed) bug in 4.13.17, or is there > something > wrong with our configuration? I am not much of an expert, and I did > recently blindly followed some performance optimization guides I > found by > Googling around a bit, maybe I did something dumb: > > > [global] > > workgroup = WORKGROUP > > server string = %h server (Samba, Ubuntu) > > log level = 1 auth_audit:3 auth_json_audit:3 > > log file = /var/log/samba/log.%m > > max log size = 1000 > > logging = file > > panic action = /usr/share/samba/panic-action %d > > server role = standalone server > > obey pam restrictions = yes > > unix password sync = yes > > passwd program = /usr/bin/passwd %u > > passwd chat = *Enter\snew\s*\spassword:* %n\n > *Retype\snew\s*\spassword:* > %n\n *password\supdated\ssuccessfully* . > > pam password change = yes > > map to guest = bad user > > usershare allow guests = yes > > allow insecure wide links = yes > > [wachtrij] > > path = /big-data/wachtrij > > valid users = processing,preprocessing > > comment > > browseable = yes > > writeable = yes > > read only = no > > force create mode = 0660 > > force directory mode = 2770 > > force user = processing > > force group = processing > > follow symlinks = yes > > wide links = yes > > store dos attributes = no > > ea support = no > > map archive = no > > map hidden = no > > map system = no > > map readonly = no > > If there's no known bugs, and I haven't made any obvious > configuration > mistakes, what's the best way to figure out what's going on, for > example if > the handles are somehow legitimate? I suppose upgrading to the latest > upstream samba release would be the first step. > > Kind regards, > TincoThere doesn't seem to be much wrong with your smb.conf (yes, there are a few defaults, but these will not affect anything). Your main problem is that 4.13.x is now EOL as far as Samba is concerned and if you have found a bug, it will not get fixed and, if it is a bug, it may have been fixed in a later version. I suggest you upgrade Samba to as near as possible to 4.16.x and see if this fixes your problem, if it doesn't then you stand a chance of getting it fixed. Rowland
L. van Belle
2022-Jun-29 13:23 UTC
[Samba] Leaking file descriptors in Version 4.13.17-Ubuntu
2 more things, - If these are office files, play a bit with the oplocking settings. - You can increase the file descriptors within the systemd service files if that?s a problem. LimitNOFILE= 65536 # should be the default but checkit. This might be 4096 The number of open file descriptors is limited by the physical memory available to the host. You can determine this limit with the following command: cat /proc/sys/fs/file-max + What Rowland said offcourse. ? Greetz, Louis> -----Oorspronkelijk bericht----- > Van: samba <samba-bounces at lists.samba.org> Namens Tinco Andringa via > samba > Verzonden: woensdag 29 juni 2022 12:24 > Aan: samba at lists.samba.org > Onderwerp: [Samba] Leaking file descriptors in Version 4.13.17-Ubuntu > > Hi, > > For about 2 weeks, I think since we updated smbd to version Version 4.13.17- > Ubuntu we have encountered a "too many files open" error. I set out to > investigate and besides the limit being relatively low (~16500) I noticed a very > strange set of directory handles allocated by smbd: > > ``` > > root at big-data-1:/proc/11929# lsof |grep smbd |grep DIR| awk '{ print $9 > }'|sort| uniq -c > > 15 / > > 16527 /big-data/wachtrij > > 2 /big-data/wachtrij/processing > > 4 /big-data/wachtrij/processing/P-CUSTOMER/Leeuwarden > > 10 /big-data/wachtrij/processing/P-CUSTOMER/WF > > 14 /big-data/wachtrij/processing/WF ``` > > > There's over 16500 handles open to the root of one of the shares. > Restarting smbd dropped it to 0, and then over a period of 10 minutes it has > already increased to a thousand. There's only about 30 machines in this > network, of which only 10 are actively using this share for rendering projects. > Most of the clients are running windows, but there are 4 linux clients as well. > The rendering clients probably make very intensive use of this samba share. > > Is there a known (hopefully fixed) bug in 4.13.17, or is there something > wrong with our configuration? I am not much of an expert, and I did recently > blindly followed some performance optimization guides I found by Googling > around a bit, maybe I did something dumb: > > > [global] > > workgroup = WORKGROUP > > server string = %h server (Samba, Ubuntu) > > log level = 1 auth_audit:3 auth_json_audit:3 > > log file = /var/log/samba/log.%m > > max log size = 1000 > > logging = file > > panic action = /usr/share/samba/panic-action %d > > server role = standalone server > > obey pam restrictions = yes > > unix password sync = yes > > passwd program = /usr/bin/passwd %u > > passwd chat = *Enter\snew\s*\spassword:* %n\n > *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* . > > pam password change = yes > > map to guest = bad user > > usershare allow guests = yes > > allow insecure wide links = yes > > [wachtrij] > > path = /big-data/wachtrij > > valid users = processing,preprocessing > > comment > > browseable = yes > > writeable = yes > > read only = no > > force create mode = 0660 > > force directory mode = 2770 > > force user = processing > > force group = processing > > follow symlinks = yes > > wide links = yes > > store dos attributes = no > > ea support = no > > map archive = no > > map hidden = no > > map system = no > > map readonly = no > > If there's no known bugs, and I haven't made any obvious configuration > mistakes, what's the best way to figure out what's going on, for example if > the handles are somehow legitimate? I suppose upgrading to the latest > upstream samba release would be the first step. > > Kind regards, > Tinco > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba