ash-samba at comtek.co.uk
2016-Aug-31 16:49 UTC
[Samba] Sudden unexplained slowness on share (caused by duplicate UID?)
Hi,
We have 5 Samba domain controllers, and 4 file servers. All are
4.2.10-Debian. Our file servers have been working fine until recently
(on the 26th), when suddenly access to the
\\users.store.example.com\user share (hostname p-cats) became extremely
slow.
Access to other shares on other machines is fine, and users is 128G
total (smaller than other shares). The machine isn't under
CPU/memory/IO/network strain, and accessing the file share via SCP/shell
is fine. Most clients are Windows 7, and are using offline files.
Eventually we gave up debugging, rsynced the share, pointed the DNS
alias at p-bats (the machine which hosts our group shares) and updated
samba. This new machine immediately exhibited exactly the same symptom
(but only for the user share - the group share, which has always been on
the new machine continued to work fine). The group share is identical,
except it has create mask = 0775, force create mode = 0660, directory
mask = 2775,force directory mode = 2770, and veto files is not set.
One thing did change on the 26th - a user was mistakenly created with
the same UID as a preexisting user. We've deleted the preexisting user
now, though. Otherwise there are no substantial changes to the user share.
I've noticed that if winbind is stopped the share is as responsive as it
used to be (on my Linux desktop it takes around 0.477s for: time find
/mnt/links/user/ash -type d). Also if I firewall off everything except
my linux desktop I see similar speeds. On the other hand it can take
minutes or longer to perform the same command with even just a couple of
windows machines able to access the server.
Another thing is that we have noticed occasional "port unreachable"
CLDAP messages from p-cats to a domain controller, (even though the DC
appeared to be sending a valid response to the same UDP port p-cats just
sent the CLDAP request from.
Can anybody offer any advice on where we can go next to debug this?
Thanks,
Ashley
testparm looks like:
Load smb config files from /etc/samba/smb.conf
Processing section "[user]"
Processing section "[group.bak]"
Processing section "[public.bak]"
Processing section "[system]"
Processing section "[wpkg.bak]"
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER
Press enter to see a dump of your service definitions
# Global parameters
[global]
workgroup = CHESTER-DC
realm = CHESTER-DC.EXAMPLE.COM
security = ADS
map to guest = Bad User
map untrusted to domain = Yes
syslog = 0
log file = /var/log/samba/log.%m
max log size = 1000
panic action = /usr/share/samba/panic-action %d
idmap config chester-dc : range = 1000-999997
idmap config chester-dc : backend = ad
idmap config * : range = 1000000-1999999
idmap config * : backend = tdb
map hidden = Yes
map system = Yes
include = /etc/samba/smb.share.conf
[user]
comment = user
path = /srv/user
read only = No
create mask = 0755
force create mode = 0600
directory mask = 02755
force directory mode = 02700
inherit permissions = Yes
delete veto files = Yes
veto files = /$RECYCLE.BIN/
[group.bak]
comment = group.bak
path = /srv/group.bak
create mask = 0775
force create mode = 0660
directory mask = 02775
force directory mode = 02770
inherit permissions = Yes
[public.bak]
comment = public.bak
path = /srv/public.bak
create mask = 0775
force create mode = 0660
directory mask = 02775
force directory mode = 02770
inherit permissions = Yes
[system]
comment = system
path = /srv/system
read only = No
create mask = 0775
force create mode = 0660
directory mask = 02775
force directory mode = 02770
inherit permissions = Yes
map archive = No
map hidden = No
map system = No
map readonly = no
[wpkg.bak]
comment = wpkg.bak
path = /srv/public.bak/admin/wpkg
write list = @it
create mask = 0775
force create mode = 0660
directory mask = 02775
force directory mode = 02770
inherit permissions = Yes
At the moment (with only 4 or 5 IT pcs permitted through the firewall)
It takes around 30 seconds to run "find /mnt/links/user/ash -type d",
which contains 976 files, and smbstatus outputs:
Samba version 4.2.10-Debian
PID Username Group Machine Protocol Version
------------------------------------------------------------------------------
7051 ash Domain Admins 10.4.1.240 (ipv4:10.4.1.240:39646)
NT1
7057 john. Domain Admins 10.4.1.242 (ipv4:10.4.1.242:65033)
SMB3_00
7057 john.doe Domain Admins 10.4.1.242 (ipv4:10.4.1.242:65033)
SMB3_00
7689 tim.doe Domain Admins 10.4.1.245 (ipv4:10.4.1.245:57526)
SMB3_00
7060 ryan Domain Admins 10.4.1.247 (ipv4:10.4.1.247:54588)
SMB2_10
Service pid machine Connected at
-------------------------------------------------------
IPC$ 7057 10.4.1.242 Wed Aug 31 16:43:51 2016
user 7057 10.4.1.242 Wed Aug 31 17:00:02 2016
user 7060 10.4.1.247 Wed Aug 31 16:43:36 2016
user 7689 10.4.1.245 Wed Aug 31 16:46:10 2016
user 7051 10.4.1.240 Wed Aug 31 16:42:41 2016
user 7057 10.4.1.242 Wed Aug 31 16:42:55 2016
Registered MSG_REQ_POOL_USAGE
Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
Locked files:
Pid Uid DenyMode Access R/W Oplock
SharePath Name Time
--------------------------------------------------------------------------------------------------
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:40 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:40 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:40 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:40 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:41 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:41 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:41 2016
7060 10160 DENY_NONE 0x1000a0 RDONLY NONE
/srv/user ryan/Private/Documents Wed Aug 31 16:44:42 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Pictures/Quotation Upddates Wed Aug 31 16:43:33
2016
7060 10160 DENY_NONE 0x100081 RDONLY NONE
/srv/user ryan/Private/Links Wed Aug 31 16:43:38 2016
7060 10160 DENY_NONE 0x100081 RDONLY NONE
/srv/user ryan/Private/Start Menu Wed Aug 31 16:44:15 2016
7060 10160 DENY_NONE 0x100081 RDONLY NONE
/srv/user ryan/Private/Desktop Wed Aug 31 16:43:42 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Desktop Wed Aug 31 16:43:24 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Desktop Wed Aug 31 16:43:25 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Favorites Wed Aug 31 16:44:29 2016
7689 10221 DENY_NONE 0x100081 RDONLY NONE
/srv/user tim.doe/Private/Desktop Wed Aug 31 16:57:38 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe Wed Aug 31 16:45:57 2016
7057 10002 DENY_NONE 0x80 RDONLY NONE
/srv/user john.doe Wed Aug 31 17:07:26 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Desktop/Files For Deletion Wed Aug 31 16:44:41
2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Documents Wed Aug 31 16:43:24 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Pictures/Icons/32x32 Wed Aug 31 16:47:33 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Pictures Wed Aug 31 16:44:39 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Music Wed Aug 31 16:43:24 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private/Pictures/Email Safety Wed Aug 31 16:44:39 2016
7060 10160 DENY_NONE 0x100081 RDONLY NONE
/srv/user ryan/Private/Music Wed Aug 31 16:44:27 2016
7060 10160 DENY_NONE 0x100081 RDONLY NONE
/srv/user ryan/Private/Music Wed Aug 31 16:44:28 2016
7057 10002 DENY_NONE 0x100080 RDONLY NONE
/srv/user john.doe/Private/Visual Studio
Projects/directory/COWS/obj/Debug/AspnetCompileMerge/Source/fonts/glyphicons-halflings-regular.ttf
Wed Aug 31 17:07:25 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user john.doe/Private Wed Aug 31 16:46:55 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user . Wed Aug 31 16:44:04 2016
7057 10002 DENY_WRITE 0x100081 RDONLY NONE
/srv/user . Wed Aug 31 16:43:32 2016
7060 10160 DENY_NONE 0x100080 RDONLY NONE
/srv/user . Wed Aug 31 16:43:36 2016
7689 10221 DENY_NONE 0x100080 RDONLY NONE
/srv/user . Wed Aug 31 16:46:10 2016
7057 10002 DENY_NONE 0x100081 RDONLY NONE
/srv/user . Wed Aug 31 17:06:33 2016
...
ash-samba at comtek.co.uk
2016-Sep-07 18:32 UTC
[Samba] Sudden unexplained slowness on share (caused by duplicate UID?) SOLVED - nogroup 65534
On 31/08/16 17:49, ash-samba--- via samba wrote:> Hi, > > We have 5 Samba domain controllers, and 4 file servers. All are > 4.2.10-Debian. Our file servers have been working fine until recently > (on the 26th), when suddenly access to the > \\users.store.example.com\user share (hostname p-cats) became > extremely slow. > > Access to other shares on other machines is fine, and users is 128G > total (smaller than other shares). The machine isn't under > CPU/memory/IO/network strain, and accessing the file share via > SCP/shell is fine. Most clients are Windows 7, and are using offline > files. > [...]Just to provide closure for future googlers: It appears that this problem was due to some files/folders which had the group "nogroup" set. Any attempts to list such files or folders caused a delay to all active connections. The unix group nogroup on Debian has the UID 65534, and did not exist within AD. We use the configuration: idmap config * : backend = tdb idmap config * : range = 1000000-1999999 idmap config EXAMPLE : backend = ad idmap config EXAMPLE : range = 1000-999997 We also discovered some files had somehow acquired posix ACLs which included nogroup, we had to delete the ACLs too (our server was supposed to be using unix UID/GID permissions only - no idea how Samba could have added ACLs!) Ash
Apparently Analagous Threads
- [4.3.11-Ubuntu] SMBD keeps Locks on NTUSER.DAT and ntuser.ini after logout
- Samba How to kill the locked files from a useraccount by multiple smbd pids
- smbstatus crazy multiple same listings
- Oplock break failed for file XXX-- replying anyway?
- Fwd: samba & Oracle ACFS Issues