We are using Samba 4.6.15. We have users not being able to login. winbindd log shows this: [2018/06/14 14:48:44.300203, 0, pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds) netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL [2018/06/14 14:53:43.896350, 0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384 [2018/06/14 14:53:43.896430, 0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384 [2018/06/14 14:53:43.896465, 0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 beyond eof at 16384 [2018/06/14 14:53:43.896500, 0, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 825242227 beyond eof at 16384 [2018/06/14 14:53:43.896523, 1, pid=3228] ../source3/lib/g_lock.c:193(g_lock_trylock) rec->store failed: NT_STATUS_UNSUCCESSFUL [2018/06/14 14:53:43.896586, 0, pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds) netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL [2018/06/14 14:54:23.899114, 1, pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) tdb(/var/vol/3/.ctera/samba/lock/mutex.tdb): tdb_lock failed on list 77 ltype=1 (Interrupted system call) [2018/06/14 14:54:23.899181, 0, pid=3228] ../source3/lib/util_tdb.c:497(tdb_chainlock_with_timeout_internal) tdb_chainlock_with_timeout_internal: alarm (40) timed out for key R-DC-CLIENT.COM in tdb /var/vol/3/.ctera/samba/lock/mutex.tdb [2018/06/14 14:54:23.899204, 1, pid=3228] ../source3/lib/server_mutex.c:97(grab_named_mutex) Could not get the lock for R-DC-CLIENT.COM [2018/06/14 14:54:23.899276, 0, pid=3228] ../source3/winbindd/winbindd_cm.c:1022(cm_prepare_connection) cm_prepare_connection: mutex grab failed for R-DC-CLIENT.COM [2018/06/14 14:54:23.899298, 1, pid=3228] ../source3/winbindd/winbindd_cm.c:1253(cm_prepare_connection) Failed to prepare SMB connection to R-DC-CLIENT.COM: NT_STATUS_POSSIBLE_DEADLOCK The issue can be reproduced manually by using "wbinfo --ping-dc" on the server machine, which produces output: wbinfo --ping-dc checking the NETLOGON for domain[CLIENT] dc connection to "" failed wbcPingDc2(CLIENT): error code was NT_STATUS_UNSUCCESSFUL (0xc0000001) Seems like g_lock.tdb is corrupted and indeed tdbdump on it returns same "beyond eof" error as in the above log. Restarting Samba fixes this and about 3 days later it would happen again. What could be the cause of the file corruption? Is there any way to bypass this flow? Thank you, Oren Kishon
On Sun, 17 Jun 2018 12:06:17 +0000 Oren Kishon via samba <samba at lists.samba.org> wrote:> We are using Samba 4.6.15. We have users not being able to login. > winbindd log shows this: > > > [2018/06/14 14:48:44.300203, 0, > pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds) > netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL [2018/06/14 > 14:53:43.896350, 0, > pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 > beyond eof at 16384 [2018/06/14 14:53:43.896430, 0, > pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 > beyond eof at 16384 [2018/06/14 14:53:43.896465, 0, > pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 1684959121 > beyond eof at 16384 [2018/06/14 14:53:43.896500, 0, > pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/vol/3/.ctera/samba/lock/g_lock.tdb): tdb_oob len 825242227 > beyond eof at 16384 [2018/06/14 14:53:43.896523, 1, > pid=3228] ../source3/lib/g_lock.c:193(g_lock_trylock) rec->store > failed: NT_STATUS_UNSUCCESSFUL [2018/06/14 14:53:43.896586, 0, > pid=3228] ../source3/rpc_client/cli_pipe.c:3292(cli_rpc_pipe_open_schannel_with_creds) > netlogon_creds_cli_get returned NT_STATUS_UNSUCCESSFUL [2018/06/14 > 14:54:23.899114, 1, > pid=3228] ../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/vol/3/.ctera/samba/lock/mutex.tdb): tdb_lock failed on list > 77 ltype=1 (Interrupted system call) [2018/06/14 14:54:23.899181, 0, > pid=3228] ../source3/lib/util_tdb.c:497(tdb_chainlock_with_timeout_internal) > tdb_chainlock_with_timeout_internal: alarm (40) timed out for key > R-DC-CLIENT.COM in tdb /var/vol/3/.ctera/samba/lock/mutex.tdb > [2018/06/14 14:54:23.899204, 1, > pid=3228] ../source3/lib/server_mutex.c:97(grab_named_mutex) Could > not get the lock for R-DC-CLIENT.COM [2018/06/14 14:54:23.899276, 0, > pid=3228] ../source3/winbindd/winbindd_cm.c:1022(cm_prepare_connection) > cm_prepare_connection: mutex grab failed for R-DC-CLIENT.COM > [2018/06/14 14:54:23.899298, 1, > pid=3228] ../source3/winbindd/winbindd_cm.c:1253(cm_prepare_connection) > Failed to prepare SMB connection to R-DC-CLIENT.COM: > NT_STATUS_POSSIBLE_DEADLOCK > > The issue can be reproduced manually by using "wbinfo --ping-dc" on > the server machine, which produces output: > > wbinfo --ping-dc checking the NETLOGON for domain[CLIENT] dc > connection to "" failed wbcPingDc2(CLIENT): error code was > NT_STATUS_UNSUCCESSFUL (0xc0000001) > > > Seems like g_lock.tdb is corrupted and indeed tdbdump on it returns > same "beyond eof" error as in the above log. > > Restarting Samba fixes this and about 3 days later it would happen > again. What could be the cause of the file corruption? > Is there any way to bypass this flow? > > Thank you, > Oren Kishon >Sorry, but there just isn't enough info provided. How are you running Samba ? On what OS ? What is in smb.conf ? Rowland
On Sun, 2018-06-17 at 14:23 +0100, Rowland Penny via samba wrote:> Sorry, but there just isn't enough info provided. > How are you running Samba ? > On what OS ? > What is in smb.conf ?I think the big clue is already in the logs: tdb(/var/vol/3/.ctera/samba/lock/mutex.tdb): tdb_lock failed on list CTERA seems to be some 'enterprise filesystem' thing. I would try again with all Samba databases on a local filesystem. If the OP wants clustered Samba, then use CTDB, but the Samba TDB files need to be on a local FS. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
On Mon, 18 Jun 2018 09:31:21 +0000 Oren Kishon <orenk at ctera.com> wrote:> We are running Samba on an embedded Linux machine, Kernel 4.9.75, > Using local XFS, nothing special. We have patches to Samba which I > havn't found to be related to the tdb engine: > https://github.com/urisimchoni/samba/commits/ctera-master.I think Uri should comment on his patches.> > Our smb.conf is below. > > PS: > This happens in several customers machines with the same AD > environment. They would not like to restart their Samba servers in > their machines. Is there any other info we can gather? Thank you. > > smb.conf: > > [global] > netbios name = R-CGW-01 > fruit:nfs_aces = no > fruit:veto_appledouble = no > server string > encrypt passwords = yes > ntlm auth = yes > server signing = disabled > log level = 1 > logging = file > debug pid = yes > read only = no > guest account = nobody > deadtime = 10 > enable core files = no > max smbd processes = 6000 > passdb backend = smbpasswd > socket options = TCP_KEEPIDLE=120 TCP_KEEPCNT=3 > TCP_KEEPINTVL=5 idmap negative cache time = 10 > rpc modify share security = no > restrict anonymous = 2 > client max protocol = SMB3 > min receivefile size = 0 > smb2 leases = yes > security=ads > workgroup=CLIENT > realm=CLIENT.COM > kerberos method = secrets only > machine password timeout = 0 > kerberos encryption types = all > allow trusted domains = yes > winbind use default domain = yes > winbind additional group sids = S-1-1-0 > client ldap sasl wrapping = sign > saf:ttl = 86400 > winbind max clients = 6100 > idmap config * : backend = passdb > idmap config * : range = 0-199999 > idmap config CLIENT:backend = rid > idmap config CLIENT:range = 200000-4999999 > idmap config CLIENT2:backend = rid > idmap config CLIENT2:range = 5000001-5999999 > map to guest = never > max log size = 50 > lock directory = /var/vol/3/.ctera/samba/lock > state directory = /var/vol/3/.ctera/samba > cache directory = /var/vol/3/.ctera/samba/cache > gencache:hash_size = 10000 > > [cloud] > path=/var/vol/cloud > comment> wide links=no > vfs objects = fruit streams_xattr acl_xattr shadow_copy2 > use sendfile = true > strict sync=yes > acl_xattr:ignore system acls = yes > force unknown acl user = yes > shadow:snapdir = .snapshots > shadow:snapdirseverywhere = yes > csc policy = manual > admin users> guest ok = no > read only = no > store dos attributes = yes > map archive = no > map hidden = no > map readonly = yes > map system = no > nt acl support=yes > map acl inherit=yes > inherit acls=yes > create mask = 0666 > directory mask = 0777 > ea support = no > valid users="+BUILTIN\Administrators" > invalid users> read list> write list="+BUILTIN\Administrators" > acl allow execute always = no > allocation roundup size = 0Can I suggest you read 'man smb.conf' there are lots of errors and default lines, but two jump out: passdb backend = smbpasswd This is old school, you should be using the 'tdbsam' backend. idmap config * : backend = passdb This is wrong, you should be using the 'tdb' backend. Finally, have you read the reply from Andrew ? From his post, it looks like you have shot yourself in the foot ;-) Rowland
Apparently Analagous Threads
- Connection problem due to tdb_lock failed
- Debugging tdb_oob log messages in samba 3.6
- Connection problem due to tdb_lock failed
- NT_STATUS_NO_LOGON_SERVERS when winbindd under large traffic.
- Mutex lock contention against Active directory domain controllers causing authentication failures