samba - Jul 2016 - Winbind process stuck at 100% after changing use_mmap to no

On 03/07/16 13:06, Volker Lendecke wrote:
> On Fri, Jul 01, 2016 at 10:00:21AM +0100, Alex Crow wrote:
>> We've had a strange issue after following the recommendations at
>> https://wiki.samba.org/index.php/Ping_pong, particularly the part
>> about mmap coherence. We are running CTDB/Samba over a MooseFS
>> clustered FS, and we'd not done the ping-pong before. After finding
>> that the mmap coherence test did not pass, we changed "use
mmap" to
>> "no" in smb.conf.
> "use mmap" only affects tdbs, which should not sit on the
clustered
> fs, they should *always* be on a local file system like ext4, assuming
> you're using Linux. The wiki page above states that it's not
> absolutely essential to provide mmap coherence, so you should not
> bother too much.
>
> Or are you putting tdb files on moosefs?
>
> BTW, the straces you're seeing are probably not real spins, that's
> traverses. With 10000 hash chains, that will be 10000 preads. That's
> just taking time. If the tdbs sit on moosefs, it might actually be
> true that it does not like that. But as I said: Never put tdbs on a
> cluster file system. If cluster file systems did that access pattern
> fine, ctdb would not exist :-)
>
> Volker
Hi Volker,

I've only put the "private dir" onto MooseFS, as instructed in the
CTDB
docs.

So, in that case, I'm assuming from your comments that it is no worry
that the mmap test does not pass on the MooseFS mounts?

The "private dir" contains these files:

-rw-------+ 1 root root   24576 Jun 23 19:48 netlogon_creds_cli.tdb
-rw-------+ 1 root root  421888 Jun 23 19:48 passdb.tdb
-rw-------+ 1 root root  430080 Jun 23 19:47 secrets.tdb
drwxr-xr-x+ 3 root root 1001200 Jun 23 19:48 smbd.tmp

It just strikes me as odd that "use mmap = no" causes such a slowdown
in
the the case I only have these files on my clustered FS. The timestamp
of all those files is the last time I joined one of the member servers
to the domain.

More worrying is that the rw ping-pong (no mmap) does not work properly
even when I mount my MooseFS with -o mfscachemode=none, which is not
supposed to do any local page/file cache and yet still I get
inconsistent results when adding and removing ping_pong tests on other
nodes (ie increment count does not always match running test nodes). I
have raised this with the MooseFS chaps and am waiting for a response.

Many thanks

Alex



--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On Sun, Jul 03, 2016 at 08:42:36PM +0100, Alex Crow
wrote:
> I've only put the "private dir" onto MooseFS, as instructed
in the CTDB
> docs.
Can you quote these docs, so that we can correct them?
> So, in that case, I'm assuming from your comments that it is no worry
> that the mmap test does not pass on the MooseFS mounts?
Yes, it is a high worry. If MooseFS is not mmap coherent, you need
"use mmap = no", and we need to fix Samba and/or MooseFS to support
tdbs better.

But as said previously, ctdb should make this unnecessary.
> The "private dir" contains these files:
> 
> -rw-------+ 1 root root   24576 Jun 23 19:48 netlogon_creds_cli.tdb
> -rw-------+ 1 root root  421888 Jun 23 19:48 passdb.tdb
> -rw-------+ 1 root root  430080 Jun 23 19:47 secrets.tdb
> drwxr-xr-x+ 3 root root 1001200 Jun 23 19:48 smbd.tmp
This is weird. ctdb-controlled files should have the node-number
appended. Are you sure that you are running ctdb?

Volker

On 03/07/16 21:47, Volker Lendecke wrote:
> On Sun, Jul 03, 2016 at 08:42:36PM +0100, Alex Crow wrote:
>> I've only put the "private dir" onto MooseFS, as
instructed in the CTDB
>> docs.
> Can you quote these docs, so that we can correct them?
>
>> So, in that case, I'm assuming from your comments that it is no
worry
>> that the mmap test does not pass on the MooseFS mounts?
> Yes, it is a high worry. If MooseFS is not mmap coherent, you need
> "use mmap = no", and we need to fix Samba and/or MooseFS to
support
> tdbs better.
>
> But as said previously, ctdb should make this unnecessary.
>
>> The "private dir" contains these files:
>>
>> -rw-------+ 1 root root   24576 Jun 23 19:48 netlogon_creds_cli.tdb
>> -rw-------+ 1 root root  421888 Jun 23 19:48 passdb.tdb
>> -rw-------+ 1 root root  430080 Jun 23 19:47 secrets.tdb
>> drwxr-xr-x+ 3 root root 1001200 Jun 23 19:48 smbd.tmp
> This is weird. ctdb-controlled files should have the node-number
> appended. Are you sure that you are running ctdb?
>
> Volker


100% sure I'm using it! :

[root at zearing ~]# ctdb status
Number of nodes:6
pnn:0 172.16.40.13     OK
pnn:1 172.16.40.14     OK (THIS NODE)
pnn:2 172.16.40.15     OK
pnn:3 192.168.21.24    OK
pnn:4 192.168.21.25    OK
pnn:5 192.168.21.27    OK
Generation:1417726848
Size:6
hash:0 lmaster:0
hash:1 lmaster:1
hash:2 lmaster:2
hash:3 lmaster:3
hash:4 lmaster:4
hash:5 lmaster:5
Recovery mode:NORMAL (0)
Recovery master:4

smb.conf:

[global]

max protocol = SMB2
host msdfs = yes
workgroup = IFA_NET
netbios name = LIZARDCLUSTER
private dir = /mfs/ctdb/private
clustering = yes
use mmap = yes

/etc/sysconfig/ctdb:

# Options to ctdbd, read by ctdbd_wrapper(1)
#
# See ctdbd.conf(5) for more information about CTDB configuration variables.

# Shared recovery lock file to avoid split brain.  No default.
#
# Do NOT run CTDB without a recovery lock file unless you know exactly
# what you are doing.
CTDB_RECOVERY_LOCK=/mfs/ctdb/recovery.lock

# List of nodes in the cluster.  Default is below.
CTDB_NODES=/mfs/ctdb/nodes

# List of public addresses for providing NAS services.  No default.
CTDB_PUBLIC_ADDRESSES=/mfs/ctdb/public_addresses_cl

# What services should CTDB manage?  Default is none.
CTDB_MANAGES_SAMBA=yes
CTDB_MANAGES_WINBIND=yes

# CTDB_MANAGES_NFS=yes

# Raise the file descriptor limit for CTDB?
# ulimit -n 10000

# Default is to use the log file below instead of syslog.
# CTDB_LOGFILE=/var/log/log.ctdb
# CTDB_SYSLOG=no

# Default log level is ERR.  NOTICE is a little more verbose.
CTDB_DEBUGLEVEL=NOTICE

# Set some CTDB tunable variables during CTDB startup?
# CTDB_SET_TraverseTimeout=60
CTDB_SAMBA_SKIP_SHARE_CHECK=yes

--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On 03/07/16 21:47, Volker Lendecke wrote:
> On Sun, Jul 03, 2016 at 08:42:36PM +0100, Alex Crow wrote:
>> I've only put the "private dir" onto MooseFS, as
instructed in the CTDB
>> docs.
> Can you quote these docs, so that we can correct them?
Here, under the lustre section. I applied the same config at it's a 
similar FS (ie distributed with a central metadata server):

https://wiki.samba.org/index.php/CTDB_Setup

"

In comparison of Lustre configurating, setting up CTDB/Samba on the two 
different versions keeps the same way. The following settings are 
recommended:

clustering = yes
idmap backend = tdb2
private dir=/mnt/lustre/ctdb
fileid:mapping = fsname
use mmap = no
nt acl support = yes
ea support = yes


"


Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On 03/07/16 21:47, Volker Lendecke wrote:
> On Sun, Jul 03, 2016 at 08:42:36PM +0100, Alex Crow wrote:
>> I've only put the "private dir" onto MooseFS, as
instructed in the CTDB
>> docs.
> Can you quote these docs, so that we can correct them?
>
>
Also Gluster have the same tip here: 
http://download.gluster.org/pub/gluster/glusterfs/doc/Gluster_CTDB_setup.v1.pdf

• Add the following lines to the [global] section of your Samba 
configuration
clustering = yes
idmap backend = tdb2
private dir = /gluster/lock

Can you confirm that "private dir" does *not* have to be set in
smb.conf
to point to a shared FS? If not, would that solve the original problem?

Thanks

Alex
--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).