samba - Apr 2020 - CTDB and locking issues reloaded

Hi there,

I'm following your list for several months, as we run a CTDB cluster of 
Samba gateways over our distributed filesystem. I decided to post my 
first message to resume an old thread 
(https://lists.samba.org/archive/samba/2016-October/204043.html), hoping 
to progress and reach some conclusion.

Essentially, we observe and can reproduce issues when attempting to lock 
a given file from two clients connecting to two DIFFERENT samba 
gateways, whereas everything works as expected when the two clients 
happen to connect to the same samba gateway.

The issue is that both clients are able to take an exclusive lock on the 
same file, as if the lock information is not propagated across the 
cluster. And indeed I could see the following on the server side:

# smbstatus -L | grep lopresti
2:8324 4179 DENY_ALL 0x3 RDWR NONE /eos home-l/lopresti/testsambalock 
Tue Apr 14 12:00:45 2020
3:29245 4179 DENY_ALL 0x3 RDWR NONE /eos home-l/lopresti/testsambalock 
Tue Apr 14 12:02:38 2020

I obtained the above by patching the smbclient code from a 4.12 branch 
following the above mentioned thread, and starting two separate sessions as:

$ ./bin/smbclient //128.142.32.44/eos
smb: \> cd home-l/lopresti
smb: \home-l\lopresti\> open testsambalock
open file \home-l\lopresti\testsambalock: for read/write fnum 1

and:

$ ./bin/smbclient //128.142.32.45/eos
smb: \> cd home-l/lopresti
smb: \home-l\lopresti\> open testsambalock
open file \home-l\lopresti\testsambalock: for read/write fnum 1

[The IPs being two of the four public IPs of our cluster]

The servers are running Samba 4.9.1-10 on CentOS 7.7. Locking settings 
are as follows:

# grep lock /etc/samba/smb.conf
     locking = yes
     posix locking = no
     strict locking = no
     oplocks = no
     level2 oplocks = no
     kernel oplocks = no

We've tried different combinations for the locking settings, following 
different (even contradicting) advices, but nothing changed with respect 
to this issue.

So the question is do you acknowledge the correct behaviour should be to 
prevent the second client from getting the lock? What would you advice 
as locking settings?

To further debug the case, I've collected full (level 10) logs of the 
two sessions on the two gateways, and can make them available, as well 
as the full configuration or anything else you may need.

Thanks in advance for your help,
Cheers,

Giuseppe

Am 4/15/20 um 10:59 AM schrieb Giuseppe Lo Presti via
samba:
> So the question is do you acknowledge the correct behaviour should be to
> prevent the second client from getting the lock?
yes. My bet: your cluster filesystem device nummer differs across nodes. Cf:

<https://wiki.samba.org/index.php/Setting_up_a_cluster_filesystem#Checking_uniformity_of_device_and_inode_numbering>

man vfs_fileid.

Hth!
-slow

-- 
Ralph Boehme, Samba Team                https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/
GPG-Fingerprint   FAE2C6088A24252051C559E4AA1E9B7126399E46

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20200415/27219357/signature.sig>

On 15/04/2020 16:57, Ralph Boehme via samba wrote:
> Am 4/15/20 um 10:59 AM schrieb Giuseppe Lo Presti via samba:
>> So the question is do you acknowledge the correct behaviour should be
to
>> prevent the second client from getting the lock?
> 
> yes. My bet: your cluster filesystem device nummer differs across nodes.
Cf:
Bingo!

Indeed inodes are (obviously) identical, but device numbers differ. For 
the record, the solution has been to include as suggested:

   vfs objects = fileid
   fileid:algorithm = fsname

in the global configuration. With this, Microsoft Office - the real use 
case - works correctly by detecting files opened by other users, no 
matter which gateway users hit.

So thanks a lot, case solved. And what would be your advice concerning 
the other locking settings? In particular:

   posix locking = no
   strict locking = no
   oplocks = no
   level2 oplocks = no
   kernel oplocks = no

Thanks again,
Cheers,
Giuseppe