samba - Jan 2023 - oplocks, kernel oplocks, kernel share modes, .. - how it all works?

I don't have that much time to explain this in glory detail, but here 
are a few pointers.

There are three different apsects:

- locking in the Windows sharemode flavor
- file change notify
- client side caching via oplocks/leases

File change notify is fully supported and will detect local changes via 
inotify on Linux. Iirc there are some caveats like recursive change 
notify but that doesn't seem to be a production problem as Windows 
clients typically don't make use of this feature.

Client side caching (oplocks/leases): somewhat limitted support by 
enabled "kernel oplocks". Kernel oplocks don't work with SMB2
Leases, so
you're restricted to SMB1 oplocks. Cf man smb.conf "kernel
oplocks".

Locking (sharemodes): we used to leverage Linux support for mandatory 
flock() locks (LOCK_MAND) to "store" and let the kernel enforce access
restrictions for local access, but unfortunately the kernel support got 
removed because it was buggy:

<https://github.com/torvalds/linux/commit/f7e33bdbd6d1bdf9c3df8bba5abcf3399f957ac3>

We still have VFS hooks for this but iirc the only filesystem that 
supports this via the VFS interface is GPFS.

Hope this helps somewhat.
-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20230120/8ff07613/OpenPGP_signature.sig>

20.01.2023 14:49, Ralph Boehme via samba wrote:
> I don't have that much time to explain this in glory detail, but here
are a few pointers.
Thank you very much for this write-up/summary. It's exactly what was needed!
> There are three different apsects:
> 
> - locking in the Windows sharemode flavor
> - file change notify
> - client side caching via oplocks/leases
> 
> File change notify is fully supported and will detect local changes via
inotify on Linux. Iirc there are some caveats like recursive change notify but
> that doesn't seem to be a production problem as Windows clients
typically don't make use of this feature.
This is the most important (I think) part, - the change notify.  Yes I
understand
there are issues with recursive inotify even without windows or extra protocols
involved.

Surprisingly for me, I haven't even looked at inotify in samba sources
before asking
this (I can't know the "support status" for it just by looking at
the sources, though).

And indeed, there's quite some code in samba/smbd to hook inotify.  And it
is actually
what I suspected long before this topic, - eg windows explorer immediately
refreshes
the directory listing once something changes on the samba server side, - it
should be
some FAM, and linux has inotify, there's nothing else like this (yes I know
dnotify too).
> Client side caching (oplocks/leases): somewhat limitted support by enabled
"kernel oplocks". Kernel oplocks don't work with SMB2 Leases, so
you're
> restricted to SMB1 oplocks. Cf man smb.conf "kernel oplocks".
And SMB1 oplocks does not work when SMB2+ protocol is negotiated/used, right?
This one I'll take a look at the source.

And here there's some interesting stuff happening. Again, I'll try to
set up some
testbed for tracing a bit later.

The thing is that smbd immediately detects the change of a linux file, and
immediately notifies windows about this change, and windows asks to re-read
the directory.

But the lease which is granted to windows on that file.. stays. And it is even
funnier - I open this file in a text viewer (read-only), win asks shared lease
for it, next I modify the file on linux, close it on windows and re-open it on
windows. And it shows me the old cached contents of this file, while in the
directory listing which windows displayed already even before second open,
this file has zero size.

It is either that samba does not try to break this lease even if it detects the
file change (ie, change notify isn't hooked into the file lease code), or
windows
ignores this. From the plain strace of smbd, it looks like there's no
"join" of
file change monitoring and leases or it doesn't work.

Is this supposed to work with current samba?
> Locking (sharemodes): we used to leverage Linux support for mandatory
flock() locks (LOCK_MAND) to "store" and let the kernel enforce access
> restrictions for local access, but unfortunately the kernel support got
removed because it was buggy:
> 
>
<https://github.com/torvalds/linux/commit/f7e33bdbd6d1bdf9c3df8bba5abcf3399f957ac3>
> 
> We still have VFS hooks for this but iirc the only filesystem that supports
this via the VFS interface is GPFS.
Oh.  Heh. Fun stuff it was :)
> Hope this helps somewhat.
It *definitely* helps, thank you very much Ralf!

I'll try to dig further to understand what's actually going on there.
It looks like this really deserves some attention, because the problem
is quite common but largely unknown.

And it deserves a separate wiki page too, at least :)

/mjt

20.01.2023 14:49, Ralph Boehme via samba wrote:
> I don't have that much time to explain this in glory detail, but here
are a few pointers.
> 
> There are three different apsects:
> 
> - locking in the Windows sharemode flavor
> - file change notify
> - client side caching via oplocks/leases
> 
> File change notify is fully supported and will detect local changes via
inotify on Linux. Iirc there are some caveats like recursive change notify but
> that doesn't seem to be a production problem as Windows clients
typically don't make use of this feature.
> 
> Client side caching (oplocks/leases): somewhat limitted support by enabled
"kernel oplocks". Kernel oplocks don't work with SMB2 Leases, so
you're
> restricted to SMB1 oplocks. Cf man smb.conf "kernel oplocks".
Hi Ralph!

I'm testing this and wonder if something's not working where it should,
or maybe I don't understand something again.

Locked files:
Pid          User(ID)   DenyMode   Access      R/W        Oplock      SharePath 
Name   Time
62347        1060       DENY_WRITE 0x120089    RDONLY     LEASE()     /ws/ws    
one    Tue Jan 24 11:32:00 2023
62347        1085       DENY_NONE  0x120089    RDONLY     LEASE()     /ws/ws    
two    Tue Jan 24 11:50:12 2023

Here we've two files open, with kernel oplocks = yes (hence it is LEASE(), -
before I enabled kernel oplocks, it was LEASE(RH) or LEASE(RWH)).

When I modify file "two" on linux, there's nothing goes on at all,
samba
does not even notice this.  I suspect this is a file cached on the client,
so effectively samba does not tell the client to drop or revalidate the
cache.

On the other hand, when I modify file "one", samba does notice
it's been
modified, - it looks like this is done using inotify mechanism, not
file lease mechanism, because it does not receive any signals, but does
receive a message over inotify FD.  Smbd when sends a message to client,
which apparently tells it to re-read the directory which it does. But
again, the file stays locked the same as before, and the client does
not notice this file is changed.

I modify it by using `cp' in both cases.

So there are 2 questions:

  1) why I don't see inotify working in case of file "two". I
guess it
   can be because the client is not subscribed to changes for this file.

and most important,

  2) where's the kernel oplocks in action? For this, I'll try to strace
   smbd for a bit more, hopefully I will see the fcntl calls in there.

Also, it looks like inotify can be used to break oplocks/leases *too*,
with or without kernel oplocks, Has this been considered?

Thanks!

/mjt