samba - Jan 2008 - Windows share modes and Linux file locking, flock & fcntl

Hi,

I am trying to understand file locking and share modes.

I am using RedHat Enterprise Linux 4, with kernel 2.6.9-55.ELsmp,
samba-3.0.10-1.4E.11 and Windows XP.

In particular I am looking at a machine running Linux exporting its
local filesystem using Samba, with a Windows client accessing the
file share.  If I use byte range locking (fcntl() on Linux, LockFile()
on Windows) things work as expected, and applications using the
file system locally and remotely see the locks set by the other.

However, if a program on the Windows client opens a file with a
share mode that denies access to other processes, I can't seem to
detect this in a local Linux process, either with fcntl() or with flock().
I included the test programs I used below. I have seen references
that this should be possible
(http://lists.samba.org/archive/samba/2004-February/080455.html
last paragraph) but I may be misunderstanding things.

strace on the smbd process handling the request from the Windows
client shows this (in part):

stat64("temp/test.txt", {st_mode=S_IFREG|0764, st_size=24, ...}) = 0
gettimeofday({1201195094, 16852}, NULL) = 0
fcntl64(12, F_SETLKW64, {type=F_WRLCK, whence=SEEK_SET,
           start=256, len=1}, 0xbff6a890) = 0
open("temp/test.txt", O_RDWR|O_LARGEFILE) = 19
flock(19, 0x20 /* LOCK_??? */)          = 0
fcntl64(19, F_SETSIG, 0x23)             = 0
fcntl64(19, 0x400 /* F_??? */, 0x1)     = 0


The second argument to flock() on the test file is 0x20, which is what
LOCK_MAND is defined as in the samba-3.0.28 source in the file
source/include/includes.h.  This is the same value I find in
/usr/include/bits/fcntl.h and /usr/include/asm/fcntl.h.

So I wrote a test program (included below) that uses flock with this
value (instad of the usual LOCK_EX) but multiple instances of this
program couldn't even detect each other's locks.  Nor did any
windows client having the file open with any share mode have an
effect. I also tried enabling mandatory locking with the mount option
and the chmod bits, since the flag is called LOCK_MAND, but that did
not change the results.

So the question is: is it supposed to be possible for a program running
locally on Linux to detect share modes applied by a Windows client,
and if so, how?

The test programs and my smb.conf file are below.

Thanks for any help,
Peter

Snippets:
1. smb.conf
2. Windows program (console app) to open with share
   mode disallowing other processes
3. Linux program to open a file and lock it with flock():
4. Linux program to lock with fcntl():
5. Windows program (console app) to lock with byte
   range locks: LockFile()
6. Linux program to lock the file using flock()
   and LOCK_MAND, instead of the standard LOCK_EX


1. smb.conf
============================================[global]
   netbios name = PFS-RHEL4-DEV
   workgroup = MYGROUP
   server string = Samba Server
   printcap name = /etc/printcap
   load printers = yes

cups options = raw

 log file = /var/log/samba/%m.log
   max log size = 50
   security = user
   socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
   dns proxy = no

   idmap uid = 16777216-33554431
   idmap gid = 16777216-33554431
   template shell = /bin/false
   winbind use default domain = no
[homes]
   comment = Home Directories
   browseable = no
   writable = yes

[printers]
   comment = All Printers
   path = /var/spool/samba
   browseable = no
   guest ok = no
   writable = no
   printable = yes
============================================



Test programs:

2. Windows program (console app) to open with share
   mode disallowing other processes
============================================#include <stdio.h>
#include <tchar.h>
#include <windows.h>

int _tmain(int argc, _TCHAR* argv[])
{
    HANDLE handle;
    TCHAR* filename = NULL;

    if(argc < 2)
    {
        printf("No argument.\n");
        return 2;
    }
    else
    {
        filename = argv[1];
    }


    printf("Try to open file with share mode\n");

    while(1)
    {

        handle = CreateFile(
            filename,
            GENERIC_READ | GENERIC_WRITE, /* desired access */
            0, /* share mode: no sharing */
            NULL, /* security attributes */
            OPEN_EXISTING, /* creation disposition - fail if not found */
            0, /* dwFlagsAndAttributes - i think we can ignore this
                   since the file exists */
            NULL /* template for attributes when creating file */
        );

        if(handle != INVALID_HANDLE_VALUE)
            break;
        else
            Sleep(1000);
    }

    printf("Acquired Lock\n");

    while(1)
        Sleep(1000);

	return 0;
}
============================================


3. Linux program to open a file and lock it with flock():
============================================#include <sys/file.h>
#include <unistd.h>
#include <stdio.h>

main(int argc, char** argv)
{
        int result;
        int fd = -1;
        char* filename = NULL;

        if(argc < 2)
        {
                printf("No argument\n");
                return 2;
        }
        else
        {
                filename = argv[1];
        }

        fd = open(filename, O_RDWR);
        if(-1 == fd)
        {
                printf("open() failed\n");
                return 1;
        }

        printf("Trying to lock...\n");

        result = flock(fd, LOCK_EX);
        if(result != 0)
        {
                printf("flock() failed\n");
                return 1;
        }

        printf("Lock acquired\n");

        while(1)
                sleep(1);
}
============================================


4. Linux program to lock with fcntl():
============================================#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>

main(int argc, char** argv)
{
    int result;
    int fd = -1;
    char* filename = NULL;

    if(argc < 2)
    {
        printf("No argument\n");
        return 2;
    }
    else
    {
        filename = argv[1];
    }

    fd = open(filename, O_RDWR);
    if(-1 == fd)
    {
        printf("open() failed\n");
        return 1;
    }

    {
        struct flock lock_opts;
        lock_opts.l_type = F_WRLCK;
        lock_opts.l_whence = SEEK_SET;
        lock_opts.l_start = 0;
        lock_opts.l_len = 0;

        printf("Trying to lock...\n");

        result = fcntl(fd, F_SETLKW, &lock_opts);
        if(result == -1)
        {
            printf("fcntl() failed\n");
            return 1;
        }
    }

    printf("Lock acquired\n");

    while(1)
        sleep(1);
}
============================================


5. Windows program (console app) to lock with byte
   range locks: LockFile()
============================================#include <stdio.h>
#include <tchar.h>
#include <windows.h>
#include <limits.h>


int _tmain(int argc, _TCHAR* argv[])
{
    HANDLE handle;
    TCHAR* filename = NULL;

    if(argc < 2)
    {
        printf("No argument.\n");
        return 2;
    }
    else
    {
        filename = argv[1];
    }

    handle = CreateFile(
        filename,
        GENERIC_READ | GENERIC_WRITE, /* desired access */
        FILE_SHARE_READ | FILE_SHARE_WRITE |
            FILE_SHARE_DELETE,
            /* share mode: no restrictions on other processes */
        NULL, /* security attributes */
        OPEN_EXISTING, /* creation disposition - fail if not found */
        0, /* dwFlagsAndAttributes - i think we can ignore this
              since the file exists */
        NULL /* template for attributes when creating file */
    );

    if(handle == INVALID_HANDLE_VALUE)
    {
        printf("Unable to open file\n");
        return 1;
    }

    printf("Trying to lock...\n");

    while(1)
    {
        int result;

        result = LockFile(handle, 0, 0, INT_MAX, 0);

        if(result)
            break;
        else
            Sleep(1000);
    }

    printf("Lock acquired\n");

    while(1)
        Sleep(1000);

	return 0;
}
============================================

6. Linux program to lock the file using flock()
   and LOCK_MAND, instead of LOCK_EX
============================================
#define _GNU_SOURCE

#include <sys/file.h>
#include <unistd.h>
#include <stdio.h>

main(int argc, char** argv)
{
        int result;
        int fd = -1;
        char* filename = NULL;

        if(argc < 2)
        {
                printf("No argument\n");
                return 2;
        }
        else
        {
                filename = argv[1];
        }

        fd = open(filename, O_RDWR);
        if(-1 == fd)
        {
                printf("open() failed\n");
                return 1;
        }

        printf("Trying to lock...\n");

        result = flock(fd, LOCK_MAND);
        if(result != 0)
        {
                printf("flock() failed\n");
                return 1;
        }

        printf("Lock acquired\n");

        while(1)
                sleep(1);
}

=============================================

> In particular I am looking at a machine running Linux exporting its
> local filesystem using Samba, with a Windows client accessing the
> file share.  If I use byte range locking (fcntl() on Linux, LockFile()
> on Windows) things work as expected, and applications using the
> file system locally and remotely see the locks set by the other.
> However, if a program on the Windows client opens a file with a
> share mode that denies access to other processes, I can't seem to
> detect this in a local Linux process, either with fcntl() or with flock().
> I included the test programs I used below. I have seen references
> that this should be possible
> (http://lists.samba.org/archive/samba/2004-February/080455.html
> last paragraph) but I may be misunderstanding things.
There seems to be a library for accessing Samba's database of share modes
in the Samba source tree (for version 3.0.28).  The relevant files are

./bin/libsmbsharemodes.so
./bin/libsmbsharemodes.a
./include/smb_share_modes.h
./libsmb/smb_share_modes.c
./libsmb/smb_share_modes.o

On the other hand, it doesn't appear to be possible to use local system
calls to detect share modes because their semantics are too different from
file locks, see threads below.

http://lists.samba.org/archive/samba/2007-March/subject.html#130305
http://lists.samba.org/archive/samba-technical/2005-February/thread.html#39425

Peter