samba - Aug 2010 - Problems related to error status_file_locking_conflict

I am running two Fedora 13 servers, one as a Samba PDC and the other as
a BDC. For the past week I have been tracing a series of issues which
may or may not be related to Samba 3.5.4. Here is one problem that I am
tracking and need help from the list with suggestions.
>From a wireshark trace between my Samba server and an XP client I am
seeing the following error status returned for up to a minute for
certain files:

error: STATUS_FILE_LOCK_CONFLICT after a SMB Write AndX Request

Specifically, this is during the login process which now takes over a
minute due to these error messages. The files in question are related to
MS credentials in /Application
Data/Microsoft/CryptNetUrlCache/Content/<long string> and /Application
Data/Microsoft/CryptNetUrlCache/Metadata/<same long string>. After a
while we get an error that pops up in XP and says that the contents of
this file could not be saved. As an aside I have had the same message
occur for /History/index.dat and /Cookies/index.dat.

Running lsof during the login process I do see the files in questions
but the lsof output does not indicate that they are locked in any way as
shown next:

smbd    6884 <user>   36u   REG       0,22       24 60784737 <some
prefix>/Application Data/Microsoft/CryptnetUrlCache/Content/<long
string>

The [homes]  and [profiles] shares are stored on the BDC with an LDAP
slave backend. The PDC provides login services and the [netlogon] share
and the primary LDAP backend.

In the wireshark trace I see the following related to the
file /Application Data/Microsoft/CryptNetUrlCache/Content/<long string>:
1) the file  gets queried with a Trans2 Request, Query Path Info, Query
File Basic Info, and return Success
2) the file is created with NT Create AndX Request, with return Success
3) the file is queried again, Trans2 Request, Query File Info, Query
File Internal Info, with return Success
4) the file is written to with, Write Request, 0 bytes at offset 24,
with return Success
5) the file is written again with, Write AndX Request, 24 bytes at
offset 0, with return ERROR: STATUS_FILE_LOCK_CONFLICT
and this last Write Andx Request continues for up to a minute or longer.

As I mentioned all of our domain login times have increased
significantly and many other aspects of general computing and network
shares have slowed. My users are complaining and frustrated that I have
not resolved this issues. Where can I look on my Samba servers and
Linux/Fedora 13 specifically if I am having file and record locking
issues related to the XP clients. (note: my wife is one of the users and
she is about to fire me or worse... ;)

Any and all suggestions are welcome.

Bob Smith
BISLink Internet Services
Private Consultant
Great Falls, VA USA

OK, so far I now understand that this is _NOT_ a general login issue
_NOR_ is it a MS credentials file issue either. This appears to be a
general locking issue in Samba and Linux. Here is a snippet of my
workstation log with another application that now no longer runs. The
SMB network transactions are the same as are the results for the user
(me, Quicken will no longer start and I am about to be hit with a late
charge $(

Note the delay between the linux_set_kernel_oplock failure and the
reporting of the SMB error NT_STATUS_FILE_LOCK_CONFLICT.
 
[2010/08/23 22:56:55.399979,  2] smbd/open.c:631(open_file)
  rwsmith opened file My Profile/Application
Data/Intuit/Quicken/Log/qw.log read=No write=Yes (numopen=7)
[2010/08/23 22:56:55.400134,  3]
smbd/oplock_linux.c:120(linux_set_kernel_oplock)
  linux_set_kernel_oplock: Refused oplock on file My Profile/Application
Data/Intuit/Quicken/Log/qw.log, fd = 23, file_id = 16:39f8d
df:0. (Invalid argument)
[2010/08/23 22:56:55.400843,  3] smbd/process.c:1485(process_smb)
  Transaction 1486 of length 76 (0 toread)
[2010/08/23 22:56:55.400885,  3] smbd/process.c:1294(switch_message)
  switch message SMBtrans2 (pid 9850) conn 0x293eac0
[2010/08/23 22:56:55.400917,  3]
smbd/trans2.c:5012(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo: TRANSACT2_QFILEINFO: level = 1006
[2010/08/23 22:56:55.400978,  3]
smbd/trans2.c:5225(call_trans2qfilepathinfo)
  call_trans2qfilepathinfo My Profile/Application
Data/Intuit/Quicken/Log/qw.log (fnum = 9996) level=1006 call=7
total_data=0
[2010/08/23 22:56:55.402033,  3] smbd/process.c:1485(process_smb)
  Transaction 1487 of length 94 (0 toread)
[2010/08/23 22:56:55.402069,  3] smbd/process.c:1294(switch_message)
  switch message SMBwriteX (pid 9850) conn 0x293eac0
[2010/08/23 22:56:55.402233,  3] lib/util.c:1846(fcntl_getlock)
  fcntl_getlock: fd 23 is returned info 0 pid 0
[2010/08/23 22:56:55.402273,  3] smbd/error.c:80(error_packet_set)
  error packet at smbd/reply.c(4375) cmd=47 (SMBwriteX)
NT_STATUS_FILE_LOCK_CONFLICT

Web searches on this status code and locking are coming up short (or too
long). Any pointers are greatly appreciated.
--bs


On Sun, 2010-08-22 at 19:55 -0400, Robert W. Smith wrote:
> I am running two Fedora 13 servers, one as a Samba PDC and the other as
> a BDC. For the past week I have been tracing a series of issues which
> may or may not be related to Samba 3.5.4. Here is one problem that I am
> tracking and need help from the list with suggestions.
> 
> >From a wireshark trace between my Samba server and an XP client I am
> seeing the following error status returned for up to a minute for
> certain files:
> 
> error: STATUS_FILE_LOCK_CONFLICT after a SMB Write AndX Request
> 
> Specifically, this is during the login process which now takes over a
> minute due to these error messages. The files in question are related to
> MS credentials in /Application
> Data/Microsoft/CryptNetUrlCache/Content/<long string> and
/Application
> Data/Microsoft/CryptNetUrlCache/Metadata/<same long string>. After a
> while we get an error that pops up in XP and says that the contents of
> this file could not be saved. As an aside I have had the same message
> occur for /History/index.dat and /Cookies/index.dat.
> 
> Running lsof during the login process I do see the files in questions
> but the lsof output does not indicate that they are locked in any way as
> shown next:
> 
> smbd    6884 <user>   36u   REG       0,22       24 60784737 <some
> prefix>/Application Data/Microsoft/CryptnetUrlCache/Content/<long
> string>
> 
> The [homes]  and [profiles] shares are stored on the BDC with an LDAP
> slave backend. The PDC provides login services and the [netlogon] share
> and the primary LDAP backend.
> 
> In the wireshark trace I see the following related to the
> file /Application Data/Microsoft/CryptNetUrlCache/Content/<long
string>:
> 1) the file  gets queried with a Trans2 Request, Query Path Info, Query
> File Basic Info, and return Success
> 2) the file is created with NT Create AndX Request, with return Success
> 3) the file is queried again, Trans2 Request, Query File Info, Query
> File Internal Info, with return Success
> 4) the file is written to with, Write Request, 0 bytes at offset 24,
> with return Success
> 5) the file is written again with, Write AndX Request, 24 bytes at
> offset 0, with return ERROR: STATUS_FILE_LOCK_CONFLICT
> and this last Write Andx Request continues for up to a minute or longer.
> 
> As I mentioned all of our domain login times have increased
> significantly and many other aspects of general computing and network
> shares have slowed. My users are complaining and frustrated that I have
> not resolved this issues. Where can I look on my Samba servers and
> Linux/Fedora 13 specifically if I am having file and record locking
> issues related to the XP clients. (note: my wife is one of the users and
> she is about to fire me or worse... ;)
> 
> Any and all suggestions are welcome.
> 
> Bob Smith
> BISLink Internet Services
> Private Consultant
> Great Falls, VA USA
> 
> 
>

On Mon, Aug 23, 2010 at 11:34:21PM -0400, Robert W. Smith
wrote:
> OK, so far I now understand that this is _NOT_ a general login issue
> _NOR_ is it a MS credentials file issue either. This appears to be a
> general locking issue in Samba and Linux. Here is a snippet of my
> workstation log with another application that now no longer runs. The
> SMB network transactions are the same as are the results for the user
> (me, Quicken will no longer start and I am about to be hit with a late
> charge $(
> 
> Note the delay between the linux_set_kernel_oplock failure and the
> reporting of the SMB error NT_STATUS_FILE_LOCK_CONFLICT.
To avoid the issue, try "kernel oplocks = no". This does not
really solve the issue, but might help you over the current
problems.

Volker

Volker, yes this "trick" worked for the current problems but agreed
that
it is not a long term solution to the "locking" problem. 
Thanks, Bob
--bs

On Tue, 2010-08-24 at 07:04 +0200, Volker Lendecke wrote:
> On Mon, Aug 23, 2010 at 11:34:21PM -0400, Robert W. Smith wrote:
> > OK, so far I now understand that this is _NOT_ a general login issue
> > _NOR_ is it a MS credentials file issue either. This appears to be a
> > general locking issue in Samba and Linux. Here is a snippet of my
> > workstation log with another application that now no longer runs. The
> > SMB network transactions are the same as are the results for the user
> > (me, Quicken will no longer start and I am about to be hit with a late
> > charge $(
> > 
> > Note the delay between the linux_set_kernel_oplock failure and the
> > reporting of the SMB error NT_STATUS_FILE_LOCK_CONFLICT.
> 
> To avoid the issue, try "kernel oplocks = no". This does not
> really solve the issue, but might help you over the current
> problems.
> 
> Volker

OK. I think I have tracked this problem down and have been able to
backout out the recommended changes below. Here is what I found.

Somehow through the PAM authentication modules the users home
directories were being mounted on top of themselves as NFS4 shares on
the Samba server (this server can also share up the home directories
using NFS4 for the Linux users). I was using pam_mount.so on this server
but am not positive of the vector that really caused the users home
directories to mount when authenticated via samba.

By fixing pam_mount to not mount itself and then unmounting any
incorrect mounts I verified that this was causing the initial reported
symptoms. I then restored kernel oplocks = yes (default) and restore the
shares to the default oplocks settings and everything works as expected
and users log in times from login to desktop is back to ~10secs.

So, yea, I guess I did step on myself with this one but in the process I
learned alot more about how locking works in Linux and Samba.

Bob
Part-time self-inflicted Linux administrator
--bs

On Tue, 2010-08-24 at 07:04 +0200, Volker Lendecke wrote:
> On Mon, Aug 23, 2010 at 11:34:21PM -0400, Robert W. Smith wrote:
> > OK, so far I now understand that this is _NOT_ a general login issue
> > _NOR_ is it a MS credentials file issue either. This appears to be a
> > general locking issue in Samba and Linux. Here is a snippet of my
> > workstation log with another application that now no longer runs. The
> > SMB network transactions are the same as are the results for the user
> > (me, Quicken will no longer start and I am about to be hit with a late
> > charge $(
> > 
> > Note the delay between the linux_set_kernel_oplock failure and the
> > reporting of the SMB error NT_STATUS_FILE_LOCK_CONFLICT.
> 
> To avoid the issue, try "kernel oplocks = no". This does not
> really solve the issue, but might help you over the current
> problems.
> 
> Volker