samba - Apr 2015 - Randomly occurring file losses on samba share

Hello,

having migrated from OSX Server with AFP filesharing, we've experienced a
problem with randomly occurring file losses on our samba server. Due to the fact
we were not able to reproduce the problem reliably, we are quite clueless what
could cause this behavior and therefore desperately looking for any hints.

The Server in question is running Ubuntu 14.04LTS with the Version of Samba
being '4.1.6+dfsg-1ubuntu2.14.04.7'. It is set up to retrieve its users
from an LDAP directory using libnss-ldapd while authentication is done via
Kerberos. File permissions are controlled by POSIX-ACLs on top of an ext4
filesystem.

On the client side, there are clients running Mac OSX 10.9.5 or Mac OSX 10.10.2.
The connection to the server is established using built-in 'Finder'
functions.

Server and clients are provided with latest updates.

The Problem was recognized while working on MS Office documents, whereby the
affected documents were first saved and closed by one client, later opened by
another client (while the first client was still connected to the share), and
disappeared shortly after the second client saved (and closed) the edited file.
The file can be restored via extundelete but then may contain content of another
file. (In a particular example a restored .docx file contained content of a
.xlsx file that was edited in a timely manner on the same share).

We're unsure whether the problem is located in Finder, MS Office or samba
itself, but quite confident it is related to the new samba environment, because
this behavior was never observed before, with nearly identical share structure,
when AFP was used. Maybe it's worth to note that normal users are not
allowed to write to the first level of a share.

Relevant config-files are written down below.

Kind regards,

Andreas Huppert



Client configuration:
#/etc/nsmb.conf
[default]
smb_neg=smb2_only


Server configuration:
#/etc/samba/smb.conf
workgroup = WORKGROUP
server string = %h server (Samba, Ubuntu)
dns proxy = no
log file = /var/log/samba/samba.log
max log size = 10000
syslog = 10
syslog only = yes
log level = all:1
ldap ssl = start tls
disable netbios = yes
usershare allow guests = no

security = ADS
realm = REALM
kerberos method = dedicated keytab
dedicated keytab file = /etc/samba/keytab.keytab

nt acl support = yes
hide unreadable = no
encrypt passwords = yes
obey pam restrictions = no
unix password sync = no
ldap ssl ads = yes

invalid users = root xxxx
inherit acls = yes
inherit permissions = yes
map acl inherit = no
inherit owner = yes

[Share]
path = /path/to/share
writeable = yes

On 08/04/15 17:34, Andreas Huppert wrote:
> Hello,
>
> having migrated from OSX Server with AFP filesharing, we've experienced
a problem with randomly occurring file losses on our samba server. Due to the
fact we were not able to reproduce the problem reliably, we are quite clueless
what could cause this behavior and therefore desperately looking for any hints.
>
> The Server in question is running Ubuntu 14.04LTS with the Version of Samba
being '4.1.6+dfsg-1ubuntu2.14.04.7'. It is set up to retrieve its users
from an LDAP directory using libnss-ldapd while authentication is done via
Kerberos. File permissions are controlled by POSIX-ACLs on top of an ext4
filesystem.
>
> On the client side, there are clients running Mac OSX 10.9.5 or Mac OSX
10.10.2. The connection to the server is established using built-in
'Finder' functions.
>
> Server and clients are provided with latest updates.
>
> The Problem was recognized while working on MS Office documents, whereby
the affected documents were first saved and closed by one client, later opened
by another client (while the first client was still connected to the share), and
disappeared shortly after the second client saved (and closed) the edited file.
The file can be restored via extundelete but then may contain content of another
file. (In a particular example a restored .docx file contained content of a
.xlsx file that was edited in a timely manner on the same share).
>
> We're unsure whether the problem is located in Finder, MS Office or
samba itself, but quite confident it is related to the new samba environment,
because this behavior was never observed before, with nearly identical share
structure, when AFP was used. Maybe it's worth to note that normal users are
not allowed to write to the first level of a share.
>
> Relevant config-files are written down below.
>
> Kind regards,
>
> Andreas Huppert
>
>
>
> Client configuration:
> #/etc/nsmb.conf
> [default]
> smb_neg=smb2_only
>
>
> Server configuration:
> #/etc/samba/smb.conf
> workgroup = WORKGROUP
> server string = %h server (Samba, Ubuntu)
> dns proxy = no
> log file = /var/log/samba/samba.log
> max log size = 10000
> syslog = 10
> syslog only = yes
> log level = all:1
> ldap ssl = start tls
> disable netbios = yes
> usershare allow guests = no
>
> security = ADS
> realm = REALM
> kerberos method = dedicated keytab
> dedicated keytab file = /etc/samba/keytab.keytab
>
> nt acl support = yes
> hide unreadable = no
> encrypt passwords = yes
> obey pam restrictions = no
> unix password sync = no
> ldap ssl ads = yes
>
> invalid users = root xxxx
> inherit acls = yes
> inherit permissions = yes
> map acl inherit = no
> inherit owner = yes
>
> [Share]
> path = /path/to/share
> writeable = yes
You seem to have set samba up as an active directory client, yet you 
don't seem to be using AD. This could have something to do with your 
problem.

Rowland

On Wed, Apr 08, 2015 at 06:34:42PM +0200, Andreas Huppert
wrote:
> having migrated from OSX Server with AFP filesharing, we've
> experienced a problem with randomly occurring file losses on our
> samba server. Due to the fact we were not able to reproduce the
> problem reliably, we are quite clueless what could cause this
> behavior and therefore desperately looking for any hints.
> 
> The Server in question is running Ubuntu 14.04LTS with the Version
> of Samba being '4.1.6+dfsg-1ubuntu2.14.04.7'. It is set up to
> retrieve its users from an LDAP directory using libnss-ldapd while
> authentication is done via Kerberos. File permissions are controlled
> by POSIX-ACLs on top of an ext4 filesystem.
> 
> On the client side, there are clients running Mac OSX 10.9.5 or Mac
> OSX 10.10.2. The connection to the server is established using
> built-in 'Finder' functions.
> 
> Server and clients are provided with latest updates.
> 
> The Problem was recognized while working on MS Office documents,
> whereby the affected documents were first saved and closed by one
> client, later opened by another client (while the first client was
> still connected to the share), and disappeared shortly after the
> second client saved (and closed) the edited file. The file can be
> restored via extundelete but then may contain content of another
> file. (In a particular example a restored .docx file contained
> content of a .xlsx file that was edited in a timely manner on the
> same share).
> 
> We're unsure whether the problem is located in Finder, MS Office or
> samba itself, but quite confident it is related to the new samba
> environment, because this behavior was never observed before, with
> nearly identical share structure, when AFP was used. Maybe it's
> worth to note that normal users are not allowed to write to the
> first level of a share.
sounds like a client side issue to me, but that's just guessing
without drilling into traces.

Is it reproducible? If yes, please file a bugreport and attach a network
trace showing the issue.

-Ralph

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de,mailto:kontakt at sernet.de

Hello,

further investigating the problem we were able to provoke an issue that might be
related to the initial problem and could be reproduced with a higher likelihood.
The core of the reproducible problem is, that sometimes a file is being created
by an application running on the client, but then cannot be written to.
Therefore the application throws permission related errors and documents cannot
be saved.

We believe, that this behavior may also has caused the problem with the
disappearing files (in connection with the weird file saving procedure of MS
Word), but of course, we can?t say that for sure. Explaining in brief: Word
creates a temp file with the new content, then renames the original file to a
temporary file, renames first temporary file to the original's name and
deletes the second temporary file afterwards.

However: While writing to a new file (saving), according to the logs, they seem
to be opened with read permissions only, but still get created with a size of 0
byte.

We tried to narrow down the problem and therefore turned off all file locking
mechanisms provided by samba:
locking = no
strict locking = no
oplocks = no
kernel oplocks = no
fake oplocks = no
blocking locks = no
level2 oplocks = no
posix locking = no

We observed that the number of occurrences decreased, but unfortunately the
problem still persists.

Next we investigated access rights. As mentioned before, permissions are
controlled via ACLs.
getfacl on the directory we were testing in, gives the following output:
# file: /path/to/folder
# owner: root
# group: root
# flags: -s-
user::rwx
user:root:rwx
group::rwx
group:root:rwx
group:group1:rwx
group:group2:rwx
mask::rwx
other::---
default:user::rwx
default:user:root:rwx
default:group::rwx
default:group:root:rwx
default:group:group1:rwx
default:group:group2:rwx
default:mask::rwx
default:other::?

Related options in smb.conf are:
invalid users = root xxxx
inherit acls = yes
inherit permissions = yes
map acl inherit = no
inherit owner = yes

According to the log files, permission don?t seem to be the problem, although we
expected returning 0770. But for the client user always having ownership of the
files, group permissions should not matter:
unix_mode(/path/to/annoy.docx) inheriting from /path/to
Apr 10 16:16:19 fileserver smbd[2148]:   unix_mode(/path/to/annoy.docx) inherit
mode 42770
Apr 10 16:16:19 fileserver smbd[2148]:   unix_mode(/path/to/annoy.docx)
returning 0760

Simply put, we are still kind of clueless and highly appreciate any hints/help.

Best regards,

Andreas Huppert

Am 08.04.2015 um 18:34 schrieb Andreas Huppert <samba-srv at
kemper-schlomski.de>:
> Hello,
> 
> having migrated from OSX Server with AFP filesharing, we've experienced
a problem with randomly occurring file losses on our samba server. Due to the
fact we were not able to reproduce the problem reliably, we are quite clueless
what could cause this behavior and therefore desperately looking for any hints.
> 
> The Server in question is running Ubuntu 14.04LTS with the Version of Samba
being '4.1.6+dfsg-1ubuntu2.14.04.7'. It is set up to retrieve its users
from an LDAP directory using libnss-ldapd while authentication is done via
Kerberos. File permissions are controlled by POSIX-ACLs on top of an ext4
filesystem.
> 
> On the client side, there are clients running Mac OSX 10.9.5 or Mac OSX
10.10.2. The connection to the server is established using built-in
'Finder' functions.
> 
> Server and clients are provided with latest updates.
> 
> The Problem was recognized while working on MS Office documents, whereby
the affected documents were first saved and closed by one client, later opened
by another client (while the first client was still connected to the share), and
disappeared shortly after the second client saved (and closed) the edited file.
The file can be restored via extundelete but then may contain content of another
file. (In a particular example a restored .docx file contained content of a
.xlsx file that was edited in a timely manner on the same share).
> 
> We're unsure whether the problem is located in Finder, MS Office or
samba itself, but quite confident it is related to the new samba environment,
because this behavior was never observed before, with nearly identical share
structure, when AFP was used. Maybe it's worth to note that normal users are
not allowed to write to the first level of a share.
> 
> Relevant config-files are written down below.
> 
> Kind regards,
> 
> Andreas Huppert
> 
> 
> 
> Client configuration:
> #/etc/nsmb.conf
> [default]
> smb_neg=smb2_only
> 
> 
> Server configuration:
> #/etc/samba/smb.conf
> workgroup = WORKGROUP
> server string = %h server (Samba, Ubuntu)
> dns proxy = no
> log file = /var/log/samba/samba.log
> max log size = 10000
> syslog = 10
> syslog only = yes
> log level = all:1
> ldap ssl = start tls
> disable netbios = yes
> usershare allow guests = no
> 
> security = ADS
> realm = REALM
> kerberos method = dedicated keytab
> dedicated keytab file = /etc/samba/keytab.keytab
> 
> nt acl support = yes
> hide unreadable = no
> encrypt passwords = yes
> obey pam restrictions = no
> unix password sync = no
> ldap ssl ads = yes
> 
> invalid users = root xxxx
> inherit acls = yes
> inherit permissions = yes
> map acl inherit = no
> inherit owner = yes
> 
> [Share]
> path = /path/to/share
> writeable = yes

Hi

On Fri, Apr 10, 2015 at 05:12:27PM +0200, Andreas Huppert
wrote:
> Simply put, we are still kind of clueless and highly appreciate any
> hints/help.
you could try 4.2 and vfs_fruit. If that doesn't work, ideally you'd
find someone who can analyze traces of the misbehaviour.

-Ralph

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de,mailto:kontakt at sernet.de

On 10/04/15 16:12, Andreas Huppert wrote:
> Hello,
>
> further investigating the problem we were able to provoke an issue that
might be related to the initial problem and could be reproduced with a higher
likelihood. The core of the reproducible problem is, that sometimes a file is
being created by an application running on the client, but then cannot be
written to. Therefore the application throws permission related errors and
documents cannot be saved.
>
> We believe, that this behavior may also has caused the problem with the
disappearing files (in connection with the weird file saving procedure of MS
Word), but of course, we can?t say that for sure. Explaining in brief: Word
creates a temp file with the new content, then renames the original file to a
temporary file, renames first temporary file to the original's name and
deletes the second temporary file afterwards.
>
> However: While writing to a new file (saving), according to the logs, they
seem to be opened with read permissions only, but still get created with a size
of 0 byte.
>
> We tried to narrow down the problem and therefore turned off all file
locking mechanisms provided by samba:
> locking = no
> strict locking = no
> oplocks = no
> kernel oplocks = no
> fake oplocks = no
> blocking locks = no
> level2 oplocks = no
> posix locking = no
>
> We observed that the number of occurrences decreased, but unfortunately the
problem still persists.
>
> Next we investigated access rights. As mentioned before, permissions are
controlled via ACLs.
> getfacl on the directory we were testing in, gives the following output:
> # file: /path/to/folder
> # owner: root
> # group: root
> # flags: -s-
> user::rwx
> user:root:rwx
> group::rwx
> group:root:rwx
> group:group1:rwx
> group:group2:rwx
> mask::rwx
> other::---
> default:user::rwx
> default:user:root:rwx
> default:group::rwx
> default:group:root:rwx
> default:group:group1:rwx
> default:group:group2:rwx
> default:mask::rwx
> default:other::?
>
> Related options in smb.conf are:
> invalid users = root xxxx
> inherit acls = yes
> inherit permissions = yes
> map acl inherit = no
> inherit owner = yes
>
> According to the log files, permission don?t seem to be the problem,
although we expected returning 0770. But for the client user always having
ownership of the files, group permissions should not matter:
> unix_mode(/path/to/annoy.docx) inheriting from /path/to
> Apr 10 16:16:19 fileserver smbd[2148]:   unix_mode(/path/to/annoy.docx)
inherit mode 42770
> Apr 10 16:16:19 fileserver smbd[2148]:   unix_mode(/path/to/annoy.docx)
returning 0760
>
> Simply put, we are still kind of clueless and highly appreciate any
hints/help.
>
> Best regards,
>
> Andreas Huppert
>
> Am 08.04.2015 um 18:34 schrieb Andreas Huppert <samba-srv at
kemper-schlomski.de>:
>
>> Hello,
>>
>> having migrated from OSX Server with AFP filesharing, we've
experienced a problem with randomly occurring file losses on our samba server.
Due to the fact we were not able to reproduce the problem reliably, we are quite
clueless what could cause this behavior and therefore desperately looking for
any hints.
>>
>> The Server in question is running Ubuntu 14.04LTS with the Version of
Samba being '4.1.6+dfsg-1ubuntu2.14.04.7'. It is set up to retrieve its
users from an LDAP directory using libnss-ldapd while authentication is done via
Kerberos. File permissions are controlled by POSIX-ACLs on top of an ext4
filesystem.
>>
>> On the client side, there are clients running Mac OSX 10.9.5 or Mac OSX
10.10.2. The connection to the server is established using built-in
'Finder' functions.
>>
>> Server and clients are provided with latest updates.
>>
>> The Problem was recognized while working on MS Office documents,
whereby the affected documents were first saved and closed by one client, later
opened by another client (while the first client was still connected to the
share), and disappeared shortly after the second client saved (and closed) the
edited file. The file can be restored via extundelete but then may contain
content of another file. (In a particular example a restored .docx file
contained content of a .xlsx file that was edited in a timely manner on the same
share).
>>
>> We're unsure whether the problem is located in Finder, MS Office or
samba itself, but quite confident it is related to the new samba environment,
because this behavior was never observed before, with nearly identical share
structure, when AFP was used. Maybe it's worth to note that normal users are
not allowed to write to the first level of a share.
>>
>> Relevant config-files are written down below.
>>
>> Kind regards,
>>
>> Andreas Huppert
>>
>>
>>
>> Client configuration:
>> #/etc/nsmb.conf
>> [default]
>> smb_neg=smb2_only
>>
>>
>> Server configuration:
>> #/etc/samba/smb.conf
>> workgroup = WORKGROUP
>> server string = %h server (Samba, Ubuntu)
>> dns proxy = no
>> log file = /var/log/samba/samba.log
>> max log size = 10000
>> syslog = 10
>> syslog only = yes
>> log level = all:1
>> ldap ssl = start tls
>> disable netbios = yes
>> usershare allow guests = no
>>
>> security = ADS
>> realm = REALM
>> kerberos method = dedicated keytab
>> dedicated keytab file = /etc/samba/keytab.keytab
>>
>> nt acl support = yes
>> hide unreadable = no
>> encrypt passwords = yes
>> obey pam restrictions = no
>> unix password sync = no
>> ldap ssl ads = yes
>>
>> invalid users = root xxxx
>> inherit acls = yes
>> inherit permissions = yes
>> map acl inherit = no
>> inherit owner = yes
>>
>> [Share]
>> path = /path/to/share
>> writeable = yes
You could try changing these line in smb.conf:

inherit acls = yes
inherit permissions = yes
inherit owner = yes

To:

vfs objects = acl_xattr
map acl inherit = Yes
store dos attributes = Yes

and change the share to:

[Share]
path = /path/to/share
read only = no
acl_xattr:ignore system acls = yes

see 'man vfs_acl_xattr' for more info

Rowland