samba - Nov 2014 - Very high system load: Samba statfs call for filesystems in /etc/mtab in: fileid_load_mount_entries()

Hello,

we recently had a big problem with a very high load on our linux (Sles11
SP3) samba server version 3.6.3.
We experienced a system load of up to 90 for 300 active samba users. The
result was that hardly any user was able to really do anything (eg
access a file via samba).

One sympton was visible in a very high share of cpu *system* time (about
25% on each of the 10 CPUs). This system load was caused by the smbd
daemons where each of these processes regurlarly did a stat as well as
as fstat on all active users directories in /home .

The home directories are mounted by automount from one of our
fileservers using NFS. From each existing fileserver (fs1, fs2) the
samba server NFS-mounts the users parent directory
under eg  /import/fs1, /import/fs2, and automount then does a
bind mount of an entry in /import/fs1/<user> to /home/<user>.
If eg "userx1" and "userx2" are located on file server fs2
then the
mount structure on the samba server is as follows:

fs2:/export/user2 on /import/user2 (nfs)
/import/user2/userx1 on /home/userx1 (bind)
/import/user2/userx2 on /home/userx2 (bind)

This way there are as many bind mounts as there are users accessing one
of the home directories. Since we have a total number of about 12000
users, after a while there usually are 1000 bind mounts
on the samba server mounted.

So if there are eg 1000 bind mounts then we saw that *each* of the 300
running smbd processes did in regular intervals
a stat on all of these 1000 bind mounted /home/<user> entries found in
/etc/mtab. This had the effect that automount was unable to umount any
of them, since they were beeing stated all the time again and on the
other hand there were so many stat() system calls that samba was unable
to really serve files to the users. Xfer speeds of 1KByte/sec was
already fast :-) So actually none of our users were able to do his work.

We solved the problem in between by a workaround, by avoiding the bind
mounts completely on this server by creating a mofified /etc/passwd with
/import/user1... entries for the users home directories. Now load is not
above 5 even with 300 active users.

My question now is, if the stat calls that are probably triggered in
sambas function  fileid_load_mount_entries() are really needed for bind
mounts? The information of the stat calls will probably be identical to
the underlying NFS mounted filesystem /import/user1, /import/user2.
Wouldn't it make sense to exclude bind mounted filesystems
from the stat call but instead limit the stat calls to the rest of the
filesystems?

Thanks
Rainer
-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse  1
56070 Koblenz, http://userpages.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html,Fax: +49261287
1001312

On Wed, Nov 12, 2014 at 02:35:59PM +0100, Rainer Krienke
wrote:
> 
> My question now is, if the stat calls that are probably triggered in
> sambas function  fileid_load_mount_entries() are really needed for bind
> mounts? The information of the stat calls will probably be identical to
> the underlying NFS mounted filesystem /import/user1, /import/user2.
> Wouldn't it make sense to exclude bind mounted filesystems
> from the stat call but instead limit the stat calls to the rest of the
> filesystems?
How can you portably tell it's a bind mount ?

Is there a flag in struct mntent ?