samba - Sep 2014 - Dtrace of smbd shows some errors in FreeBSD 9.2 with Samba 4.1.11 possibly related to talloc

Opening folders containing 10K files totaling 20GB is slow (takes 1.5 - 2.0
minutes) in FreeBSD 9.2 with Samba 4.1.11 on a system with 32GB RAM and a
newer Xeon processor.

Server is a standalone server that I'm using for testing. I am using nfsv4
acls.

I used the procsystime dtrace script from here:
http://www.brendangregg.com/DTrace/procsystime with the following syntax:
procsystime -n smbd

Top offenders were as follows:
          fcntl            64334950
extattr_list_file          88368438
            stat          305050713
   getdirentries          480578501
            poll        60233510027
extattr_get_file        63198610831

I decided to poke around a bit since I am a noob and noticed the following
errors in my dtrace output:

trace -n 'syscall:::return /execname=="smbd"/ {@[ustack)] =
count(); }'

libc.so.7`__sys_close+0xc
              libsmbd_base.so`smb_vfs_call_closedir+0x50
              libsmbd_base.so`0x800b27a92
              libtalloc.so.2`0x8036b205e
              dtrace: ERROR: open  failed: No such file or directory
`0x8115768c0
              dtrace: ERROR: open  failed: No such file or directory
`0x811576680
              dtrace: ERROR: open  failed: No such file or directory
`0x81149e200
              dtrace: ERROR: open  failed: No such file or directory
`0x811499b70

dtrace -n 'syscall:::entry /execname=="smbd"/ {@[ustack()] =
count(); }'

              libc.so.7`__sys_close+0xc
              libsmbd_base.so`smb_vfs_call_closedir+0x50
              libsmbd_base.so`0x800b27a92
              libtalloc.so.2`0x8036b205e
              dtrace: ERROR: gelf_getehdr() failed: No error: 0
passdb.tdb`0x81157a1c0
              dtrace: ERROR: gelf_getehdr() failed: No error: 0
passdb.tdb`0x81154d280
              dtrace: ERROR: gelf_getehdr() failed: No error: 0
passdb.tdb`0x81149e280
              dtrace: ERROR: gelf_getehdr() failed: No error: 0
passdb.tdb`0x8115780c0
              dtrace: ERROR: gelf_getehdr() failed: No error: 0
passdb.tdb`0x81149e300

I'm not familiar with dtrace / the innards of samba. Is this normal
behavior? Is this because passdb.tdb is located in a different location
than the rest of samba's tdb files?

On Tue, Sep 16, 2014 at 07:56:10PM -0500, Andrew Walker
wrote:
> Opening folders containing 10K files totaling 20GB is slow (takes 1.5 - 2.0
> minutes) in FreeBSD 9.2 with Samba 4.1.11 on a system with 32GB RAM and a
> newer Xeon processor.
> 
> Server is a standalone server that I'm using for testing. I am using
nfsv4
> acls.
If you have 10k files in a folder, you probably
want to configure the ZFS filesystem as case
insensitive.