samba - Apr 2017 - Slow population of Properties in Windows Explorer

Hello,

We recently deployed Samba 4.5 on Debian Jessie-derived Linux with kernel v4.5.7
sharing an XFS filesystem.

We are migrating a large data set from old NetApp filers to the Samba share.
While attempting to monitor the progress of the migration, we are observing very
slow population of filesystem metadata in the Properties window (specifically
the General tab for info like "Size" and "Contains: n Files, n
Folders") for top-level directories via Windows Explorer and very slow
recursive directory listing on the command line in PowerShell on Windows Server
2k8 and 2k12 clients mounting the share. Much slower than the same clients
querying the same top-level directories via the same methods on the data
sets' old home on the NetApp.

"Very slow" means tens of minutes to complete for the Samba share vs
tens of seconds for the same operation on the NetApp share.

smbstatus shows these Windows clients are mounting via SMB2_10 protocol version.
The Linux clients available to us to test with are too old to support any
protocol greater than SMB 1, so we don't yet know if this behavior is the
same on Linux mounts of the same share via the same protocol doing an 'ls
-R' of the same directories.

I've found several topics related to slow browsing of heavily populated
directories (over 10,000 files per directory), but that is not the case here. I
have not found much with respect to any kind of performance tuning that can be
done to improve the feedback of filesystem metadata through Samba to clients. Is
this something the vfs_readahead module can help with? Its man page only
mentions Windows Vista, but it isn't clear if the issue it addresses remains
in newer versions of Windows Explorer.

Thanks,
Andrew Richards
Senior Systems Engineer
keepertechnology

On Wed, Apr 05, 2017 at 02:21:48PM -0400, Andrew Richards via samba
wrote:
> Hello,
> 
> We recently deployed Samba 4.5 on Debian Jessie-derived Linux with kernel
v4.5.7 sharing an XFS filesystem.
> 
> We are migrating a large data set from old NetApp filers to the Samba
share. While attempting to monitor the progress of the migration, we are
observing very slow population of filesystem metadata in the Properties window
(specifically the General tab for info like "Size" and "Contains:
n Files, n Folders") for top-level directories via Windows Explorer and
very slow recursive directory listing on the command line in PowerShell on
Windows Server 2k8 and 2k12 clients mounting the share. Much slower than the
same clients querying the same top-level directories via the same methods on the
data sets' old home on the NetApp.
> 
> "Very slow" means tens of minutes to complete for the Samba share
vs tens of seconds for the same operation on the NetApp share.
> 
> smbstatus shows these Windows clients are mounting via SMB2_10 protocol
version. The Linux clients available to us to test with are too old to support
any protocol greater than SMB 1, so we don't yet know if this behavior is
the same on Linux mounts of the same share via the same protocol doing an
'ls -R' of the same directories.
> 
> I've found several topics related to slow browsing of heavily populated
directories (over 10,000 files per directory), but that is not the case here. I
have not found much with respect to any kind of performance tuning that can be
done to improve the feedback of filesystem metadata through Samba to clients. Is
this something the vfs_readahead module can help with? Its man page only
mentions Windows Vista, but it isn't clear if the issue it addresses remains
in newer versions of Windows Explorer.
Can you do any performance tests on the local
filesystem to see if this is due to XFS xattr
performance ? If you're migrating a lot of ACLs
across we will probably hit the xattr code
very hard. NetApp may be reference counting
the ACLs on WAFL.

I should have noted; we did try an 'ls -R' on the Samba server itself on
the same directory to try to rule out the local filesystem being behind the lag.
That command returned almost immediately, which led us to believe that something
with how Samba is relaying the filesystem metadata to the Windows clients was to
blame.

Is Windows Explorer querying all the subordinate ACLs recursively along with
simply doing a count on the number of file and their aggregate size when a
Properties window is opened? I wouldn't expect it to try to query the
complete ACL state recursively, though I could see it having to request access
to each and every item it is counting.

Is Samba doing any kind of translation of the xattrs for the sake of Windows?
We're using 'vfs objects = acl_xattr streams_xattr' in our smb.conf.
The streams_xattr module steers ACL data into the XFS xattrs, should we expect
significant computational overhead reading them back out? Is Samba having to
interpret each xattr entry and translate them on the fly to return the values
the Windows clients are expecting? It stands to reason it could be, and that
doing so en masse would result in what we're seeing.

Does anyone have insight on what exactly Windows Explorer's Properties
window is doing when opened on a top-level directory on a large mounted Samba
share?

Thanks,
Andrew Richards
Senior Systems Engineer
keepertechnology
> On Apr 7, 2017, at 11:53 AM, Jeremy Allison <jra at samba.org> wrote:
> 
> On Wed, Apr 05, 2017 at 02:21:48PM -0400, Andrew Richards via samba wrote:
>> Hello,
>> 
>> We recently deployed Samba 4.5 on Debian Jessie-derived Linux with
kernel v4.5.7 sharing an XFS filesystem.
>> 
>> We are migrating a large data set from old NetApp filers to the Samba
share. While attempting to monitor the progress of the migration, we are
observing very slow population of filesystem metadata in the Properties window
(specifically the General tab for info like "Size" and "Contains:
n Files, n Folders") for top-level directories via Windows Explorer and
very slow recursive directory listing on the command line in PowerShell on
Windows Server 2k8 and 2k12 clients mounting the share. Much slower than the
same clients querying the same top-level directories via the same methods on the
data sets' old home on the NetApp.
>> 
>> "Very slow" means tens of minutes to complete for the Samba
share vs tens of seconds for the same operation on the NetApp share.
>> 
>> smbstatus shows these Windows clients are mounting via SMB2_10 protocol
version. The Linux clients available to us to test with are too old to support
any protocol greater than SMB 1, so we don't yet know if this behavior is
the same on Linux mounts of the same share via the same protocol doing an
'ls -R' of the same directories.
>> 
>> I've found several topics related to slow browsing of heavily
populated directories (over 10,000 files per directory), but that is not the
case here. I have not found much with respect to any kind of performance tuning
that can be done to improve the feedback of filesystem metadata through Samba to
clients. Is this something the vfs_readahead module can help with? Its man page
only mentions Windows Vista, but it isn't clear if the issue it addresses
remains in newer versions of Windows Explorer.
> 
> Can you do any performance tests on the local
> filesystem to see if this is due to XFS xattr
> performance ? If you're migrating a lot of ACLs
> across we will probably hit the xattr code
> very hard. NetApp may be reference counting
> the ACLs on WAFL.