samba - May 2004 - Poor performance with Mac OS X Panther clients

Here's my setup: 

        - Single Mac OS X Panther client w/ latest patches.
        - Single Windows NT4 w/ latest SPs client.
        - Debian stable Samba server(2.2.3a-13) w/ custom 2.4.26 kernel.
        Oplocks enabled.
        - Cisco 2950
        - All nodes 100Mb and everything on the same switch.
        - HP DL360 w/ 2.8GHz hyper-threaded processor and 1GB of memory.
        - HP MSA1000 SAN.
        - XFS filesystems w/ ACL support.
        
My client is moving their data over to a fail-over Samba setup backended
with the HP SAN. They have several directories with approx. 5000 to
10000 files. The files were moved into the directories on the new file
server from a Mac OS X client in order to preserve the Mac OS metadata.

Viewing these directories from explorer.exe on the Windows client, from
the Terminal on OSX using ls, and from a Linux laptop using smbclient
and ls is stellar. Very quick. Viewing these directories from within the
OSX Finder causes the smbd process to spike to approx. 45% to 55% and
stay there for approx. 10 seconds or however long it takes the Finder to
render all of the icons for the files in the directory. Scrolling the
Finder while the icons are being rendered causes the CPU to jump again.
Depending on how many screens you scroll through in Finder, you can
cause smbd to jump to 99% and stay there for several minutes. We are
still in pre-deployment but the CPU spike is making the client nervous
about what will happen when we start to load many Mac OS X client onto
the server. 

I'm not an expert on the SMB protocol but I've captured packet dumps
from a fresh share mount and directory view from both the Windows NT4
client and the Mac OS X client and it looks to me like the NT4 client is
doing the directory contents lookups with a single "FIND_FIRST2, \*"
and
a corresponding return packet from the server with the listing of the
directory contents whereas the Mac OS X Finder client appears to be
doing a FIND_FIRST2 call for every file, including metadata files, in
the directory. This would seem to be supported by a strace of the smbd
process which shows tons and tons of getdents32 calls. 

My gut says that this is a problem with OSX's Finder and not Samba but,
understandably, my client doesn't care...they just want it fixed. Is
there a known work-around for the performance issue with Mac OS X Finder
clients? Do I need to tweak something on the server? On the client? I've
search the Samba archives and not found much mention of this issue and
I've Googled around for information from Apple's end and drawn a blank
there too.

Thanks ahead of time for any insight that you can provide. 

-- 
Nathan R. Valentine <nathan@nathanvalentine.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.samba.org/archive/samba/attachments/20040511/4dd4f3b1/attachment.bin

Hi Nathan,

	all I can say is we are also testing Mac OS X with Samba (on Solaris)
and with slightly more typical numbers of files per directory are not seeing
any simliar issues.
In principal it does not surprise me that it is much slower on your OS X
client than on your Linux or Windows clients. When you open a folder in
your Linux or Windows file browser all they will need to do is request a 
file listing. On the other hand an OS X client will read the resourse fork 
(meta data) for each file to give it the correct application association and 
icon in the finder.
Does your customer not see a similar problem when the files are stored on
a Windows file server? I would expect they would (that is if they are
mounting from the Windows server over SMB not AFP which may prove quicker).

no answer but hope this helps,

	thanks Andy.



Here's my setup: 

        - Single Mac OS X Panther client w/ latest patches.
        - Single Windows NT4 w/ latest SPs client.
        - Debian stable Samba server(2.2.3a-13) w/ custom 2.4.26 kernel.
        Oplocks enabled.
        - Cisco 2950
        - All nodes 100Mb and everything on the same switch.
        - HP DL360 w/ 2.8GHz hyper-threaded processor and 1GB of memory.
        - HP MSA1000 SAN.
        - XFS filesystems w/ ACL support.
        
My client is moving their data over to a fail-over Samba setup backended
with the HP SAN. They have several directories with approx. 5000 to
10000 files. The files were moved into the directories on the new file
server from a Mac OS X client in order to preserve the Mac OS metadata.

Viewing these directories from explorer.exe on the Windows client, from
the Terminal on OSX using ls, and from a Linux laptop using smbclient
and ls is stellar. Very quick. Viewing these directories from within the
OSX Finder causes the smbd process to spike to approx. 45% to 55% and
stay there for approx. 10 seconds or however long it takes the Finder to
render all of the icons for the files in the directory. Scrolling the
Finder while the icons are being rendered causes the CPU to jump again.
Depending on how many screens you scroll through in Finder, you can
cause smbd to jump to 99% and stay there for several minutes. We are
still in pre-deployment but the CPU spike is making the client nervous
about what will happen when we start to load many Mac OS X client onto
the server. 

I'm not an expert on the SMB protocol but I've captured packet dumps
from a fresh share mount and directory view from both the Windows NT4
client and the Mac OS X client and it looks to me like the NT4 client is
doing the directory contents lookups with a single "FIND_FIRST2, \*"
and
a corresponding return packet from the server with the listing of the
directory contents whereas the Mac OS X Finder client appears to be
doing a FIND_FIRST2 call for every file, including metadata files, in
the directory. This would seem to be supported by a strace of the smbd
process which shows tons and tons of getdents32 calls. 

My gut says that this is a problem with OSX's Finder and not Samba but,
understandably, my client doesn't care...they just want it fixed. Is
there a known work-around for the performance issue with Mac OS X Finder
clients? Do I need to tweak something on the server? On the client? I've
search the Samba archives and not found much mention of this issue and
I've Googled around for information from Apple's end and drawn a blank
there too.

Thanks ahead of time for any insight that you can provide. 

-- 
Nathan R. Valentine <nathan@nathanvalentine.org>

BBCi at http://www.bbc.co.uk/

This e-mail (and any attachments) is confidential and may contain personal views
which are not the views of the BBC unless specifically
stated.
If you have received it in error, please delete it from your system. Do not use,
copy or disclose the information in any way nor act in
reliance on it and notify the sender immediately. Please note that the BBC
monitors e-mails sent or received.
Further communication will signify your consent to this.

I would strongly suspect that the Mac OS finder is simply hammering the
file server because, in your case, it as to do 10000 seperate read opertions
just to display the folder contents. Try with an NFS share, this would
identify whether the SMB support in either the client or server is
ineffecient or if you just have unrealistic expectations of performance
from this type of configuration.
If you are using an OS X server have you tried connected via Apple Talk
instead? This has native support for resource fork data and may give
better performance,

	thanks Andy.
> I will test the results from both Windows and Mac OS X clients with the
> data hosted on a Windows server to measure the results. From experience,
> I expect that the Mac OS X clients will still perform worse but not to
> the same degree as when the data was hosted on Samba. 
Turns out that we are seeing similar CPU spikes on the server whether
the server is a Windows box of a *nix box running Samba. To me, this
would indicate that this is a problem with the Mac OS X client. Based on
the packet dumps, I would say that the Mac OS X Samba system is doing
the the directory contents lookups in a very inefficient manner. Since
OS X is running a version of Samba... Anyone have any tips? 


-- 
Nathan R. Valentine <nathan@nathanvalentine.org>

On Sat, 2004-05-15 at 03:34, Nathan R. Valentine wrote:
> > I will test the results from both Windows and Mac OS X clients with
the
> > data hosted on a Windows server to measure the results. From
experience,
> > I expect that the Mac OS X clients will still perform worse but not to
> > the same degree as when the data was hosted on Samba. 
> 
> Turns out that we are seeing similar CPU spikes on the server whether
> the server is a Windows box of a *nix box running Samba. To me, this
> would indicate that this is a problem with the Mac OS X client. Based on
> the packet dumps, I would say that the Mac OS X Samba system is doing
> the the directory contents lookups in a very inefficient manner. Since
> OS X is running a version of Samba... Anyone have any tips? 
Actually, the OSX client is the smbfs originally from FreeBSD.  (And
unrelated to the Linux smbfs).  

Andrew Bartlett

-- 
Andrew Bartlett                                 abartlet@pcug.org.au
Manager, Authentication Subsystems, Samba Team  abartlet@samba.org
Student Network Administrator, Hawker College   abartlet@hawkerc.net
http://samba.org     http://build.samba.org     http://hawkerc.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.samba.org/archive/samba/attachments/20040515/12cbe1e8/attachment.bin