samba - Nov 2002 - Samba is fast with slow hardware, but slow with fast hardware.

Samba consistently runs faster loading directory (browsing) listings from a 
5400 rpm IDE disk than it does loading from a) raid-1 LVD scsi array b) 
independent scsi devices on a different, non-raid controller. This is truly 
baffling.

I tried to use the scientific method as much as possible. I have tested 
using the following hardware: 2-way SMP Piii 550/512 mb. 900mhz, 
athlon/512mb. Mylex DAC960 lvd single channel scsi controller, AIC7xxx 
Adaptec 3940 controller. I am using the SGI XFS on both the IDE disks, SCSI 
disks and raid array. The scsi disks individually are a mix of Maxtor Atlas 
IV and Seagate Barracuda 10k. The 5400 rpm drive is come crap-o Seagate 10 
gb. Ata III I think.

Client hardware: Generic, but patched up, Windows XP/Win2k workstations. 
Generally, 550MHz, 256mb ram.

Software: I tried kernels 2.4.18-xfs and 2.4.19-xfs. I am not using ACLs or 
anything fancy at this point. I have tried Samba 2.2.3a, 2.2.6, 3.0alpha3, 
3.0alpha20. I am using the debian distribution, though I have also 
(briefly) tried compiling source packages. Same result in either cases.

It does not seem to be an smb.conf or winbind/wins problem. Browsing shares 
on machines, etc, is very fast. Browsing directories inside shares becomes 
very slow if the directory contains more than about 1200 files and 
directories. But only on the scsi or raid disks. On the 5400 rpm IDE disk, 
browsing the same shares is very fast. They all have the same filesystem. 
(I thought it might be a free-space issue so I used the smb.conf to limit 
the apparent freespace to 100 mb. Didn't fix it.)

I used a tcpdump to discover that when browsing inside shares, packets are 
generated that contain names of directories. One packet contains a lot of 
directory names. ("Find Next2" or something similar). Between each
packet
that contains directory names there is about .4 seconds, with no other 
network activity (other than ACKs and related). When browsing the same 
directories from the crapo IDE disk, this .4 second delay is not present. 
There is hardly any real disk activity with the scsi setups or the ide 
setup, as I do this so many times it tends to get cached.  It does this 
consistently on the two different hardware setups (Athlon 900 and SMP PIII 
550).

One other curious thing: During the delay, while waiting for SMB to xmit 
the directories, the client CPU utilization goes to 100% and stays there 
for the 6-20 seconds it takes to fully load the directory. With the IDE 
disk, it spikes at about 35% for the .25 seconds it takes to load the same 
directory. CPU usage of the server during this time is consistently <10% 
with scsi and ide. I have shares set up with identical parameters, except 
for path.

For example something like [clients] path = /mnt/clients and [test] 
path=/mnt/ide/copyofclients in my smb.conf, both XFS, both containing the 
exact same files with the exact same permissions, the files listing on the 
IDE disk are loaded several orders of magnitude faster. For testing 
purposes, I always use the file -> Open dialog in notepad and MS Word to 
test directory load times.

I will provide packet dumps and log level 10 info, if this helps.

The DAC960 is a hardware raid controller, and has 16 mb cache. Its driver 
doesn't seem to use the kernel's scsi subsystem at all. (E.g. the DAC960
works just fine with no kernel scsi support). I also tried a different 
controller that uses the aic7xxx driver, with roughly the same result. 
Bonnie says the scsi setup is ~70 mb/sec and the ide setup is ~6 mb/sec..

I am somewhat inclined to think this problem may be kernel related? But I 
have run into a wallI can't do anything more myself. Can anyone give me any 
hints about where to look next, or has anyone encountered similar behavior? 
Questions about the test rig, or setups or new tests to try.. or anything?

Thank you very much, Samba Team.
Wendell