samba - Jul 2003 - Win2k - samba-2.2.8a - can't get above 4MB/sec

I'm trying to maximize throughput between my Samba server and my Win2k (SP3 
and 4) clients. Samba's configured to allow up to 64k packets and the 
server is on a GigE network (3com card, acenic driver) with SCSI UW2 drives 
(software RAID1 and RAID5 volumes). All clients are 100MB/sec FDX on a 
switched LAN. 
 
I can get sustained 35MB/sec transfers (read and write) on the disk 
subsystem so I know that's not a limiting factor. (not bad for software 
RAID, SCSI UW2 max theoretical throughput is 40MB/sec!) 
 
I have read the various TCP/IP and throughput tuning documentation for Win2k 
(and WinNT) found on various sites, including MSDN and the MSKB. My maximum 
window size is 128k, MTU is 1500 (well auto-discover but packet traces show 
that this is where it was autonegotiated at)... TCP window scaling is on 
and so on and so forth... I don't need TCP/IP tuning, I need CIFS and NBT 
tuning. 
 
No matter what I do I can't get Win2k to give up limiting the maximum NBT 
packet size to the MSS. All NBT packets (coming or going) max out at 1460 
bytes and all packets have the DF bit set. I have extra CPU to spare so I'd 
like to have 64k NBT packets that are fragmented to fit into the ethernet 
frames. 

A clip from tcpdump during a transfer:
18:46:07.770734 192.168.1.169.1038 > 192.168.1.2.139: . ack 2375572 win 
65535 (DF)
18:46:07.770942 192.168.1.169.1038 > 192.168.1.2.139: . ack 2378492 win 
65535 (DF)
18:46:07.771002 192.168.1.2.139 > 192.168.1.169.1038: . 
2385792:2387252(1460) ack 4719 win 5840 NBT Packet (DF)
18:46:07.771008 192.168.1.2.139 > 192.168.1.169.1038: . 
2387252:2388712(1460) ack 4719 win 5840 NBT Packet (DF)
18:46:07.771013 192.168.1.2.139 > 192.168.1.169.1038: . 
2388712:2390172(1460) ack 4719 win 5840 NBT Packet (DF)
18:46:07.771017 192.168.1.2.139 > 192.168.1.169.1038: . 
2390172:2391632(1460) ack 4719 win 5840 NBT Packet (DF)
 
I've also read that CIFS can work "raw" (i.e. outside of NBT) but
I'm not
sure how to do it or if it'll help.

Testing with NFS and FTP has shown that I can hit 9.6MB/sec with larger 
packets, but am stuck down around 4MB/sec with these measly 1-2k packets. I 
am pretty sure it's not a Samba-side issue because if I start four or five 
transfers from different workstations I can easily aggregate their 
individual transfer speeds and get 20MB/sec on the GigE interface.  I am 
drawing up blanks so I'm hoping one of the samba gurus can help here.  :-)
 
What I've done so far:
- read over Speed.txt and Speed2.txt
- tried compiling with sendfile and enabling it in smb.conf
- tried -DUSE_MMAP (and turning read raw off)
- setting the maxrecvsize registry parameter (and rebooting)
- probably a few more things I've forgotten.  :-)

Regards,
Andrew

> I'm trying to maximize throughput between my Samba 
> server and my Win2k (SP3 and 4) clients. Samba's 
> configured to allow up to 64k packets and the server 
> is on a GigE network (3com card, acenic driver) with 
> SCSI UW2 drives (software RAID1 and RAID5 volumes). 
> All clients are 100MB/sec FDX on a switched LAN. 
I can't imagine 64k TCP packets. You probably mean the
window size.
> I can get sustained 35MB/sec transfers (read and 
> write) on the disk subsystem so I know that's not a 
> limiting factor. (not bad for software RAID, SCSI 
> UW2 max theoretical throughput is 40MB/sec!)
Not bad if you don't have high expectations. My 4 IDE
LVM volume yields 115 MB/s continuous with dd. It nearly saturates the PCI.
However, if something more
complex like tar is used I get:

  # time tar cbf 64 - . | dd of=/dev/null obs=64k
  4141517+0 records out  # that's 271,418,458,112 bytes
  real    46m49.755s     # that's 2,809.755 seconds

only 96.6 megabytes per second.

but even throwing things through a GigE network into 
/dev/null on another server yields respectable 36 MB/s:

  #time tar cbf 64 - . | rsh p9g "dd of=/dev/null obs=64k"
  4141517+0 records out   # again 271,418,458,112 bytes
  real    125m48.650s     # that's 7,548.650 seconds

which only goes to show that 32-bit 33 MHz PCI bus is 
a bottle-neck.
 
> I have read the various TCP/IP and throughput tuning 
> documentation for Win2k (and WinNT) found on various 
> sites, including MSDN and the MSKB. My maximum 
> window size is 128k, MTU is 1500 (well auto-discover 
> but packet traces show that this is where it was 
> autonegotiated at)... TCP window scaling is on and 
> so on and so forth... I don't need TCP/IP tuning, I 
> need CIFS and NBT tuning. 
There's very little scope for tuning CIFS and NBT, 
other than having a faster server and disks. Using 
special programs to measure transfer rates by 
creating and then copying files with windows API 
calls, I do get about 9,5 MB/s from most clients but 
the actual real-life average is closer to 3-5 MB/s in 
my experience. One of my samba servers is backup 
server collecting data from a dozen Windows clients so 
I have lots of data showing that Windows clients are 
unable to exceed about 4 MB/s sending data to the 
backup server, most often going at 2.2-3.5 MB/s. All 
of them W2K with 2-3 GHz processors, new IDE disks and 
plenty of memory. You know what? I always start all 
the backups simultaneously to save time. It takes just 
a little longer than backing up any one of them alone.
 
> No matter what I do I can't get Win2k to give up 
> limiting the maximum NBT packet size to the MSS. All 
> NBT packets (coming or going) max out at 1460 bytes 
> and all packets have the DF bit set. I have extra 
> CPU to spare so I'd like to have 64k NBT packets 
> that are fragmented to fit into the ethernet frames.
You can't have 64k NBT packets just the same as you
can't have TCP packets (in which NetBEUI packets are 
encapsulated for TCP/IP transport) longer than about 
1514 bytes at most. 1460 sounds good to me.
> I've also read that CIFS can work "raw" (i.e. 
> outside of NBT) but I'm not sure how to do it or if 
> it'll help.
It wouldn't bring much anyway.
> Testing with NFS and FTP has shown that I can hit 
> 9.6MB/sec with larger packets, but am stuck down 
> around 4MB/sec with these measly 1-2k packets. I 
> am pretty sure it's not a Samba-side issue because 
> if I start four or five transfers from different 
> workstations I can easily aggregate their individual 
> transfer speeds and get 20MB/sec on the GigE 
> interface.  I am drawing up blanks so I'm hoping one 
> of the samba gurus can help here.  :-)
You are mixing up the NFS mount options read/write 
block size with packets. These are different things. 
The packet size is always <= 1514 bytes on 
Ethernet/802.3. If you want slightly more (i.e. 4k), 
get FDDI or TokenRing/100, but it won't make it any
more efficient. In fact, on HP-UX 11.0 it was often
optimal to set the block size of NFS shares to 1500,
so go figure.
 
IMO the best way to speed up your smb traffic is to
get rid of those lame scuzzies. Try a solid Opteron
board with 64-bit/66 MHz PCI slots if you're so keen
on speed. (I am too, but it's still a bit expensive:-)

On the other hand, try to think out of box. Perhaps
reducing the packet size to something like 776 or 93 
bytes could do the trick. Who knows, perhaps there's a 
magic number somewhere in the range 41-1460. I've 
never tried but it's sure a lot easier to do than 
increasing the ethernet packet size to 64k.


____________________________________________________________
Get advanced SPAM filtering on Webmail or POP Mail ... Get Lycos Mail!
http://login.mail.lycos.com/r/referral?aid=27005