samba - May 2004 - Samba 3 write performance drop on files > 16 MB

> I have been running performance tests on our Debian-Samba 3 file
> server.  It is running the 2.4 kernel on a ProLiant DL380 G3 server
> (Full Specs below).  For a windows client, we are running Windows 2003
> server on the exact same hardware.  They are both running at Gigabit
> speed.  
> 
> What we have found is that on files 16MB and smaller we can get better
> performance Windows client to Linux Server through SMB than we can get
> Windows to local disk.  (Both servers have the same disk subsystem)
> We get around 120,000KB/sec throughput.  But after 16MB, the
> performance drops until it flatlines from 32MB on at 5000 to
> 7000KB/sec throughput.  This drop does not occur on either server
> running the tests locally, which shows the drop is not due to memory
> caching or controller caching.  Additionally, SMB reads stay constant
> at around 50 to 60,000KB per second.
> I ran ethereal and found one surprising difference between the 16MB
> and 32MB transfers.  For approximately .8 seconds before the transfer
> begins, the following conversation occurs over and over:
> 
> Windows>Linux Write AndX Request
> Linux>Windows Write AndX Response
> Windows>Linux Trans2 Request, QUERY_FILE_INFO, FID: 0x212a, Query File
> Standard Info
> Linux>Windows Trans2 Response, QUERY_FILE_INFO
> 
> I have tried all of the recommended SAMBA performance tweaks that I
> can find, but none seem to affect this dropoff.  Has anyone seen this?
> It appears to be a bug in either Samba or Windows 2003.
> 
> Thanks,
> David
> 
> HEWLETT PACKARD COMPUTER: ProLiant DL380 G3 
> Dual P4 2.8 GHz Xeon Socket Processors with 512KB cache (FULL
> processor speed) 
> 1GB RAM PC2100 DDR SDRAM 200MHz
> 400MHz System Bus Speed
> Dual NC7781 PCI-X Gigabit NICs
> Integrated 5i Plus 64MB cache 
> RAID 5 Hard disk array: 6x 146GB 1.0" Hot-Plug Universal U320 SCSI 10K
> RPM
>

> I have been running performance tests on our Debian-Samba 3 file
> server.  It is running the 2.4 kernel on a ProLiant DL380 G3 server
> (Full Specs below).  For a windows client, we are running Windows 2003
> server on the exact same hardware.  They are both running at Gigabit
> speed.  
> What we have found is that on files 16MB and smaller we can get better
> performance Windows client to Linux Server through SMB than we can get
> Windows to local disk.  (Both servers have the same disk subsystem)
> We get around 120,000KB/sec throughput.  But after 16MB, the
> performance drops until it flatlines from 32MB on at 5000 to
> 7000KB/sec throughput.  This drop does not occur on either server
> running the tests locally, which shows the drop is not due to memory
> caching or controller caching.  Additionally, SMB reads stay constant
> at around 50 to 60,000KB per second.
> I ran ethereal and found one surprising difference between the 16MB
> and 32MB transfers.  For approximately .8 seconds before the transfer
> begins, the following conversation occurs over and over:
> 
> Windows>Linux Write AndX Request
> Linux>Windows Write AndX Response
> Windows>Linux Trans2 Request, QUERY_FILE_INFO, FID: 0x212a, Query File
> Standard Info
> Linux>Windows Trans2 Response, QUERY_FILE_INFO
> 
> I have tried all of the recommended SAMBA performance tweaks that I
> can find, but none seem to affect this dropoff.  Has anyone seen this?
> It appears to be a bug in either Samba or Windows 2003.
> 
> Thanks,
> David
> 
> HEWLETT PACKARD COMPUTER: ProLiant DL380 G3 
> Dual P4 2.8 GHz Xeon Socket Processors with 512KB cache (FULL
> processor speed) 
> 1GB RAM PC2100 DDR SDRAM 200MHz
> 400MHz System Bus Speed
> Dual NC7781 PCI-X Gigabit NICs
> Integrated 5i Plus 64MB cache 
> RAID 5 Hard disk array: 6x 146GB 1.0" Hot-Plug Universal U320 SCSI 10K
> RPM
>

On Sat, May 22, 2004 at 10:40:48AM -0400, DKrause@jw.org
wrote:
> > I have been running performance tests on our Debian-Samba 3 file
> > server.  It is running the 2.4 kernel on a ProLiant DL380 G3 server
> > (Full Specs below).  For a windows client, we are running Windows 2003
> > server on the exact same hardware.  They are both running at Gigabit
> > speed.  
> > 
> > What we have found is that on files 16MB and smaller we can get better
> > performance Windows client to Linux Server through SMB than we can get
> > Windows to local disk.  (Both servers have the same disk subsystem)
> > We get around 120,000KB/sec throughput.  But after 16MB, the
> > performance drops until it flatlines from 32MB on at 5000 to
> > 7000KB/sec throughput.  This drop does not occur on either server
> > running the tests locally, which shows the drop is not due to memory
> > caching or controller caching.  Additionally, SMB reads stay constant
> > at around 50 to 60,000KB per second.
> > I ran ethereal and found one surprising difference between the 16MB
> > and 32MB transfers.  For approximately .8 seconds before the transfer
> > begins, the following conversation occurs over and over:
> > 
> > Windows>Linux Write AndX Request
> > Linux>Windows Write AndX Response
> > Windows>Linux Trans2 Request, QUERY_FILE_INFO, FID: 0x212a, Query
File
> > Standard Info
> > Linux>Windows Trans2 Response, QUERY_FILE_INFO
> > 
> > I have tried all of the recommended SAMBA performance tweaks that I
> > can find, but none seem to affect this dropoff.  Has anyone seen this?
> > It appears to be a bug in either Samba or Windows 2003.
Have you tried this against a W2K3 server ? If it behaves
the same way then it's a client issue, if not we need to
examine how Samba differs.

Jeremy.

I tried W2K3 to W2K3 server, still on same hardware.  It does not get as
high of performance initially, but it also does not drop off after 16MB.
It stayed at about 25 to 30,000KB/sec.

-----Original Message-----
From: Jeremy Allison [mailto:jra@samba.org] 
Sent: Monday, May 24, 2004 11:49 AM
To: Krause, David
Cc: samba@lists.samba.org
Subject: Re: [Samba] Samba 3 write performance drop on files > 16 MB


On Sat, May 22, 2004 at 10:40:48AM -0400, DKrause@jw.org
wrote:
> > I have been running performance tests on our Debian-Samba 3 file
> > server.  It is running the 2.4 kernel on a ProLiant DL380 G3 server
> > (Full Specs below).  For a windows client, we are running Windows
2003
> > server on the exact same hardware.  They are both running at Gigabit
> > speed.
> >
> > What we have found is that on files 16MB and smaller we can get
better
> > performance Windows client to Linux Server through SMB than we can
get
> > Windows to local disk.  (Both servers have the same disk subsystem)
> > We get around 120,000KB/sec throughput.  But after 16MB, the
> > performance drops until it flatlines from 32MB on at 5000 to
> > 7000KB/sec throughput.  This drop does not occur on either server
> > running the tests locally, which shows the drop is not due to memory
> > caching or controller caching.  Additionally, SMB reads stay
constant
> > at around 50 to 60,000KB per second.
> > I ran ethereal and found one surprising difference between the 16MB
> > and 32MB transfers.  For approximately .8 seconds before the
transfer
> > begins, the following conversation occurs over and over:
> >
> > Windows>Linux Write AndX Request
> > Linux>Windows Write AndX Response
> > Windows>Linux Trans2 Request, QUERY_FILE_INFO, FID: 0x212a, Query
File
> > Standard Info
> > Linux>Windows Trans2 Response, QUERY_FILE_INFO
> >
> > I have tried all of the recommended SAMBA performance tweaks that I
> > can find, but none seem to affect this dropoff.  Has anyone seen
this?
> > It appears to be a bug in either Samba or Windows 2003.
Have you tried this against a W2K3 server ? If it behaves
the same way then it's a client issue, if not we need to
examine how Samba differs.

Jeremy.

Jeremy,
I have the protocol traces but they are about 44MB and 70MB.  Is there a
way I can send them to you?  As for the testing, we are running Windows
2003 advanced server with no service packs.  From the 2003 client, we
map a drive to the server (Samba or W2003).  Then from that drive we
have been running the IO-zone basic suite of tests.  Running (from the
command line in the mapped drive, I.e. Y:\...) "iozone -s 16m -i 0 -A -R
-f report.txt" should show normal results and "iozone -s 32m -i 0 -A
-R
-f report2.txt" should show the problem.

Here is an example of a full report.  File size is the left column, and
record size is the top row.  Speed is in KB/sec.  You can see where the
speed drops dramatically at 32MB.

Thanks,
David

"Writer report"
        "4" "8" "16" "32" "64"
"128" "256" "512" "1024"
"2048" "4096"
"8192" "16384"
"64"	 75628  78294  77721  78513  74395 
"128 "  90751  95369  96809  95184  95865  93212 
"256 "  107823  107711  107092  109354  107665  105674  105504 
"512 "  114706  115112  115430  115644  114250  110561  112454  112175
"1024 "  115586  118498  119247  118106  118558  117669  117109 
115713
113378 
"2048 "  117543  117730  119502  118905  120940  117782  120144 
118108
118950  118211 
"4096 "  119136  117897  121224  120629  121265  120565  120701 
118823
119519  119297  119750 
"8192 "  119864  120137  121090  120940  121612  120158  120953 
119481
118914  119388  120234  120022 
"16384 "  119758  120410  121239  121283  121382  121003  121197 
49943
120290  120309  71590  120519  120383 
"32768 "  8815  8517  8410  8390  7580  7907  7489  7784  8075  7372
7526  7374  8710 
"65536 "  5923  6028  6118  6053  6147  6109  6125  6161  6061  6040
6118  6182  6058 
"131072 "  5355  5320  5329  5324  5337  5340  5305  5316  5295  5328
5316  5358  5121 
"262144 "  5000  5008  5010  5022  5010  5015  5032  5016  5006  5002
5010  5016  5025 
"524288 "  4855  4868  4777  4857  4869  4842  4842  4865  4869  4836
4864  4864  4852 

-----Original Message-----
From: Jeremy Allison [mailto:jra@samba.org] 
Sent: Tuesday, May 25, 2004 8:03 PM
To: Krause, David
Cc: jra@samba.org; samba@lists.samba.org
Subject: Re: [Samba] Samba 3 write performance drop on files > 16 MB


On Mon, May 24, 2004 at 11:56:11AM -0400, DKrause@jw.org
wrote:
> I tried W2K3 to W2K3 server, still on same hardware.  It does not get
as
> high of performance initially, but it also does not drop off after
16MB.
> It stayed at about 25 to 30,000KB/sec.
Ok, I'd really like to see some protocol traces of it going
fast (to w2k3) and slow (to samba). These might be quite large,
or can you tell me *exactly* how to reproduce this on a system
here ? I'd need W2k3 SP level, file size and exaclty what client
operations you do to reproduce it (ie. is it a mapped drive
or unc pathname, do you use a command line copy or explorer etc.).

Thanks,

Jeremy.