samba - May 2014 - Samba 4 + Windows XP very slow - especially noticeable with many files

Hi,

After our upgrade from Samba 3.x to 4.1.6 we are having major 
performance issues. Today, I've been able to nail it down to a simple 
test case.
What strikes me is that Windows XP takes about 25 seconds for one job, 
while it only costs 3 seconds on Windows 7.

I have two tests:

1000 files of 10k each (total ~10Mb):
* Linux on localhost or a remote host (1GE or 10GE): 5.5 seconds
* Windows XP - Machine #1: 24.5 seconds
* Windows XP - Machine #2: 60 sec
* Windows 7 - Identical hardware as XP machine #1: 3 seconds

One 500MB file:
* Expected result: 4-5 seconds on 1Gbit (100-125Mbyte/s)
* Linux on localhost: 0.5 sec
* Other Linux machine with 10GE connect: 0.9 sec
* Remote Linux machine (1Gbit): 4.6 sec
* Windows XP - Machine #1: 10.4 sec
* Windows XP - Machine #2: 11 sec
* Windows 7 - Identical hardware as XP machine #1: 7 sec

The setup is a Samba 4.1.6 server on a high school serving 200 - 250 
computers, (at the moment) all Windows XP clients. Server runs 
virtualized in a KVM and has a 10GE connection to the switch.
Traffic stats: OUT: 110Mbit peak (5m avg) and 30Mbit average
Traffic stats: IN: 20Mbit peak (5m avg) and <5Mbit average

The slower-than-usual-speed for the 500M file I can live with for now, 
but the "many small files" case is a real problem. User profiles and 
desktops are full of small files so this causes a slow login. In fact, 
when like 30 computers log in at the same time it becomes so slow it 
takes many minutes for just a 10Mb profile (some even freezing forever), 
students can't start their lesson, etc... it's quite a problem.
Previously, with the same Windows XP clients the performance on the 
Samba 3.x server was good. Users were able to log in quickly. We had no 
issues whatsoever.

The files are served from Samba 4.1.6 on a share which points to 
'tmpfs' storage (ramdisk).
All clients in this test have ssd's. In case of the 500M file I copied 
the file to 'NUL' to avoid storage altogether.
Test files .tar.gz can be found on 
http://www.vulnscan.org/tmp/samba-testcase-slow-files.tar.gz
Test code I used was:
echo.|time
copy \\green\xyz\500M nul
echo.|time
And for the small files case:
echo.|time
xcopy \\green\ramdisk\10k-1000x /S/E/K/H c:\local\10k-1000x\ >nul
echo.|time
Explorer gives very similar timings by the way. I repeated the tests on 
all clients multiple times.
I also tried the "100 files in 10 directories" test case to see if 
maybe the problem was many files in one directory, this made no 
difference so was ruled out.

I can provide log files on request. Sorry, I didn't want to post them 
in public. I have log level 3 ones available plus packet dumps. If 
that's not sufficient I can re-run the test case on a higher log level, 
just let me know.

I already saw a number of 'Samba 4 is slow' threads here, didn't see
much of a resolution, though. Also makes me wonder if these are also XP.

Finally, here's my smb.conf:
[global]
         workgroup = XXXXXXXX
         realm = jnet.xxxxxxxxxxxx.nl
         netbios name = GREEN
         server role = active directory domain controller
         idmap_ldb:use rfc2307 = yes
         dns forwarder = 8.8.8.8
         netbios aliases = STATLER ATHENA THORTON
         interfaces = jnet
         logon script = connectu.bat
         log file = /usr/local/samba/var/log/samba.log
         log level = 1
         max log size = 250000
         debug prefix timestamp = yes
         debug uid = yes
         reset on zero vc = yes
#       socket options = IPTOS_LOWDELAY TCP_NODELAY
         socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE 
TCP_KEEPIDLE=120 TCP_KEEPINTVL=10 TCP_KEEPCNT=5 SO_SNDBUF=65536 
SO_RCVBUF=65536
         kernel oplocks = no
         kernel share modes = no
         mangling method         = hash2
         mangle prefix           = 6
         lanman auth             = yes
         time server = yes
         admin users = admin bram root xyz fog
         guest account = nobody
         map to guest            = Never
         # ON OR OFF??? (default = no)
         winbind enum users = yes
         winbind enum groups = yes
         wins support            = yes
         deadtime = 20
         # Make wide symlinks work
         unix extensions = no
         wide links = yes

[ramdisk]
         comment = "Ramdisk - for testing"
         read only = no
         browseable = no
         public = no
         guest ok = no
         valid users = bram, xyz, zzz, admin
         path = /data/ramdisk


I already tried a number of configuration changes, like not having any 
socket options, or only LOWDELAY NODELAY, etc.. no difference (on XP at 
least).
Also commented out various options to see if that made a difference... 
not that I could find.

Hopefully someone can help us out here. We've always been proud to use 
Samba at our school instead of Windows servers, but right now it's not 
really showing the best of it ;).

Thanks in advance, and like I said.. if I need to provide anything.. 
let me know,

Bram Matthys.

Hi,

I am actually seeing something very similar in another email thread (which I
created) in which I am yet to provide some logs - I am not having enough time to
do testing :(. I myself migrated from Samba 3 to Samba 4 just this weekend. My
clients are all Windows XPs.

Could you please try adding the option force user = root to the definition of
the share you are testing with? I know that this is a serious security risk, BUT
for me it seems to have worked around the speed issue, although I realize that
we will have to get rid of that eventually :).

Have a nice day,
Petr MOTEJLEK
________________________________________
Od: samba-bounces at lists.samba.org <samba-bounces at lists.samba.org> za
u?ivatele Bram Matthys <syzop at vulnscan.org>
Odesl?no: 20. kv?tna 2014 15:39
Komu: samba at lists.samba.org
P?edm?t: [Samba] Samba 4 + Windows XP very slow - especially noticeable with
many files

Hi,

After our upgrade from Samba 3.x to 4.1.6 we are having major
performance issues. Today, I've been able to nail it down to a simple
test case.
What strikes me is that Windows XP takes about 25 seconds for one job,
while it only costs 3 seconds on Windows 7.

I have two tests:

1000 files of 10k each (total ~10Mb):
* Linux on localhost or a remote host (1GE or 10GE): 5.5 seconds
* Windows XP - Machine #1: 24.5 seconds
* Windows XP - Machine #2: 60 sec
* Windows 7 - Identical hardware as XP machine #1: 3 seconds

One 500MB file:
* Expected result: 4-5 seconds on 1Gbit (100-125Mbyte/s)
* Linux on localhost: 0.5 sec
* Other Linux machine with 10GE connect: 0.9 sec
* Remote Linux machine (1Gbit): 4.6 sec
* Windows XP - Machine #1: 10.4 sec
* Windows XP - Machine #2: 11 sec
* Windows 7 - Identical hardware as XP machine #1: 7 sec

The setup is a Samba 4.1.6 server on a high school serving 200 - 250
computers, (at the moment) all Windows XP clients. Server runs
virtualized in a KVM and has a 10GE connection to the switch.
Traffic stats: OUT: 110Mbit peak (5m avg) and 30Mbit average
Traffic stats: IN: 20Mbit peak (5m avg) and <5Mbit average

The slower-than-usual-speed for the 500M file I can live with for now,
but the "many small files" case is a real problem. User profiles and
desktops are full of small files so this causes a slow login. In fact,
when like 30 computers log in at the same time it becomes so slow it
takes many minutes for just a 10Mb profile (some even freezing forever),
students can't start their lesson, etc... it's quite a problem.
Previously, with the same Windows XP clients the performance on the
Samba 3.x server was good. Users were able to log in quickly. We had no
issues whatsoever.

The files are served from Samba 4.1.6 on a share which points to
'tmpfs' storage (ramdisk).
All clients in this test have ssd's. In case of the 500M file I copied
the file to 'NUL' to avoid storage altogether.
Test files .tar.gz can be found on
http://www.vulnscan.org/tmp/samba-testcase-slow-files.tar.gz
Test code I used was:
echo.|time
copy \\green\xyz\500M nul
echo.|time
And for the small files case:
echo.|time
xcopy \\green\ramdisk\10k-1000x /S/E/K/H c:\local\10k-1000x\ >nul
echo.|time
Explorer gives very similar timings by the way. I repeated the tests on
all clients multiple times.
I also tried the "100 files in 10 directories" test case to see if
maybe the problem was many files in one directory, this made no
difference so was ruled out.

I can provide log files on request. Sorry, I didn't want to post them
in public. I have log level 3 ones available plus packet dumps. If
that's not sufficient I can re-run the test case on a higher log level,
just let me know.

I already saw a number of 'Samba 4 is slow' threads here, didn't see
much of a resolution, though. Also makes me wonder if these are also XP.

Finally, here's my smb.conf:
[global]
         workgroup = XXXXXXXX
         realm = jnet.xxxxxxxxxxxx.nl
         netbios name = GREEN
         server role = active directory domain controller
         idmap_ldb:use rfc2307 = yes
         dns forwarder = 8.8.8.8
         netbios aliases = STATLER ATHENA THORTON
         interfaces = jnet
         logon script = connectu.bat
         log file = /usr/local/samba/var/log/samba.log
         log level = 1
         max log size = 250000
         debug prefix timestamp = yes
         debug uid = yes
         reset on zero vc = yes
#       socket options = IPTOS_LOWDELAY TCP_NODELAY
         socket options = IPTOS_LOWDELAY TCP_NODELAY SO_KEEPALIVE
TCP_KEEPIDLE=120 TCP_KEEPINTVL=10 TCP_KEEPCNT=5 SO_SNDBUF=65536
SO_RCVBUF=65536
         kernel oplocks = no
         kernel share modes = no
         mangling method         = hash2
         mangle prefix           = 6
         lanman auth             = yes
         time server = yes
         admin users = admin bram root xyz fog
         guest account = nobody
         map to guest            = Never
         # ON OR OFF??? (default = no)
         winbind enum users = yes
         winbind enum groups = yes
         wins support            = yes
         deadtime = 20
         # Make wide symlinks work
         unix extensions = no
         wide links = yes

[ramdisk]
         comment = "Ramdisk - for testing"
         read only = no
         browseable = no
         public = no
         guest ok = no
         valid users = bram, xyz, zzz, admin
         path = /data/ramdisk


I already tried a number of configuration changes, like not having any
socket options, or only LOWDELAY NODELAY, etc.. no difference (on XP at
least).
Also commented out various options to see if that made a difference...
not that I could find.

Hopefully someone can help us out here. We've always been proud to use
Samba at our school instead of Windows servers, but right now it's not
really showing the best of it ;).

Thanks in advance, and like I said.. if I need to provide anything..
let me know,

Bram Matthys.

--
To unsubscribe from this list go to the following URL and read the
instructions:  https://lists.samba.org/mailman/options/samba

Hi,

Bram Matthys wrote, on 20-5-2014 15:39:
> After our upgrade from Samba 3.x to 4.1.6 we are having major performance
> issues. Today, I've been able to nail it down to a simple test case.
> What strikes me is that Windows XP takes about 25 seconds for one job,
while
> it only costs 3 seconds on Windows 7.
> 
> I have two tests:
> 
> 1000 files of 10k each (total ~10Mb):
> * Linux on localhost or a remote host (1GE or 10GE): 5.5 seconds
> * Windows XP - Machine #1: 24.5 seconds
> * Windows XP - Machine #2: 60 sec
> * Windows 7 - Identical hardware as XP machine #1: 3 seconds
> 
> One 500MB file:
> * Expected result: 4-5 seconds on 1Gbit (100-125Mbyte/s)
> * Linux on localhost: 0.5 sec
> * Other Linux machine with 10GE connect: 0.9 sec
> * Remote Linux machine (1Gbit): 4.6 sec
> * Windows XP - Machine #1: 10.4 sec
> * Windows XP - Machine #2: 11 sec
> * Windows 7 - Identical hardware as XP machine #1: 7 sec 
I also tried the following, on the server-side:
0) Installed a new server
1) Tried version 4.1.7 (was: 4.1.6)
2) Tried latest samba from git (master)
3) As domain member
4) Also, not in a domain: security = user and map to guest = bad user
5) compiling without --enable-debug

These all made no difference. Okay, without --enable-debug it's maybe 20%
faster, but nowhere near the Windows 7 performance (still 6-7 times slower).

On the client-side I reset some TCP/IP settings to Windows default, reboot,
test. Also later tried other 'optimal' values, reboot, test. Made no
difference. Would also be strange when the Samba 3 performance was OK and
when FTP goes at near-full wire speed, see next.

Some more benchmarking / testing:
Although I'm more concerned about the small files case, I also tested
FTP'ing the (same) 500Mb file. This took 4.5 seconds (115Mbyte/sec). In
contrast to 10.4 seconds via Samba (~50Mbyte/sec data transfer, with
protocol overhead I see ~59Mbyte/sec on the LAN). smbd uses 15-20% CPU
during the transfer.

Kinda out of ideas now.

Regards,

Bram.

-- 
Bram Matthys
Software developer/IT consultant        syzop at vulnscan.org
Website:                                  www.vulnscan.org
PGP key:                       www.vulnscan.org/pubkey.asc
PGP fp: EBCA 8977 FCA6 0AB0 6EDB  04A7 6E67 6D45 7FE1 99A6