samba - Mar 2014 - suggestions for a "fast" fileserver - 1G / 10G

Hi,

I'm looking for some recommendations for a "fast" fileserver
regarding
the hardware you use.

We have different fileservers as our requirements changed over time.

The "main" problem we are faced with is, that with smb (windows 7 and
OS
X) clients we never get really close to GBit speed on reads or writes.
Using the same servers/storages with ftp, ssh, rsync, nfs we are on the
max. of GBit or with the 10Gbit Storage/server on the max of the storage
we currently own. (about 400MB/s)

E.g. from my Mac Pro I get smb r/w +- 40MB/s, with ftp I get 90MB/s on a
1Gbit Server.

So I try to eliminate some bottle necks. But where are they?

I know there are some protocol overheads etc. comparing smb and e.g. ftp.

So what kind of setup serverside may get better performance? What kind
of Hardware are you using?

Faster NICs (usually I have INTEL ones) faster cpu? Different generation?

	Any feedback and suggestions are welcome. Regards . G?tz
-- 
G?tz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
Fax  +49 7141 969 55 420
E-Mail goetz.reinicke at filmakademie.de

Filmakademie Baden-W?rttemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats:
J?rgen Walter MdL
Staatssekret?r im Ministerium f?r Wissenschaft,
Forschung und Kunst Baden-W?rttemberg

Gesch?ftsf?hrer:
Prof. Thomas Schadt

>
>
> E.g. from my Mac Pro I get smb r/w +- 40MB/s, with ftp I get 90MB/s on a
> 1Gbit Server.
>
> So I try to eliminate some bottle necks. But where are they?
>
> I know there are some protocol overheads etc. comparing smb and e.g. ftp.
>
What samba version are you running ? I'm using the latest (or close to
latest) samba sernet 4.1.x . I think you're running into protocol
overheads. I got the same max 40MB/s on reads and writes over a gig
connection to our server which has 2x10gb LACP bond from a linux client
running RHEL6/CENTOS6 the kernel for which only supports smb protocol ver=1
mounts. Between the client and the server there are 2 10gbps hops, we're
getting 0.2ms ping latency from the client to the server. Using smbclient
-m 3 I was able to get ~110MB/s on writes and reads from the same client
(using smbclient from the same samba-sernet 4.1.x packages). From a windows
7 and 8 client I was able to get the same 110MB/s .

Anyone know how to determine the smb protocol version of connected clients
on the server side without turning up connection logging, i.e. is there
some command like smbstatus which might list this information (I couldn't
find it in the output of smbstatus even with -v -d 10) and wasn't able to
find it through the net commands.

I would say posting your smb4.conf minus any information that you need to
keep private would be a must.  This way others could see what tuning
options you may already have set or may need.

Joe Maloney



On Sat, Mar 22, 2014 at 6:37 AM, G?tz Reinicke - IT Koordinator <
goetz.reinicke at filmakademie.de> wrote:
> Hi,
>
> I'm looking for some recommendations for a "fast" fileserver
regarding
> the hardware you use.
>
> We have different fileservers as our requirements changed over time.
>
> The "main" problem we are faced with is, that with smb (windows 7
and OS
> X) clients we never get really close to GBit speed on reads or writes.
> Using the same servers/storages with ftp, ssh, rsync, nfs we are on the
> max. of GBit or with the 10Gbit Storage/server on the max of the storage
> we currently own. (about 400MB/s)
>
> E.g. from my Mac Pro I get smb r/w +- 40MB/s, with ftp I get 90MB/s on a
> 1Gbit Server.
>
> So I try to eliminate some bottle necks. But where are they?
>
> I know there are some protocol overheads etc. comparing smb and e.g. ftp.
>
> So what kind of setup serverside may get better performance? What kind
> of Hardware are you using?
>
> Faster NICs (usually I have INTEL ones) faster cpu? Different generation?
>
>         Any feedback and suggestions are welcome. Regards . G?tz
> --
> G?tz Reinicke
> IT-Koordinator
>
> Tel. +49 7141 969 82 420
> Fax  +49 7141 969 55 420
> E-Mail goetz.reinicke at filmakademie.de
>
> Filmakademie Baden-W?rttemberg GmbH
> Akademiehof 10
> 71638 Ludwigsburg
> www.filmakademie.de
>
> Eintragung Amtsgericht Stuttgart HRB 205016
>
> Vorsitzender des Aufsichtsrats:
> J?rgen Walter MdL
> Staatssekret?r im Ministerium f?r Wissenschaft,
> Forschung und Kunst Baden-W?rttemberg
>
> Gesch?ftsf?hrer:
> Prof. Thomas Schadt
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

If it's clean 10GBE end to end, have you maxed out the MTU? (use the same
setting on all devices in the same layer2 domain)
I suspect this is the #1 limitation. I see many 10G networks still using
1500ish MTU. (some vendors calculate different parts of the overhead +/-
into MTU. Test test test and test...) Some inferior nics can borrow lots of
CPU usage ( Realtek's are notorious, but hopefully they didn't stick
their
nose in 10G space yet). There could be some driver and nic settings like
memory rings/buffers etc, but this either lives on the card or sucks imo.

What kind of cabling, optics, and SFP/XFP's are in use end to end?

Have you monitored cpu usage during this time?
Have you reviewed the interconnected hardware and layer1 specifications
between the server and host(s) in question?
Have you tried other performance testing, such as apache benchmark, iperf,
etc?
Are there any firewalls, even iptables/equiv with permit all? (esp if cpu
usage is in question)
There is some encoding overhead at layer2 (before TCP/IP is even part of
the equation).

HTH, a few things to chase/confirm before focusing on Samba too deeply.

Regards,


Scott.


On Sat, Mar 22, 2014 at 4:37 AM, G?tz Reinicke - IT Koordinator <
goetz.reinicke at filmakademie.de> wrote:
> Hi,
>
> I'm looking for some recommendations for a "fast" fileserver
regarding
> the hardware you use.
>
> We have different fileservers as our requirements changed over time.
>
> The "main" problem we are faced with is, that with smb (windows 7
and OS
> X) clients we never get really close to GBit speed on reads or writes.
> Using the same servers/storages with ftp, ssh, rsync, nfs we are on the
> max. of GBit or with the 10Gbit Storage/server on the max of the storage
> we currently own. (about 400MB/s)
>
> E.g. from my Mac Pro I get smb r/w +- 40MB/s, with ftp I get 90MB/s on a
> 1Gbit Server.
>
> So I try to eliminate some bottle necks. But where are they?
>
> I know there are some protocol overheads etc. comparing smb and e.g. ftp.
>
> So what kind of setup serverside may get better performance? What kind
> of Hardware are you using?
>
> Faster NICs (usually I have INTEL ones) faster cpu? Different generation?
>
>         Any feedback and suggestions are welcome. Regards . G?tz
> --
> G?tz Reinicke
> IT-Koordinator
>
> Tel. +49 7141 969 82 420
> Fax  +49 7141 969 55 420
> E-Mail goetz.reinicke at filmakademie.de
>
> Filmakademie Baden-W?rttemberg GmbH
> Akademiehof 10
> 71638 Ludwigsburg
> www.filmakademie.de
>
> Eintragung Amtsgericht Stuttgart HRB 205016
>
> Vorsitzender des Aufsichtsrats:
> J?rgen Walter MdL
> Staatssekret?r im Ministerium f?r Wissenschaft,
> Forschung und Kunst Baden-W?rttemberg
>
> Gesch?ftsf?hrer:
> Prof. Thomas Schadt
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

Le Sat, 22 Mar 2014 12:37:15 +0100 vous ?criviez:
> I'm looking for some recommendations for a "fast" fileserver
regarding
> the hardware you use.
> 
What is your disk subsystem like? What filesystem, with which mount
options? Any server I've setup in the past 3 years routinely achieve
110 MB/s with samba on GigE, 600 to 800 MB/s using 10 GigE. 

The very very most important option is :

1? don't ever use CFQ on a file server. Use noop or eventually
deadline. 

CFQ absolutely sucks wind on servers. CFQ is only good for workstation
use of rotating disks (it absolutely sucks with SSDs, too). On servers
with a hardware RAID controller, use noop. Using md raid, noop or
deadline may be better depending upon your workflow.

2? Then comes the hardware: your disk subsystem should be about twice as
fast as you want your network to be. To achieve smooth, 100 MB/s
throughput, aim for 200 MB/s disk speed. Any pair of modern drives
should come close, anyway. Any decent RAID controller (not HP smartshit)
should achieve 500 GB/s in RAID 6 with 6 or 8 drives.

3? the filesystem is important. Don't use ext3, it's slow as snails
crawling in molasses. ext4 is pretty slow, too.  If you want to get
closer to bare metal performance, XFS is the best choice unless you
need more than 25 ACLs per file.

4? filesystem options: if you have a RAID controller with BBU or like
to live dangerously, use nobarrier option or barrier=0.  Don't use
data=ordered (well, if you really want to use ext4 try not to tie its
short legs).

5? queue, read ahead options are very  important too. Default linux
options are all thought out for single, 2004-vintage ATA disks. They're
far from  optimal on modern hardware. They're even farther for RAID
arrays. Play with the nr_requests and read_ahead_kb options
from /sys/block/sdXX/queue . On modern RAID arrays, typical value I use
are

 cat /sys/block/sda/queue/nr_requests 
1024

 cat /sys/block/sda/queue/read_ahead_kb 
32768

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------