samba - Apr 2016 - poor samba performance with many smaller files

Hi,

I have problem with my samba. I have shared folder which contains many
smaller files (few MB each), this files are stored on very fast disk array.
Locally, from that machine I can read this files in parallel with speed
about 500-1000 MB/s, but when I read them through samba, I get only 50-100
MB/s on a 10 gigabit network. I tried to find the problem, so I analyzed
strace logs and found out, that files are open synchronous and it takes
about 15 ms to open one file, so most of the time is spend with openings of
files. Now my question is if I can do somethings about that, is there some
settings which turns on asynchronous open of files?

Thanks

On 2016-04-26 15:49, Peter Bulin wrote:
> Hi,
> 
> I have problem with my samba. I have shared folder which contains many
> smaller files (few MB each), this files are stored on very fast disk array.
> Locally, from that machine I can read this files in parallel with speed
> about 500-1000 MB/s, but when I read them through samba, I get only 50-100
> MB/s on a 10 gigabit network. I tried to find the problem, so I analyzed
> strace logs and found out, that files are open synchronous and it takes
> about 15 ms to open one file, so most of the time is spend with openings of
> files. Now my question is if I can do somethings about that, is there some
> settings which turns on asynchronous open of files?
It's a whole family of settings, look at options starting with
"aio" in
the man page (aio read size/aio write size/…). Going by the man page and
changelog, you'll want Samba 4.4 to not be limited to 10 AIO threads and
various other performance improvements; older versions IIRC don't really
manage to saturate 10G networking.

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas
Systemadministrator
TAO Beratungs- und Management GmbH | Lendplatz 45 | A - 8020 Graz
Mail/XMPP: sven.schwedas at tao.at | +43 (0)680 301 7167
http://software.tao.at

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 648 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20160426/3a54d9dd/signature.sig>

On Tue, Apr 26, 2016 at 03:49:44PM +0200, Peter Bulin
wrote:
> Hi,
> 
> I have problem with my samba. I have shared folder which contains many
> smaller files (few MB each), this files are stored on very fast disk array.
> Locally, from that machine I can read this files in parallel with speed
> about 500-1000 MB/s, but when I read them through samba, I get only 50-100
> MB/s on a 10 gigabit network. I tried to find the problem, so I analyzed
> strace logs and found out, that files are open synchronous and it takes
> about 15 ms to open one file, so most of the time is spend with openings of
> files. Now my question is if I can do somethings about that, is there some
> settings which turns on asynchronous open of files?
If you're on Linux then use the module vfs_aio_pthread
and set "aio_pthread:aio open = true"

This makes open(O_CREAT|O_EXCL) calls async on Linux
only (as it allows per-thread credentials).

If it's opening existing files then this module
currently doesn't help (although removal of a
couple of lines of code would make it work).

Are you *sure* it's the open() calls that
take the time ? Or is it file name lookups ?

Hi,
I need to to open existing readonly files. Yes you are right, not the open
call takes the time, but stat() system call.
I looked at aio_pthread source, the lines which needs to be removed to make
it work are just the lines which tests for O_CREAT|O_EXCL? What problems
can cause such solution for readonly share?

On Tue, Apr 26, 2016 at 8:42 PM, Jeremy Allison <jra at samba.org> wrote:
> On Tue, Apr 26, 2016 at 03:49:44PM +0200, Peter Bulin wrote:
> > Hi,
> >
> > I have problem with my samba. I have shared folder which contains many
> > smaller files (few MB each), this files are stored on very fast disk
> array.
> > Locally, from that machine I can read this files in parallel with
speed
> > about 500-1000 MB/s, but when I read them through samba, I get only
> 50-100
> > MB/s on a 10 gigabit network. I tried to find the problem, so I
analyzed
> > strace logs and found out, that files are open synchronous and it
takes
> > about 15 ms to open one file, so most of the time is spend with
openings
> of
> > files. Now my question is if I can do somethings about that, is there
> some
> > settings which turns on asynchronous open of files?
>
> If you're on Linux then use the module vfs_aio_pthread
> and set "aio_pthread:aio open = true"
>
> This makes open(O_CREAT|O_EXCL) calls async on Linux
> only (as it allows per-thread credentials).
>
> If it's opening existing files then this module
> currently doesn't help (although removal of a
> couple of lines of code would make it work).
>
> Are you *sure* it's the open() calls that
> take the time ? Or is it file name lookups ?
>