samba - Apr 2016 - poor samba performance with many smaller files

The directory is small, it contains only one file. stat() takes 15ms and if
I read 1000 files it sums to 15 seconds. So the issue is really that open
file is synchronous.

On Tue, Apr 26, 2016 at 9:57 PM, Jeremy Allison <jra at samba.org> wrote:
> On Tue, Apr 26, 2016 at 09:44:57PM +0200, Peter Bulin wrote:
> > Hi,
> > I need to to open existing readonly files. Yes you are right, not the
> open
> > call takes the time, but stat() system call.
> > I looked at aio_pthread source, the lines which needs to be removed to
> make
> > it work are just the lines which tests for O_CREAT|O_EXCL? What
problems
> > can cause such solution for readonly share?
>
> So ignore the aio_pthread advice - that really isn't relevent.
> What's causing the problem is the lookup time for cache misses
> with a really large directory.
>
> See here:
>
> https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/largefile.html
>
> for details.
>

On Tue, Apr 26, 2016 at 10:08:41PM +0200, Peter Bulin
wrote:
> The directory is small, it contains only one file. stat() takes 15ms and if
> I read 1000 files it sums to 15 seconds. So the issue is really that open
> file is synchronous.
Oh I misunderstood - I though you said there were many
files in the directory.

Why is stat() so slow ?

The vfs_aio_pthread code can speed up the open()
calls, but not the stat.

Oh, I see :(

I think the problem with slow stat() is that, the file is actually a
symlink to another file in another part of filesystem, so resolving the
symlink takes most of the time.

On Tue, Apr 26, 2016 at 10:21 PM, Jeremy Allison <jra at samba.org> wrote:
> On Tue, Apr 26, 2016 at 10:08:41PM +0200, Peter Bulin wrote:
> > The directory is small, it contains only one file. stat() takes 15ms
and
> if
> > I read 1000 files it sums to 15 seconds. So the issue is really that
open
> > file is synchronous.
>
> Oh I misunderstood - I though you said there were many
> files in the directory.
>
> Why is stat() so slow ?
>
> The vfs_aio_pthread code can speed up the open()
> calls, but not the stat.
>

On Tue, Apr 26, 2016 at 10:08:41PM +0200, Peter Bulin
wrote:
> The directory is small, it contains only one file. stat() takes 15ms and if
> I read 1000 files it sums to 15 seconds. So the issue is really that open
> file is synchronous.
What do you exactly mean by "open file is synchronous"? At which level
does the synchronly kick in? Is it the client? Or is it smbd doing the
different steps it takes in a synchronous fashion, not allowing
multiple overlapping opens to make progress simultaneously?

If it's the client that waits for the open replies in sequence, there
is not much we can do. If it's smbd blocking parallel opens, we could
do something, but it will quite some work.

Thanks,

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de

Hi,
I did use wireshark to verify that the client (window7) is not the problem,
client is sending out multiple "SMB2 Create request" packets before he
gets
some answer.
On the server side samba is processing this requests in a synchronous way,
in order which they were send, that is what I see from strace output.

On Wed, Apr 27, 2016 at 8:31 AM, Volker Lendecke <Volker.Lendecke at
sernet.de>
wrote:
> On Tue, Apr 26, 2016 at 10:08:41PM +0200, Peter Bulin wrote:
> > The directory is small, it contains only one file. stat() takes 15ms
and
> if
> > I read 1000 files it sums to 15 seconds. So the issue is really that
open
> > file is synchronous.
>
> What do you exactly mean by "open file is synchronous"? At which
level
> does the synchronly kick in? Is it the client? Or is it smbd doing the
> different steps it takes in a synchronous fashion, not allowing
> multiple overlapping opens to make progress simultaneously?
>
> If it's the client that waits for the open replies in sequence, there
> is not much we can do. If it's smbd blocking parallel opens, we could
> do something, but it will quite some work.
>
> Thanks,
>
> Volker
>
> --
> SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
> phone: +49-551-370000-0, fax: +49-551-370000-9
> AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
> http://www.sernet.de, mailto:kontakt at sernet.de
>