samba - Apr 2016 - poor samba performance with many smaller files

Hi,
I did use wireshark to verify that the client (window7) is not the problem,
client is sending out multiple "SMB2 Create request" packets before he
gets
some answer.
On the server side samba is processing this requests in a synchronous way,
in order which they were send, that is what I see from strace output.

On Wed, Apr 27, 2016 at 8:31 AM, Volker Lendecke <Volker.Lendecke at
sernet.de>
wrote:
> On Tue, Apr 26, 2016 at 10:08:41PM +0200, Peter Bulin wrote:
> > The directory is small, it contains only one file. stat() takes 15ms
and
> if
> > I read 1000 files it sums to 15 seconds. So the issue is really that
open
> > file is synchronous.
>
> What do you exactly mean by "open file is synchronous"? At which
level
> does the synchronly kick in? Is it the client? Or is it smbd doing the
> different steps it takes in a synchronous fashion, not allowing
> multiple overlapping opens to make progress simultaneously?
>
> If it's the client that waits for the open replies in sequence, there
> is not much we can do. If it's smbd blocking parallel opens, we could
> do something, but it will quite some work.
>
> Thanks,
>
> Volker
>
> --
> SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
> phone: +49-551-370000-0, fax: +49-551-370000-9
> AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
> http://www.sernet.de, mailto:kontakt at sernet.de
>

On Wed, Apr 27, 2016 at 01:02:54PM +0200, Peter Bulin
wrote:
> I did use wireshark to verify that the client (window7) is not the problem,
> client is sending out multiple "SMB2 Create request" packets
before he gets
> some answer.
Ok, that's what I wanted to know. In this case Samba in theory has a
chance. Not that we would jump on this immediately and get a solution next
month, but this is a workload that many people have pressure to improve.

Unfortunately opening a file for SMB is a pretty complex task, so making
that fully parallel without too much context switch overhead is difficult.

I'd have loved to blame this on the client, but not this time :-)

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de

On Wed, Apr 27, 2016 at 01:02:54PM +0200, Peter Bulin
wrote:
> Hi,
> I did use wireshark to verify that the client (window7) is not the problem,
> client is sending out multiple "SMB2 Create request" packets
before he gets
> some answer.
> On the server side samba is processing this requests in a synchronous way,
> in order which they were send, that is what I see from strace output.
Just to check if parallelizing the open() call helps,
then remove the restrictions in vfs_aio_pthread that
insist on O_CREAT|O_EXCL and test it out (wouldn't
run in production that way without me re-evaluating
this code). If that speeds it up then we have an
idea.

I removed the O_CREAT|O_EXCL checks and added check to exclude O_DIRECTORY,
but it is not working, opening of files fails and I don't know why.
I can see in strace logs that openat syscall in vfs_aio_pthread was
successful, but it still fails and this messages are logged:

[2016/04/28 13:28:59.691782,  5, pid=5623, effective(32697, 10513),
real(32697, 0)] ../source3/smbd/open.c:2551(open_file_ntcreate)
  open_file_ntcreate: FILE_OPEN requested for file subor and file doesn't
exist.

[2016/04/28 13:28:59.693969, 10, pid=5623, effective(32697, 10513),
real(32697, 0)] ../source3/smbd/open.c:4806(create_file_unixpath)
  create_file_unixpath: NT_STATUS_OBJECT_NAME_NOT_FOUND

Any idea what is wrong?

And if stat() is the problem, could SMB_VFS_STAT be implemented in
vfs_aio_pthread module the same way SMB_VFS_OPEN is implemented?




On Wed, Apr 27, 2016 at 6:49 PM, Jeremy Allison <jra at samba.org> wrote:
> On Wed, Apr 27, 2016 at 01:02:54PM +0200, Peter Bulin wrote:
> > Hi,
> > I did use wireshark to verify that the client (window7) is not the
> problem,
> > client is sending out multiple "SMB2 Create request" packets
before he
> gets
> > some answer.
> > On the server side samba is processing this requests in a synchronous
> way,
> > in order which they were send, that is what I see from strace output.
>
> Just to check if parallelizing the open() call helps,
> then remove the restrictions in vfs_aio_pthread that
> insist on O_CREAT|O_EXCL and test it out (wouldn't
> run in production that way without me re-evaluating
> this code). If that speeds it up then we have an
> idea.
>