samba - Sep 2016 - No increased throughput with SMB Multichannel and two NICs

On Tue, Sep 06, 2016 at 08:06:48PM +0200, Volker Lendecke via samba
wrote:
> On Tue, Sep 06, 2016 at 07:58:27PM +0200, Daniel Vogelbacher via samba
wrote:
> > I don't have these options in my smb.conf.
> > Do you recommend any specific values?
> 
> aio read size = 1
> aio write size = 1
> 
> You might try with current master. There we have improved async I/O
> handling significantly.
Yep. Without those values smbd reads/writes synchronously, meaning
issuing two requests down different TCP sockets still get serialized
in the server. With aio turned on they'll get handed to a pthread
pool to complete, allowing as many requests as your client will send
to be outstanding (and your disk and network bandwith will allow of
course :-).

On 06.09.2016 20:15, Jeremy Allison via samba wrote:
> On Tue, Sep 06, 2016 at 08:06:48PM +0200, Volker Lendecke via samba wrote:
>> On Tue, Sep 06, 2016 at 07:58:27PM +0200, Daniel Vogelbacher via samba
wrote:
>>> I don't have these options in my smb.conf.
>>> Do you recommend any specific values?
>>
>> aio read size = 1
>> aio write size = 1
>>
>> You might try with current master. There we have improved async I/O
>> handling significantly.
> 
> Yep. Without those values smbd reads/writes synchronously, meaning
> issuing two requests down different TCP sockets still get serialized
> in the server. With aio turned on they'll get handed to a pthread
> pool to complete, allowing as many requests as your client will send
> to be outstanding (and your disk and network bandwith will allow of
> course :-).
> 
Oh great, now it works much better!

I've added:

vfs objects = aio_pthread # very important!!!
aio read size = 1
aio write size = 1
read raw = Yes
write raw = Yes
strict locking = No
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072
SO_SNDBUF=131072
min receivefile size = 16384
use sendfile = Yes

Transferring a file from share(linux,tmpfs) to Windows SSD hits >2GBit/s
now.

But transferring from Windows SSD to linux-tmpfs share still only hits 1
GBit/s (~500MBit per interface).

The SSD is fast enough to deliver 2GBit/s and on samba-side no disks
involved (tmpfs).

Is there maybe another option required?


Regards
Daniel Vogelbacher

On Tue, Sep 06, 2016 at 10:58:01PM +0200, Daniel Vogelbacher via samba
wrote:
> On 06.09.2016 20:15, Jeremy Allison via samba wrote:
> > On Tue, Sep 06, 2016 at 08:06:48PM +0200, Volker Lendecke via samba
wrote:
> >> On Tue, Sep 06, 2016 at 07:58:27PM +0200, Daniel Vogelbacher via
samba wrote:
> >>> I don't have these options in my smb.conf.
> >>> Do you recommend any specific values?
> >>
> >> aio read size = 1
> >> aio write size = 1
> >>
> >> You might try with current master. There we have improved async
I/O
> >> handling significantly.
> > 
> > Yep. Without those values smbd reads/writes synchronously, meaning
> > issuing two requests down different TCP sockets still get serialized
> > in the server. With aio turned on they'll get handed to a pthread
> > pool to complete, allowing as many requests as your client will send
> > to be outstanding (and your disk and network bandwith will allow of
> > course :-).
> > 
> 
> Oh great, now it works much better!
> 
> I've added:
> 
> vfs objects = aio_pthread # very important!!!
^^^^
That's only used for async open. Are you doing many
opens per second ?
> aio read size = 1
> aio write size = 1
> read raw = Yes
> write raw = Yes
The two above are also not needed. Not for about 15 years.
> strict locking = No
The above might help.
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072
> SO_SNDBUF=131072
Remove all the above. Don't try and second guess the
kernel tcp tuning.
> min receivefile size = 16384
> use sendfile = Yes
2 above not needed. With aio enabled sendfile is disabled.
Receive file to kernel isn't implemented in Linux.
> Transferring a file from share(linux,tmpfs) to Windows SSD hits >2GBit/s
> now.
> 
> But transferring from Windows SSD to linux-tmpfs share still only hits 1
> GBit/s (~500MBit per interface).
> 
> The SSD is fast enough to deliver 2GBit/s and on samba-side no disks
> involved (tmpfs).
> 
> Is there maybe another option required?
Delete all the crap above first :-).

Then start trying to copy locally to the tmpfs share to see what
the max local copy speed it.