samba - Sep 2016 - No increased throughput with SMB Multichannel and two NICs

On Tue, Sep 06, 2016 at 10:58:01PM +0200, Daniel Vogelbacher via samba
wrote:
> On 06.09.2016 20:15, Jeremy Allison via samba wrote:
> > On Tue, Sep 06, 2016 at 08:06:48PM +0200, Volker Lendecke via samba
wrote:
> >> On Tue, Sep 06, 2016 at 07:58:27PM +0200, Daniel Vogelbacher via
samba wrote:
> >>> I don't have these options in my smb.conf.
> >>> Do you recommend any specific values?
> >>
> >> aio read size = 1
> >> aio write size = 1
> >>
> >> You might try with current master. There we have improved async
I/O
> >> handling significantly.
> > 
> > Yep. Without those values smbd reads/writes synchronously, meaning
> > issuing two requests down different TCP sockets still get serialized
> > in the server. With aio turned on they'll get handed to a pthread
> > pool to complete, allowing as many requests as your client will send
> > to be outstanding (and your disk and network bandwith will allow of
> > course :-).
> > 
> 
> Oh great, now it works much better!
> 
> I've added:
> 
> vfs objects = aio_pthread # very important!!!
^^^^
That's only used for async open. Are you doing many
opens per second ?
> aio read size = 1
> aio write size = 1
> read raw = Yes
> write raw = Yes
The two above are also not needed. Not for about 15 years.
> strict locking = No
The above might help.
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072
> SO_SNDBUF=131072
Remove all the above. Don't try and second guess the
kernel tcp tuning.
> min receivefile size = 16384
> use sendfile = Yes
2 above not needed. With aio enabled sendfile is disabled.
Receive file to kernel isn't implemented in Linux.
> Transferring a file from share(linux,tmpfs) to Windows SSD hits >2GBit/s
> now.
> 
> But transferring from Windows SSD to linux-tmpfs share still only hits 1
> GBit/s (~500MBit per interface).
> 
> The SSD is fast enough to deliver 2GBit/s and on samba-side no disks
> involved (tmpfs).
> 
> Is there maybe another option required?
Delete all the crap above first :-).

Then start trying to copy locally to the tmpfs share to see what
the max local copy speed it.

On 06.09.2016 23:11, Jeremy Allison via samba wrote:
> On Tue, Sep 06, 2016 at 10:58:01PM +0200, Daniel Vogelbacher via samba
wrote:
>> On 06.09.2016 20:15, Jeremy Allison via samba wrote:
>>> On Tue, Sep 06, 2016 at 08:06:48PM +0200, Volker Lendecke via samba
wrote:
>>>> On Tue, Sep 06, 2016 at 07:58:27PM +0200, Daniel Vogelbacher
via samba wrote:
>>>>> I don't have these options in my smb.conf.
>>>>> Do you recommend any specific values?
>>>>
>>>> aio read size = 1
>>>> aio write size = 1
>>>>
>>>> You might try with current master. There we have improved async
I/O
>>>> handling significantly.
>>>
>>> Yep. Without those values smbd reads/writes synchronously, meaning
>>> issuing two requests down different TCP sockets still get
serialized
>>> in the server. With aio turned on they'll get handed to a
pthread
>>> pool to complete, allowing as many requests as your client will
send
>>> to be outstanding (and your disk and network bandwith will allow of
>>> course :-).
>>>
>>
>> Oh great, now it works much better!
>>
>> I've added:
>>
>> vfs objects = aio_pthread # very important!!!
> ^^^^
> That's only used for async open. Are you doing many
> opens per second ?
> 
>> aio read size = 1
>> aio write size = 1
> 
>> read raw = Yes
>> write raw = Yes
> 
> The two above are also not needed. Not for about 15 years.
> 
>> strict locking = No
> 
> The above might help.
> 
>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072
>> SO_SNDBUF=131072
> 
> Remove all the above. Don't try and second guess the
> kernel tcp tuning.
> 
>> min receivefile size = 16384
>> use sendfile = Yes
> 
> 2 above not needed. With aio enabled sendfile is disabled.
> Receive file to kernel isn't implemented in Linux.
> 
>> Transferring a file from share(linux,tmpfs) to Windows SSD hits
>2GBit/s
>> now.
>>
>> But transferring from Windows SSD to linux-tmpfs share still only hits
1
>> GBit/s (~500MBit per interface).
>>
>> The SSD is fast enough to deliver 2GBit/s and on samba-side no disks
>> involved (tmpfs).
>>
>> Is there maybe another option required?
> 
> Delete all the crap above first :-).
> 
> Then start trying to copy locally to the tmpfs share to see what
> the max local copy speed it.
> 
Now I have:

server multi channel support = yes
vfs objects = aio_pthread,recycle
aio read size = 1
aio write size = 1
strict locking = No

But read (linux->windows) transfer rate is now again 1GBit/s instead of
2GBit/s?!


Regards
Daniel Vogelbacher

On Tue, Sep 06, 2016 at 11:53:04PM +0200, Daniel Vogelbacher via samba
wrote:
> > Delete all the crap above first :-).
> > 
> > Then start trying to copy locally to the tmpfs share to see what
> > the max local copy speed it.
> > 
> 
> Now I have:
> 
> server multi channel support = yes
> vfs objects = aio_pthread,recycle
> aio read size = 1
> aio write size = 1
> strict locking = No
> 
> But read (linux->windows) transfer rate is now again 1GBit/s instead of
> 2GBit/s?!
OK, very strange. But at least you can now
add in the things I told you to remove one
by one to see which one I was wrong about :-).

Don't keep adding, add one - then remove and
add another until we discover which makes
the difference (if it's indeed one, which
I'm guessing) !