Hi Jeremy,
Thank you for your time; Let me spend some time with Wireshark analyzing the
packets and I will get back to you
Carter Sheehan
Cloud Engineer
o: +1 (781) 224-7753<tel:+1%20(781)%20224-7753>
e: csheehan at vestmark.com<mailto:csheehan at vestmark.com>
This e-mail and any attachments hereto, are intended for use by the addressee(s)
only and may contain information that is confidential information of Vestmark,
Inc. If you are not the intended recipient of this e-mail, or if you have
otherwise received this e-mail in error, please immediately notify me by
telephone or by e-mail, and please permanently delete the original, any print
outs and any copies of the foregoing. Any dissemination, distribution or copying
of this e-mail is strictly prohibited.
________________________________
From: Jeremy Allison <jra at samba.org>
Sent: Wednesday, February 1, 2023 2:09 PM
To: Carter Sheehan <csheehan at vestmark.com>
Cc: samba at lists.samba.org <samba at lists.samba.org>
Subject: Re: [Samba] SMB Multichannel not working?
External Email
This email was NOT sent from someone at Vestmark
On Wed, Feb 01, 2023 at 06:49:07PM +0000, Carter Sheehan via samba
wrote:>Hello,
>
>I, like many before me, am having an issue with SMB multichannel. I am
running
> Samba v4.10.16 (package is provided by Amazon) on an Amazon Linux 2 EC2
instance.
> I am aware that SMB multichannel is an experimental feature until 4.15, but
the documentation states it is supported so I would like to evaluate it.
>
>We have had reports of performance degradation when certain workloads (heavy
write workloads) are using this Samba server. Upon further investigation, I
believe we are not getting the benefits of SMB multichannel. According to the
documentation the feature can be enabled by setting "server multi channel
support = true" and also an option to force it on. There are also many
articles or posts where it is suggested to set interfaces along with their IP,
speed, and capabilities (such as RSS). All of these things are set in my
configuration, but SMB multichannel does not engage.
>On the client side, I am running another EC2 instance but it is a Windows
Server 2019 instance. SMB3 is supported and multichannel support is enabled by
default, along with BandwidthThrottling being disabled. When I connect to an SMB
share hosted on another Windows EC2 instance, the results of the
Get-SmbMultichannelConnection -IncludeNotSelected command returns that the
connection is a multichannel connection. However, when connected to an SMB share
hosted on my Samba EC2 instance, I get no results for it at all. The results of
Get-SmbConnection does return a result showing a connection is established to my
Samba server.
>
>I have done a LOT of digging including trying a newer Samba version
(v4.16.5), which is after 4.15 where multichannel is no longer experimental, and
I got the same results. I am aware that SMB multichannel from a hardware
perspective relies on multiple physical CPU cores and either multiple NICs or a
single RSS-capable NIC on both sides. I have tried EC2 instances (both client
and server) that have had up to 16 physical cores each (32 vCPUs with
hyperthreading), 32 GB of memory, and NICs capable of 25 Gbps. AWS has concepts
of CPU and Network burst credits but for short bursts it should not be a problem
here.
>
>I have tried multiple storage types including EBS, EFS, and FSx (ZFS). For
EBS, with 16k provisioned IOPS and a 1 GB/s throughput max, I was only getting
300-400 MB/s average and 500-600 MB/s max throughput at the beginning of
transfers through samba. No results returned in powershell for multichannel
connections. I got very similar performance from the other solutions even when I
have configured the resources with a specific performance configuration trying
to test peak performance.
>
>My goal here is to ultimately confirm that SMB multichannel is working, but
> I have a feeling that the results of the PowerShell commands on the client
side prove it is not. Thank you for taking the time to read this and if you
provide feedback I greatly appreciate it!
Firstly, test using v4.16.x - multichannel
is experimental before that (and ther were
many bugs fixes after 4.10.x).
Quick check - in a wireshark trace to this server,
ensure it returns SMB2_CAP_MULTI_CHANNEL (0x00000008)
in the returned capabilities in the SMB2_OP_NEGPROT
packet. That will tell you if you've configured
things right on the server side and the client is
seeing the server's multichannel capability.
If the PowerShell commands on the client are
not showing multichannel use, then it's not
using multichannel. Now we have to figure out
why :-).